SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yingbin Bai<sup>1</sup> Zhongyi Han<sup>2</sup> Erkun Yang<sup>3</sup> Jun Yu<sup>4</sup> Bo Han<sup>5</sup> Dadong Wang<sup>6</sup> Tongliang Liu<sup>1</sup> *
$^{1}$ Sydney AI Centre, University of Sydney;  $^{2}$ Mohamed bin Zayed University of Artificial Intelligence;  $^{3}$ Xidian University;  $^{4}$ University of Science and Technology of China;  $^{5}$ Hong Kong Baptist University;  $^{6}$ CSIRO
In this paper, we empirically investigate a previously overlooked and widespread type of label noise, subclass-dominant label noise (SDN). Our findings reveal that, during the early stages of training, deep neural networks can rapidly memorize mislabeled examples in SDN. This phenomenon poses challenges in effectively selecting confident examples using conventional early stopping techniques. To address this issue, we delve into the properties of SDN and observe that long-trained representations are superior at capturing the high-level semantics of mislabeled examples, leading to a clustering effect where similar examples are grouped together. Based on this observation, we propose a novel method called NoiseCluster that leverages the geometric structures of long-trained representations to identify and correct SDN. Our experiments demonstrate that NoiseCluster outperforms state-of-the-art baselines on both synthetic and real-world datasets, highlighting the importance of addressing SDN in learning with noisy labels. The code is available at https://github.com/tml1lab/2023_NeurIPS_SDN.
The popularity of giant neural networks in recent years has emphasized the importance of vast amounts of annotated data [42, 11, 41]. However, manually annotating large amounts of data can be prohibitively expensive, leading researchers to explore alternative methods for collecting data such as web crawling [57] and crowdsourcing [45] that can be more cost-effective. Unfortunately, these methods may produce inaccurate data, particularly when noisy labels are present, thereby undermining the performance of deep neural networks (DNNs). As a result, the research community has increasingly focused on developing approaches to mitigate the impact of noisy labels and prevent the degeneration of DNNs [16, 25, 13].
Most existing approaches for dealing with noisy labels broadly fall into two categories: noise-modeling-based methods and memorization-effects-based methods. Noise-modeling-based methods [51, 53, 32, 47, 55] usually construct a transition matrix to exploit connections between given noisy labels and underlying clean labels. While these methods offer strong theoretical guarantees that the corrected risk closely approximates the risk associated with training on clean data, accurate estimation of the transition matrix relies on numerous assumptions that are difficult to satisfy in real-world scenarios [40, 56]. Memorization-effects-based methods [17, 43] mainly employ early stopping to identify confident examples based on the phenomenon that DNNs will fit the easy examples (i.e., the majority of examples with similar characteristics) first and then memorize the complex examples (i.e., the minority of examples with diverse characteristics) [58, 1, 22]. Recently, some of them [30, 34, 3], combined with semi-supervised learning techniques [5, 23], have achieved state-of-the-art results.
(a) Sym
(b) SDN
(c) SDN 10th epoch
Figure 1: (a, b) It is evident that early stopping can effectively separate symmetric label noise (Sym), but it proves ineffective for subclass-dominant label noise (SDN) as the network initially memorizes mislabeled examples.(c, d) Using the same settings as in (b), it becomes apparent that representations from the 100th epoch exhibit superior geometric structures compared to those from the 10th epoch. Specifically, the boundaries between classes become clearer, allowing for better distinguishability of correctly labeled and mislabeled examples.
(d) SDN 100th epoch
In this paper, we present a new type of label noise, subclass-dominant label noise (SDN), which is overlooked by existing studies. SDN refers to label noise in which mislabeled examples dominate at least one subclass, whose examples exhibit distinct characteristics compared to others in the same class. Real-world datasets are prone to experiencing SDN due to several factors. The likelihood of human mislabeling is influenced by various object characteristics such as shape, context, and part [50, 33, 7, 44, 6]. Moreover, classification tasks usually define classes based on typical or common examples. However, in reality, numerous special sub-classes can exist, sharing common characteristics with other classes. For instance, while whiskers typically play a vital role in distinguishing between cats and dogs, Sphynx cats, unlike other cat breeds, do not possess whiskers. An inexperienced annotator may mislabel the majority of Sphynx cats as dogs, leading to SDN in the dog class.
To evaluate the detrimental effects of SDN, we construct an SDN dataset and conduct an experiment on it. Specifically, we subdivide the aircraft class in CIFAR-10 [27] into two sub-classes based on their status: landed and flying. We generate the SDN dataset by flipping the labels of all landed airplane examples to the automobile class, leading to an overall noise rate of  $2.7\%$ . Our experiment involves training a ResNet-18 model [19] for 100 epochs using cross-entropy loss and extracting confident examples at each epoch. We identify confident examples based on the criteria described in [43]. To quantify the extent to which the network has memorized correctly labeled and mislabeled data, we compute the ratio of confident examples that include mislabeled data to the total number of mislabeled examples in the training dataset. As shown in Figure 1(a, b), early stopping fails to identify mislabeled examples because the network has already memorized the mislabeled examples in SDN. This outcome can be attributed to the fact that DNNs excel at learning from the majority of examples that share similar characteristics first. When the majority of examples in a clean subclass are mislabeled, these mislabeled examples often dominate the gradient and are learned first by DNNs, thereby posing challenges for methods that rely on early stopping.
To overcome this challenge, we explore a novel method named NoiseCluster, which is capable of distinguishing correctly labeled and mistrabeled examples in SDN. The fundamental principle of NoiseCluster is rooted in the concept of later stopping. Contrary to the conventional belief that noisy data increasingly degrade representations over time, our findings suggest that the long-trained representations obtained from later stopping are more effective at capturing the high-level semantics of noisy examples. This leads to a clustering effect where the embeddings of noisy examples with similar characteristics are drawn closer together, as shown in Figure 1(c, d). Motivated by this observation, NoiseCluster utilizes the geometric structures of long-trained representations to identify and correct mistrabeled examples of SDN. Specifically, we first extract features from the penultimate layer of the network after halting the training progress with later stopping. We then group these features by class, based on feature density, and identify the largest group as clean due to its high likelihood of being correctly labeled, while treating the remaining groups as potentially mistrabeled. To determine the label of a potentially mistrabeled group, we assess its similarity to surrounding examples according to set distance and assign it the class with the most similar examples.
The key contributions are summarized as follows:
- We introduce subclass-dominant label noise (SDN) for the first time, highlighting its prevalence in real-world datasets and its potential to render early stopping ineffective.
- We propose a novel method, NoiseCluster, which leverages the geometric structures of long-trained representations to identify and correct mislabeled examples in SDN.
- Empirical results demonstrate the effectiveness of NoiseCluster on both synthetic and real-world datasets, establishing it as a solid baseline for future studies.
Learning with noisy labels is a vibrant research area aimed at improving model robustness. Two main avenues of research have emerged: noise-modeling-based and memorization-effects-based methods.
Noise-modeling-based methods. From a noise modeling perspective, label noise can be categorized into three types: random classification noise (RCN), class-conditional label noise (CCN), and instance-dependent label noise (IDN). RCN maintains a static flip rate for binary classification [35]. CCN assumes that the noise generation is solely dependent on classes and independent of instances [40, 56, 48, 9]. The generation of IDN depends on both classes and instances, making IDN the most general type of label noise [50, 10, 62, 36]. The common philosophy behind noise modeling is to estimate the noise transition matrix. Xia et al. [50] assume annotators may make mistakes on the parts of instances rather than on whole instances, and replace the estimation of the instance-dependent transition matrix with the transition matrices for parts; Yang et al. [55] propose a method to generate Bayes optimal labels and estimate the transition matrix between Bayes optimal labels and noisy labels. Although noise-modeling-based methods do not theoretically require early stopping, many of them implicitly employ it to accurately estimate the transition matrix [40, 51, 53, 56, 55]. As a result, many existing methods struggle to handle SDN.
Memorization-effects-based methods. Arpit et al. [1] find that Deep Neural Networks (DNNs) tend to learn simple (clean) examples first, then gradually memorize complex (noisy) examples. This observed phenomenon has inspired a subset of methodologies grounded in understanding and leveraging memorization effects [17, 2, 38, 22, 24]. JointOptim [43] introduces a pseudo-labeling technique known as hard-label, which regards the model predictions consistent with noisy labels as confident examples and uses them to train the model. DivideMix [30] utilizes Gaussian Mixture Model (GMM) to identify confident examples as labeled data and treat the remaining examples as unlabeled data for semi-supervised learning (SSL). Furthermore, some methods focus on the latent representations extracted from previous layers, as the final layer is particularly vulnerable to the influence of noisy examples [29, 46, 26, 3]. TopoFilter [46] utilizes k-NN to explore the topological information of the latent representations and then selects the largest component as clean data. Kim et al. [26] propose a noise detection method called FINE that utilizes Eigen decomposition and the principal component to select clean examples. PES [3] takes advantage of the fact that the effects of noisy examples tend to decrease as the layer number increases and improves early stopping by re-learning later layers based on the latent representations. NoiseCluster also exploits the geometric structures of the latent representations. However, unlike previous methods that rely on early stopping, NoiseCluster takes advantage of later stopping to extract long-trained representations, enabling the identification and correction of mislabeled examples in SDN.
3 SDN Analysis
In this section, we begin by defining subclass-dominant label noise (SDN). Next, we introduce a specially crafted dataset, CIFAR20-SDN, for exploring SDN, and compare it with two established real-world datasets. Finally, we delve into the characteristics of SDN from two distinct perspectives: its adverse effects on early stopping and its negative impact across various layers.
Definition of SDN. Let  $\widetilde{D} = \{(x_i, \widetilde{y}_i)\}_{i=1}^n$  be the noisy dataset, where  $x_i \in X$  denotes input data,  $\widetilde{y}_i \in \{1, \dots, C\}$  denote a noisy label, and  $C$  is the total number of classes.  $x_i$  is associated with a latent (not provided) clean subclass label  $b_i \in \{1, \dots, B\}$ , and  $B$  is the total number of sub-classes in this dataset. It is assumed that each clean subclass  $b$  corresponds to a single clean class  $c$ . Subclass-dominant noise (SDN) refers to a type of label noise in which mislabeled examples dominate at least one subclass. Specifically, within a clean subclass  $b$ , examples could be flipped into different classes,
(a) CIFAR20-SDN
(b) Clothing1M
(c) WebVision
Figure 2: Comparison of feature distribution between CIFAR20-SDN and two real-world noisy datasets, with colors corresponding to clean labels. For Clothing1M, we utilize a provided evaluation set containing both clean and noisy labels for each example, employing the early stopping process described in Section 5.1. For WebVision, a DivideMix model is employed to generate features for the first ten classes; "clean" labels for these are inferred using models pretrained on ImageNet.
Table 1: Comparison of noisy impacts on different layers with different types of label noise. Training with clean labels terms as Clean; "Sym" is short for symmetric label noise; "IDN" is short for instance-dependant label noise; and "Pair" is short for pairflip label noise. "-XX" indicates the noise rate. The mean and standard deviation (percentage) computed over five runs are presented.
NOISE TYPE	CLEAN	SYM-20	IDN-20	PAIR-20	SDN-20
FINAL	85.17±0.32	72.25±0.21	72.41±0.22	73.39±0.48	68.50±0.19
PENULTIMATE	85.28±0.35	73.43±0.67	76.51±0.89	78.77±0.34	81.67±0.17
including its own clean class. If the class  $j \in \{1, \dots, C\}$  with the most flipped examples does not equal to the clean class  $c$ , then the subclass  $b$  is considered to be dominated by the examples flipped into noisy class  $j$ .
CIFAR20-SDN. To facilitate research on SDN, we introduce CIFAR20-SDN, a representative SDN dataset built from CIFAR-100, which provides 20 class labels and 100 subclass labels. The generation of CIFAR20-SDN is similar to that of Pairflip label noise [17]. Specifically, for each class, a certain percentage of labels in the last subclass are flipped to the subsequent class. For example, the first class in CIFAR20 is termed "aquatic mammals", which comprises five sub-classes: "beaver", "dolphin", "otter", "seal", and "whale". Labels from the "whale" subclass are randomly flipped to the subsequent "fish" class. As a result, any "whale" labeled as "fish" is deemed an instance of SDN. Note that only the class labels are utilized for model training and evaluation.
To assess the similarity between CIFAR20-SDN and real-world noisy datasets, we compare the feature distribution of CIFAR20-SDN with two other real-world noisy datasets, Clothing1M [52] and WebVision [31]. Specifically, we directly visualize the features of CIFAR20-SDN, as it exclusively contains SDN, while for the two real-world datasets, we visualize the features of confident examples extracted using early stopping methods. As shown in Figure 2, it is clear that CIFAR20-SDN successfully emulates the properties of the two real-world datasets, with unlabeled examples clustering together and maintaining a distance from their noisy classes.
Negative effects of SDN on different layers. To thoroughly study the negative impacts of SDN on various layers, we train a PreAct ResNet-18 model on CIFAR20-SDN with four types of label noise. We report the highest test accuracy achieved during training as an indicator of the quality of the representations from the final layer. Additionally, to assess the representation quality from the penultimate layer, we freeze the other layers, re-initialize and retrain the final layer using clean labels.
Table 1 reveals that SDN significantly impacts the final layer more detrimentally than other types of label noise, resulting in the lowest test accuracy, which is roughly  $80\%$  of the accuracy achieved on the clean dataset. Conversely, when considering the penultimate layer, the outcomes achieved with SDN surpass those obtained with any other type of label noise, indicating that previous layers are less adversely impacted even after 300 epochs of training. These findings underscore the importance of utilizing the representations extracted from the penultimate layer in addressing SDN.
(a) SDN intra-class
(b) Sym-50 intra-class
(c) SDN inter-class
Figure 3: (a, b) illustrates the intra-class relationships between correctly labeled and mislabeled examples in a noisy class, while (c, d) demonstrates the inter-class relationships between mislabeled examples and their corresponding latent clean classes. To better visualize the intra-class feature relationships, we use the features from a noisy automobile class and visualize them with clean labels. Similarly, to effectively visualize inter-class feature relationships, we use features from four classes, as 2D images cannot adequately represent high-dimensional relationships.
(d) Sym-50 inter-class
Advantages of long-trained representations. We extend the experiments depicted in Figure 1 and visualize the features extracted from the networks trained for 200 epochs. As shown in Figure 3(a, b), features within a noisy class are clustered based on their semantics, which suggests that networks can learn the semantics of mislabeled examples even when trained with noisy labels. Moreover, Figure 3(c, d) shows that features of mislabeled examples tend to be close to the features with corresponding clean classes, implying that the geometric structures of the features can be utilized to identify their clean classes. Finally, a comparison of the four sub-figures reveals that the features extracted from SDN exhibit superior geometric structures compared to those extracted from symmetric label noise. This observation suggests that the properties of long-trained representations are particularly advantageous when dealing with SDN.
To overcome the challenges posed by SDN, we introduce a novel method called NoiseCluster, which leverages the geometric structures of long-trained representations. NoiseCluster consists of two main steps: initially, it identifies potentially mislabeled examples by clustering similar instances into multiple sets. Subsequently, it performs label correction by computing set distance and reassigning the labels of potentially mislabeled examples.
4.1 Identify SDN
NoiseCluster begins with training on  $\widetilde{D}(X, \widetilde{Y})$  and then halts the training with later stopping, whose stopping point is much later than that of early stopping, allowing for thorough exploitation of knowledge in noisy data. We then extract the latent feature  $Z$  of the input data  $X$  from the penultimate layer and reduce its dimensionality to two dimensions using t-SNE [39], resulting in a new feature denoted as  $\hat{Z}$ . To separate mislabeled data, we explore the geometric structures of  $\hat{Z}_c$ , where  $c \in \{1, \dots, C\}$  and  $C$  is the total number of classes. To handle the unknown number of clusters in each class, we employ DBSCAN [12], a feature density-based clustering algorithm, to cluster the feature  $\hat{Z}_c$  into  $K$  groups  $U_K^c$ . The largest group, which is most likely to contain correctly labeled examples, is identified as the clean group, while the remaining groups are regarded as potentially mislabeled groups and denoted as  $U_{K-1}^c$ .
Stopping point selection. Although NoiseCluster can be applied to a fully trained model, we find that further training on the original noisy data does not constantly improve the representations after the training becomes stable. Therefore, to reduce training time, we incorporate our method into the middle stage of the training process.
4.2 Label Correction
Simply removing all examples in  $U_{K - 1}^{c}$  can result in the loss of many important examples. To address this challenge, we treat examples in a potentially mislabeled group  $U_{k}^{c}$  as a cohesive unit, where all
(a) Set distance
(b) Mean distance
Figure 4: The figure compares two methods for quantifying the similarity between two sets. Compared to mean distance shown in (b), set distance can more accurately measure the similarity between a potentially mislabeled group and a subset of similar examples within the class sets, particularly when the classes have high diversity.
examples share the same label. We then assess the similarity between this group and the surrounding examples by computing distances and assigning examples in the group to the class with the most similar examples. Specifically, due to the clustering effects of long-trained representations, we treat the examples in  $U_{k}^{c}$  as a set, and form multiple class sets  $O_{b}$  from the surrounding examples, where  $b \in \{1,\dots,C\}$ . We measure the distance between  $U_{k}^{c}$  and  $O_{b}$  using set distance, which is defined as the minimum Euclidean distance between any feature  $z_{q},z_{p}$  from sets  $U_{k}^{c}$  and  $O_{b}$ , as depicted in Figure 4. The formula for set distance is defined as follows:
$$
d \left(U _ {k} ^ {c}, O _ {b}\right) = \inf  _ {p \in U _ {k} ^ {c}, q \in O _ {b}} \left\{\left\| z _ {q} - z _ {p} \right\| _ {2} \right\}, \tag {1}
$$
where  $\| \cdot \|_2$  indicates  $L2$  Norm. In practice, to mitigate the potential impacts of mislabeled examples close to the set  $U_k^c$ , we utilize a specified number of closest distances, denoted as ClosePoint, and compute their average. We then correct the labels in  $U_k^c$  with the category of the class set with the closest distance,  $d_{kb}$ , which is defined as follows:
$$
d _ {k b} = \inf  _ {b \in \{1, \dots , C \}} \left\{d \left(U _ {k} ^ {c}, O _ {b}\right) \right\}. \tag {2}
$$
After acquiring the corrected data, we use them to continue training the network for the rest of the epochs. The whole process of NoiseCluster is summarized in Algorithm 1.
Combine with SSL. Semi-supervised learning (SSL) has enabled numerous methods to attain impressive benchmarks in LNL [30, 34, 26]. NoiseCluster can also take advantage of SSL techniques to enhance its performance. Specifically, for each class  $c$ , DBSCAN divides features  $\hat{Z}_c$  into  $K$  distinct groups  $U_K^c$  and assigns a group index  $g_i^c$  to each feature  $\hat{z}_i^c$ . Examples that cannot be clustered into any group are regarded as "noise". To integrate SSL, we treat the DBSCAN noise as the unlabeled data and the remaining examples as the labeled data. Accordingly, we can summarize the labeled and unlabeled data sets as follows:
$$
\left\{ \begin{array}{l} \mathcal {D} _ {l} = \left\{\left(x _ {i}, \widetilde {y} _ {i}\right) \mid g _ {i} ^ {c} = 1, \dots , K _ {c} \right\} \\ \mathcal {D} _ {u} = \left\{x _ {i} \mid g _ {i} ^ {c} \neq 1, \dots , K _ {c} \right\} \end{array} . \right. \tag {3}
$$
Combine with early stopping. Although NoiseCluster is primarily designed to address SDN, it can be seamlessly integrated with early stopping. To tackle a noisy dataset containing multiple types of label noise, we propose a two-stage training process. In the first stage, we employ early stopping to select confident data, thereby filtering out label noise that is sensitive to early stopping. In the second stage, we apply NoiseCluster to the confident data to identify and correct mislabeled examples in SDN. This two-stage strategy allows us to effectively address the challenges posed by a mixture of various types of label noise, resulting in enhanced performance on noisy datasets.
Datasets. We evaluate the performance of NoiseCluster on a synthetic dataset, CIFAR20-SDN, and a real-world dataset, Clothing-1M [52]. To ensure a fair comparison with existing methods, we follow the settings of the previous methods [30, 34, 3] and conduct two sets of experiments on the synthetic