SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1: NoiseCluster
Input: Network  $f_{\theta}$  ; Final layer  $f_{\xi}$  ; Noisy training dataset  $\widetilde{D} (X,\widetilde{Y})$  ; Number of epochs for long-trained N; Class number C; DBSCAN Eps and MinPts.   
for  $i = 1,\dots ,N$  do Train  $f_{\theta}$  and  $f_{\xi}$  on  $\widetilde{D} (X,\widetilde{Y})$  // standard training   
 $\hat{Z}\gets tSNE(f_{\theta}(X))$    
for  $c\gets 1$  to  $C$  do   
 $U_K^c\gets DBSCAN(\hat{Z}_c,Eps,MinPts)$  // identify SDN for  $U_{k}^{c}$  in  $U_{K}^{c}$  do if  $U_{k}^{c}\neq$  largest then Compute set distance with Eq. (1); Update  $\tilde{Y}$  in  $U_{k}^{c}$  with Eq. (2);  $\mathcal{D}_l\gets \mathcal{D}_l\cup U_k^c;$    
Continually train  $f_{\theta}$  and  $f_{\xi}$  on  $\mathcal{D}_l$  for the rest of the epochs.
Table 2: Comparison with state-of-the-art methods without SSL on CIFAR20-SDN. "-XX" indicates the noise rate. The mean and standard deviation computed over five runs are presented.
METHODS	SDN-12	SDN-16	SDN-18	SDN-20
CE [19]	70.42±0.32	67.38±1.31	66.28±0.51	65.23±0.60
JOINTOPTIM [43]	71.48±0.33	67.69±0.37	66.51±0.28	65.45±0.20
DMI [53]	70.67±0.64	66.95±0.61	65.86±0.32	64.35±0.52
TOPOFILTER [46]	69.04±0.47	64.78±0.49	63.52±0.36	62.97±0.62
PTD-R-V [50]	70.85±0.44	66.45±0.40	65.29±0.47	64.41±0.20
VOLMINNET [32]	72.45±0.38	67.78±0.43	66.20±0.24	65.20±0.41
CDR [49]	72.61±0.36	68.87±0.16	67.79±0.28	67.00±0.25
BLTM-V [55]	70.46±0.47	66.57±0.50	64.76±0.42	63.96±0.46
NOISECLUSTER	80.20±0.80	78.77±1.01	77.01±0.34	72.67±0.68
dataset: one with semi-supervised learning (SSL) and one without SSL. In experiments without SSL, we reserve  $10\%$  of the training data as the validation set, while we utilize the entire training data for experiments with SSL. Clothing-1M contains over one million images gathered from the Internet. Additional details on implementation can be found in Appendix B.
Network structure and optimization. For CIFAR20-SDN, we employ ResNet-34 [19] for experiments without SSL and PreAct ResNet-18 [20] for experiments with SSL. During optimization, we train the model for 300 epochs, using a learning rate of  $2 \times 10^{-2}$ , a single cycle of cosine annealing [37], a momentum of 0.9, and a weight decay of  $5 \times 10^{-4}$ . We utilize a batch size of 128 and a stopping epoch of 80, with a ClosePoint value of 20. For DBSCAN hyperparameters, Eps and MinPts are set to 0.02 and 100, respectively.
For Clothing1M, we employ a ResNet-50 pre-trained on ImageNet[28], and train the network for only one epoch. Specifically, we first utilize  $95\%$  of the training data to train the network. Then, we use early stopping to select confident data from the remaining  $5\%$  of the training data and apply NoisyCluster to this data. For optimization, we use SGD with a learning rate of 0.01, a momentum of 0.9, a weight decay of  $10^{-4}$ , and a batch size of 64. ClosePoint, Eps, and MinPts are set to 20, 0.04, and 100, respectively.
Approaches without SSL. To begin our comparison, we evaluate our method against state-of-the-art (SOTA) approaches that do not employ SSL. As shown in Table 2, it becomes evident that many of the methods struggle with SDN, with some even underperforming in comparison to standard training with Cross-Entropy (CE) loss. CDR [49] performs better than its counterparts, but it requires a known noise rate, which remains challenging to accurately estimate in SDN. Our method outperforms all baselines, delivering superior performance. Specifically, our proposed method surpasses CDR by  $5.67\%$  in classification accuracy when a noise rate is  $20\%$ , and the margin further extends to
Table 3: Comparison with state-of-the-art methods with SSL on CIFAR20-SDN. "-XX" indicates the noise rate. The mean and standard deviation computed over five runs are presented.
METHODS	SDN-12	SDN-16	SDN-18	SDN-20
CE [20]	74.27±0.40	70.53±0.38	69.27±0.12	68.50±0.19
MIXUP [59]	75.83±0.41	71.65±0.17	70.29±0.24	69.37±0.17
ELR+ [34]	75.22±0.37	71.09±0.30	70.18±0.23	69.55±0.15
DIVIDEMIX [30]	79.06±0.26	72.80±0.30	70.66±0.51	70.27±0.11
F-DIVIDEMIX [26]	79.71±0.32	74.00±0.83	71.69±0.22	69.59±0.43
PES SEMI [3]	80.27±0.30	73.95±0.24	72.78±0.22	71.72±0.45
NOISECLUSTER+	82.20±0.93	80.35±1.23	77.59±0.86	73.54±0.72
Figure 5: Evaluation memorized percentage per class obtained with DivideMix on CIFAR20-SDN (SDN) and symmetric label noise (Sym). "-XX" indicates the noise rate.
Table 4: Classification accuracy (%) on Clothing1M. Baseline results are taken from the original papers. * indicates the results obtained with an ensemble model.
CE	DMI	JOINTOPTIM	TOPOFILER	PTD-R-V	VOLMINNET
69.21	71.67	72.16	74.10	72.46	72.42
BLTM-V	F-DIVIDEMIX*	PES	DIVIDEMIX*	ELR+*	NOISECLUSTER
73.39	74.37	74.64	74.76	74.81	75.51
$7.59\%$  for a noise rate of  $12\%$ . Furthermore, in comparison with noise-modeling-based methods, our approach outperforms the state-of-the-art VolMinNet by over  $7\%$  across four noise rates.
Approaches with SSL. We integrate our method with semi-supervised learning (SSL) techniques, denoted as NoiseCluster+ and compare it against SOTA approaches that also leverage SSL. To ensure a fair comparison, we include the results of MixUp [59], which is used by all baselines to boost performance. The results are summarized in Table 3. Interestingly, our observations indicate that SSL contributes marginally to performance enhancement on CIFAR20-SDN. Notably, DivideMix [30] shows only a modest advancement over MixUp. In contrast, methods based on representations, such as F-DivideMix [26] and PES semi [3], deliver better outcomes by avoiding the use of the final layer, which is prone to overfitting to SDN. Conclusively, our method, NoiseCluster+, consistently outperforms all baselines by a substantial margin, underscoring that the proposed techniques are the primary factors leading to the improvements.
To better understand why current SOTA methods face difficulties when dealing with SDN, we conduct experiments with DivideMix and compute the ratio of confident examples that include unlabeled data to the total number of unlabeled examples in the training dataset at the end of training. As shown in Figure 5, DivideMix performs well on merely three classes. For the remaining classes, we can see that the confident data contains a substantial number of unlabeled examples, confirming that early stopping and other techniques utilized in DivideMix cannot effectively address SDN issues.
Real-world dataset. We perform experiments on a real-world dataset, Clothing1M, which is widely used as a benchmark due to its challenges. Many existing methods, though effective on noisy synthetic datasets, exhibit limited impact on Clothing1M. Our investigation suggests that this issue stems
(a) Sleep T-Shirt(X:0.45, Y:0.1)
(b) Leather Jacket(X:0.75, Y:0.6)
(c) Down Vest(X:0.9, Y:0.3)
Figure 6: SDN in Clothing1M. (a) The first row displays images from the "T-Shirt" class. The second row shows images from the "Underwear" class. The third and fourth rows present images that are mislabeled as "Underwear". (b) The row order is as follows: "Jacket", "Windbreaker" (correct name:"trench coat"), "Leather Jacket". (c) The row order is as follows: "Down coat", "Vest" (correct name:"singlet" or "tank top"), "Down Vest". The coordinate values correspond to Figure 2 (b). It is worth noting that some label names within Clothing1M are inaccurate due to translation issues. After comparing images in the test set and Chinese label names, we put the correct names in parentheses.
Figure 7: Ablation analyses of MinPts, Eps, and ClosePoint.
from the presence of SDN in Clothing1M, leading to suboptimal performance on the test data. As illustrated in Table 4, NoiseCluster significantly outperforms SOTA methods, achieving an accuracy of  $75.51\%$  without resorting to self/semi-supervised learning or ensemble network techniques.
To delve into the existence and underlying causes of SDN in real-world datasets, we manually generated t-SNE images, strictly adhering to the experimental process outlined in Section 5.1. We then showcase examples within clusters that are predominantly mislabeled. As Figure 6 illustrates, these mislabeled examples clearly belong to specific sub-classes, yet they share common characteristics with their associated noisy classes. These observations confirm the pervasive presence of SDN in real-world datasets. For additional details and further results, please refer to Appendix D.1.
Sensitivity of hyper-parameters. We perform ablation studies on CIFAR20-SDN with noise rates of  $12\%$  and  $20\%$ . NoiseCluster employs DBSCAN [12], which involves two hyperparameters, Eps and MinPts. We find that the selection of these two hyperparameters is relatively straightforward when applied to feature representations due to the substantial margin between classes. Additionally, to reduce the negative impacts of mislabeled examples, we introduce a hyperparameter named ClosePoint, which controls the number of points used to compute the set distance. The results shown in Figure 7 indicate that ClosePoint, along with Eps and MinPts, has a negligible impact on the final outcomes, indicating robustness against hyperparameter changes.
Self-supervised pretrained model. As self-supervised Learning methods continue to mature [18, 15, 61, 4], the utilization of pretrained models from this approach has emerged as a promising strategy for addressing noisy labels, particularly since they are unaffected by such labels during the pre-training
(a) Scratch
(b) MoCo V2[18]
Figure 8: Examining memorization trends when training from scratch compared to training with self-supervised pretrained models on CIFAR20-SDN with an  $18\%$  noise rate. These models were pretrained with MoCo V2 and RSA on the same dataset for 200 epochs, resulting in linear classification accuracies of  $63.02 \pm 0.31$  and  $77.67 \pm 0.27$ , respectively. However, as the performance of pretrained models increases, the speed of overfitting to unlabeled data also becomes faster. The mean and standard deviation (percentage) computed over three runs are presented.
(c) RSA[4]
Table 5: Comparison with/without correcting labels on CIFAR20-SDN. "-XX" indicates the noise rate. The mean and standard deviation (percentage) computed over five runs are presented.
METHODS	SDN-12	SDN-20
OURS	80.20±0.80	72.67±0.68
OURS W/O CORRECTION	78.68±0.61	71.42±0.54
stage [14, 54, 21, 60]. This raises the pivotal question: Does employing an self-supervised pretrained model simplify the addressing of SDN? While a thorough investigation will be reserved for future research work, the succinct answer is no. As shown in Figure 8, self-supervised pretrained models offer initial advantages by facilitating rapid learning of clean examples. However, this rapid learning also leads to the quick memorization of unlabeled examples. Furthermore, after only training for several epochs, the gaps between the memorization rates for clean and unlabeled examples become much smaller than those observed when training from scratch, suggesting that early stopping with self-supervised pretrained models is less effective. We hope that our research will inspire a re-thinking of utilizing self-supervised pretrained models in noisy datasets.
Importance of label correction. To mitigate the bias introduced by learning with SDN, we employ a label correction technique. Given that DNNs have already memorized many mislabeled examples during the early learning phase, label correction becomes indispensable. In Table 5, we present a comparison of results with and without applying label correction and observe a performance improvement of over  $1\%$  for both low and high noise rates. These findings emphasize the importance of the label correction technique in addressing SDN.
In this paper, we empirically examine subclass-dominant label noise (SDN), a new type of label noise that commonly occurs in real-world datasets. SDN presents a significant challenge to conventional methods that rely on early stopping. To overcome this issue, we reveal that long-trained representations are more effective at capturing the semantic relationships between noisy examples. Building on this insight, we propose a novel approach, NoiseCluster, to identify and correct unlabeled examples in SDN. We hope our work will inspire the research community and pave the way for addressing label noise in real-world applications.
The primary limitation is that our proposed method is effective in addressing SDN cases where mislabeled examples have notably different semantics from correctly labeled examples. However, it could struggle in situations where correctly labeled and mislabeled examples exhibit close semantics, especially without any prior knowledge. We leave it as an open research topic for future investigation.
The authors would like to thank the anonymous reviewers and the meta-reviewer for their constructive feedback and encouraging comments on this work. Yingbin Bai was supported by CSIRO Data61. Erkun Yang was supported in part by the National Natural Science Foundation of China under Grant 62202365, Guangdong Basic and Applied Basic Research Foundation (2021A1515110026), and Natural Science Basic Research Program of Shaanxi (Program No.2022JQ-608). Jun Yu was supported by the Natural Science Foundation of China (62276242), National Aviation Science Foundation (2022Z071078001), CAAI-Huawei MindSpore Open Fund (CAAIXSJLJJ-2021-016B, CAAIXSJLJJ-2022-001A), Anhui Province Key Research and Development Program (202104a05020007), USTC-IAT Application Sci. & Tech. Achievement Cultivation Program (JL06521001Y), Sci. & Tech. Innovation Special Zone (20-163-14-LZ-001-004-01). Bo Han was supported by the NSFC Young Scientists Fund No. 62006202, NSFC General Program No. 62376235, and Guangdong Basic and Applied Basic Research Foundation No. 2022A1515011652. Tongliang Liu was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031.
[1] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In ICML, pages 233-242, 2017.
[2] Yingbin Bai and Tongliang Liu. Me-momentum: Extracting hard confident examples from noisily labeled data. In ICCV, 2021.
[3] Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. In NeurIPS, pages 24392-24403, 2021.
[4] Yingbin Bai, Erkun Yang, Zhaoqing Wang, Yuxuan Du, Bo Han, Cheng Deng, Dadong Wang, and Tongliang Liu. Rsa: Reducing semantic shift from aggressive augmentations for self-supervised learning. In NeurIPS, pages 21128-21141, 2022.
[5] David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS, pages 5050-5060, 2019.
[6] Irving Biederman. Recognition-by-components: a theory of human image understanding. Psychological review, page 115, 1987.
[7] Irving Biederman, Robert J Mezzanotte, and Jan C Rabinowitz. Scene perception: Detecting and judging objects undergoing relational violations. Cognitive psychology, pages 143-177, 1982.
[8] Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In ICML, pages 1062-1070, 2019.
[9] De Cheng, Yixiong Ning, Nannan Wang, Xinbo Gao, Heng Yang, Yuxuan Du, Bo Han, and Tongliang Liu. Class-dependent label-noise learning with cycle-consistency regularization. In NeurIPS, 2022.
[10] Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependent label noise: A sample sieve approach. In ICLR, 2021.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[12] Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD-96), pages 226–231, 1996.
[13] Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Gläser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Trans. Intell. Transp. Syst., pages 1341-1360, 2021.
[58] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
[59] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[60] Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M. Bronstein, and Or Litany. Contrast to divide: Self-supervised pre-training for learning with noisy labels. In WACV, pages 387-397, 2022.
[61] Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Ressl: Relational self-supervised learning with weak augmentation. 2021.
[62] Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance-dependent label noise. In CVPR, pages 10113-10123, 2021.
A CIFAR20-SDN
A.1 Construction Details
Subclass-dominant label noise (SDN) is designed to mimic human labeling errors, which often emerge from distinct characteristics of examples. To identify such examples, we employ CIFAR-100, which provides two types of labels: class labels and sub-class labels. Samples belonging to the same subclass are considered to possess distinct characteristics. During the generation of the SDN dataset, we randomly flip a specified number of the last subclass labels within each class to the subsequent class. For instance, the labels of the "whale" subclass within the "aquatic mammals" class are probabilistically flipped to the "fish" class. Table 6 illustrates the flipped sub-classes and the relationships between classes and sub-classes in CIFAR-100. For simplicity, we refer to this newly generated SDN dataset as CIFAR20-SDN.
Table 6: The table outlines the comprehensive structure of the classes within CIFAR20-SDN. Subclasses marked in bold denote those from which examples are selectively flipped.
Class name	Abbreviation	Sub-classes name
aquatic mammals	aquatic	beaver, dolphin, otter, seal, whale
fish	fish	aquarium fish, flatfish, ray, shark,trout
flowers	flowers	orchids, poppies, roses, sunflowers, tulips
food containers	containers	bottles, bowls, cans, cups, plates
fruit and vegetables	fruit	apples, mushrooms, oranges, pears, sweet peppers
household electrical devices	electrical	clock, computer keyboard, lamp, telephone, television
household furniture	furniture	bed, chair, couch, table, wardrobe
insects	insects	bee, beetle, butterfly, caterpillar, cockroach
large carnivores	carnivores	bear, leopard, lion, tiger, wolf
large man-made outdoor things	man-made	bridge, castle, house, road, skyscraper
large natural outdoor scenes	natural	cloud, forest, mountain, plain, sea
large omnivores and herbivores	omnivores	camel, cattle, chimpanzee, elephant, kangaroo
medium-sized mammals	m-mammals	fox, porcupine, possum, raccoon, skunk
non-insect invertebrates	invertebrates	crab, lobster, snail, spider, worm
people	people	baby, boy, girl, man, woman
reptiles	reptiles	crocodile, dinosaur, lizard, snake, turtle
small mammals	s-mammals	hamster, mouse, rabbit, shrew, squirrel
trees	trees	maple, oak, palm, pine, willow
vehicles 1	vehicles 1	bicycle, bus, motorcycle, pickup truck, train
vehicles 2	vehicles 2	lawn-mower, rocket, streetcar, tank, tractor
A.2 Diverse Challenges across Different Classes
Figure 9: The figure shows the effectiveness of early stopping across different classes and various noise rates. When the noise rate in the dataset is small, early stopping performs well in all classes. However, when it is over  $50\%$ , memorized unlabeled samples increases rapidly.
CIFAR20-SDN presents a multifaceted landscape with its array of sub-classes, paving the way for a comprehensive study of SDN and a deeper insight into the intricacies of early stopping. In our experiments, we adjust the overall noise rate from  $0\%$  to  $20\%$ . This implies that within each specific subclass, the noise rate varies from  $0\%$  to  $100\%$ .
As illustrated in Figure 9, the effectiveness of early stopping in addressing SDN is notably inconsistent, heavily contingent on the specific subclass. Our analysis identifies two primary factors that significantly influence the success of early stopping. First, when the mislabeled subclass shares substantial semantic similarities with the clean class, distinguishing mislabeled examples from the clean class becomes challenging. For instance, mislabeling an instance from the "whale" subclass as a "fish" class is difficult to discern from other "fish" instances. Second, when classes have low diversity, the network tends to identify outliers, thus facilitating the detection of mislabeled examples, such as "trees" and "flowers". Conversely, distinguishing between correctly labeled and mislabeled examples becomes considerably more challenging in classes with higher diversity, such as "insects".
A.3 Clusters Effects
We train a ResNet-18 on CIFAR20-SDN for 200 epochs with an overall noise rate of  $18\%$  (SDN-18). Afterward, we utilized t-SNE to visualize all 45,000 training examples. By comparing Figure 10 (a) and (b), we can pinpoint the noise locations, specifically evident in the small clusters. Notably, the vast majority of the mislabeled examples tend to cluster together and usually maintain a distance from their noisy class. This pattern suggests that clustering algorithms, such as DBSCAN, can effectively identify them. However, when the semantics of mislabeled examples closely match those of the noisy class, these mislabeled examples' clusters may not be distinct from the main cluster of the noisy class, which is a limitation of our proposed method.
(a) CIFAR20-SDN with noisy labels
(b) CIFAR20-SDN with clean labels
Figure 10: The figure shows the distribution of training examples after 200 epochs on CIFAR20-SDN. We can find that the large clusters represent examples with correct labels, whereas the smaller clusters indicate those with incorrect labels.
B More Details about Experimental Settings
Data preprocessing. For all experiments, we employ simple data augmentation, including random crop and horizontal flip. For the Clothing-1M dataset, images are initially resized to a resolution of  $256 \times 256$  pixels. Subsequently, a random crop is applied to produce images of  $224 \times 224$  pixels, which are then followed by a random horizontal flip.
Hyper-parameters of semi-supervised learning. We follow [30, 3], and combine NoiseCluster with the semi-supervised learning method, MixMatch [5]. For experiments with SSL, we set  $K = 2$ ,  $T = 0.5$ , and  $\lambda_u = 0$ .  $\alpha$  begins at 4 and drops to 0.75 after half of the total number of epochs.
Clothing1M. Due to the large size of the Clothing1M dataset [52], which consists of one million training examples, the computational time for t-SNE is significantly extended. This volume of data also results in changes to the hyper-parameters. To address these issues, we split the Clothing1M training data into two parts: a larger subset containing 950,000 examples and a smaller one with 50,000 examples. We initially employ the larger subset for training a model with early stopping. Subsequently, based on the trained model, we select and correct confident data from the smaller subset. This corrected data is then used for the remainder of the training process. Note that the model used in the Clothing1M experiments has been pre-trained with the ImageNet-1k dataset [28], leading to the generation of separable representations. As a result, late stopping is omitted in this case. When applying NoiseCluster to a model trained from scratch, it is advisable to employ late stopping.