SlowGuess/NeurIPS_2023_OCR huggingface dataset:
C.1 Ablation Study for Stopping Epoch
The effectiveness of NoiseCluster relies on long-trained representations derived from a model trained with later stopping, which stops the training process significantly later than what is common with early stopping. This raises two pertinent questions: How is the stopping epoch determined for later stopping? And does it display the same sensitivity as early stopping? To address these questions, we conduct a series of ablation experiments, adjusting the stopping epoch from 20th to 200th epochs.
(a) Noise rate  $12\%$
(b) Noise rate  $20\%$
Figure 11: Ablation analyses of stopping epoch. The mean and standard deviation (percentage) computed over five runs are presented.
As shown in Figure 11, NoiseCluster does not perform well with representations obtained at the 20th epoch, indicating that the effectiveness of NoiseCluster hinges on long-trained representations. Additionally, we can observe that NoiseCluster maintains a stable performance between the 50th and 100th epochs. This span is considerably wider compared to early stopping, which typically require a precise stopping epoch or operate within a relatively narrow range [30, 34, 3]. To delve deeper into the impact of the stopping epoch, we expand the training period for later stopping, and stop the training up to the 200th epoch. Despite a slight drop in the performance of NoiseCluster, attributable to the degradation of representation quality, it remains superior to the best baseline, CDR[49]. These experiments suggest that the choice of the stopping epoch in NoiseCluster can be made with considerable flexibility, displaying minimal sensitivity and negligible impact on the performance of NoiseCluster over a broad range.
C.2 Explore Noise-modeling-based Approaches on SDN
In this section, we further investigate into why noise-modeling-based approaches are ineffective for SDN. Methods based on noise modeling build a transition matrix between the clean distribution and noisy distribution [40, 51]. We utilize the state-of-the-art method, VolMinNet [32], on CIFAR20-SDN with a noise rate  $18\%$ . Following the original paper [32], we set  $\hat{T}$  to -2 and -4.5, respectively.
(a)  $\hat{T} = -2$
(b)  $\hat{T} = -4.5$
Figure 12: (a, b) display the best estimated transition matrices during the training process with  $\hat{T} = -2$  and  $\hat{T} = -4.5$ . The estimated noisy class-posterior probabilities are seen to be widespread across all classes, which is inconsistent with the generation of CIFAR20-SDN.
As depicted in Figure 12, it is evident that  $\hat{T}$  holds the capability to modulate the extent of estimated noisy class-posterior probabilities within the transition matrix. For example, with  $\hat{T} = -2$ , VolMinNet predicts higher noisy class-posterior probabilities in the transition matrix compared to that with  $\hat{T} = -4.5$ . However, neither of these  $\hat{T}$  settings successfully enable an accurate estimation of the transition matrix. Consequently, these inaccurately estimated transition matrices result in inferior performance on SDN.
C.3 Training Time Comparison
Table 7: Training time comparison on CIFAR20-SDN with SDN-12. All methods run on four core CPU and a single Nvidia V100.
NOISECLUSTER	JOINTOPTIM	TOPOFILER	CDR
1.7H	2.5H	1.5H	5.5H
NOISECLUSTER+	DIVIDEMIX	PES SEMI	F-DIVIDEMIX
2H	5.5H	3.1H	14.2H
We evaluate the efficiency of our proposed method with state-of-the-art baselines. Table 7 displays the total training time on CIFAR20-SDN. We can see that the training time of NoiseCluster is comparable to that of TopoFiler [46] and faster than JointOptim [43] and CDR[49]. Furthermore, NoiseCluster+ exhibits substantial efficiency compared with other baselines that employ SSL. For instance, it requires less than half the time needed by DivideMix [30].
D Exploring SDN in Real-world Datasets
Due to space constraints in the main paper, we presented only a limited number of SDN examples. In this subsection, we provide additional visual evidence to further demonstrate the presence of SDN in the Clothing1M and WebVision datasets.
(a) All examples, noisy labels
(b) Confident examples, noisy labels
(c) Confident examples, clean labels
Figure 13: (a) We visualize the features of all the examples in the evaluation set with noisy labels. (b) By removing unconfident examples, the features of confident examples are better separated. (c) We visualize the features of confident examples with clean labels. (d) We visualize the features of mislabeled examples in the confident set with clean labels. It is evident that the majority of unlabeled examples cluster together, confirming our observation of the clustering phenomenon in long-trained representations.
(d) Mislabeled examples in confident set, clean labels
D.1 SDN in Clothing1M
Figure 13 presents a step-by-step exploration process for the Clothing1M dataset, demonstrating how features of unlabeled examples are clustered together based on their semantics. Given the plethora of terms available for different clothing items, we can use accurate names to pinpoint these sub-classes. We highlight three sub-classes prominently impacted by label noise: "Sleep T-Shirt," "Leather Jacket," and "Down Vest," as shown in Figures 14, 15, and 16.
Figure 14: Subclass: Sleep T-Shirt, positioned at coordinates (X:0.45, Y:0.1). The first row displays images from the "T-Shirt" class. The second row showcases those from the "Underwear" class, while the third and fourth rows present images mislabeled as "Underwear". For the mislabeled pictures, it's evident that these images depict a type of shirt meant for home use, Sleep T-Shirt. After comparing images in the test set and Chinese label names, we put the correct names in parentheses.
Figure 15: Subclass: Leather Jacket, positioned at coordinates (X:0.75, Y:0.6). The first row images are from the "Jacket" class, and the second row presents images correctly labeled as the "Windbreaker" class (correct name:"trench coat"). Clearly, these mislabeled images belong to the "Leather Jacket" subclass within the "Jacket" category.
We can also observe that some mislabeled examples are not clustered together, but are instead evenly dispersed in some classes. It is important to understand that in the real world, clothing categories are not solely distinguished by visual features. For instance, the features of "Sweater" and "Knitwear" are intertwined, with the main distinction being the material, not the visual appearance. Similarly, the categories "Underwear" and "Vest" (correct name:"singlet" or "tank top") lead to confusion since some garments can serve multiple purposes.
Removing correctly labeled examples. Early stopping not only helps remove mislabeled examples but also inadvertently excludes those that are correctly labeled when SDN exists. A closer inspection of instances within the "Leather Jacket" cluster confirms the issue: out of the total examples, 313 are mislabeled, with only 23 correctly labeled. This finding indicates that more than  $90\%$  of the "Leather
Figure 16: Subclass: Down Vest, positioned at coordinates (X:0.9, Y:0.3). The first row displays images from the "Down coat" class, while the second row showcases those from the "Vest" class (correct name:"singlet" or "tank top"). When the "Vest" class is absent, these unlabeled images should be classified as "Down Vest" within the "Down coat" class.
Figure 17: We discover 23 instances of "Leather Jacket" within the unconfident set. Despite being correctly labeled, these images are omitted from the confident set due to using early stopping. This process results in a purer cluster that predominantly contains mislabeled examples.
"Jacket" examples are incorrectly labeled in the original dataset. Furthermore, the implementation of early stopping leads to the removal of the small fraction of correctly labeled examples from the confident set, as shown in Figure 17, pushing the noise rate from over  $90\%$  to close to  $100\%$ .
D.2 SDN in WebVision
Table 8: Top-1 performance of our re-implemented DivideMix models alongside the pre-trained models, as assessed on the ImageNet and WebVision validation datasets. By employing an ensemble of pre-trained models, we generate pseudo clean labels to effectively identify unlabeled examples.
Methods	Output Class	WebVision	ImageNet
DivideMix	50	77.34	75.2
DivideMix (reproduce)	50	77.04	74.24
Regnet_y_128gf (IMAGENET1K_SWAG_E2E_V1)	1000	86.64	91.12
ViT_H_14 (IMAGENET1K_SWAG_E2E_V1)	1000	86.08	91.2
Ensemble pretrained models	1000	86.96	91.6
To extend our evaluation of SDN prevalence, we engage in experiments using another real-world dataset: WebVision V1.0 [31]. Following the setting presented in [8], we utilize the first 50 classes of WebVision, known as "mini WebVision". To demonstrate the flexibility of integrating with existing
early stopping methods, we employ DivideMix to select confident examples and extract long-trained representations from two networks, which have been trained for 80 epochs. The extracted features are presented in Figure 18. We utilize an ensemble of pre-trained models to produce pseudo clean labels, and their performance is depicted in Table 8.
tench, Tinca tinca
stingray 
#
#
03
(a) Noisy labels
Figure 18: Due to the absence of clean labels for the training data in WebVision, we employ two pre-trained models from PyTorch to generate pseudo-clean labels. While the pseudo-clean labels may not be accurate, they suffice to provide a general overview of the distribution of mislabeled examples. We then visualize confident features for the first ten classes using t-SNE, comparing noisy labels with the pseudo-clean ones. The category "other" represents examples where predictions from the pre-trained models are not in the first ten classes.
tench-Tinca tinca
stingray 
#
Cock
other
(b) Pseudo clean labels
In Figures 19, 20, and 21, it is evident that, despite clear discrepancies between mislabeled and correctly labeled images, methods based on early stopping struggle to effectively identify these mislabeled examples. Moreover, these mislabeled samples do not fall into conventional categories such as "human" or "cars". Instead, they cluster according to different criteria, sharing similarities in attributes like shape and color with their correctly labeled counterparts. By considering out-of-domain examples as a single, unified class—termed "other", the mislabeled examples within each class can be interpreted as sub-classes of this "other" category.
Figure 19: Explore images categorized under the "Electric Ray" class. The first row showcases images sourced from the ImageNet validation set, while the second row features images from the WebVision validation set. Correctly labeled training images are in the third row, whereas the fourth and fifth rows illustrate instances of mislabeling in the training set. We can find majority of mislabeled images are in "black and white" pattern, which are close to some examples in the third row, so we call it "black and white" subclass.
Figure 20: The row structure remains consistent with the previous format. Notably, numerous cars are incorrectly classified in this category, alongside the presence of wheels and ships. The leading attribute contributing to this misclassification is likely the presence of "curves".
Figure 21: The row structure remains consistent with the previous format. The mislabeled subclass may derive from color attributes, such as pink or white, which serve as the decisive classification features for the "hen" category.