Subclass-Dominant Label Noise: A Counterexample for the Success of Early Stopping
Authors: Yingbin Bai
Zhongyi Han
Erkun Yang
Jun Yu
Bo Han
Dadong Wang
Tongliang Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: learning with noisy labels
weakly supervised learning

Abstract: In this paper, we empirically investigate a previously overlooked and widespread type of label noise, subclass-dominant label noise (SDN). Our findings reveal that, during the early stages of training, deep neural networks can rapidly memorize mislabeled examples in SDN. This phenomenon poses challenges in effectively selecting confident examples using conventional early stopping techniques. To address this issue, we delve into the properties of SDN and observe that long-trained representations are superior at capturing the high-level semantics of mislabeled examples, leading to a clustering effect where similar examples are grouped together. Based on this observation, we propose a novel method called NoiseCluster that leverages the geometric structures of long-trained representations to identify and correct SDN. Our experiments demonstrate that NoiseCluster outperforms state-of-the-art baselines on both synthetic and real-world datasets, highlighting the importance of addressing SDN in learning with noisy labels. The code is available at https://github.com/tmllab/2023_NeurIPS_SDN.

Subclass-Dominant Label Noise: A Counterexample for the Success of Early Stopping
Yingbin Bai
Zhongyi Han Erkun Yang²
Jun Yu*
Bo Han Dadong Wang Tongliang Liu²
SSdney AI Centre, University of Sydney: "Mohamed bin Zayed University of Artificial Intelligence: Xidiaa University; *University of Science and Technology of China; Hong Kong Baptist University; "CSIRO
Abstract
In this paper, we empirically investigate previously overlooked and widespread type of label noise, subclassddominan label noise (SDN), Our findings reveal that, during the early stages of training, deep neural networks can rapidly memorize mislabeled examples in SDN. This phenomenon poses challenges in effectively selecting confident examples using conventional early stopping techniques. To ad- dress this issue, we delve into the properties of SDN and observe that traietrained representations are superior capturing the high-level semanties of mislabeled examples, leading to a clustering effect where similar examples are grouped to- gether. Based on this observation, we propose a novel method called NoiseCluster that leverages the geometric structures of long-trained representations to identify and correct SDN, Our experiments demonstrate that NoiseCluster outperforms state-of-the-art baselines both synthetic and real-world datasets, highlighting the importance addressing SDN in learning with noisy labels. The code available at hhttpp::/gittee waaaa//mlllabbaaaaamaaa 2ttp://2222_NeeeS_P____0.P.
Introduction
The popularity of giant neural networks in recent years has emphasized the importance of vast amounts of annotated data 42
However, manually annotating large amounts of data can be prohibitively expensive, leading researchers to explore alternative methods for collecting data such as web crawling [57] and crowdsoureing (45) that can be more cost-effective. Unfortunately. these methods may produce inaccurate data, particularly when noisy labels are present, thereby undermining the performance of deep neural networks (DNNs).
result, the research community has increasingly focused on developing approaches to mitigate the impact of noisy labels and prevent the degeneration of DNNs [T6] 25 [3].
Most existing approaches for dealing with noisy labels broadly fall into two categories: noise- modeling-based methods and cianoriiation-fffett---ooeo
methods. Noise-modeling- 47 55 usually construct transition matrix to exploit connections between given noisy labels and underlying clean labels. While these methods offer strong theoretical guarantees that the corrected risk closely approximates the risk associated with training on clean data, accurate estimation of the transition matrix relies on numerous assumptions that are difficult to satisfy real-world scenarios [40 [56.. Neemoriation-eeffett-
methods
51 53 32
mainly employ early stopping to identify confident examples based on the phenomenon that DNNs will fit the easy examples (i.e., the majority of examples with similar characteristics) first and then memorize the complex examples (i.e.,
methods 17 43
the minority of examples with diverse characteristics) (58 I 22 Recently, some them [30 34 3)
combined with semi-supervised learning techniques 5 23 have achieved tate-off-the-at results.
"Correspondence to Tongliang Liu (tongliang.lu@@syynny..ddo @sydncy.cdu.auu))
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

100
100
Nemorzed percentage
Memorizd percentage
00
00
Clean Mistabeleed
Clean
Mistabeled
100
Epochs
Epochs
(a) Sym
(b) SDN
(c) SDN 10th epoch
(d) SDN 100th epoch
Figure I: (a, b) It is evident that early stopping can effectively separate symmetric label noise (Sym), but it proves ineffective for subclass-dominant label noise (SDN) the network initially memorizes mislabeled examples. (c, d) Using the same settings as in (b), it becomes apparent that representations from the 100th epoch exhibit superior geometric structures compared to those from the 10th epoch. Specifically, the boundaries between classes become clearer, allowing for better distinguishabiity of correctly labeled and mislabeled examples.
In this paper, we present a new type of label noise, subclass-dominant label noise (SDN), which is overlooked by existing studies. SDN refers label noise in which mislabeled examples dominate at least one subclass, whose examples exhibit distinct characteristics compared to others in the same class. Real-world datasets are prone experiencing SDN due to several factors. The likelihood of human mislabeling is influenced by various object characteristics such as shape, context, and part Moreover, classification tasks usually define classes based on typical or common examples. However, in reality, numerous special sub-classes can exist, sharing common characteristics with other classes. For instance, while whiskers typically play a vital role distinguishing between cats and dogs, Sphynx cats, unlike other cat breeds, do not possess whiskers. An inexperienced annotator may mislabel the majority of Sphynx cats as dogs, leading to SDN in the dog class.
50 33
To evaluate the detrimental effects of SDN, we construct an SDN dataset and conduct an experiment on it. Specifically. we subdivide the aircraft class in CIFAR-10 (27 into two sub-classes based on their status: landed and flying. We generate the SDN dataset by flipping the labels of all landed airplane examples to the automobile class, leading to an overall noise rate of 2.7%. Our experiment involves training a ResNet-18 18 model [9 for 100 epochs using cross-entropy loss and extracting confident examples at each epoch. We identify confident examples based on the criteria described in
[4). To quantify the extent to which the network has memorized correctly labeled and mislabeled data, we compute the ratio of confident examples that include mislabeled data to the total number of mislabeled examples in the training dataset. As shown in Figure Ja. a. b), early stopping fails to identify mislabeled examples because the network has already memorized the mislabeled examples in SDN. This outcome can be attributed to the fact that DNNs excel at learning from the majority of examples that share similar characteristics first. When the majority of examples in a clean subclass mislabeled, these mislabeled examples often dominate the gradient and are learned first by DNNs, thereby posing challenges for methods that rely on early stopping.
To overcome this challenge, we explore a novel method named NoiseCluster, which is capable of distinguishing correctly labeled and mislabeled examples in SDN. The fundamental principle of NoiseCluster is rooted in the concept of later stopping. Contrary to the conventional belief that noisy data increasingly degrade representations over time, our findings suggest that the long- representations obtained from later stopping are more effective at capturing the high-level semantics of noisy examples. This leads to a clustering effect where the embeddings of noisy examples with similar characteristics are drawn closer together, as shown in Figure [c. d). Motivated by this observation, NoiseCluster utilizes the geometric structures of long-trained representations to identify and correct mislabeled examples of SDN. Specifically, we first extract features from the penultimate layer of the network after halting the training progress with later stopping. We then group these features by class, based on feature density, and identify the largest group clean due to its high likelihood of being correctly labeled, while treating the remaining groups as potentially mislabeled. To determine the label of a potentially mislabeled group, we assess its similarity to surrounding examples according to set distance and assign it the class with the most similar examples.
The key contributions are summarized as follows:

---

We introduce subclass- dominant label noise (SDN) for the first time, highlighting its preva- lence in real-world datasets and its potential to render early stopping ineffective.
We propose a novel method, NoiseCluster, which leverages the geometric structures of long-trained representations to identify and correct mislabeled examples in SDN.
Empirical results demonstrate the effectiveness of NoiseCluster on both synthetic and real-world datasets, establishing it as a solid baseline for future studies.
2
Related Work
Learning with noisy labels is vibrant research area aimed improving model robustness. Two main avenues of research have emerged: noise-modeling-bast -based and memorization--ffect---ee
methods.
Noise-modeling-basee based methods. From noise modeling perspective, label noise can be categorized into thrce types: random classification noise (RCN), class-conditional label noise (CCN), and instance-dependent label noise (IDN). RCN maintains static flip rate for binary classification B5 CCN assumes that the noise generation is solely dependent on classes and independent instances The generation of IDN depends
both classes and instances, making IDN the most general type of label noise [50] 10 62 36). The common philosophy behind noise modeling is to estimate the noise transition matrix. Xia et al. [50] assume annotators may make mistakes the parts of instances rather than on whole instances, and replace the estimation of the instance-dependent transition matrix with the transition matrices for parts; Yang et al. (55) propose method generate Bayes optimal labels and estimate the transition matrix between Bayes optimal labels and noisy labels. Although moiie-modeling-baae based methods do not theoretically require early stopping, many of
40
56
48 9
them implicitly employ it to accurately estimate the transition matrix [40 51 53 56 55). As result, many existing methods struggle handle SDN.
methods. Arpit et al. I find that Deep Neural Networks (DNNs) tend to learn simple (clean) examples first, then gradually memorize complex (noisy) examples. This observed phenomenon has inspired a subset of methodologies grounded in understanding and leveraging memorization effects [17 2 38 22 24]. JointOptim 43 introduces a pseudo-labeling
Memorization--ffetts--
technique known as hard-label, which regards the model predictions consistent with noisy labels confident examples and uses them train the model. DivideMix (30) utilizes Gaussian Mixture Model (GMM) to identify confident examples as labeled data and treat the remaining examples as unlabeled data for semi-supervised learning (SSL). Furthermore, some methods focus on the latent representations extracted from previous layers, as the final layer is particularly vulnerable to the TopoFilter 446) utilizes K-NN to explore the topological information of the latent representations and then selects the largest component clean data. Kim al. [26] propose a noise detection method called FINE that utilizes Eigen decomposition and the principal component to select clean examples. PES B takes advantage of the fact that the effects of noisy examples tend to decrease as the layer number increases and improves early stopping by re-learning later layers based on the latent representations. NoiseCluster also exploits the geometric structures of the latent representations. However, unlike previous methods that rely on early stopping, NoiseCluster takes advantage of later stopping to extract long- long-trained representations, enabling the identification and correction of mislabeled examples in SDN.
influence of noisy examples 22
3 SDN Analysis
label noise (SDN). Next, we introduce a specially crafted dataset, CIFAR20-SDN, for exploring SDN, and compare it with two established real-world datasets. Finally, we delve into the characteristies of SDN from two distinct perspectives: its adverse effects on early stopping and its negative impact across various layers.
In this section, we begin by defining
Definition of SDN. Let D = { (2,, Fo be the noisy dataset, where x; X denotes input data, J: E {1....,C) C} denote noisy label, and C' is the total number of classes. z; is associated with latent (not provided) clean subclass label by E (1, B). and B is the total number of sub-classes in this dataset. It is assumed that each clean subclass b corresponds to single clean class Subclass- dominant noise (SDN) refers to a type of label noise in which mislabeled examples dominate at least one subclass. Specifically, within clean subclass b, examples could be flipped into different classes,
3

---

(a) CIFAR20-SDD SDD
(b) Clothing IM
(c) WebVision
Figure 2: Comparison of feature distribution between CIFAR20-SDN and two real-world noisy datasets, with colors corresponding to clean labels. For Clothing IM, we utilize a provided evaluation set containing both clean and noisy labels for each example, employing the early stopping process model is employed to generate features for the first ten classes; "clean" labels for these are inferred using models pretrained on ImageNet.
described in Section 5.1 For WiiVinnon, DivideMix
Table 1:
Comparison of noisy impacts on different layers with different types of label noise. Training with clean labels terms as Clean; "Sym" is short for symmetric label noise: "IDN" is short for instance-depen deneedand label noise; and "Pair" short for pairflip label noise. "-XX" indicates the noise rate. The mean and standard deviation (percentage) computed over five runs are presented.
NOISE TYPE
CLEAN
SYM-20
IDN-20
20
SDN-20
FINAL PENULTIMATE
72.2511.221 85.28
85.17:0..32
72.4110.22
73.3910.48 78.77:10.34
68.50110.19
73.43-10.67
76.51
81.67
including its clean class. If the class (1,....C) with the most flipped examples does equal to the clean class c, then the subclass b is considered to be dominated by the examples flipped into noisy class j.
CIFAR20-SDN.
To facilitate research on SDN, we introduce CIFAR20-S2 SDN, a representative SDN dataset built from CIFAR-100, which provides 20 class labels and 100 subelass labels. The generation of CIFAR20-SDD is similar to that of Pairflip label noise 17]. Specifically, for each class, a certain percentage of labels in the last subclass flipped to the subsequent class. For example, the first class in CIFAR20 is termed "aquatic mammals", ", which comprises five sub-classes: "beaver", "dolphin", "otter", "seal", and "whale". Labels from the "whale" subclass randomly flipped to the subsequent "fish" class. As result, any "whale" labeled as "fish" is deemed an instance of SDN. Note that only the class labels utilized for model training and evaluation.
To assess the similarity between CIFAR20-SDN and real-world noisy datasets, we compare the feature distribution of CIFAR20-SDN with two other real-world noisy datasets, Clothing LM [52) and WebVision [31] Specifically, we directly visualize the features of CIFAR20-SDN, as it exclusively contains SDN, while for the real-world datasets, we visualize the features of confident examples extracted using early stopping methods. As shown in Figure 2 it is clear that CIFAR20- successfully emulates the properties of the two real-world datasets, with mislabeled examples clustering together and maintaining a distance from their noisy classes.
Negative effects of SDN on different layers. To thoroughly study the negative impacts of SDN on various layers, we train a PreAct ResNet-18 model on CIFAR20-SDN with four types of label noise. We report the highest test accuracy achieved during training an indicator of the quality of the representations from the final layer. Additionally, to assess the representation quality from the penultimate layer, we freeze the other layers, re-initialize and retrain the final layer using clean labels.
Table reveals that SDN significantly impacts the final layer more detrimentally than other types of label noise, resulting in the lowest test accuracy, which is roughly 80% of the accuracy achieved on the clean dataset. Conversely, when considering
penultimate layer, the outcomes achieved with SDN surpass those obtained with any other type of label noise, indicating that previous layers are less adversely impacted even after 300 epochs of training. These findings underscore the importance of utilizing the representations extracted from the penultimate layer in addressing SDN.

---

(a) SDN intra-class
(b) Sym-50 intra-class
(c) SDN inter-class
(d) Sym-50 inter-class
Figure 3: (a, b) illustrates the intra-class relationships between correctly labeled and mislabeled examples in a noisy class, while (c, d) demonstrates the inter-class relationships between mislabeled examples and their corresponding latent clean classes. To better visualize the intra-class feature relationships, we use the features from a noisy automobile class and visualize them with clean labels. Similarly, to effectively visualize inter-class feature relationships, we use features from four classes, as 2D images cannot adequately represent high-dimensions relationships.
Ad Advantages of long-trained representations, We extend the experiments depicted in Figure | and visualize the features extracted from the networks trained for 200 epochs. As shown in Figure 3|a, b), features within a noisy class are clustered based on their semantics, which suggests that networks can learn the semantics of mislaheled examples even when trained with noisy labels. Moreover, Figure d) shows that features of mislabeled examples tend to be close to the features with corresponding clean classes, implying that the geometric structures of the features can be utilized to identify their clean classes. Finally, comparison of the four sub-figures reveals that the features extracted from SDN exhibit superior geometric structures compared to those extracted from symmetric label noise. This observation suggests that the properties of long- eeainted representations are particularly advantageous when dealing with SDN.
Methodology
To overcome the challenges posed by SDN, we introduce novel method called NoiseCluster, which leverages the geometric structures of long-trained trained representations. NoiseCluster consists of two main steps: initially, it identifies potentially mislabeled examples by clustering similar instances into multiple sets. Subsequently. it performs label correction by computing set distance and reassigning the labels of potentially mislabeled examples.
4.1 Identify SDN
NoiseCluster begins with training on (Xx, Y) and then halts the training with later stopping, whose stopping point is much later than that of early stopping, allowing for thorough exploitation of knowledge in noisy data. We then extract the latent feature Z of the input data X from the penultimate layer and reduce its dimensionality to two dimensions using t-SNE (39], resulting in new feature denoted as Ż. To separate mislabeled data, explore the geometric structures of 2.. where с E {1.... C}} and is the total number of classes. To handle the unknown number of clusters in each class, we employ DBSCAN [12, feature density-based clustering algorithm, to cluster the feature 2, into K groups U%. The largest group, which is most likely to contain correctly labeled examples, is identified as the clean group, while the remaining groups are regarded as potentially mislabeled groups and denoted --
Stopping point selection. Although NoiseCluster can applied to fully trained model, we find that further training on the original noisy data does not constantly improve the representations after the training becomes stable. Therefore, to reduce training time, we incorporate our method into the middle stage of the training process.
4.2 Label Correction
Simply removing examples U can result in the loss of many important examples. address this challenge, treat examples in potentially mislabeled group Uf cohesive unit, where all
5

---

(a) Set distance
(b) Mean distance
Figure The figure compares two methods for quantifying the similarity between two sets. Compared to mean distance shown in (b), set distance can more accurately measure the similarity between potentially mislabeled group and subset of similar examples within the class sets, particularly when the classes have high diversity.
examples share the same label. We then assess the similarity between this group and the surrounding examples by computing distances and assigning examples in the group to the class with the most similar examples. Specifically, due the clustering effects of long-traines representations, we treat the examples in UC as set, and form multiple class sets O₂, from the surrounding examples, where b E (1, ..., C}. We measure the distance between US and O₂ using set distance, which is defined as the minimum Euclidean distance between any feature 2g: Zp from sets Ug and Os, as depicted in Figure The formula for set distance is defined as follows:
d(U Ou) = inf {ll2g 2p112)
pEUF,-QEO,
where I1-I12 indicates L2 Norm. In practice, to mitigate the potential impacts of mislabeled examples close to the set Ug. we utilize a specified number of closest distances, denoted ClosePoint and compute their average. We then correct the labels in Ug with the category of the class set with the closest distance, d,s, which is defined as follows:
dRb inf
(d(U), 0&))-
bE(L.....C)
After acquiring the corrected data, we use them continue training the network for the rest of the epochs. The whole process NoiseCluster is summarized in Algorithm I
Combine with SSL. Semi Semi-supervised learning (SSL) has enabled numerous methods to attain impressive benehmarks in LNL. 30 26
NoiseCluster can also take advantage of SSL techniques
to enhance its performance. Specifically, for each class c, DBSCAN divides features 2, into K distinct groups Uj and assigns group index of to each feature ξf Examples that cannot be clustered into any group are regarded as "noise". To integrate SSL, we treat the DBSCAN noise as the unlabeled data and the remaining examples as the labeled data. Accordingly, unlabeled data sets as follows:
summarize the labeled and
{
D: = (1r Vil of = K.} of # 1, Kg)
(3)
D. (2i
Combine with early stopping. Although NoiseCluster is primarily designed to address SDN, it can be seamlessly integrated with early stopping. To tackle a noisy dataset containing multiple types of label noise, we propose a two-stage training process. In the first stage, we employ early stopping to select confident data, thereby filtering out label noise that is sensitive early stopping. In the second stage, we apply NoiseCluster to the confident data to identify and correct mislabeled examples in SDN. This two-stage strategy allows us to effectively address the challenges posed by a mixture of various types of label noise, resulting in enhanced performance on noisy datasets.
5 Experiments
5.1 Experimental Setting
Datasets. We evaluate the performance of NoiseCluster synthetic dataset, CIFAR20-SDN, and real-world dataset, Clothing 1M [52]. To ensure fair comparison with existing methods, we follow the settings of the previous methods 33 34 and conduct two sets of experiments on the synthetic

---

Algorithm 1: NoiseCluster
Input: Network fo; Final layer fc; Noisy training dataset D(X, Y); Number of epochs for long-trained N: Class number C: DBSCAN Eps and Min for i .,...,.. do
Train fo and f& D(X,Y); 2 + LSN ((x())( for cr C do
// standard training
UK (BSCAN( Eps, Mts)
// /ideettffy SDN
for Ug in UK do if Ug + largest then Compute set distance with Eq. Update Y in Ug with Eq. (2)
Di f D UU
Continually train fo and f& on Dj for the rest of the epochs.
Table 2: Comparison with state-of-the-arr methods without SSL CIFAR20-SDN. "-XX" indicates the noise rate. The mean and standard deviation computed over five runs are presented.
METHODS
SDN-12
SDN-16
SDN-18
SDN-20
CE [9
70.42=0.32 0.32
67.38
66.28+0.51
65.23+0.66 0.60
JOINTOPTIM 43
0..48=0.33
67.69+0.37
06.51
65.45+0.20
DMI 53
70.67=0.64
66.95+0.61
65.86+0.32
6..35+0.52
TOPOFILTER
69.0411044
64.78:10.49
63.52.10.36
62.9710.62
PTD-R-V 50
70.85110.44
66.45:10.00
65.2910.477
64.4110.20
VOLMINNET 32
72.45-0.388
67.78110.43
66.2011.24
65.20.10441
CDR 4
72.61=0.36
68.87
67.7940.28
67.00
BLTM-V
155
70.46=0.47
66.5720.50
64.76100.2
63.9611046
NOISECLUSTTER
80.20:0.80
78.77+1.01
77.01
72.67+10.68
dataset: with semi- supervise learning (SSL) and one without SSL. In experiments without SSL, we reserve 10% of the training data as the validation set, while we utilize the entire training data for experiments with SSL. Clothing-IM contains over one million images gathered from the Internet. Additional details on implementation can be found in Appendix
Network structure and optimization. For CIFAR20-SDN,
employ ResNet-34 19 for experi-
ments without SSL and PreAct R8 (20) for experiments with SSL. During optimization, we train the model for 300 epochs, using learning rate of 2 x 10 single cycle of cosine annealing 337. momentum of 0.9, and a weight decay of 5 x 10-4, We utilize a batch size of 128 and a stopping epoch of 80, with ClosePoint value of 20. For DBSCAN hyperparameters, Eps and MinPts are set to 0.02 and 100, respectively.
For Clothing MM, employ ResNet-50 pre-trained on ImageNet[29 28 and train the network for only one epoch. Specifically, we first utilize 95% of the training data to train network. Then, we use early stopping to select confident data from the remaining 5% of the training data and apply NoisyCluster to this data, For optimization, we use SGD with a learning rate of 0.01, a momentum of 0.9, weight decay of 10-4, and a batch size of 64, ClosePoint, Eps, and MinPts are set 20, 0.04, and 100, respectively.
5.2 Comparison with State-of-the-art Methods
Approaches without SSL. To begin our comparison, we evaluate our method against state-of-the- (SOTA) approaches that do employ SSL. As shown in Table it becomes evident that many of the methods struggle with SDN, with some even underperforming in comparison to standard training with Ponso-En (CE) loss. CDR 49 performs better than its counterparts, but requires known noise rate, which remains challenging to accurately estimate in SDN. Our method outperforms all baselines, delivering superior performance. Specifically, our proposed method surpasses CDR by 5.67% in classification accuracy when noise rate is 20%, and the margin further extends to

---

Table Comparison with state-of-the-an methods with SSL on CIFAR20-SDN. "-XX" indicates the noise rate. The mean and standard deviation computed over five runs are presented.
METHODS
SDN-12 12
SDN-16
SDN-18
SDN-20
CE 120
74.27
70.53
69.27
68.50:10.11 19
MIXUP
55
75.83:10.41
71.65:0.117
70.29=0.24
69.37 17
ELR+ 14
75.22=10.7
71.0910.30
70.1810.23
69.55=0.15
DIVIDEMIX BU
79.06=0.226
72.80+0.30
70.66:0.51
70.27
F-DIVIDEMIX [26
79.71
74.00+0.83
71.6910.22
69.59=0,44
PES SEMI 3
80.27+0.30
73.95+0024
72.78+0.22
71.72+0.45
NOISFCLUSTER+
82.20+00.33
80.35+1.22
0.85
73.54+0.72
92
SON-18
Sym-18
Memoroed percantage
4000050
Someet
vesides
Figure 5: Evaluation memorized percentage per class obtained with DivideMix CIFAR20-SDN (SDN) and symmetric label noise (Sym). "-XX" indicates the noise rate.
Table 4: Classification accuracy (%) on Clothing M.. Baseline results are taken from the original papers. indicates the results obtained with ensemble model.
CE
DMI
JOINTOPTIM
TOPOFILER
PTD-R-V
VOLMINNET
69.21
71.67
72.16
74.10
72.46
72.42
BLTM-V F-DIVIDEMIx*
PES
DIVIDEMIX
ELR+*
NOISECLUSTER
73.39
74.37
74.64
74.76
74.81
75.51
7.59% for noise rate of 12%. Furthermore, in comparison with poise-modeling-basee methods, our approach outperforms the state-of-the-art VolMinNet by over 7% across four noise rates.
Approaches with SSL. We integrate method with semi-supervised learning (SSL) techniques, denoted as NoiseCluster+ and compare against SOTA approaches that also leverage SSL. To ensure fair comparison, include the results of MixUp [59], which is used by all basclines to boost performance. The results are summarized Table Interestingly. our observations indicate that SSL contributes marginally performance enhancement [3 shows only a modest advancement over MixUp. In contrast, methods based on representations, such F-DivideMix [26] and PES semi [3, deliver better outcomes by avoiding the use of the final layer, which is prone to overfitting to SDN. Conclusively, our method, NoiseCluster+, consistently outperforms all baselines by substantial margin, underscoring that the proposed techniques the primary factors leading to the improvements.
CIFAR20-SDY Notably, DivideMix
To better understand why current SOTA methods face difficulties when dealing with SDN, conduct experiments with DivideMix and compute the ratio of confident examples that include mislabeled data to the total number of mislabeled examples in the training dataset at the end of training. As shown in Figure 5 DivideMix performs well on merely three classes. For the remaining classes, we that the confident data contains substantial number of mislabeled examples, confirming that early stopping and other techniques utilized in DivideMix cannot effectively address SDN issues.
Real-world dataset. We perform experiments on real-world dataset, Clothing M, which is widely used as a benchmark due its challenges. Many existing methods, though effective on noisy synthetic datasets, exhibit limited impact Clothing M.. Our investigation suggests that this issue stems

---

(a) Sleep T-Shirt(X:0.45, I:O.I) (b) Leather Jacket(X:0.75, Y:0.6) (c) Down Vest(X:0.9, Y:0.3)
Figure SDN in Clothing M. (a) The first row displays images from the "T-Shirt" class. The second row shows images from the "Underwear" class. The third and fourth rows present images that are mislabeled as "Underwear".. (b) The row order is as follows: "Jacket", "Windbreaker" (correct name:"trench coat"), "Leather Jacket". (c) The row order is as follows: "Down coat", "Vest" (correct name:" singlet" or "tank top"). "Down Vest". The coordinate values correspond to Figure (b). is worth noting that some label names within ClothingIM M are inaccurate due translation issues. After comparing images in the test set and Chinese label names, we put the correct names in parentheses.
90
Tesn Accursey
5D8-12
SDN
SDN
SDN-20
50
75
300 MinPss
125
130
0.008
0,000
0.003
0.224
30
20
50
Eas
CossPoint
Figure 7: Ablation analyses of MinPts, Eps, and ClosePoint.
from the presence of SDN in Clothing IM, leading suboptimal performance the test data. As illustrated in Table 4 NoiseCluster significantly outperforms SOTA methods, achieving an accuracy of 75.51% without resorting to talf/semi-superviie learning ensemble network techniques.
To delve into the existence and underlying causes of SDN real-world datasets, we manually generated t-SNE images, strictly adhering to the experimental process outlined in Section We then showcase examples within clusters that are predominantly mislabeled. As Figure iillustrates these mislabeled examples clearly belong to specific sub-classes, yet they share common characteristics with their associated noisy classes. These observations confirm the pervasive presence of SDN in real-world datasets. For additional details and further results, please refer to Appendix D.I
5.3 Ablation Study
Sensitivity of hyper-parameter We perform ablation studies on CIFAR20-SDN with noise rates of 12% and 20%. NoiseCluster employs DBSCAN [2], which involves two hyperparameters, Eps and MinPts. We find that the selection of these two hyperparameters is relatively straightforward when applied to feature representations due to the substantial margin between classes. Additionally, introduce a hyperparameter named
to reduce the negative impacts of mislabeled examples, ClosePoint, which controls the number of points used to compute the set distance. The results shown in Figure indicate that ClosePoint, along with Eps and MinPts, has negligible impact the final outcomes, indicating robustness against hyperparameter changes.
Self-supervised pretrained model. As self-supervised Learning methods continue to mature 18 15
61 4. the utilization of pretrained models from this approach has emerged as a promising strategy for addressing noisy labels, particularly since they are unaffected by such labels during the pre-training

---

100
100
100
Nemortes percensage
Memortee
Clean
Milebeledd
Milebeleed
Mislabeled
Epocht
Epocha
Scratch
(b) MoCo V2
(c) RSA
Figure 8: Examining memorization trends when training from scratch compared to training with self-supervised pretrained models on CIFAR20-SDY, with an 18% noise rate. These models were pre- trained with MoCo V2 and RSA on the same dataset for 200 epochs, resulting in linear classification accuracies of 63.02-4 0.31 and 77.67 = 0.27, respectively. However, as the performance of pretrained models increases, the speed of overfitting to mislabeled data also becomes faster. The mean and standard deviation (percentage) computed over three runs are presented.
Table 5: Comparison with/without correcting labels on CIFAR20-SHP " "-XX" indicates the noise rate. The mean and standard deviation (percentage) computed over five runs are presented.
METHODS
SDN-12
SDN-20
OURS
80.20
72.67=10.68 78.68:10.61
OURS W/O CORRECTION
71.42:10.54
This raises the pivotal question: Does employing self-supervised pretrained model simplify the addressing of SDN? While thorough investigation will be reserved for future research work, the succinct answer is no. As shown in Figure self-supervised pretrained models offer initial advantages by facilitating rapid learning of clean examples. However, this rapid learning also leads to the quick memorization of mislabeled examples. Furthermore, after only training for several epochs, the gaps between the memorization rates for clean and mislabeled examples become much smaller than those observed when training from scratch, suggesting that early stopping with seff-supervised pretrained models less effective. We hope that our research will inspire a re-thinking of utilizing self supervueee pretrained models in noisy datasets.
stage [1 54 21 60
Importance of label correction. mitigate the bias introduced by learning with SDN, we employ label correction technique. Given that DNNs have already memorized many mislabeled examples during the early learning phase, label correction becomes indispensable. In Table [5] we present comparison of results with and without applying label correction and observe performance improvement of over 1% for both low and high noise rates. These findings emphasize the importance of the label correction technique in addressing SDN.
6 Conclusion
In this paper, empirically examine subclass-dominant label noise (SDN), new type of label noise that commonly occurs in real-world datascts. SDN presents a significant challenge to conventional methods that rely early stopping. To overcome this issue, we reveal that long- traiitrined representa- tions are more effective at capturing the semantic relationships between noisy examples. Building on this insight, we propose novel approach, NoiseCluster, to identify and correct mislabeled examples in SDN. We hope our work will inspire the research community and pave the way for addressing label noise in real-world applications.
The primary limitation is that our proposed method effective in addressing SDN cases where mislabeled examples have notably different semantics from correctly labeled examples. However, it could struggle situations where correctly labeled and mislabeled examples exhibit close semanties, especially without any prior knowledge. We leave it as open research topic for future investigation.
10

---

Acknowledgments and Disclosure of Funding
The authors would like to thank the anonymous reviewers and the meta-reviewer for their constructive feedback and encouraging comments on this work. Yingbin Bai was supported by CSIRO Data61. Erkun Yang was supported in part by the National Natural Science Foundation of China under Grant 62202365, Guangdong Basic and Applied Basic Research Foundation (2021 A1515 10026), and Nat- ural Science Basic Research Program of Shaanxi (Program No.2022JQ-608), Jun was supported by the Natural Science Foundation of China (62276242), National Aviation Science Foundation 2222207107800111 CAAI-Huawei MindSpore Open Fund (CAAIXSJLJJ-200) 1-2211000-02--1..... CAAIXSJLU- 2022-001A), Anhui Province Key Research and Development Program (202 1021000200007)) USTC- IAT Application Sci. & Tech. Achievement Cultivation Program JL065210011 Y). Sci. & Tech. Innovation Special Zone
Bo Han was supported by the NSFC Young Scientists Fund No. 62006202, NSFC General Program No. 62376235, and Guangdong Basic and Applied Basic Research Foundation No. 2022A151 1652. Tongliang Liu was partially supported by the following Australian Research Council projects: FT220100318, DP220102 LP220100527, LP220220949, and IC190100031.
100---1---00110004---0
References
[1] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj. Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In ICML, pages 233-242, 2017.
[2] Yingbin Bai and Tongliang Liu. Mc-momentum: Extracting hard confident examples from noisily labeled data, In ICCV, 2021.
[3] Yingbin Bai, Erkun Yang. Bo Han, Yanhua Yang. Jiatong Li. Yinian Mao. Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. In NeurlPS, pages 14392-24403. 2021.
[4] Yingbin Bai, Erkun Yang, Zhaoqing Wang, Yuxuan Du, Bo Han, Cheng Deng, Dadong Wang, and Tongliang Liu. Rsa: Reducing semantic shift from aggressive augmentations for self-supervised learning. In NeurlPS, pages 21 21122-21 141, 2022.
[5] David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurlPS, pages 5050-5060, 2019.
[6] Irving Biederman. Recognitio-bby-compnnnns a theory of human image understanding. Psychological review, page 115, 1987.
[7] Irving Biederman, Robert J Mezzanotte, and Jan C Rabinowitz. Scene perception: Detecting and judging objects undergoing relational violations. Cognitive psychology. pages 143-177, 1982.
[8] Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In ICMI., pages 1062-1070. 2019.
[9] De Cheng, Yixiong Ning, Nannan Wang, Xinbo Gao, Heng Yang. Yuxuan Du, Bo Han, and Tongliang Liu. Class-dependent label-noise learning with cycle-consistency regularization. NeurlPS, 2022.
[10] Hao Cheng, Zhaowei Zhu, Xingyu Li. Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. ICLRR, 2021.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
[12] Martin Ester, Hans- Kriegel, Jörg Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD-96), pages 226-231, 1996.
[13] Di Feng. Christian Haase Schüitz, Lars Rosenbaum, Heinz Hertlein, Claudius Gläser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer. Deep multi-modas object detection and semantic segmentation for autonomous driving: Datasets, methods. and challenges. IEEE Trans. Intell. Transp. Syst., pages 13600, 2021.
11

---

[14] Aritra Chosh and Andrew S. Lan. Contrastive learning improves model robustness under label noise. In CVPR, pages 2703-2708. Computer Vision Foundation / IEEE, 2021.
[15] MannBBatien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent- A new approach to self- supervised learning. In NeurlPS, pages 21271-21284, 2020.
[16] Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor Tsang, James T. Kwok, and Masashi Sugiyama. A survey of label-noise representation learning: Past, present and future. CoRR, abs/201 1.04406, 2020.
[17] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS, pages 8527-8 5527-8537, 2018.
[18] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsuper- vised visual representation learning. In CVPR, pages 9726-9733 2020.
[19] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages
2016.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, pages 630-645, 2016.
[21] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning improve model robustness and uncertainty. In NeurlPS, pages 15637-15648, 2019.
[22] Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing out-of-distribution examples via augmenting content and style. In ICLRR, 2023.
[23] Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong, Universal semi-supervised learning. In Advances in Neural Information Processing Systems, volume 34, pages 26714 267251 2021.
[24] Zhuo Huang, Miaxxi Zhu, Xiaoho Xia, Li Shen, Jun Yu, Chen Gong, Bo Han, Bo Du, and Tongliang Liu. Robust generalization against photon-limice corruptions via worst-case sharpness minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16175- 16185, 2023.
[25] Davood Karimi, Haoran Dou, Simon K. Warfield, and Ali Gholipour. Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis. Medical Image Anol., page 101759, 2020.
[26] Tachyeon Kim, Jongwoo Ko, Sangwook Cho. Jinhwan Choi, and Se-Young Yun. FINE samples for learning with noisy labels. In NewrlPS, pages 24137-24149, 2021.
[27] Alex Krizhevsky, Geoffrey Hinton, ct al. Learning multiple layers of features from tiny images. Technical report, 2009.
[28] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, pages 1097-1105, 2012.
[29] Jingling Li, Mozhi Zhang, Keyulu Xu, John Dickerson, and Jimmy Ba. How does a neural network's architecture impact its rohustness to noisy labels? In NeurlPS, pages 9788-9803, 2021.
[30] Junnan Li, Richard Socher, and Steven C. H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In ICLR, 2020.
[31] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Lue Van Gool. Webvision database: Visual learning and understanding from web data. CoRR, abs/1 708.0222. 2017.
[32] Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. ICML, pages 6403-6444 6413, 2021.
[33] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, pages 740-755, 2014.
[34] Sheng Liu, Jonathan Niles Weed, Narges Razavian, and Carlos Fernandez- Early-learning regular- ization prevents memorization of noisy labels. In NeurIPS, pages 20331-20343 2020.
[35] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. JEEE Transactions on pattern analysis and machine intelligence, 38(3):447 461, 2016.
12

---

[36] Yang Liu. Understanding instance-level label noise: Disparate impacts and treatments. In ICML, pages 6725-6735, 2021.
[37] Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts, In ICLR, 2017.
[38] Yangdi Lu, Yang Bo, and Wenbo He. Noise attention learning: Enhancing noise robustness by gradient scaling. In NeurlPS, 2022.
[39] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11):2579-2600 2008.
[40] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, pages 2233 2241, 2017.
[41] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Chriistiano. Learning summarize with human feedback. In NeurlPS, pages 3008-3021, 2020.
[42] Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICMI, pages 6105-61 14, 2019.
[43] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In CVPR, pages 5552-5566 2018.
[44] Anne Treisman and Garry Gelade. feature-integration theory of attention. Cognitive psychology. pages 97-136, 1980.
[45] Peter Welinder, Steve Branson, Serge Belongic, and Pictro Perona. The multidimensional wisdom of crowds. In NeurlPS, pages 2424 2432, 2010.
[46] Pengxiang Wu, Songzhu Zheng. Mayank Goswami, Dimitris Metaxas, and Chao Chen. topological filter for learning with label noise. In NeurlPS, pages 21382-21393, 2020.
[47] Songhua Wu, Tongliang Liu, Bo Han, Jun Yu, Gang Niu, and Masashi Sugiyama. Learning from noisy pairwise similarity and unlabeled data. Journal of Machine Learning Research, 23(307):1-3,, 2022.
[48] Songhua Wu, Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Nannan Wang. Haifeng Liu, and Gang Niu. Class2simi: A noise reduction perspective on learning with noisy labels. In ICML, pages 11285-12295 1195.. PMLR, 2021.
[49] Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust carly-leaming: Hindering the memorization of noisy labels. In /CLR, 2021.
[50] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In NeurlPS, pages 1597-7610 2020.
[51] Xiaobo Xia. Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurlPS, pages 6835-6846 2019.
[52] Tong Xiao, Tian Xia, Yi Yang. Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. CVPR, pages 2691-2699, 2015.
[53] Yilun Xu, Peng Cao. Yuqing Kong, and Yizhou Wang. L dmi: A novel information- loss function for training deep nets robust label noise. In NewrlPS, pages 2223,6233, 2019.
[54] Yihao Xue, Kyle Whitecross, and Baharan Mirzasoleiman. Investigating why contrastive leamning henefits robustness against label noise. In ICMI., volume 162, pages 24851-24871, 2022.
[55] Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance- dependent babes-label transition matrix using a deep neural network. In ICML, pages 25302 25312, 2022.
[56] Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual T: reducing estimation error for transition matrix in label-noise learning. In NeurlPS, pages 7260-7271, 2020.
[57] Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with biased complementary labels. In ECCV, pages 69 85, 2018.
13

---

[58] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
[59] Hongyi Zhang. Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[60] Evgenii Zheltonozhskii. Chaim Baskin, Avi Mendelson, Alex M. Bronstein, and Or Litany. Contrast to divide: Self-supervised pre-training for learning with noisy labels. In WACV, pages 387-397, 2022.
[61] Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang. Xiaogang Wang, and Chang Xu.
Ressl: Relational self-supervised learning with weak augmentation. 2021.
[62] Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance-dependent label noise. In CVPR, pages 101 13-10123, 2021.
14

---

A CIFAR20-SDN
A.1 Construction Details
dubelass-dominan label noise (SDN) is designed to mimic human labeling errors, which often emerge from distinct characteristics of examples. To identify such examples, we employ CIFAR- which provides two types of labels: class labels and sub-class labels. Samples belonging to the same subclass are considered to possess distinct characteristics. During the generation of the SDN dataset, randomly flip specified number of the last subclass labels within each class to the subsequent class. For instance, the labels of the "whale" subclass within the "aquatic mammals" class probabilistically flipped to the "fish" class. Table 6 relationships between classes and sub-classes in CIFAR-100. For simplicity, we refer to this newly generated SDN dataset as CIFAR20-SDN.
illustrates the flipped sub-classes and the
Table 6: The table outlines the comprehensive structure of the classes within CIFAR20-SDN. Sub- classes marked in bold denote those from which examples are selectively flipped.
Class name
Abhreviation
Sub-classes name
aquatic mammals fish
aquatic
beaver, dolphin, otter, scal, whale aquarium fish, flatfish, ray, shark, trout orchids, poppies, roses, sunflowers, tulips bottles, bowls, cans, cups, plates apples, mushrooms, oranges, pears, sweet peppers clock, computer keyboard, lamp, telephone, television bed, chair, couch, table, wardrobe bee, beetle, butterfly, caterpillar, cockroach bear, leopard, lion, tiger, wolf bridge, castle, house, road, skyscraper cloud, forest, mountain, plain, sea camel, cattle, chimpanzee, elephant, kangaroo fox, porcupine, possum, raccoon, skunk crab, lobster, snail, spider, worm baby, boy, girl, man, woman crocodile, dinosaur, lizard, snake, turtle
fish
flowers
flowers
food containers
containers
fruit and vegetables
fruit
household electrical devices household furniture
electrical
furniture
insects
insects
large carnivores
carnivores
large man-made outdoor things large natural outdoor scenes large omnivores and herbivores medium-sized mammals
man-made
natural
omnivores
m-mammals invertebrates people
non-insect invertebrates people
reptiles
reptiles
small mammals trees
s-mammals
hamster, mouse, rabbit, shrew, squirrel maple, oak, palm, pine, willow bicycle, bus, motorcycle, pickup truck, train lawn-mower rocket, streetcar, tank, tractor
trees
vehicles
vehicles
vehicles
vehicles 2
A.2 Diverse Challenges across Different Classes
Rewers
34
15
36
55 26
04
15 04
tHe Pareeraag
Madte rate diitre
20
25 29
21
w.. 6600oo
Manne Fote cillort
vahidaa
Mutic r0be 11 druuee
Nove: rio: AMMroo
Noxt F40 ColAeC
No 900 14
Masic 1o deleve
Musc ruli cilee)
Figure 9: The figure shows the effectiveness of early stopping across different classes and various noise rates. When the noise rate in the dataset is small, early stopping performs well in all classes. However, when it is over 50%, memorized mislabeled samples increases rapidly.
15

---

CIFAR2O-SDN presents multifaceted landscape with its array of sub-classes, paving the way for comprehensive study of SDN and a deeper insight into the intricacies of early stopping, In our experiments, adjust the overall noise rate from 0% to 20%. This implies that within each specific subclass, the noise rate varies from 0% to 100%.
As illustrated in Figure D the effectiveness of early stopping in addressing SDN is notably incon- sistent, heavily contingent on the specific subclass. Our analysis identifies two primary factors that significantly influence the success of early stopping. First, when the mislabeled subclass shares substantial semantic similarities with the clean class, distinguishing mislabeled examples from the clean class becomes challenging. For instance, mislabeling an instance from the "whale" subclass as a "fish" class is difficult discern from other "fish" instances. Second, when classes have low diversity, the network tends to identify outliers, thus facilitating the detection of mislabeled examples, such as "trees" " and "flowers". Conversely, distinguishing between correctly labeled and mislabeled examples becomes considerably more challenging in classes with higher diversity, such as "insects".
A.3 Clusters Effects
We train a ResNet-18 on CIFARZO- SDN for 200 epochs with an overall noise rate of 18% (SDN-18) Afterward, we utilized t-SNE to visualize all 45, 5,,000 training examples. By comparing Figure T0 (a) and (b), we can pinpoint the noise locations, specifically evident in the small clusters. Notably, the vast majority of the mislabeled examples tend to cluster together and usually maintain distance from their noisy class. This pattern suggests that clustering algorithms, such as DBSCAN, can effectively identify them. However, when the semanties of mislabeled examples closely match those of the noisy class, these mislabeled examples' clusters may not be distinct from the main cluster of the noisy class, which is limitation of our proposed method.
(a) CIFAR20-SDN with noisy labels
(b) CIFAR20-SDN with clean labels
Figure 10: The figure shows the distribution of training examples after 200 epochs on CIFAR20-SDN. We can find that the large clusters represent examples with correct labels, whereas the smaller clusters indicate those with incorrect labels.
B More Details about Experimental Settings
Data preprocessing. For all experiments, employ simple data augmentation, including random crop and horizontal flip. For the Clothing- 1M dataset, images are initially resized to a resolution of 256 x 256 pixels. Subsequently, random crop is applied to produce images of 224 224 pixels, which are then followed by a random horizontal flip.
Hyper-parameters of semi-supervised learning. We follow 30 3 and combine NoiseCluster with the semi-supee learning method, MixMatch [5). For experiments with SSL, we set K 2, T = 0.5, and X, = O. begins at and drops to 0.75 after half of the total number of epochs.
16

---

Clothing Due to the large size of the Clothing M dataset [52], which consists of one million training examples. the computational time for t-SNE is significantly extended. This volume of data also results in changes to the sypeer-praametes. To address these issues, we split the Clothing M training data into two parts: a larger subset containing 950,000 examples and smaller one with 50,000 examples. We initially employ the larger subset for training model with early stopping. Subsequently, based on the trained model, we select and correct confident data from the smaller subset. This corrected data is then used for the remainder of the training process. Note that the model used in the Clothing M experiments has been pre-trained with the ImageNet-Ik Ik dataset [28], leading to the generation of separable representations. As result, late stopping is omitted in this case. When applying NoiseCluster to a model trained from scratch, it is advisable to employ late stopping.
C
Additional Experiments
C.1 Ablation Study for Stopping Epoch
The effectiveness of NoiseCluster relies on long-trained representations derived from model trained with later stopping. which stops the training process significantly later than what is common with early stopping. This raises two pertinent questions: How is the stopping epoch determined for later stopping? And does it display the same sensitivity early stopping? To address these questions, we conduct a series of ablation experiments, adjusting the stopping epoch from 20th to 200th epochs.
50
80
80
Test Accurccyyc
Test Test Accuracy
70
70
60
60
NoiseCluster
NolseCluster
Best Baseline
150
150
Stopping Epoch
Stopping Epoch
(a) Noise rate 12%
(b) Noise rate 20%
Figure 11: Ablation analyses of stopping epoch. The mean and standard deviation (percentage) computed over five runs are presented.
As shown in Figure I1 NoiseCluster does not perform well with representations obtained at the 20th epoch, indicating that the effectiveness of NoiseCluster hinges on long-trained representations. Additionally, we can observe that NoiseCluster maintains a stable performance between the 50th and 100th epochs. This span considerably wider compared to early stopping, which typically require precise stopping epoch operate within relatively narrow range 30 34 3
To delve
deeper into the impact of the stopping epoch, we expand the training period for later stopping, and stop the training up to the 200th epoch. Despite a slight drop in the performance of NoiseCluster, attributable to the degradation of representation quality, it remains superior to the best baseline, CDR These experiments suggest that the choice of the stopping epoch in NoiseCluster can be made with considerable flexibility, displaying minimal sensitivity and negligible impact on the performance of NoiseCluster over broad range.
C.2 Explore Noise-modeling- based Approaches on SDN
In this section, we further investigate into why noise- wwwemoogodoeesssseeeees approaches are ineffective for SDN. Methods based on noise modeling build a transition matrix between the clean distribution and noisy distribution 440 51). We utilize the state-of-the-art method, VolMinNet (32), CIFAR20-SDN with noise rate 18%. Following the original paper 32), set 7 to -2 and -4.5, respectively.
17

---

[[0.15 8.88 0.82
9.02
0.02
6.63
0.01
8.83
0.62
8.85
8.83
8.
8:8
[6. A8 0.40 0.65
[9.62 8,
8.
[e. 82 Ra
82
[6.02 8. 82 8
62
82
8.82
6.02
[8.82 8.82 8
8:88
82
82
8.82
[6.69 8.89
[6.68
[9.63 8,
[6, R.
[6.64 8.82
8. 6.82]
[8.64 8.82 8.
88
8. 8.82]
[0.09
[6.02
[0. 84
08
[6.82 8
8.17 6.82
[8. 84 8. 82
8.5
(a)
[[ovs 0.81
8.01
0.01
0.61
0.02
6.61
0.61
8.01
6.81
9 9.
9.91
0.01
0.61
0.80
8,81
8,61
8681
0.01
0.01]
[6.01 8. a.81
0.81
8.
81
A1
8.81
8.81
8
81
8
8.
8.81
0.81
0
0.81
0.01]
[e.e1 8.
8.85
0.
01
0.
61
81
8.
8.81
8 81
8 81
6
8.
8.
8.
8.81 0.01
[e.e1
8.
6 11
6
8
9 8688
[6.61 8.81
8.81
6.61
81
62
86
81
8
81
88
88
81
8,
8.
88 8681 6.01
[6.01 8. 2.31
9.
81
81
8
AM
81
81
8 81
01
8.
M
8. M1
8.81
8.81
a.
a.a1 0.a1]
[6.01 8.81 8.81
0.
81
88
81
81
8.81
8.81
8. 0.81 6.01
[e.e1
8.81
[6.61 8.81
8.81
6.61
81
0.
6.
6.61
8.
8.81
8.89
8.
8.
6.61
8.81
8.81
[e.01 a. 2.31
0.
01
01
8.
8.
8.61
8.81
a.a1
[6.01 8.81 8.81
0.
0.61
8.
8.61
6.61
[8.61
6.61
81
[6.61 81 81
6.
8.61
6.61
6.
8.81
8.81
8.81
8.81
8.
8.81
8.61
8.81
8.36
6.61
[8.01
81
8.
8.
81
81
8.
8:81
8.81
8.81
8
8.81
(b) 4.5
Figure 12: (a, b) display the best estimated transition matrices during the training process with π -= 2 and T -4.5. The estimated noisy class paast-postrrior probabilities are seen to be widespread across all classes, which inconsistent with the generation of CIFAR20-SDN.
As depicted Figure is evident that T holds the capability to modulate the extent of estimated noisy pass--oottrrio probabilities within the transition matrix. For example, with π 22 VolMin- Net predicts higher noisy class-posterior probabilities in the transition matrix compared to that with T = -4.5. However, neither of these 7' settings successfully enable an accurate estimation of the transition matrix. Consequently, these inaccurately estimated transition matrices result in inferior performance on SDN.
C.3 Training Time Comparison
Table Training time comparison on CIFAR20-SDN with SDN-12, All methods run on four core CPU and single Nvidia V100.
NOISECLUSTER
JOINTOPTIM
TOPOFILER
CDR
1.7H
2.5H
1.5H
5.5H
NOISECLUSTER+
DIVIDEMIX
PES SEMI F-DIVIDEMIX
2H
5.5H
3.1H
14.2H
We evaluate the efficiency of our proposed method with of-te- baselines. 7 the total training time on CIFAR20-SDN. We can see that training time of NoiseCluster comparable to that of TopoFiler [46] and faster than JointOptim 43 and CDR [49]. Furthermore, NoiseCluster+ exhibits substantial efficiency compared with other baselines that employ SSL. For instance, it requires less than half the time needed by DivideMix (30].
18

---

D Exploring SDN in Real-world Datasets
Due to space constraints in the main paper, we presented only a limited number of SDN examples. In this subsection, we provide additional visual evidence to further demonstrate the presence of SDN in the Clothing IM and WebVision datasets.
04
30
14
20
300t Greweor
Crllen
O.ifinn 1o0ds
Madinooo Dove Cups
Houes
Down Cast
2010
Drus
(a) All examples, noisy labels
(b) Confident examples, noisy labels
18
Citorer
Swortt
Jedtet
(c) Confident examples, clean lahels
(d) Mislabeled examples in confident set, clean labels
Figure 13: (a) We visualize the features of all the examples in the evaluation set with noisy labels. (b) By removing unconfident examples, the features of confident examples are better separated. (c) We visualize the features of confident examples with clean labels. (d) We visualize the features of
mislabeled examples in the confident set with clean labels. It is evident that the majority of mislabeled examples cluster together, confirming our observation of the clustering phenomenon long-traines representations.
D.I SDN in Clothing
presents step-by-step exploration process for the Clothing IM dataset, demonstrating how features of mislabeled examples are clustered together based on their semantics. Given the plethora of terms available for different clothing items, we can use accurate names to pinpoint these sub-classes. We highlight three sub-classes prominently impacted by label noise: "Sleep T-Shirt,"
Figure 13
"Leather Jacket," and "Down Vest," shown in Figures 14
15 and 16
19

---

vt
Figure 14: Subclass: Sleep T-Shirt, positioned coordinates (X:0.45, Y.O.1). The first displays images from the "T-Shirt" class. The second row showcases those from the "Underwear" class, while the third and fourth rows present images mislabeled as "Underwear". For the mislabeled pictures, it's evident that these images depict a type of shirt meant for home use, Sleep T-Shirt. After comparing images the test and Chinese label names, we put the correct names in parentheses.
Figure 15: Subclass: Leather Jacket, positioned at coordinates (X:0.75, Y:0.6). The first row images are from the "Jacket" class, and the second row presents images correctly labeled as the "Windbreaker" class (correct name: "trench coat"). Clearly, these mislabeled images belong to the "Leather Jacket" subclass within the "Jacket" category.
We can also observe that some mislabeled examples are not clustered together, but are instead evenly dispersed in some classes. It is important to understand that in the real world, clothing categories are not solely distinguished by visual features. For instance, the features of "Sweater" and "Knitwear" intertwined, with the main distinction being the material, not the visual appearance. Similarly, the categories "Underwear" and "Vest" (correct name: ssngleet or "tank top") lead to confusion since some garments can multiple purposes.
Removing correctly labeled examples. Early stopping not only helps remove mislabeled examples but also inadvertently excludes those that are correctly labeled when SDN exists. A closer inspection of instances within the "Leather Jacket" cluster confirms the issue: out of the total examples, 313 are mislabeled, with only 23 correctly labeled. This finding indicates that more than 90% of the "Leather
20

---

LEVE BO
Figure 16: Subclass: Down Vest, positioned at coordinates (X:0.9, Y:0.3). The first row displays images from the "Down coat" class, while the second row showcases those from the "Vest" class (correct name:" singlet" "tank top"). When the "Vest" class is absent, these mislabeled images should be classified as "Down Vest" within the "Down coat" class.
Figure 17: We discover 23 instances of "Leather Jacket" within the unconfident set. Despite being correctly labeled, these images are omitted from the confident set due to using early stopping. This process results in purer cluster that predominantly contains mislabeled examples.
Jacket" examples are incorrectly labeled the original dataset. Furthermore, the implementation of early stopping leads to the removal of the small fraction of correctly labeled examples from the confident set, as shown in Figure I7| pushing the noise rate from over 90% close 100%.
D.2 SDN in WisVin
Table 8: Top-1 performance of re-implemented DivideMix models alongside the pre-trained models, assessed on the ImageNet and WebVision validation datasets. By employing an ensemble of pre-trained models, we generate pseudo clean labels to effectively identify mislabeled examples.
Methods
Output Class
WebVision
ImageNet
DivideMix
50
77.34
75.2
DivideMix (reproduce)
50
77.04
74.24
Regnet_y Leanne___12____ 128gf (IMAGENETIK SWAG (AG__2EEVV)
1000
86.64
91.12
ViT ViT_H_14 14 (IMAGENETIK SWWGSWWAEEE.E..S (2E_VI)
1000
86.08
91.2
Ensemble pretrained models
1000
86.96
91.6
To extend our evaluation of SDN prevalence, engage in experiments using another real-world dataset: WeiVision V1.0 BI. Following the setting presented [8) we utilize the first 50 classes of WebVision. known as "mini WebVision". To demonstrate the flexibility of integrating with existing
21

---

early stopping methods, we employ DivideMix to select confident examples and extract long- trained representations from two networks, which have been trained for 80 epochs. The extracted features are presented in Figure [8] We utilize an ensemble of pre-trained models to produce pseudo clean labels, and their performance is depicted in Table
Lagada
GRTE Drablo condia
03717 Soucls cancka
(a) Noisy labels
(b) Pseudo clean labels
Figure 18: Due to the absence of clean labels for the training data in WebVision, we employ two pre-trained models from PyTorch to generate pseudo-clean labels. While the pseudo-clean labels may be accurate, they suffice to provide general overview of the distribution of mislabeled examples. We then visualize confident features for the first ten classes using t-SNE, comparing noisy labels with the pseudo-clean ones. The category "other" represents examples where predictions from the pre-traine pre-trained models are not in the first ten classes.
In Figures 19 20 and 21 is evident that, despite clear discrepancies between mislabeled and correctly labeled images, methods based on early stopping struggle to effectively identify these mislabeled examples. Moreover, these mislabeled samples do not fall into conventional categories such as "human" or "cars". Instead, they cluster according to different criteria, sharing similarities in attributes like shape and color with their correctly labeled counterparts. By considering oft-oo-domain examples as single, unified class-termee "other", the mislabeled examples within each class can be interpreted as sub-classes of this "other" category.
22

---

Figure 19: Explore images categorized under the "Electric Ray" class. The first row showcases images sourced from the ImageNet validation set, while the second row features images from the WebVision validation set. Correctly labeled training images are in the third row, whereas the fourth and fifth rows illustrate instances of mislabeling in the training We find majority of mislabeled images are "black and white" pattern, which are close to some examples in the third row, so we call it "black and white" subelass.
Figure 20: The row structure remains consistent with the previous format. Notably, numerous cars are incorrectly classified in this category, alongside the presence of wheels and ships. The leading attribute contributing to this misclassification is likely the presence of "curves".
23

---

Figure 21: The row structure remains consistent with the previous format. The mislabeled subclass may derive from color attributes, such as pink or white, which serve as the decisive classification features for the "hen" category.
24


Reviews:
[{"review_id": "9pONRnRdcL", "reviewer": "Reviewer_7AfS", "summary": "In the paper, the authors uncover the phenomenon that mislabeled examples are quickly learned during the initial stages of training when Subclass-Dominant Label Noise (SDN) is present. This behavior hinders the effectiveness of early stopping-based robust methods. To address this issue, the authors propose an approach that does not rely on early stopping. The method involves identifying mislabeled examples by clustering the penultimate layer features and correcting them by assigning them labels of the closest class. Experiments are conducted to demonstrate the superiority of their method compared to existing approaches.", "questions": "In addition to the questions raised in Weaknesses, I have the following additional questions:\n1. Regarding Cloth1M, why was only one epoch used for training? Furthermore, what is the rationale behind training on 95% of the data first and then applying SDN with only 5%? Currently it is unclear why these modifications are made. \n2. The method doesn't appear to be specifically designed for SDN alone, as identifying mislabeled examples and the label correction part make sense for general noise as well. How does the proposed method perform under normal noise?\n3. Since the authors mentioned incorporating self-supervised learning, it would be valuable to discuss self-supervised pretraining, which has been shown to enhance robust methods [1][2][3][4]. As features are initially learned without labels, starting from a pretrained model might lower the chance of mislabeled examples in SDN being learned early. Therefore, I wonder if early stopping could potentially be effective with this pretraining even under SDN. \n\n[1] Hendrycks, Dan, et al. \"Using self-supervised learning can improve model robustness and uncertainty.\" Advances in neural information processing systems 32 (2019).\n\n[2] Zheltonozhskii, Evgenii, et al. \"Contrast to divide: Self-supervised pre-training for learning with noisy labels.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.\n\n[3] Xue, Yihao, Kyle Whitecross, and Baharan Mirzasoleiman. \"Investigating why contrastive learning benefits robustness against label noise.\" International Conference on Machine Learning. PMLR, 2022.\n\n[4] Ghosh, Aritra, and Andrew Lan. \"Contrastive learning improves model robustness under label noise.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "limitations": "The authors have discussed limitations in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.\tThe finding of the failure of early stopping under SDN is intriguing with practical implications, as SDN can be a common occurrence in practice.\n2.\tThe proposed method has significant improvements over existing methods under SDN.", "weaknesses": "Overall, while I appreciate the first part of the paper that discusses the failure of early stopping under SDN, I have some concerns and suggestions regarding the other parts:\n1. It would be beneficial to delve deeper into the phenomenon of early stopping’s failure under SDN, e.g., exploring why subclass dominance leads to wrong labels being learned quickly.\n2. The advantage of long-trained representations may not be surprising, as training progress can naturally lead to more distinguishable clusters. It seems that this property is not specific to SDN.\n3. Regarding the combination with SSL, it is unclear why there would be features that are not assigned any group index. Are these features the same as $U^c_{k-1}$ mentioned in section 4.1?\n4. The Noisecluster+ method, where labels are not corrected but instead discarded (as those examples are treated as unlabeled), achieves better performance than the class-correction based method. Does this mean label correction is not as effective?\n5. I am curious about the percentage of examples whose labels were successfully corrected by the label-correction method.\n6. In Table 5, is the method without label correction simply vanilla training? If that is the case, vanilla training already accomplishes the majority of the performance, and the performance improvements attributed to label correction appear relatively modest (1%). So the proposed method does not contribute much. \n7. ELR should also be included in Table 2, as it does not require SSL (only ELR+ does).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">** **\nQ1: It would be beneficial to delve deeper ... dominance leads to wrong labels being learned quickly.\n\nA1: Thanks for the insightful question. To explain why SDN is learned quickly, we must recall why a machine learning model can learn a generalized function and why the early stopping phenomenon exists in learning with noisy labels. In real life, ``class\" denotes a group of objects with similar characteristics, defined by a set of rules. Deep networks, designed to identify these characteristics, facilitate the classification of real-world classes. In supervised learning, the classification task involves learning a function from prior labeled samples that can accurately predict unobserved samples, in accordance with the characteristics.\n\nThis viewpoint sheds light on the early stopping phenomenon. A network can quickly learn a generalized function to recognize correctly labeled samples within a noisy dataset, as both observed and unobserved examples with clean labels share the same characteristics. On the other hand, the network encounters difficulty in finding a generalized function for mislabeled samples, especially if their generation method (e.g., random selection) fails to align with any real-world rule. As a result, the network is forced to memorize mislabeled samples individually to distinguish them from correctly labeled ones. This leads to the earlier learning of correctly labeled samples.\n\nHowever, real life provides various ways to organize objects, such as by status, color, shape, etc. For instance, in the experiments shown in Figure 1, examples within the airplane class can be divided into flying or landed categories based on their status. We flip the labels of landed airplanes to automobiles. The network can then quickly learn a generalized function to distinguish landed airplanes from flying ones by recognizing special features in landed airplanes, such as wheels or land. Overall, this may explain why SDN is learned quickly.\n\nAdditionally, we have visualized SDN examples from both Clothing1M and WebVision datasets and included them in the attached one-page PDF. We believe these images will provide more hints on why SDN can be learned rapidly.\n\n\n>** **\nQ2: The advantage of long-trained representations may not be surprising, as training progress can naturally lead to more distinguishable clusters. It seems that this property is not specific to SDN.\n\nA2: (1) Thank you for your observation. In the self-supervised learning community, it has been widely recognized that long-trained representations become more distinguishable in alignment with their inherent features. In contrast, supervised learning typically sees representations of examples forming more distinct clusters based on their labels. To the best of my knowledge, no existing paper in the field of learning with noisy labels has discussed that long-trained representations will gravitate towards their inherent features (e.g., images) rather than conforming to their (noisy) labels, when using supervised learning. \n\n(2) Yes, this property is not unique to SDN. We have demonstrated that symmetric label noise exhibits a similar phenomenon, as shown in Figure 3(d). However, the representations resulting from symmetric label noise are significantly more complex and cannot be effectively clustered in a 2-dimensional space. Research methods, including TopoFilter and ELR, have highlighted the vital importance of the early learning phase in preventing the degradation of representations trained over extended periods. Thus, long-trained representations may not be suitable for all types of label noise.\n\n\n>** **\nQ3: Regarding the combination with SSL, it is unclear why there would be features that are not assigned any group index. Are these features the same as mentioned in section 4.1?\n\nA3: Thank you for bringing up this point. There are some peripheral points in t-SNE images, which are too sparse to constitute a new cluster and are distant from established groups. Then, these points of the features are not in any group, and are regarded as unlabeled examples for SSL. For a more comprehensive understanding of DBSCAN Noise, we recommend referring to the original DBSCAN paper.\n\n\n>** **\nQ4: The Noisecluster+ method, where labels are not corrected ... label correction is not as effective?\n\nA4: There is a potential confusion. Label correction is used in both NoiseCluster and NoiseCluster+. The term \"NoiseCluster+\" denotes NoiseCluster plus MixMatch. \n\n\n>** **\nQ5: I am curious about the percentage of examples whose labels were successfully corrected by the label-correction method.\n\nA5: The mean and standard deviation computed over five runs are presented in the below table. \n\n|Method| SDN-12 | SDN-16 | SDN-18 | SDN-20 |\n|:-----|:----:|:----:|:----:|:----:|\n|confident examples                  | 41536 (361) | 41484 (455)  | 41812 (501)  | 42245 (129) |\n|unconfident examples (DBSCAN Noise) | 3464 (361)  | 3516 (455)   | 3188 (501)   | 2755 (129)  |\n|total corrected examples            | 4774 (761)  | 4914 (544)   | 5958 (562)   | 6112 (609)  |\n|successful corrected examples       | 4153 (446)  | 4422 (763)   | 5040 (800)   | 3272 (762)  |\n|successful correct rate (\\%)        | 87.50 (4.47) | 89.52 (5.65) | 84.25 (6.16) | 53.17 (8.47) |\n\n>** **\nResponses to questions Q6-Q10 can be found in General Questions due to character limitations."}, {"review_id": "0c7XvFZkC9", "reviewer": "Reviewer_t1fi", "summary": "In this work, the authors present a new type of label noise: subclass-dominant label noise. The authors show that the model trained over time can better capture such label noise in the feature space, and based on this idea, a clustering then correcting pseudo labels algorithm, NoiseCluster, is designed to identify and correct SDN. The proposed method can be combined with current label noise and semi-supervised learning methods, and is validated on the authors' constructed dataset, cifar20-SDN, and another real-world dataset, Clothing 1M.", "questions": "- It will be more clear to describe the class and sub-class relations briefly, or giving a concrete example of CIFAR20-SDN in the main paper.\n    \n- Clustering Result: More analysis of clusting effects and results could make the proposed SDn and method more convincing. For example, on the CIFAR20-SDN dataset, does the clustering result consist with the manually created SDN noise? And what about the real-world dataset Clothing 1M?", "limitations": "The authors have mentioned their limitations in the conclusion section.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is well-organized. The Introduction and related work sections are clear and well introduces the studied problem.\n    \n- According to the proposed subclass-dominant label noise characteristics, the authors constructed a corresponding dataset and proposed a methodology addressing it. The proposed approach is heuristically reasonable and applicable.", "weaknesses": "- My main concern is the existence of proposed subclass-dominant label noise (SDN) in the real world scenarios.\n    \n- The proposed method works well on the constructed dataset, CIFAR20-SDN. However, the performance improvement on real-world dataset, Clothing 1M is marginal. This makes the existence of SDN questionable.\n    \n- Lack of some analysis of clusting effects and results in experiments. Please refer to the Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">** **\nQ1: My main concern is the existence of proposed subclass-dominant label noise (SDN) in the real world scenarios.}\n\nA1: Your concern is important. The existence of SDN in real-world scenarios is fundamental to the study of SDN. Since this question is so important, we reponse it in General Question 2.\n\nAfter the publication of this paper, we plan to create a website to showcase the collections of SDN we've identified. The site will also offer an upload feature, allowing other users to share SDN instances they've found. This initiative aims to address this issue in real-world scenarios. Additionally,  on the website, we will acknowledge all the reviewers for their invaluable suggestions.\n\n>** **\nQ2: The proposed method works well on the constructed dataset, CIFAR20-SDN. However, the performance improvement on real-world dataset, Clothing1M is marginal. This makes the existence of SDN questionable.\n\nA2: We concede that the improvements on Clothing1M are not as much as those on CIFAR20-SDN, given the complexity of Clothing1M. However, when compared with SOTA methods such as ELR+ and DivideMix, which both leverage dual networks, MixUp, and semi-supervised techniques (techniques known to enhance performance in clean settings). The improvements made by NoiseCluster are remarkable, considering it doesn't rely on these supplemental techniques. Furthermore, a 0.7\\% rise is also considerable, constituting over 10\\% of the total 6.3\\% improvement in comparison to the CE result. Given the straightforward approach of NoiseCluster, it's evident that this enhancement is purely due to its adept handling of SDN.\n\n\n>** **\nQ3: It will be more clear to describe the class and sub-class relations briefly, or giving a concrete example of CIFAR20-SDN in the main paper.\n\n\nA3: We agree that providing a concrete example of class and sub-class relations, particularly in the context of the CIFAR20-SDN dataset, would help in understanding the concept of Subclass-Dominant Label Noise (SDN). Here's a potential addition to the manuscript:\n\nThe first category in CIFAR20 is termed \"aquatic mammals,\" which is subdivided into five sub-classes: \"beaver\", \"dolphin\", \"otter\", \"seal\", and \"whale\". To introduce SDN, we randomly flip a significant proportion of labels within the \"whale\" subclass to the subsequent category, \"fish\". Subsequently, \"whales\" labeled as \"fish\" are considered as SDN.\n\nWe hope this example makes the concept of class and subclass relations, as well as the Subclass-Dominant Label Noise, clearer. We will incorporate this explanation into the revised manuscript to facilitate better understanding for readers.\n\n>** **\nQ4: Clustering Result: More analysis of clusting effects and results could make the proposed SDn and method more convincing. For example, on the CIFAR20-SDN dataset, does the clustering result consist with the manually created SDN noise? And what about the real-world dataset Clothing 1M?\n\nA4: Thank you for your insightful suggestions. We agree that a more thorough analysis of the clustering effects and results could strengthen our claims and make our proposed SDN and method more convincing. \n\nTo this end, we have added t-SNE images from CIFAR20-SDN, Clothing1M, and WebVision datasets in the attached one-page PDF. A comparison of t-SNE images from these datasets with the manually created CIFAR-10 SDN in Figure 1 in the main paper clearly demonstrates a consistent clustering phenomenon across all four datasets."}, {"review_id": "jFa6AP7DKV", "reviewer": "Reviewer_Agy8", "summary": "This paper investigates the impact of early stopping on models trained with noisy labels and introduces a new type of label noise called subclass-dominant label noise (SDN). The experiments reveal that later stopping during training can better capture the high-level semantics of noisy examples. Building upon this finding, the authors propose a novel approach named NoiseCluster, which utilizes the geometric structures of long-trained representations to detect and correct SDN. The experimental results demonstrate that NoiseCluster outperforms several label noise robust methods on both SDN and Clothing-1M datasets.", "questions": " Could the authors provide further clarification on the differences between SDN and IDN? Moreover, given that there have been existing studies on IDN, what is the significance of specifically investigating and studying SDN?", "limitations": "please refer to the Weaknesses", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper introduces a novel type of label noise called subclass-dominant label noise (SDN), providing a well-motivated and insightful perspective on the limitations of early stopping in the presence of SDN. By demonstrating the inapplicability of early stopping for training with this type of label noise, the authors shed light on the challenges faced by early stopping in handling real world label noise.\n\n- Experimental results showcase the superior performance of the proposed method on both synthetic and real-world datasets, highlighting its effectiveness in addressing the issues posed by SDN. \n", "weaknesses": "- The paper would benefit from a comparison with instance-dependent label noise robust methods, which could provide a comprehensive evaluation of the proposed approach.\n- The evaluation of methods in this study is limited to a single synthetic dataset and one real-world dataset. Assessing the proposed approach on a wider range of datasets can further demonstrate the generalizability and robustness of their method in various practical scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">** **\nQ1: The paper would benefit from a comparison with instance-dependent label noise robust methods, which could provide a comprehensive evaluation of the proposed approach.\n\nA1: Thank you for your insightful suggestion. In this work, we have included 12 baselines, many of which have been demonstrated to perform well on IDN. For example, PTD-R-V and BLTM-V are two methods specifically designed to address IDN, and both DivideMix and PES(semi) show strong performance on the CIFAR-N dataset, which contains human annotations. Despite their success in these settings, all of these baselines encounter serious difficulties when dealing with SDN, which validates the effectiveness of NoiseCluster in handling SDN. \n\n>** **\nQ2: Assessing the proposed approach on a wider range of datasets can further demonstrate the generalizability and robustness of their method in various practical scenarios.\n\nA2: We highly agree with the importance of evaluating our method on a broader range of real-world datasets. In pursuit of this, we undertook additional experiments on mini WebVision and revisited our experiments on Clothing1M to confirm the presence of SDN and the noise clustering phenomenon. Our observations indicate that SDN is widespread across classes in both datasets. Moreover, clustering effects inherent to SDN are clearly observable, affirming the efficacy of our proposed method in detecting SDN. We've pinpointed six unique SDN cases from WebVision and Clothing1M and closely examined their corresponding images. These findings are detailed in the attached one-page PDF.\n\nFor other types of synthetic label noise, such as symmetric or asymmetric, we response it in General Question 1.\n\n>** **\nQ3: Could the authors provide further clarification on the differences between SDN and IDN? Moreover, given that there have been existing studies on IDN, what is the significance of specifically investigating and studying SDN?\n\nA3: We truly appreciate this insightful question. Since Instance-Dependent Noise (IDN) is still an area under active research, multiple explanations for IDN may exist. Below, we offer our perspective.\n\nIDN is derived from a noise modelling perspective and its definition is exceptionally flexible, encompassing all types of label noise, including SDN. This comprehensive approach enables the pursuit of a universal solution that can address all forms of label noise. However, this broad approach also presents challenges, such as defining a generation method that accurately mimics all real-world label noise types, or identifying typical real-world datasets that include all forms of label noise.\n\nOn the other hand, SDN is a distinct type of label noise we have identified empirically. We came across specific mislabeled examples resistant to early stopping yet exhibiting recurring patterns. Based on these findings, we deduced the underlying causes, articulated the concept of SDN, and embarked on thorough experiments to explore its properties, all of which are elaborated upon in our paper. The value of our SDN study stems from its emulation of a prevalent type of real-world label noise, notably observed after early stopping. \n\nOverall, since current methods may struggle with SDN, we believe specifying SDN will aid in the study of IDN.\n\n"}, {"review_id": "qae4AcBwuq", "reviewer": "Reviewer_tXVq", "summary": "This paper proposes a new type of noisy labels called subclass-dominant noisy labels and introduces an algorithm called NoiseCluster based on this noisy label modeling. In the experimental section, the authors demonstrate the superiority of NoiseCluster over previous approaches in the subclass-dominant noisy label cases. Additionally, NoiseCluster exhibits better performance than previous algorithms on the real-world Clothing1M dataset.", "questions": "-", "limitations": "The limitations of this paper are summarized in the \"Question\" and \"Weakness\" sections", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Introduces a new type of noisy label modeling.\n- Well-written paper.\n- Conducts extensive experiments.\n", "weaknesses": "- The justification for the proposed subclass-dominant noisy label model is lacking. It is unclear whether this type of noisy labels frequently occurs in the real-world. For instance, it would be helpful to investigate if the Clothing1M dataset exhibits similar phenomena.\n- The analysis should include the performance of NoiseCluster on other types of noisy label models such as symmetric, instance, or asymmetric. This would provide a more comprehensive evaluation of the algorithm.\n- Further analysis on real-world datasets is necessary. While the authors evaluate NoiseCluster on the Clothing1M dataset, it would be beneficial to assess its performance on other real-world benchmarks such as WebVision, Food101N, and similar datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">** **\nW1: The justification for the proposed subclass-dominant noisy label model is lacking ... investigate if the Clothing1M dataset exhibits similar phenomena.\n\nA1: Thank you for your insightful feedback. We highly agree with the justifying the presence of SDN and the noise clustering phenomena is pivotal for our study. For more existence of SDN, we response it in General Question 2. Detailed results are available in the attached one-page PDF.\n\n>** **\nW2: The analysis should include the performance of NoiseCluster on other types of noisy label models such as symmetric, instance, or asymmetric. This would provide a more comprehensive evaluation of the algorithm.\n\nA2: In this study, our primary focus is on examining a specific type of real-world label noise: Subclass Dominant label Noise (SDN). NoiseCluster has been specifically designed to tackle SDN. \n\nFor other types of synthetic label noise, such as symmetric, instance,  or asymmetric, we response it in General Question 1.\n\n>** **\nW3: Further analysis on real-world datasets is necessary ... assess its performance on other real-world benchmarks such as WebVision, Food101N, and similar datasets.\n\nA3: We agree that evaluating our method, NoiseCluster, on additional real-world benchmarks would strengthen our findings and provide a more comprehensive understanding of SDN. Following your suggestion, we delve into SDN within the WebVision dataset. We find that SDN is prevalent across classes within the mini WebVision dataset. Moreover, clustering effects, specific to SDN, are distinctly evident therein. We pinpointed three unique SDN cases in WebVision and have provided visualizations of their associated images. These can be viewed in the attached one-page PDF.\n"}]

Author rebuttal:
>** **
General Questions

>** **
Q1: What is NoiseCluster's performance across different noisy label models, like symmetric, instance, and asymmetric?

A1: NoiseCluster has been specifically designed to tackle SDN, utilizing our proposed late stopping strategy. As evidenced in Table 1, representations under symmetric, instance, or asymmetric noise largely degrade over extended training periods. In such cases, early stopping is the typical strategy to prevent this degradation. Therefore, NoiseCluster might not match the performance of other SOTA methods for these specific noise types.

For a comprehensive algorithm to manage various label noises, we've proposed a combined strategy in Line 212. This involves pairing early stopping-based methods with NoiseCluster. Initially, the early stopping method tackles label noise such as symmetric, instance, or asymmetric. Subsequently, NoiseCluster continues to address SDN. By adopting this integrated approach, we ensure competitive performance against symmetric, instance, or asymmetric label noise, while also tackling a new type of real-world label noise.

>** **
Q2: Is the proposed subclass-dominant label noise (SDN) prevalent in real-world scenarios?

A2: We first address this question through logical reasoning, followed by showcasing examples of SDN from real-world datasets.

In real-world situations, label noise is not uniformly random (e.g., symmetric noise) but tends to follow certain patterns. For instance, samples from specific sub-classes may frequently be mislabeled, especially if annotators are unfamiliar with them. When humans annotate ambiguous samples, our approach isn't to assign arbitrary labels. Rather, we label based on our existing understanding. Hence, if an example is mistakenly labelled, other closely related examples (in the same subclass) could be similarly mislabeled. Labels generated from models may lead to this issue more serious, misclassifying similar samples in a consistent manner. This consistent mislabeling within sub-classes results in the prevalence of SDN in real-world datasets.

To confirm the presence of SDN in real-world datasets, we carry out experiments on Clothing1M and WebVision by carefully examining the images in clusters. Our observations indicate that SDN is widespread across classes in both datasets. These findings are showed in the attached one-page PDF.

>** **
More response to Reviewer 7AfS Question 6-10

>** **
Q6: In Table 5, is the method without label correction simply vanilla training?

A6: The result of 'Ours W/O CORRECTION' is not equivalent to vanilla training, which is denoted as 'CE' in Table 2. Our method consists of two crucial components: identification and label correction. When we say 'Ours W/O CORRECTION,' we mean that we do not correct the noisy labels, but we still identify potentially mislabeled examples. Once identified, we can exclude them from the confident examples. This removal is a simple yet effective strategy that results in a 6-8\% improvement. It is noted that the improvement of removal is built on the success of identifying SDN. 

However, label correction remains a risky action, which may not work in a more complex situation. We will provide further clarification in Table 5 and Section 5.3. Thank you for pointing this out. 

>** **
Q7: ELR should also be included in Table 2, as it does not require SSL (only ELR+ does).

A7: It's worth noting that ELR still retains some semi-supervised techniques, such as temporal ensembling and target probabilities. Since the performance of ELR+ falls below that of NoiseCluster (without semi-supervised techniques), the results might not differ significantly.

>** **
Q8: Regarding Cloth1M, why was only one epoch used for training? ... why these modifications are made.

A8: (1) Clothing1M is comprised of one million images with noisy labels. Even a single training epoch provides enough data for fine-turning a pretrained model, equivalent to 20 epochs on CIFAR-10. As a result, we only train the network for one epoch, making NoiseCluster not only the state-of-the-art method but also the fastest method on Clothing1M.

(2) Clothing1M is a real world noisy dataset, containing complex types of label noise, not only SDN. So, we employ a simple early stopping method, training on 95\% of the data, to remove other types of label noise, then apply NoiseCluster on the filtered dataset to continue to identify and correct SDN.

(3) The primary rationale for applying NoiseCluster to 5\% of the training data is to maintain the number of training examples on a par with CIFAR20-SDN, thereby minimizing the need for tuning DBSCAN hyperparameters. An additional consideration is the computational cost: the calculation of t-SNE for one million features takes around one hour, whereas it takes only about five minutes for 50k features.

>** **
Q9: The method doesn't appear to be specifically designed for SDN alone, ... under normal noise?

A9: NoiseCluster has been specifically designed to tackle SDN. For the performance of other types of synthetic label noise, such as symmetric or asymmetric, we response it in General Question 1.

>** **
Q10: it would be valuable to discuss self-supervised pretraining, ... even under SDN.

A10: It should be noted that our findings do not conflict with previous works. When using self-supervised pretrained models, the challenge of early stopping may be intensified. Since self-supervised learning is capable of uncovering underlying characteristics, it might enlarge the feature distances between sub-classes. This could lead to an acceleration in the speed of overfitting to SDN, making early stopping more ineffective. Supporting evidence can be drawn from the performance metrics of C2D (ELR+ with SimCLR) [2] on the Clothing1M dataset. C2D lags behind NoiseCluster by 0.93\% and even trails ELR+ by 0.23\%, despite the fact that the initialization of self-supervised pretrained models surpasses supervised learning. 


Meta-review: The paper studies subclass-dominant label noise, a type of noise where mislabeled examples dominate at least one class. The authors study the evolution of representations generated by long trained network, and show that similar examples can be clustered together. Based on this observation, they suggest a new algorithm for combating subclass-dominant label noise, by clustering examples together and then eliminating noisy examples based on their distance to other clustered examples.

It seems that, after discussion with the authors, all the reviewers are supporting the paper for acceptance. While some of the reviewers were concerned about the applicability of the noise model studied in the paper to real-world datasets, to my understanding the thorough response of the authors convinced the reviewers that this study is a worthy contribution.

In this situation, I recommend that the paper is accepted, assuming that the authors make the proper changes to the paper suggested in the reviews.
Decision: Accept (poster)