Title: Subclass-Dominant Label Noise: A Counterexample for the Success of Early Stopping
Venue: NeurIPS 2023
Accepted: yes
Keywords: learning with noisy labels

weakly supervised learning
Abstract: In this paper, we empirically investigate a previously overlooked and widespread type of label noise, subclass-dominant label noise (SDN). Our findings reveal that, during the early stages of training, deep neural networks can rapidly memorize mislabeled examples in SDN. This phenomenon poses challenges in effectively selecting confident examples using conventional early stopping techniques. To address this issue, we delve into the properties of SDN and observe that long-trained representations are superior at capturing the high-level semantics of mislabeled examples, leading to a clustering effect where similar examples are grouped together. Based on this observation, we propose a novel method called NoiseCluster that leverages the geometric structures of long-trained representations to identify and correct SDN. Our experiments demonstrate that NoiseCluster outperforms state-of-the-art baselines on both synthetic and real-world datasets, highlighting the importance of addressing SDN in learning with noisy labels. The code is available at https://github.com/tmllab/2023_NeurIPS_SDN.

Subclass-Dominant Label Noise: A Counterexample for the Success of Early Stopping

Abstract

In this paper, we empirically investigate a previously overlooked and widespread type of label noise, subclass-dominant label noise (SDN). Our findings reveal that, during the early stages of training, deep neural networks can rapidly memorize mislabeled examples in SDN. This phenomenon poses challenges in effectively selecting confident examples using conventional early stopping techniques. To address this issue, we delve into the properties of SDN and observe that long-trained representations are superior at capturing the high-level semantics of mislabeled examples, leading to a clustering effect where similar examples are grouped together. Based on this observation, we propose a novel method called NoiseCluster that leverages the geometric structures of long-trained representations to identify and correct SDN. Our experiments demonstrate that NoiseCluster outperforms state-of-the-art baselines on both synthetic and real-world datasets, highlighting the importance of addressing SDN in learning with noisy labels. The code is available at https://github.com/tmllab/2023_NeurIPS_SDN .

1 Introduction

The popularity of giant neural networks in recent years has emphasized the importance of vast amounts of annotated data [42, 11, 41]. However, manually annotating large amounts of data can be prohibitively expensive, leading researchers to explore alternative methods for collecting data such as web crawling [57] and crowdsourcing [45] that can be more cost-effective. Unfortunately, these methods may produce inaccurate data, particularly when noisy labels are present, thereby undermining the performance of deep neural networks (DNNs). As a result, the research community has increasingly focused on developing approaches to mitigate the impact of noisy labels and prevent the degeneration of DNNs [16, 25, 13].

Most existing approaches for dealing with noisy labels broadly fall into two categories: noisemodeling-based methods and memorization-effects-based methods. Noise-modeling-based methods [51, 53, 32, 47, 55] usually construct a transition matrix to exploit connections between given noisy labels and underlying clean labels. While these methods offer strong theoretical guarantees that the corrected risk closely approximates the risk associated with training on clean data, accurate estimation of the transition matrix relies on numerous assumptions that are difficult to satisfy in real-world scenarios [40, 56]. Memorization-effects-based methods [17, 43] mainly employ early stopping to identify confident examples based on the phenomenon that DNNs will fit the easy examples (i.e., the majority of examples with similar characteristics) first and then memorize the complex examples (i.e., the minority of examples with diverse characteristics) [58, 1, 22]. Recently, some of them [30, 34, 3], combined with semi-supervised learning techniques [5, 23], have achieved state-of-the-art results.

(b) SDN

(c) SDN 10th epoch

(d) SDN 100th epoch

100

100

airplane

airplane

Memorized percentage

Memorized percentage

bird

automobile

automobile

bird

80

80

trog

dog

cat

deer

cat

deer

dog

tog

60

60

horse

horse

ship

ship

truck

truck

40

40

20

Clean

20

Clean

Mislabeled

Mislabeled

20

40

60

80

100

20

40

60

80

100

Epochs

Epochs

In this paper, we present a new type of label noise, subclass-dominant label noise (SDN), which is overlooked by existing studies. SDN refers to label noise in which mislabeled examples dominate at least one subclass, whose examples exhibit distinct characteristics compared to others in the same class. Real-world datasets are prone to experiencing SDN due to several factors. The likelihood of human mislabeling is influenced by various object characteristics such as shape, context, and part [50, 33, 7, 44, 6]. Moreover, classification tasks usually define classes based on typical or common examples. However, in reality, numerous special sub-classes can exist, sharing common characteristics with other classes. For instance, while whiskers typically play a vital role in distinguishing between cats and dogs, Sphynx cats, unlike other cat breeds, do not possess whiskers. An inexperienced annotator may mislabel the majority of Sphynx cats as dogs, leading to SDN in the dog class.

To evaluate the detrimental effects of SDN, we construct an SDN dataset and conduct an experiment on it. Specifically, we subdivide the aircraft class in CIFAR-10 [27] into two sub-classes based on their status: landed and flying. We generate the SDN dataset by flipping the labels of all landed airplane examples to the automobile class, leading to an overall noise rate of 2.7%. Our experiment involves training a ResNet-18 model [19] for 100 epochs using cross-entropy loss and extracting confident examples at each epoch. We identify confident examples based on the criteria described in [43]. To quantify the extent to which the network has memorized correctly labeled and mislabeled data, we compute the ratio of confident examples that include mislabeled data to the total number of mislabeled examples in the training dataset. As shown in Figure 1(a, b), early stopping fails to identify mislabeled examples because the network has already memorized the mislabeled examples in SDN. This outcome can be attributed to the fact that DNNs excel at learning from the majority of examples that share similar characteristics first. When the majority of examples in a clean subclass are mislabeled, these mislabeled examples often dominate the gradient and are learned first by DNNs, thereby posing challenges for methods that rely on early stopping.

To overcome this challenge, we explore a novel method named NoiseCluster, which is capable of distinguishing correctly labeled and mislabeled examples in SDN. The fundamental principle of NoiseCluster is rooted in the concept of later stopping. Contrary to the conventional belief that noisy data increasingly degrade representations over time, our findings suggest that the long-trained representations obtained from later stopping are more effective at capturing the high-level semantics of noisy examples. This leads to a clustering effect where the embeddings of noisy examples with similar characteristics are drawn closer together, as shown in Figure 1(c, d). Motivated by this observation, NoiseCluster utilizes the geometric structures of long-trained representations to identify and correct mislabeled examples of SDN. Specifically, we first extract features from the penultimate layer of the network after halting the training progress with later stopping. We then group these features by class, based on feature density, and identify the largest group as clean due to its high likelihood of being correctly labeled, while treating the remaining groups as potentially mislabeled. To determine the label of a potentially mislabeled group, we assess its similarity to surrounding examples according to set distance and assign it the class with the most similar examples.

The key contributions are summarized as follows:

We introduce subclass-dominant label noise (SDN) for the first time, highlighting its prevalence in real-world datasets and its potential to render early stopping ineffective.

We propose a novel method, NoiseCluster, which leverages the geometric structures of long-trained representations to identify and correct mislabeled examples in SDN.

Empirical results demonstrate the effectiveness of NoiseCluster on both synthetic and real-world datasets, establishing it as a solid baseline for future studies.

2 Related Work

Learning with noisy labels is a vibrant research area aimed at improving model robustness. Two main avenues of research have emerged: noise-modeling-based and memorization-effects-based methods.

Noise-modeling-based methods. From a noise modeling perspective, label noise can be categorized into three types: random classification noise (RCN), class-conditional label noise (CCN), and instance-dependent label noise (IDN). RCN maintains a static flip rate for binary classification [35]. CCN assumes that the noise generation is solely dependent on classes and independent of instances [40, 56, 48, 9]. The generation of IDN depends on both classes and instances, making IDN the most general type of label noise [50, 10, 62, 36]. The common philosophy behind noise modeling is to estimate the noise transition matrix. Xia et al. [50] assume annotators may make mistakes on the parts of instances rather than on whole instances, and replace the estimation of the instance-dependent transition matrix with the transition matrices for parts; Yang et al. [55] propose a method to generate Bayes optimal labels and estimate the transition matrix between Bayes optimal labels and noisy labels. Although noise-modeling-based methods do not theoretically require early stopping, many of them implicitly employ it to accurately estimate the transition matrix [40, 51, 53, 56, 55]. As a result, many existing methods struggle to handle SDN.

Memorization-effects-based methods. Arpit et al. [1] find that Deep Neural Networks (DNNs) tend to learn simple (clean) examples first, then gradually memorize complex (noisy) examples. This observed phenomenon has inspired a subset of methodologies grounded in understanding and leveraging memorization effects [17, 2, 38, 22, 24]. JointOptim [43] introduces a pseudo-labeling technique known as hard-label, which regards the model predictions consistent with noisy labels as confident examples and uses them to train the model. DivideMix [30] utilizes Gaussian Mixture Model (GMM) to identify confident examples as labeled data and treat the remaining examples as unlabeled data for semi-supervised learning (SSL). Furthermore, some methods focus on the latent representations extracted from previous layers, as the final layer is particularly vulnerable to the influence of noisy examples [29, 46, 26, 3]. TopoFilter [46] utilizes k-NN to explore the topological information of the latent representations and then selects the largest component as clean data. Kim et al. [26] propose a noise detection method called FINE that utilizes Eigen decomposition and the principal component to select clean examples. PES [3] takes advantage of the fact that the effects of noisy examples tend to decrease as the layer number increases and improves early stopping by re-learning later layers based on the latent representations. NoiseCluster also exploits the geometric structures of the latent representations. However, unlike previous methods that rely on early stopping, NoiseCluster takes advantage of later stopping to extract long-trained representations, enabling the identification and correction of mislabeled examples in SDN.

3 SDN Analysis

In this section, we begin by defining subclass-dominant label noise (SDN). Next, we introduce a specially crafted dataset, CIFAR20-SDN, for exploring SDN, and compare it with two established real-world datasets. Finally, we delve into the characteristics of SDN from two distinct perspectives: its adverse effects on early stopping and its negative impact across various layers.

Definition of SDN. Let ˜ D = { ( x i , ˜ y i ) } n i =1 be the noisy dataset, where x i ∈ X denotes input data, ˜ y i ∈ { 1 , ..., C } denote a noisy label, and C is the total number of classes. x i is associated with a latent (not provided) clean subclass label b i ∈ { 1 , ..., B } , and B is the total number of sub-classes in this dataset. It is assumed that each clean subclass b corresponds to a single clean class c . Subclassdominant noise (SDN) refers to a type of label noise in which mislabeled examples dominate at least one subclass. Specifically, within a clean subclass b , examples could be flipped into different classes,

(a) CIFAR20-SDN

(b) Clothing1M

(c) WebVision

1.0

1.0

1.0

0.8

80

80

0.6

0.6

0.4

0.4

0.4

0.2

0 2

02

00

0.0

fish

containers

aquatic

fowers

fruit

insects

electrical

angwny piouasnou

0.2

0.4

large man-made

large natural

omnivores

large camivores

0.6

people

reptiles

medium mammals

invertebrates

0.8

small mammals

trees

vehicles 1

vehicles 2

1.0

0.0

0.0

Knitwear

TShirt

Shirt

0.2

Hoodie

Chiffon

Sweater

0.4

Wndbreaker

Down Coat

Jacket

0.6

Suit

Dress

Shawl

0.8

Vest

Underwear

1.0

00

0.0

tench

great white shark

goldfish

0.2

0.4

tiger shark

electric ray

hammerhead

0.6

cock

hen

stingray

ostrich

others

0.8

1.0

including its own clean class. If the class j ∈ { 1 , ..., C } with the most flipped examples does not equal to the clean class c , then the subclass b is considered to be dominated by the examples flipped into noisy class j .

CIFAR20-SDN. To facilitate research on SDN, we introduce CIFAR20-SDN, a representative SDN dataset built from CIFAR-100, which provides 20 class labels and 100 subclass labels. The generation of CIFAR20-SDN is similar to that of Pairflip label noise [17]. Specifically, for each class, a certain percentage of labels in the last subclass are flipped to the subsequent class. For example, the first class in CIFAR20 is termed "aquatic mammals", which comprises five sub-classes: "beaver", "dolphin", "otter", "seal", and "whale". Labels from the "whale" subclass are randomly flipped to the subsequent "fish" class. As a result, any "whale" labeled as "fish" is deemed an instance of SDN. Note that only the class labels are utilized for model training and evaluation.

To assess the similarity between CIFAR20-SDN and real-world noisy datasets, we compare the feature distribution of CIFAR20-SDN with two other real-world noisy datasets, Clothing1M [52] and WebVision [31]. Specifically, we directly visualize the features of CIFAR20-SDN, as it exclusively contains SDN, while for the two real-world datasets, we visualize the features of confident examples extracted using early stopping methods. As shown in Figure 2, it is clear that CIFAR20-SDN successfully emulates the properties of the two real-world datasets, with mislabeled examples clustering together and maintaining a distance from their noisy classes.

Negative effects of SDN on different layers. To thoroughly study the negative impacts of SDN on various layers, we train a PreAct ResNet-18 model on CIFAR20-SDN with four types of label noise. We report the highest test accuracy achieved during training as an indicator of the quality of the representations from the final layer. Additionally, to assess the representation quality from the penultimate layer, we freeze the other layers, re-initialize and retrain the final layer using clean labels.

Table 1 reveals that SDN significantly impacts the final layer more detrimentally than other types of label noise, resulting in the lowest test accuracy, which is roughly 80% of the accuracy achieved on the clean dataset. Conversely, when considering the penultimate layer, the outcomes achieved with SDN surpass those obtained with any other type of label noise, indicating that previous layers are less adversely impacted even after 300 epochs of training. These findings underscore the importance of utilizing the representations extracted from the penultimate layer in addressing SDN.

(a) SDN intra-class

(b) Sym-50 intra-class

(c) SDN inter-class

(d) Sym-50 inter-class

airplane

airplane

airplane

airplane

automobille

automobile

automobile

automobile

bird

Pq

Pq

cat

cat

cat

deer

dog

trog

horse

ship

truck

Figure 3: (a, b) illustrates the intra-class relationships between correctly labeled and mislabeled examples in a noisy class, while (c, d) demonstrates the inter-class relationships between mislabeled examples and their corresponding latent clean classes. To better visualize the intra-class feature relationships, we use the features from a noisy automobile class and visualize them with clean labels. Similarly, to effectively visualize inter-class feature relationships, we use features from four classes, as 2D images cannot adequately represent high-dimensional relationships.

Advantages of long-trained representations . We extend the experiments depicted in Figure 1 and visualize the features extracted from the networks trained for 200 epochs. As shown in Figure 3(a, b), features within a noisy class are clustered based on their semantics, which suggests that networks can learn the semantics of mislabeled examples even when trained with noisy labels. Moreover, Figure 3(c, d) shows that features of mislabeled examples tend to be close to the features with corresponding clean classes, implying that the geometric structures of the features can be utilized to identify their clean classes. Finally, a comparison of the four sub-figures reveals that the features extracted from SDN exhibit superior geometric structures compared to those extracted from symmetric label noise. This observation suggests that the properties of long-trained representations are particularly advantageous when dealing with SDN.

4 Methodology

To overcome the challenges posed by SDN, we introduce a novel method called NoiseCluster, which leverages the geometric structures of long-trained representations. NoiseCluster consists of two main steps: initially, it identifies potentially mislabeled examples by clustering similar instances into multiple sets. Subsequently, it performs label correction by computing set distance and reassigning the labels of potentially mislabeled examples.

4.1 Identify SDN

NoiseCluster begins with training on ˜ D ( X, ˜ Y ) and then halts the training with later stopping, whose stopping point is much later than that of early stopping, allowing for thorough exploitation of knowledge in noisy data. We then extract the latent feature Z of the input data X from the penultimate layer and reduce its dimensionality to two dimensions using t-SNE [39], resulting in a new feature denoted as ˆ Z . To separate mislabeled data, we explore the geometric structures of ˆ Z c , where c ∈ { 1 , ..., C } and C is the total number of classes. To handle the unknown number of clusters in each class, we employ DBSCAN [12], a feature density-based clustering algorithm, to cluster the feature ˆ Z c into K groups U c K . The largest group, which is most likely to contain correctly labeled examples, is identified as the clean group, while the remaining groups are regarded as potentially mislabeled groups and denoted as U c K -1 .

Stopping point selection . Although NoiseCluster can be applied to a fully trained model, we find that further training on the original noisy data does not constantly improve the representations after the training becomes stable. Therefore, to reduce training time, we incorporate our method into the middle stage of the training process.

4.2 Label Correction

Simply removing all examples in U c K -1 can result in the loss of many important examples. To address this challenge, we treat examples in a potentially mislabeled group U c k as a cohesive unit, where all

.

(a) Set distance

(b) Mean distance

examples share the same label. We then assess the similarity between this group and the surrounding examples by computing distances and assigning examples in the group to the class with the most similar examples. Specifically, due to the clustering effects of long-trained representations, we treat the examples in U c k as a set, and form multiple class sets O b from the surrounding examples, where b ∈ { 1 , ..., C } . We measure the distance between U c k and O b using set distance, which is defined as the minimum Euclidean distance between any feature z q , z p from sets U c k and O b , as depicted in Figure 4. The formula for set distance is defined as follows:

where ∥·∥ 2 indicates L 2 Norm. In practice, to mitigate the potential impacts of mislabeled examples close to the set U c k , we utilize a specified number of closest distances, denoted as ClosePoint , and compute their average. We then correct the labels in U c k with the category of the class set with the closest distance, d kb , which is defined as follows:

After acquiring the corrected data, we use them to continue training the network for the rest of the epochs. The whole process of NoiseCluster is summarized in Algorithm 1.

Combine with SSL . Semi-supervised learning (SSL) has enabled numerous methods to attain impressive benchmarks in LNL [30, 34, 26]. NoiseCluster can also take advantage of SSL techniques to enhance its performance. Specifically, for each class c , DBSCAN divides features ˆ Z c into K distinct groups U c K and assigns a group index g c i to each feature ˆ z c i . Examples that cannot be clustered into any group are regarded as "noise". To integrate SSL, we treat the DBSCAN noise as the unlabeled data and the remaining examples as the labeled data. Accordingly, we can summarize the labeled and unlabeled data sets as follows:

̸

Combine with early stopping . Although NoiseCluster is primarily designed to address SDN, it can be seamlessly integrated with early stopping. To tackle a noisy dataset containing multiple types of label noise, we propose a two-stage training process. In the first stage, we employ early stopping to select confident data, thereby filtering out label noise that is sensitive to early stopping. In the second stage, we apply NoiseCluster to the confident data to identify and correct mislabeled examples in SDN. This two-stage strategy allows us to effectively address the challenges posed by a mixture of various types of label noise, resulting in enhanced performance on noisy datasets.

5 Experiments

5.1 Experimental Setting

Datasets . We evaluate the performance of NoiseCluster on a synthetic dataset, CIFAR20-SDN, and a real-world dataset, Clothing-1M [52]. To ensure a fair comparison with existing methods, we follow the settings of the previous methods [30, 34, 3] and conduct two sets of experiments on the synthetic

̸

Continually train f θ and f ξ on D l for the rest of the epochs.

dataset: one with semi-supervised learning (SSL) and one without SSL. In experiments without SSL, we reserve 10% of the training data as the validation set, while we utilize the entire training data for experiments with SSL. Clothing-1M contains over one million images gathered from the Internet. Additional details on implementation can be found in Appendix B.

Network structure and optimization . For CIFAR20-SDN, we employ ResNet-34 [19] for experiments without SSL and PreAct ResNet-18 [20] for experiments with SSL. During optimization, we train the model for 300 epochs, using a learning rate of 2 × 10 -2 , a single cycle of cosine annealing [37], a momentum of 0.9, and a weight decay of 5 × 10 -4 . We utilize a batch size of 128 and a stopping epoch of 80, with a ClosePoint value of 20. For DBSCAN hyperparameters, Eps and MinPts are set to 0.02 and 100, respectively.

For Clothing1M, we employ a ResNet-50 pre-trained on ImageNet[28], and train the network for only one epoch. Specifically, we first utilize 95% of the training data to train the network. Then, we use early stopping to select confident data from the remaining 5% of the training data and apply NoisyCluster to this data. For optimization, we use SGD with a learning rate of 0.01, a momentum of 0.9, a weight decay of 10 -4 , and a batch size of 64. ClosePoint, Eps, and MinPts are set to 20, 0.04, and 100, respectively.

5.2 Comparison with State-of-the-art Methods

Approaches without SSL. To begin our comparison, we evaluate our method against state-of-the-art (SOTA) approaches that do not employ SSL. As shown in Table 2, it becomes evident that many of the methods struggle with SDN, with some even underperforming in comparison to standard training with Cross-Entropy (CE) loss. CDR [49] performs better than its counterparts, but it requires a known noise rate, which remains challenging to accurately estimate in SDN. Our method outperforms all baselines, delivering superior performance. Specifically, our proposed method surpasses CDR by 5.67% in classification accuracy when a noise rate is 20%, and the margin further extends to

natural

100

91

93

89

91

89

93

SDN-18

Memorized percentage

75

80

83

85

79

84

82

72

Sym-18

50

49

52

25

6

3

1

3

2

2

3

3

3

1

1

1

0

0

L

aquatic

fish

flowers

containersfruit

electrical

funiture

insects

canivores

man-made

omnivores

m-mammals

reptiles

vehicles 1

vehicles 2

7.59% for a noise rate of 12%. Furthermore, in comparison with noise-modeling-based methods, our approach outperforms the state-of-the-art VolMinNet by over 7% across four noise rates.

Approaches with SSL. We integrate our method with semi-supervised learning (SSL) techniques, denoted as NoiseCluster+ and compare it against SOTA approaches that also leverage SSL. To ensure a fair comparison, we include the results of MixUp [59], which is used by all baselines to boost performance. The results are summarized in Table 3. Interestingly, our observations indicate that SSL contributes marginally to performance enhancement on CIFAR20-SDN. Notably, DivideMix [30] shows only a modest advancement over MixUp. In contrast, methods based on representations, such as F-DivideMix [26] and PES semi [3], deliver better outcomes by avoiding the use of the final layer, which is prone to overfitting to SDN. Conclusively, our method, NoiseCluster+, consistently outperforms all baselines by a substantial margin, underscoring that the proposed techniques are the primary factors leading to the improvements.

To better understand why current SOTA methods face difficulties when dealing with SDN, we conduct experiments with DivideMix and compute the ratio of confident examples that include mislabeled data to the total number of mislabeled examples in the training dataset at the end of training. As shown in Figure 5, DivideMix performs well on merely three classes. For the remaining classes, we can see that the confident data contains a substantial number of mislabeled examples, confirming that early stopping and other techniques utilized in DivideMix cannot effectively address SDN issues.

Real-world dataset. We perform experiments on a real-world dataset, Clothing1M, which is widely used as a benchmark due to its challenges. Many existing methods, though effective on noisy synthetic datasets, exhibit limited impact on Clothing1M. Our investigation suggests that this issue stems

Figure 6: SDN in Clothing1M. (a) The first row displays images from the "T-Shirt" class. The second row shows images from the "Underwear" class. The third and fourth rows present images that are mislabeled as "Underwear". (b) The row order is as follows: "Jacket", "Windbreaker" (correct name:"trench coat"), "Leather Jacket". (c) The row order is as follows: "Down coat", "Vest" (correct name:"singlet" or "tank top"), "Down Vest". The coordinate values correspond to Figure 2 (b). It is worth noting that some label names within Clothing1M are inaccurate due to translation issues. After comparing images in the test set and Chinese label names, we put the correct names in parentheses.

90

06

90

80

80

80

TestAccuracy

TestAccuracy

TestAccuracy

70

70

70

60

60

60

SDN-12

SDN-12

SDN-12

SDN-20

SDN-20

SDN-20

50

50

75

100

125

150

500.016

0.018

0.020

0.022

0.024

50

5

10

20

30

50

MinPts

Eps

ClosePoint

from the presence of SDN in Clothing1M, leading to suboptimal performance on the test data. As illustrated in Table 4, NoiseCluster significantly outperforms SOTA methods, achieving an accuracy of 75.51% without resorting to self/semi-supervised learning or ensemble network techniques.

To delve into the existence and underlying causes of SDN in real-world datasets, we manually generated t-SNE images, strictly adhering to the experimental process outlined in Section 5.1. We then showcase examples within clusters that are predominantly mislabeled. As Figure 6 illustrates, these mislabeled examples clearly belong to specific sub-classes, yet they share common characteristics with their associated noisy classes. These observations confirm the pervasive presence of SDN in real-world datasets. For additional details and further results, please refer to Appendix D.1.

5.3 Ablation Study

Sensitivity of hyper-parameters. We perform ablation studies on CIFAR20-SDN with noise rates of 12% and 20%. NoiseCluster employs DBSCAN [12], which involves two hyperparameters, Eps and MinPts. We find that the selection of these two hyperparameters is relatively straightforward when applied to feature representations due to the substantial margin between classes. Additionally, to reduce the negative impacts of mislabeled examples, we introduce a hyperparameter named ClosePoint, which controls the number of points used to compute the set distance. The results shown in Figure 7 indicate that ClosePoint, along with Eps and MinPts, has a negligible impact on the final outcomes, indicating robustness against hyperparameter changes.

Self-supervised pretrained model. As self-supervised Learning methods continue to mature [18, 15, 61, 4], the utilization of pretrained models from this approach has emerged as a promising strategy for addressing noisy labels, particularly since they are unaffected by such labels during the pre-training

(a) Scratch

(b) MoCo V2[18]

(c) RSA[4]

100

100

100

Memorized percentage

80

Memorized percentage

80

Memorizedpercentage

80

60

60

60

40

40

40

20

Clean

20

Clean

20

Clean

Mislabeled

Mislabeled

Mislabeled

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

Epochs

Epochs

Epochs

stage [14, 54, 21, 60]. This raises the pivotal question: Does employing an self-supervised pretrained model simplify the addressing of SDN? While a thorough investigation will be reserved for future research work, the succinct answer is no. As shown in Figure 8, self-supervised pretrained models offer initial advantages by facilitating rapid learning of clean examples. However, this rapid learning also leads to the quick memorization of mislabeled examples. Furthermore, after only training for several epochs, the gaps between the memorization rates for clean and mislabeled examples become much smaller than those observed when training from scratch, suggesting that early stopping with self-supervised pretrained models is less effective. We hope that our research will inspire a re-thinking of utilizing self-supervised pretrained models in noisy datasets.

Importance of label correction. To mitigate the bias introduced by learning with SDN, we employ a label correction technique. Given that DNNs have already memorized many mislabeled examples during the early learning phase, label correction becomes indispensable. In Table 5, we present a comparison of results with and without applying label correction and observe a performance improvement of over 1% for both low and high noise rates. These findings emphasize the importance of the label correction technique in addressing SDN.

6 Conclusion

In this paper, we empirically examine subclass-dominant label noise (SDN), a new type of label noise that commonly occurs in real-world datasets. SDN presents a significant challenge to conventional methods that rely on early stopping. To overcome this issue, we reveal that long-trained representations are more effective at capturing the semantic relationships between noisy examples. Building on this insight, we propose a novel approach, NoiseCluster, to identify and correct mislabeled examples in SDN. We hope our work will inspire the research community and pave the way for addressing label noise in real-world applications.

The primary limitation is that our proposed method is effective in addressing SDN cases where mislabeled examples have notably different semantics from correctly labeled examples. However, it could struggle in situations where correctly labeled and mislabeled examples exhibit close semantics, especially without any prior knowledge. We leave it as an open research topic for future investigation.

Acknowledgments and Disclosure of Funding

The authors would like to thank the anonymous reviewers and the meta-reviewer for their constructive feedback and encouraging comments on this work. Yingbin Bai was supported by CSIRO Data61. Erkun Yang was supported in part by the National Natural Science Foundation of China under Grant 62202365, Guangdong Basic and Applied Basic Research Foundation (2021A1515110026), and Natural Science Basic Research Program of Shaanxi (Program No.2022JQ-608). Jun Yu was supported by the Natural Science Foundation of China (62276242), National Aviation Science Foundation (2022Z071078001), CAAI-Huawei MindSpore Open Fund (CAAIXSJLJJ-2021-016B, CAAIXSJLJJ2022-001A), Anhui Province Key Research and Development Program (202104a05020007), USTCIAT Application Sci. & Tech. Achievement Cultivation Program (JL06521001Y), Sci. & Tech. Innovation Special Zone (20-163-14-LZ-001-004-01). Bo Han was supported by the NSFC Young Scientists Fund No. 62006202, NSFC General Program No. 62376235, and Guangdong Basic and Applied Basic Research Foundation No. 2022A1515011652. Tongliang Liu was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031.

References

Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In ICML , pages 233-242, 2017.

Yingbin Bai and Tongliang Liu. Me-momentum: Extracting hard confident examples from noisily labeled data. In ICCV , 2021.

Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. In NeurIPS , pages 24392-24403, 2021.

Yingbin Bai, Erkun Yang, Zhaoqing Wang, Yuxuan Du, Bo Han, Cheng Deng, Dadong Wang, and Tongliang Liu. Rsa: Reducing semantic shift from aggressive augmentations for self-supervised learning. In NeurIPS , pages 21128-21141, 2022.

David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS , pages 5050-5060, 2019.

Irving Biederman. Recognition-by-components: a theory of human image understanding. Psychological review , page 115, 1987.

Irving Biederman, Robert J Mezzanotte, and Jan C Rabinowitz. Scene perception: Detecting and judging objects undergoing relational violations. Cognitive psychology , pages 143-177, 1982.

Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In ICML , pages 1062-1070, 2019.

De Cheng, Yixiong Ning, Nannan Wang, Xinbo Gao, Heng Yang, Yuxuan Du, Bo Han, and Tongliang Liu. Class-dependent label-noise learning with cycle-consistency regularization. In NeurIPS , 2022.

Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instancedependent label noise: A sample sieve approach. In ICLR , 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD-96) , pages 226-231, 1996.

Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Gläser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Trans. Intell. Transp. Syst. , pages 1341-1360, 2021.

Aritra Ghosh and Andrew S. Lan. Contrastive learning improves model robustness under label noise. In CVPR , pages 2703-2708. Computer Vision Foundation / IEEE, 2021.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent - A new approach to selfsupervised learning. In NeurIPS , pages 21271-21284, 2020.

Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W. Tsang, James T. Kwok, and Masashi Sugiyama. A survey of label-noise representation learning: Past, present and future. CoRR , abs/2011.04406, 2020.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS , pages 8527-8537, 2018.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , pages 9726-9735, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV , pages 630-645, 2016.

Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In NeurIPS , pages 15637-15648, 2019.

Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing out-of-distribution examples via augmenting content and style. In ICLR , 2023.

Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. In Advances in Neural Information Processing Systems , volume 34, pages 26714-26725, 2021.

Zhuo Huang, Miaoxi Zhu, Xiaobo Xia, Li Shen, Jun Yu, Chen Gong, Bo Han, Bo Du, and Tongliang Liu. Robust generalization against photon-limited corruptions via worst-case sharpness minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1617516185, 2023.

Davood Karimi, Haoran Dou, Simon K. Warfield, and Ali Gholipour. Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis. Medical Image Anal. , page 101759, 2020.

Taehyeon Kim, Jongwoo Ko, Sangwook Cho, Jinhwan Choi, and Se-Young Yun. FINE samples for learning with noisy labels. In NeurIPS , pages 24137-24149, 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS , pages 1097-1105, 2012.

Jingling Li, Mozhi Zhang, Keyulu Xu, John Dickerson, and Jimmy Ba. How does a neural network's architecture impact its robustness to noisy labels? In NeurIPS , pages 9788-9803, 2021.

Junnan Li, Richard Socher, and Steven C. H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In ICLR , 2020.

Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. CoRR , abs/1708.02862, 2017.

Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. In ICML , pages 6403-6413, 2021.

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV , pages 740-755, 2014.

Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. In NeurIPS , pages 20331-20342, 2020.

Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence , 38(3):447-461, 2016.

Yang Liu. Understanding instance-level label noise: Disparate impacts and treatments. In ICML , pages 6725-6735, 2021.

Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In ICLR , 2017.

Yangdi Lu, Yang Bo, and Wenbo He. Noise attention learning: Enhancing noise robustness by gradient scaling. In NeurIPS , 2022.

Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11):2579-2605, 2008.

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR , pages 2233-2241, 2017.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In NeurIPS , pages 3008-3021, 2020.

Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML , pages 6105-6114, 2019.

Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In CVPR , pages 5552-5560, 2018.

Anne M Treisman and Garry Gelade. A feature-integration theory of attention. Cognitive psychology , pages 97-136, 1980.

Peter Welinder, Steve Branson, Serge J. Belongie, and Pietro Perona. The multidimensional wisdom of crowds. In NeurIPS , pages 2424-2432, 2010.

Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris N. Metaxas, and Chao Chen. A topological filter for learning with label noise. In NeurIPS , pages 21382-21393, 2020.

Songhua Wu, Tongliang Liu, Bo Han, Jun Yu, Gang Niu, and Masashi Sugiyama. Learning from noisy pairwise similarity and unlabeled data. Journal of Machine Learning Research , 23(307):1-34, 2022.

Songhua Wu, Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Nannan Wang, Haifeng Liu, and Gang Niu. Class2simi: A noise reduction perspective on learning with noisy labels. In ICML , pages 11285-11295. PMLR, 2021.

Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust early-learning: Hindering the memorization of noisy labels. In ICLR , 2021.

Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In NeurIPS , pages 7597-7610, 2020.

Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurIPS , pages 6835-6846, 2019.

Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR , pages 2691-2699, 2015.

Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L_dmi: A novel information-theoretic loss function for training deep nets robust to label noise. In NeurIPS , pages 6222-6233, 2019.

Yihao Xue, Kyle Whitecross, and Baharan Mirzasoleiman. Investigating why contrastive learning benefits robustness against label noise. In ICML , volume 162, pages 24851-24871, 2022.

Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instancedependent bayes-label transition matrix using a deep neural network. In ICML , pages 25302-25312, 2022.

Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual T: reducing estimation error for transition matrix in label-noise learning. In NeurIPS , pages 7260-7271, 2020.

Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with biased complementary labels. In ECCV , pages 69-85, 2018.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR , 2017.

Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR , 2018.

Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M. Bronstein, and Or Litany. Contrast to divide: Self-supervised pre-training for learning with noisy labels. In WACV , pages 387-397, 2022.

Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Ressl: Relational self-supervised learning with weak augmentation. 2021.

Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance-dependent label noise. In CVPR , pages 10113-10123, 2021.

A CIFAR20-SDN

A.1 Construction Details

Subclass-dominant label noise (SDN) is designed to mimic human labeling errors, which often emerge from distinct characteristics of examples. To identify such examples, we employ CIFAR-100, which provides two types of labels: class labels and sub-class labels. Samples belonging to the same subclass are considered to possess distinct characteristics. During the generation of the SDN dataset, we randomly flip a specified number of the last subclass labels within each class to the subsequent class. For instance, the labels of the "whale" subclass within the "aquatic mammals" class are probabilistically flipped to the "fish" class. Table 6 illustrates the flipped sub-classes and the relationships between classes and sub-classes in CIFAR-100. For simplicity, we refer to this newly generated SDN dataset as CIFAR20-SDN.

A.2 Diverse Challenges across Different Classes

Memorized percentage

10

Memorized percentage

10

aquatic

Memorized percentage

10

fish

Memorized percentage

10

flowers

containers

fruit

electrical

6

0.8

0.8

.81

0.6

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

02

0.2

.2

0.0

020.40.60.8

1.0

0.0

0.2

2040.60.8

1.0

0.0

020.40.60.8

1.0

0.0

0.2

040.60.8

1.0

0.0

0.2

0.40.60.8

10

0.0

0.2

0.4

0.60.8

10

0.0

02040.60.8

1.0

Noise rate in dataset

Noise rate in dataset

Noise rate in dataset

Noise rate in dataset

Noise rate in dataset

Noise rate in dataset

Noise rate in dataset

furmiture

insects

carnivores

1.0

man-made

natural

omnivores

m-mammals

0.8

0.8

0.8

06-

0.6

0.6

0.6

0.6

0.6

0.6

0.4

0.4

0.4

Memorized

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.2

0.2

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.60.8

1.0

0.0

0.2

0.4

0.60.8

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.60.8

1.0

Noise rate in dataset

Noise rate in dataset

Noise rate in dataset

Noise rate in dataset

Noise rate in dataset

Noise rate in dataset

Noise rate in dataset

Memorized percentage

10

invertebrates

Memorizedpercentage

10

people

Memorized percentage

10

reptiles

percentage

10

s-mammals

trees

Memorizedpercentage

10

vehicles 1

Memorized percentage

10

vehicles 2

0.8

0.8

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.6

0.6

0.4

0.4

0.4

Memorized

0.4

0.4

0.4

0.4

0.2

0.2

0.2

02

0.2

021

0.2

0.0

0.204

0.6

80

1.0

0.0

0.2

0.4

0.6

80

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.60.8

10

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.60.8

1.0

Noise rate in dataset

Noise rate in dataset

Noise rate in dataset

Noise rate in dataset

Noise rate in dataset

Noise rate in dataset

Noise rate in dataset

CIFAR20-SDN presents a multifaceted landscape with its array of sub-classes, paving the way for a comprehensive study of SDN and a deeper insight into the intricacies of early stopping. In our experiments, we adjust the overall noise rate from 0% to 20%. This implies that within each specific subclass, the noise rate varies from 0% to 100%.

As illustrated in Figure 9, the effectiveness of early stopping in addressing SDN is notably inconsistent, heavily contingent on the specific subclass. Our analysis identifies two primary factors that significantly influence the success of early stopping. First, when the mislabeled subclass shares substantial semantic similarities with the clean class, distinguishing mislabeled examples from the clean class becomes challenging. For instance, mislabeling an instance from the "whale" subclass as a "fish" class is difficult to discern from other "fish" instances. Second, when classes have low diversity, the network tends to identify outliers, thus facilitating the detection of mislabeled examples, such as "trees" and "flowers". Conversely, distinguishing between correctly labeled and mislabeled examples becomes considerably more challenging in classes with higher diversity, such as "insects".

A.3 Clusters Effects

We train a ResNet-18 on CIFAR20-SDN for 200 epochs with an overall noise rate of 18% (SDN-18). Afterward, we utilized t-SNE to visualize all 45,000 training examples. By comparing Figure 10 (a) and (b), we can pinpoint the noise locations, specifically evident in the small clusters. Notably, the vast majority of the mislabeled examples tend to cluster together and usually maintain a distance from their noisy class. This pattern suggests that clustering algorithms, such as DBSCAN, can effectively identify them. However, when the semantics of mislabeled examples closely match those of the noisy class, these mislabeled examples' clusters may not be distinct from the main cluster of the noisy class, which is a limitation of our proposed method.

1.0

1.0

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.0

0.2

0.4

0.6

0.8

1.0

aquatic

fruit

large carnivores

medium mammals

small mammals

genbe

fruit

large carnivores

medium mammals

small mammals

fish

electrical

large man-made

invertebrates

trees

fish

electrical

large man-made

invertebrates

trees

flowers

containers

household furniture

insects

large natural

omnivores

1people

reptiles

vehicles 1

vehicles 2

fowers

containers

insects

household furniture

large natural

omnivores

people

reptiles

vehicles 2

vehicles 1

(a) CIFAR20-SDN with noisy labels

(b) CIFAR20-SDN with clean labels

B More Details about Experimental Settings

Data preprocessing . For all experiments, we employ simple data augmentation, including random crop and horizontal flip. For the Clothing-1M dataset, images are initially resized to a resolution of 256 x 256 pixels. Subsequently, a random crop is applied to produce images of 224 x 224 pixels, which are then followed by a random horizontal flip.

Hyper-parameters of semi-supervised learning . We follow [30, 3], and combine NoiseCluster with the semi-supervised learning method, MixMatch [5]. For experiments with SSL, we set K = 2 , T = 0 . 5 , and λ u = 0 . α begins at 4 and drops to 0 . 75 after half of the total number of epochs.

Clothing1M . Due to the large size of the Clothing1M dataset [52], which consists of one million training examples, the computational time for t-SNE is significantly extended. This volume of data also results in changes to the hyper-parameters. To address these issues, we split the Clothing1M training data into two parts: a larger subset containing 950,000 examples and a smaller one with 50,000 examples. We initially employ the larger subset for training a model with early stopping. Subsequently, based on the trained model, we select and correct confident data from the smaller subset. This corrected data is then used for the remainder of the training process. Note that the model used in the Clothing1M experiments has been pre-trained with the ImageNet-1k dataset [28], leading to the generation of separable representations. As a result, late stopping is omitted in this case. When applying NoiseCluster to a model trained from scratch, it is advisable to employ late stopping.

C Additional Experiments

C.1 Ablation Study for Stopping Epoch

The effectiveness of NoiseCluster relies on long-trained representations derived from a model trained with later stopping, which stops the training process significantly later than what is common with early stopping. This raises two pertinent questions: How is the stopping epoch determined for later stopping? And does it display the same sensitivity as early stopping? To address these questions, we conduct a series of ablation experiments, adjusting the stopping epoch from 20th to 200th epochs.

(a) Noise rate 12%

(b) Noise rate 20%

90

90

80

80

Test Accuracy

Test Accuracy

70

70

60

60

50

NoiseCluster

50

NoiseCluster

BestBaseline

BestBaseline

40

20

50

80100

150

200

40

20

50

80100

150

200

Stopping Epoch

Stopping Epoch

As shown in Figure 11, NoiseCluster does not perform well with representations obtained at the 20th epoch, indicating that the effectiveness of NoiseCluster hinges on long-trained representations. Additionally, we can observe that NoiseCluster maintains a stable performance between the 50th and 100th epochs. This span is considerably wider compared to early stopping, which typically require a precise stopping epoch or operate within a relatively narrow range [30, 34, 3]. To delve deeper into the impact of the stopping epoch, we expand the training period for later stopping, and stop the training up to the 200th epoch. Despite a slight drop in the performance of NoiseCluster, attributable to the degradation of representation quality, it remains superior to the best baseline, CDR[49]. These experiments suggest that the choice of the stopping epoch in NoiseCluster can be made with considerable flexibility, displaying minimal sensitivity and negligible impact on the performance of NoiseCluster over a broad range.

C.2 Explore Noise-modeling-based Approaches on SDN

In this section, we further investigate into why noise-modeling-based approaches are ineffective for SDN. Methods based on noise modeling build a transition matrix between the clean distribution and noisy distribution [40, 51]. We utilize the state-of-the-art method, V olMinNet [32], on CIFAR20-SDN with a noise rate 18%. Following the original paper [32], we set ˆ T to -2 and -4.5, respectively.

(a)

ˆ

T

=

-

2

(b)

ˆ

T

=

-

4

.

5

[[0.450.040.020.020.020.020.020.020.040.020.020.030.040.030.020.040.050.030.020.03]

[0.080.490.030.020.020.020.020.020.020.020.020.020.020.020.030.030.060.020.020.02]

[0.02 0.030.5

0.130.030.030.020.030.020.020.020.020.020.020.020.020.020.020.020.02]

[0.020.02

0.030.470.036

[0.020.02

0.030.060.51

0.070.020.026

0.020.020.020.026

0.020.030.036

0.020.020.02

0.020.02]

[0.020.02

0.020.050.04

0.480.03

0.03

0.026

0.020.02

0.02

0.020.030.03

0.030.020.02

0.020.02]

[0.020.02

0.020.030.02

0.056

0.53

0.03

0.02

0.020.02

0.02

0.020.02]

[0.020.02

0.030.030.02

0.03

0.03

0.5

0.03

0.020.02

0.02

0.020.050.02

0.030.030.02

0.020.02]

[0.030.02

0.020.020.02

0.020.02

0.04(

0.5

0.030.020.03

0.040.030.02

0.030.030.026

0.020.02]

[0.030.02

0.020.020.02

0.020.02

0.02

0.020.520.040.03

0.020.020.02

0.020.020.02

0.030.04]

[0.020.03

0.020.02

0.02

0.020.02

0.02

0.026

0.040.540.076

0.030.03]

[0.04 0.02

0.020.020.02

0.020.020.020.060.030.030.496

0.040.020.030.030.030.030.020.02]

[0.040.02

0.020.020.02

0.020.020.020.050.050.020.04

0.460.030.020.030.060.040.020.02]

[0.030.02

0.020.020.03

0.030.020.040.030.020.020.02

0.040.480.030.040.030.030.020.02]

[0.020.02

0.020.020.02

0.02 0.020.470.170.020.020.020.02]

[0.040.03

0.030.020.02

0.030.020.040.030.030.020.030.040.050.030.430.050.030.020.03]

[0.040.030.020.020.026

0.020.020.030.030.020.020.030.050.030.030.030.450.050.020.02]

[0.020.020.020.020.020.020.020.020.020.020.030.020.020.020.020.020.020.470.170.02]

0.05]

[0.040.020.020.020.020.030.020.020.020.040.020.020.020.020.020.020.020.020.050.52]]

[[0.850.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.01]

[0.01

0.86

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.010.01]

[0.01

0.01

0.85

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.010.01]

[0.01

0.01

0.01

0.86

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01]

[0.01

0.01

0.01

0.01

0.86

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01]

[0.01

0.01

0.01

0.01

0.01

0.86

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01]

[0.01

0.01

0.01

0.01

0.01

0.01

0.86

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01]

[0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.86

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01]

[0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.86

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.011

[0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.86

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.011

[0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.86

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01]

[0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.86

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01]

[0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.85

0.01

0.01

0.01

0.01

0.01

0.01

0.01]

[0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.85

0.01

0.01

0.01

0.01

0.01

0.011

[0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.86

0.01

0.01

0.01

0.01

0.01]

[0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.85

0.01

0.01

0.01

0.01]

[0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.85

0.01

0.01

0.01]

[0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.86

0.01

0.01]

[0.016

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.01

0.860.01]

[0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.86]]

As depicted in Figure 12, it is evident that ˆ T holds the capability to modulate the extent of estimated noisy class-posterior probabilities within the transition matrix. For example, with ˆ T = -2 , VolMinNet predicts higher noisy class-posterior probabilities in the transition matrix compared to that with ˆ T = -4 . 5 . However, neither of these ˆ T settings successfully enable an accurate estimation of the transition matrix. Consequently, these inaccurately estimated transition matrices result in inferior performance on SDN.

C.3 Training Time Comparison

We evaluate the efficiency of our proposed method with state-of-the-art baselines. Table 7 displays the total training time on CIFAR20-SDN. We can see that the training time of NoiseCluster is comparable to that of TopoFiler [46] and faster than JointOptim [43] and CDR[49]. Furthermore, NoiseCluster+ exhibits substantial efficiency compared with other baselines that employ SSL. For instance, it requires less than half the time needed by DivideMix [30].

D Exploring SDN in Real-world Datasets

Due to space constraints in the main paper, we presented only a limited number of SDN examples. In this subsection, we provide additional visual evidence to further demonstrate the presence of SDN in the Clothing1M and WebVision datasets.

1.0

1.0

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0

0.2

0.4

0.6

80

1.0

0.0

0.0

0.2

0.4

0.6

80

1.0

T-Shirt

Chiffon

Windbreaker

Suit

vest

T-Shirt

Chiffon

Windbreaker

1Suit

Vest

Shirt

Knitwear

Hoodie

Sweater

Down Coat

Jacket

Shawl

Dress

Underwear

Shirt

Knitwear

Hoodie

Sweater

Down Coat

Jacket

Shawl

Dress

Underwear

1.0

1.0

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0

0.2

0.4

0.6

80

1.0

0.0

0.0

0.2

0.4

0.6

80

1.0

T-Shirt

Chiffon

Windbreaker

Suit

Vest

Correct

Knitwear

Hoodie

Down Coat

Dress

Sweater

Jacket

Shawl

Underwear

T-Shirt

Chiffon

Windbreaker

Suit

Vest

Knitwear

Hoodie

Down Coat

Dress

Shirt

Sweater

Jacket

Shawl

Underwear

(c) Confident examples, clean labels

(d) Mislabeled examples in confident set, clean labels

D.1 SDN in Clothing1M

Figure 13 presents a step-by-step exploration process for the Clothing1M dataset, demonstrating how features of mislabeled examples are clustered together based on their semantics. Given the plethora of terms available for different clothing items, we can use accurate names to pinpoint these sub-classes. We highlight three sub-classes prominently impacted by label noise: "Sleep T-Shirt," "Leather Jacket," and "Down Vest," as shown in Figures 14, 15, and 16.

YES

D

We can also observe that some mislabeled examples are not clustered together, but are instead evenly dispersed in some classes. It is important to understand that in the real world, clothing categories are not solely distinguished by visual features. For instance, the features of "Sweater" and "Knitwear" are intertwined, with the main distinction being the material, not the visual appearance. Similarly, the categories "Underwear" and "Vest" (correct name:"singlet" or "tank top") lead to confusion since some garments can serve multiple purposes.

Removing correctly labeled examples. Early stopping not only helps remove mislabeled examples but also inadvertently excludes those that are correctly labeled when SDN exists. A closer inspection of instances within the "Leather Jacket" cluster confirms the issue: out of the total examples, 313 are mislabeled, with only 23 correctly labeled. This finding indicates that more than 90% of the "Leather

EAVET

BO

ALO

善

Jacket" examples are incorrectly labeled in the original dataset. Furthermore, the implementation of early stopping leads to the removal of the small fraction of correctly labeled examples from the confident set, as shown in Figure 17, pushing the noise rate from over 90% to close to 100%.

D.2 SDN in WebVision

To extend our evaluation of SDN prevalence, we engage in experiments using another real-world dataset: WebVision V1.0 [31]. Following the setting presented in [8], we utilize the first 50 classes of WebVision, known as "mini WebVision". To demonstrate the flexibility of integrating with existing

early stopping methods, we employ DivideMix to select confident examples and extract long-trained representations from two networks, which have been trained for 80 epochs. The extracted features are presented in Figure 18. We utilize an ensemble of pre-trained models to produce pseudo clean labels, and their performance is depicted in Table 8.

(a) Noisy labels

(b) Pseudo clean labels

1.0

1.0

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0

0.2

0.4

0.6

80

1.0

0.0

0.0

0.2

0.4

0.6

80

tench, Tinca tinca

tench, Tinca tinca

goldfish, Carassius auratus

goldfish, Carassius auratus

great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias

great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias

hammerhead, hammerhead shark

hammerhead, hammerhead shark

electric ray, crampfish, numbfish, torpedo

electric ray, crampfish, numbfish, torpedo

stingray

stingray

cock

cock

hen

hen

ostrich, Struthio camelus

ostrich, Struthio camelus

others

others

In Figures 19, 20, and 21, it is evident that, despite clear discrepancies between mislabeled and correctly labeled images, methods based on early stopping struggle to effectively identify these mislabeled examples. Moreover, these mislabeled samples do not fall into conventional categories such as "human" or "cars". Instead, they cluster according to different criteria, sharing similarities in attributes like shape and color with their correctly labeled counterparts. By considering out-of-domain examples as a single, unified class-termed "other", the mislabeled examples within each class can be interpreted as sub-classes of this "other" category.

Peer reviews:
comments: SUMMARY:
This paper proposes a new type of noisy labels called subclass-dominant noisy labels and introduces an algorithm called NoiseCluster based on this noisy label modeling. In the experimental section, the authors demonstrate the superiority of NoiseCluster over previous approaches in the subclass-dominant noisy label cases. Additionally, NoiseCluster exhibits better performance than previous algorithms on the real-world Clothing1M dataset.

STRENGTHS:
- Introduces a new type of noisy label modeling.
- Well-written paper.
- Conducts extensive experiments.

WEAKNESSES:
- The justification for the proposed subclass-dominant noisy label model is lacking. It is unclear whether this type of noisy labels frequently occurs in the real-world. For instance, it would be helpful to investigate if the Clothing1M dataset exhibits similar phenomena.
- The analysis should include the performance of NoiseCluster on other types of noisy label models such as symmetric, instance, or asymmetric. This would provide a more comprehensive evaluation of the algorithm.
- Further analysis on real-world datasets is necessary. While the authors evaluate NoiseCluster on the Clothing1M dataset, it would be beneficial to assess its performance on other real-world benchmarks such as WebVision, Food101N, and similar datasets.

QUESTIONS:
-

LIMITATIONS:
The limitations of this paper are summarized in the "Question" and "Weakness" sections
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the impact of early stopping on models trained with noisy labels and introduces a new type of label noise called subclass-dominant label noise (SDN). The experiments reveal that later stopping during training can better capture the high-level semantics of noisy examples. Building upon this finding, the authors propose a novel approach named NoiseCluster, which utilizes the geometric structures of long-trained representations to detect and correct SDN. The experimental results demonstrate that NoiseCluster outperforms several label noise robust methods on both SDN and Clothing-1M datasets.\n\nSTRENGTHS:\n- This paper introduces a novel type of label noise called subclass-dominant label noise (SDN), providing a well-motivated and insightful perspective on the limitations of early stopping in the presence of SDN. By demonstrating the inapplicability of early stopping for training with this type of label noise, the authors shed light on the challenges faced by early stopping in handling real world label noise.\n\n- Experimental results showcase the superior performance of the proposed method on both synthetic and real-world datasets, highlighting its effectiveness in addressing the issues posed by SDN.\n\nWEAKNESSES:\n- The paper would benefit from a comparison with instance-dependent label noise robust methods, which could provide a comprehensive evaluation of the proposed approach.\n- The evaluation of methods in this study is limited to a single synthetic dataset and one real-world dataset. Assessing the proposed approach on a wider range of datasets can further demonstrate the generalizability and robustness of their method in various practical scenarios.\n\nQUESTIONS:\nCould the authors provide further clarification on the differences between SDN and IDN? Moreover, given that there have been existing studies on IDN, what is the significance of specifically investigating and studying SDN?\n\nLIMITATIONS:\nplease refer to the Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors present a new type of label noise: subclass-dominant label noise. The authors show that the model trained over time can better capture such label noise in the feature space, and based on this idea, a clustering then correcting pseudo labels algorithm, NoiseCluster, is designed to identify and correct SDN. The proposed method can be combined with current label noise and semi-supervised learning methods, and is validated on the authors' constructed dataset, cifar20-SDN, and another real-world dataset, Clothing 1M.\n\nSTRENGTHS:\n- The paper is well-organized. The Introduction and related work sections are clear and well introduces the studied problem.\n    \n- According to the proposed subclass-dominant label noise characteristics, the authors constructed a corresponding dataset and proposed a methodology addressing it. The proposed approach is heuristically reasonable and applicable.\n\nWEAKNESSES:\n- My main concern is the existence of proposed subclass-dominant label noise (SDN) in the real world scenarios.\n    \n- The proposed method works well on the constructed dataset, CIFAR20-SDN. However, the performance improvement on real-world dataset, Clothing 1M is marginal. This makes the existence of SDN questionable.\n    \n- Lack of some analysis of clusting effects and results in experiments. Please refer to the Questions.\n\nQUESTIONS:\n- It will be more clear to describe the class and sub-class relations briefly, or giving a concrete example of CIFAR20-SDN in the main paper.\n    \n- Clustering Result: More analysis of clusting effects and results could make the proposed SDn and method more convincing. For example, on the CIFAR20-SDN dataset, does the clustering result consist with the manually created SDN noise? And what about the real-world dataset Clothing 1M?\n\nLIMITATIONS:\nThe authors have mentioned their limitations in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper, the authors uncover the phenomenon that mislabeled examples are quickly learned during the initial stages of training when Subclass-Dominant Label Noise (SDN) is present. This behavior hinders the effectiveness of early stopping-based robust methods. To address this issue, the authors propose an approach that does not rely on early stopping. The method involves identifying mislabeled examples by clustering the penultimate layer features and correcting them by assigning them labels of the closest class. Experiments are conducted to demonstrate the superiority of their method compared to existing approaches.\n\nSTRENGTHS:\n1.\tThe finding of the failure of early stopping under SDN is intriguing with practical implications, as SDN can be a common occurrence in practice.\n2.\tThe proposed method has significant improvements over existing methods under SDN.\n\nWEAKNESSES:\nOverall, while I appreciate the first part of the paper that discusses the failure of early stopping under SDN, I have some concerns and suggestions regarding the other parts:\n1. It would be beneficial to delve deeper into the phenomenon of early stopping’s failure under SDN, e.g., exploring why subclass dominance leads to wrong labels being learned quickly.\n2. The advantage of long-trained representations may not be surprising, as training progress can naturally lead to more distinguishable clusters. It seems that this property is not specific to SDN.\n3. Regarding the combination with SSL, it is unclear why there would be features that are not assigned any group index. Are these features the same as $U^c_{k-1}$ mentioned in section 4.1?\n4. The Noisecluster+ method, where labels are not corrected but instead discarded (as those examples are treated as unlabeled), achieves better performance than the class-correction based method. Does this mean label correction is not as effective?\n5. I am curious about the percentage of examples whose labels were successfully corrected by the label-correction method.\n6. In Table 5, is the method without label correction simply vanilla training? If that is the case, vanilla training already accomplishes the majority of the performance, and the performance improvements attributed to label correction appear relatively modest (1%). So the proposed method does not contribute much. \n7. ELR should also be included in Table 2, as it does not require SSL (only ELR+ does).\n\nQUESTIONS:\nIn addition to the questions raised in Weaknesses, I have the following additional questions:\n1. Regarding Cloth1M, why was only one epoch used for training? Furthermore, what is the rationale behind training on 95% of the data first and then applying SDN with only 5%? Currently it is unclear why these modifications are made. \n2. The method doesn't appear to be specifically designed for SDN alone, as identifying mislabeled examples and the label correction part make sense for general noise as well. How does the proposed method perform under normal noise?\n3. Since the authors mentioned incorporating self-supervised learning, it would be valuable to discuss self-supervised pretraining, which has been shown to enhance robust methods [1][2][3][4]. As features are initially learned without labels, starting from a pretrained model might lower the chance of mislabeled examples in SDN being learned early. Therefore, I wonder if early stopping could potentially be effective with this pretraining even under SDN. \n\n[1] Hendrycks, Dan, et al. \"Using self-supervised learning can improve model robustness and uncertainty.\" Advances in neural information processing systems 32 (2019).\n\n[2] Zheltonozhskii, Evgenii, et al. \"Contrast to divide: Self-supervised pre-training for learning with noisy labels.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.\n\n[3] Xue, Yihao, Kyle Whitecross, and Baharan Mirzasoleiman. \"Investigating why contrastive learning benefits robustness against label noise.\" International Conference on Machine Learning. PMLR, 2022.\n\n[4] Ghosh, Aritra, and Andrew Lan. \"Contrastive learning improves model robustness under label noise.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nLIMITATIONS:\nThe authors have discussed limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}