CS-Isolate: Extracting Hard Confident Examples by Content and Style Isolation
Authors: Yexiong Lin
Yu Yao
Xiaolong Shi
Mingming Gong
Xu Shen
Dong Xu
Tongliang Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: learning with label errors

Abstract: Label noise widely exists in large-scale image datasets. To mitigate the side effects of label noise, state-of-the-art methods focus on selecting confident examples by leveraging semi-supervised learning. Existing research shows that the ability to extract hard confident examples, which are close to the decision boundary, significantly influences the generalization ability of the learned classifier.
In this paper, we find that a key reason for some hard examples being close to the decision boundary is due to the entanglement of style factors with content factors. The hard examples become more discriminative when we focus solely on content factors, such as semantic information, while ignoring style factors. Nonetheless, given only noisy data, content factors are not directly observed and have to be inferred.
To tackle the problem of inferring content factors for classification when learning with noisy labels, our objective is to ensure that the content factors of all examples in the same underlying clean class remain unchanged as their style information changes.
To achieve this, we utilize different data augmentation techniques to alter the styles while regularizing content factors based on some confident examples. By training existing methods with our inferred content factors, CS-Isolate proves their effectiveness in learning hard examples on benchmark datasets. The implementation is available at https://github.com/tmllab/2023_NeurIPS_CS-isolate.

CS-Isolate: Extracting Hard Confident Examples by Content and Style Isolation
Yexiong Lin' Yu Yao Xiaolong Shi
Mingming Gong' Xu Shen Dong Xu
Tongliang Liu-*
'The University of Sydney; MMohamed bin Zayed University of Artificial Intelligence; CCrneeiie Mellon University: "The University of Melbourne; 6 Alibaba DAMO Academy; Thhe University of Hong Kong.
Abstract
Label noise widely exists in large-scale image datasets. To mitigate the side effects of label noise, state-of-the-ar methods focus on selecting confident examples by leveraging semi-supervised learning. Existing research shows that the ability to extract hard confident examples, which are close to the decision boundary, significantly influences the generalization ability of the learned classifier, In this paper, we find that a key reason for some hard examples being close to the decision boundary due to the entanglement of style factors with content factors. The hard examples become more discriminative when we focus solely on content factors, such as semantic information, while ignoring style factors. Nonetheless, given only noisy data, content factors are not directly observed and have to be inferred. To tackle the problem of inferring content factors for classification when learning with noisy labels, our objective is to ensure that the content factors of all examples the same underlying clean class remain unchanged as their style information changes. To achieve this, we utilize different data augmentation techniques to alter the styles while regularizing content factors based on some confident examples. By training existing methods with our inferred content factors, CS Isolate proves their effectiveness in learning hard examples on benchmark datasets. The implementation is available at https: /et///gttttt com/tmllabi 2023_Neur Neur IPS CS isolate.
1 Introduction
Large-scale machine learning datasets frequently contain noisy labels, in datasets like ImageNet [9] and Clothing M [55]. Training deep neural networks with such noisy data would result in poor generalization ability, these networks memorize incorrect labels [13, 3].
To mitigate the side effects of label noise, different methods have been proposed [33, 13, 37, 61, 30, 31, 49]. Major existing of methods are based confident examples selection [61, 2]. Intuitively, those methods first exploit the memorization effect, enabling deep neural networks to learn simple patterns shared by the majority of training examples [60, 1, 19]. Since clean labels typically constitute the majority in each noisy class [7, 39], deep neural networks initially fit the training data with correct labels and then progressively fit examples with incorrect labels [13]. To prevent the model from fitting incorrect labels, early stopping is usually employed [44, 3, 50, 4]. Then the small-loss trick is used to extract the confident examples with high certainty [21, 35, 28, 38]. extracted examples have high quality, the performance of classifier can be enhanced.
"Corresponding author: Tongliang Liu (tongliang, liu @sw.ncyccdd..uu).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Figure 1: Illustrating the entanglement of con- tent and style factors. The circles on the left side of the dotted line represent examples with the underlying clean label Y O. while the ones on the right side represent examples with clean label Y 1. The blue filling corresponds to the noisy label Y 0, and the yellow filling corre- sponds to the noisy label Ý = 1. Black outlines indicate that the labels of examples are correct, whereas red outlines indicate that the labels of examples are incorrect. In (a), we visualize the underlying style factor Z₂ and the underlying To improve the performance of a classifier, it is crucial to ensure that the confident examples have a distribution similar to that of clean data [36, 321. Specifically, it is necessary to extract not only confident examples that are far from the decision boundary but those that are close to it. The former examples are easy to be identified and extracted. However, the latter examples, which are close to the decision boundary, often become entangled with mislabeled examples, making them challenging to be identified or extracted. In this paper, we discover that on image datasets, the entanglement of unhelpful style factors with useful content factors is key reason that leads to the hard examples becoming hard to be classified, and thus these examples are close to the decision boundary.
In Fig. 1, we provide an intuitive understand- ing of the relationship between hard examples and the entanglement of style factor Z, and con- tent factor Ze- Let us first assume the under- lying content factor Ze and style factor Z, are given. In Fig. 1a, by visualizing Z, and Ze. we can observe separation between the examples belonging to underlying clean classes and 1. In real-world scenarios, however, Zo and Z. are not directly available. Instead, existing methods [13, 28] could learn representations Z1 and Ze from noisy data to extract confident examples. These learned representations, Z₁ and Zz. are nonlinear transformations of style factor Z5 and content factor Ze. They do not ensure the dis- entanglement of Z, and Ze [24, 20, 59]. As the example illustrated in Fig. 1b, both Zi and Zz contain the information of the style factor Z. and the content factor Ze. In the illustrated represen- tation space, examples within the region encircled by red dashed lines are close together. The con- fident examples in this region are entangled with mislabeled examples, making them challenging to be extracted by existing methods without isolating (disentangling style and content information in learned representations.
Z.
(a)
(b)
content factor Ze. (b) shows the impact of a non- linear transformation on Z, and Ze, leading to their entanglement in the new space defined by representations Zi and Zz.
We have discussed that the entanglement of style factor Z. and content factor Z, in the representation space can create hard examples. However, if we can isolate content from styles, then by focusing solely on the content fac- tor Ze, these hard examples become more discriminative. For example, let's 's consider projecting the examples onto the Ze-axis by ignoring style factor Z, in Fig. 1a. The data points on the Zaiix in Fig. 1 illustrate that even after the nonlinear transformation, the examples remain distinguishable. This motivates us to tackle the problem of inferring content factors for extracting hard examples when learning with noisy labels.
1.05
AuC lolate 0.855, DivideMlic 0.560, Momeramc
Precision: 0.954
Precision: 0.957
Recall: 0.966
Recall: 0,978
1.00
0.95
0.90
Precision: 0,896
Rocall: O, 9.94
0.85
AUG of CS-Isclate
AUG of DivisoMix
0.80
Mo-Momentum
0
50
100
160
200
250
300
Epochs
To isolate style and content information in learned repre- sentations, it is crucial to learn invariant representations of all examples as their styles change. However, when data contains label noise, we can not access clean classes, making it challenging to achieve this isolation. To address this issue, we introduce a method built upon identifiable Variational Autoencoder (iVAE) [23]. The core intuition behind our approach is to construct different styles for each example through various data augmentation techniques. We then encourage both the original and the augmented examples to possess different style factors but maintain
Figure 2: Illustrate the comparison of the performance of confident examples selection of our method CS-Isolate with DivideMix and Me-Momentum on CI- FARION Worst [48]. The highest points on AUCs are indicated by arrows. High precision indicates that most of the con- fident examples are correctly labeled, while recall indicates the fraction of cor- rectly labeled examples out of all exam- ples that are correctly labeled.
2

---

identical content factors, which is self-supervised learning manner. This initial step could achieve preliminary level of isolation between style and content factors. As the training process progresses. we can identify and select certain confident examples based on the content factors. Leveraging these confident examples, we further enhance the isolation process by encouraging all examples sharing the same label to share identical content factors. By isolating content and style factors, our method, CS- ES-Isolate, effectively helps existing sample selection methods to extract hard examples typically overlooked by existing sample selection methods. In Fig 2. we evaluate the performance of confident examples selection on CIFAR ION with the noise type "worst" [48] by using two metrics: precision and recall. When comparing these two metrics, CS-Isolate consistently outperforms DivideMix [28] and Me- Mle-Momentum [2]. This result demonstrates that our method can not only select confident examples more accurately, shown by the higher precision, but also capture a larger portion of the correctly labeled examples from the entire dataset, as evidenced by the higher recall. By improving the quality of confident examples with the isolation of the content and style factors, method effectively helps improve the test accuracy of existing methods. We have also theoretically analyzed the identifiability of content and style factors in Appendix A.
2 Background and Related Work
Problem setup. Let's denote D as the distribution of a noisy example (X, Y) from the set X' x {1.2...., C}. where X denotes the variable of instances, Y represents the variable of noisy labels, X' is the feature space, 21, C) is the label space, and C' is the total number of classes. In the learning scenario with noisy labels, clean labels are not observed. Given a noisy training sample 5= (z,, d 1. independently drawn from D, the objective is to leverage this sample to learn a classifier robust against label noise.
Sample selection methods for learning with noisy labels. In learning with noisy labels, major current state-of-the-art (SOTA) methodologies predominantly involve sample selection strategies. These strategies seek to divide the dataset into confident and unconfident examples. The basis of these strategies is the exploitation of the memorization effect inherent in deep neural networks. The memorization effect enables the networks to initially grasp and learn the simple patterns, and then learn the complex patterns gradually [60, 1]. Given that clean labels usually form the majority within each noisy class [7, 391, these networks would initially fit the examples with accurate labels, and subsequently fit the examples with incorrect labels over time [13].
Preventing the learning model from fitting incorrect labels is crucial to ensure sample selection quality. To achieve this, strategies such as early stopping are often employed [44, 3]. Additionally, the small- loss trick is utilized to identify and extract confident examples with high certainty [28, 38, 51, 18). Some variation has also been proposed, e.g., some methods opt to reweight examples, thereby decreasing the contribution of mislabeled samples to the overall loss [41, 13]. To guarantee the statistical consistency of algorithms, Jiacheng et al. [7] introduces active learning acquire the labels of randomly chosen examples from unconfident examples to mitigate the bias introduced by sample selection.
Moreover, the set of confident examples often undergoes dynamic changes during the training stage. This is achieved by leveraging semi-supervised learning methods to relabel training instances and reselect confident and unconfident examples using the small loss trick. Various techniques for this purpose have been proposed and empirically demonstrated superior performance, including consistency regularization [27] adopted by [10], MixMatch [5] used by [28], co-regularization by [47], and contrastive learning by [30. 29, 45, 8, 11, 57. 61]. Self- Self-training and co-training have been used by [2] and [35, 13, 21], respectively.
Z
x
(a)
z
(b)
Figure 3: The different
data generative processes with or without auxiliary variables.
Representation learning by generative model. Consider data gener- ation in Fig. 3a, X is the observed data, and Z is the unknown underlying representation to generate X. Variational autoencoder framework [24] can be used to learn the latent representation. In this process, standard normal distribution is utilized as a prior for the latent variables, and variational posterior q( x is employed to approximate the unknown underlying posterior X(). Disentangled representation is very important, which
3

---

can allow a rich class of properties to be imposed on the learned representation, such as sparsity and clustering. To disentangle the representation. the variational autoencoder framework has been further expanded by modifying the original loss function, resulting in various algorithms. 3-VAF [16] suggests an adaptation framework that adjusts the weight of the KI. term to balance the independence of disentangled factors and reconstruction performance. 3-TCVAE [6] further analyzes the KI term of B- VAV [16] and only adjusts the total correlation term to achieve disentanglement. HOOD [17] uses the clean labels and domain labels to disentangle content and style factors, but the clean labels and domain labels are unknown in the setting of learning with noisy labels. These methods are toward the goal of disentanglement but do not have theoretical guarantees of identifiability of their inferred latent representations.
Disentangled latent factors via auxiliary variable. Recent studies [20, 23] show that theoretical guarantees of identifiability can be achieved if an auxiliary variable related to the representation can be obtained. Khemakhem et al. [23] provide identifiability of iVAE with additional inputs by employing the theory of nonlinear Independent Component Analysis (nonlinear ICA) [20]. Intuitively, given a additional auxiliary variable U, i.e., the class label and the time index a time series, the latent factors are identifiable up to certain degree. The data generation processing is changed to Fig. 3b. To disentangle the representation, the method assumes that each factor of the representation is independent.
factorized prior distribution over the latent variables that are conditional
3 Content and Style Isolation When Learning with Noisy Labels
In this section, we present CS-Isolate, a method designed to help select hard confident examples by content and style isolation.
Us
3.1 Preliminaries
Noisy data generative process. To learn latent factors by leveraging generative models, the generative model has to model the noisy data generative process. Firstly, introduce the noisy data generative process. We denote observed variables with gray circles and latent variables with white circles. Specifically, the content factor Ze is generated by the latent label Y. The different style domains U. give rise to the different style factor Z.. Then, the image X is generated by the combined influence of the style factor Z, and the content factor Z... Noisy labels Y are then generated based on the image X. In general cases, Ze and Ze can also have statistical or causal dependencies. We follow existing work that assumes the content factors unchanged across different styles [46].
Ze
Zs
X
Figure 4: The noisy data gener- ative process.
Challenge in isolating content from styles without labels. Isolating content from style without labels is challenging task [46]. We first introduce the assumptions made by existing methods, as well as difficulties that may be encountered in practice.
Existing methods [26, 23, 46] for isolating content from styles without labels assume that the data augmentation for controlling each style factor can be designed, which means that we can intervene (control) all style factors. For instance, the data augmentations for controlling rotation angle and scaling of images can designed by using affine transformation [40]. When we apply a data augmentation that rotates an image, the style factor for the rotation angle changes in the augmented image compared to the original. Similarly, when we scale an image, the style factor for scaling becomes different in the augmented image. If we have the data augmentation to control each style factor, then by training generative model with the augmented images, the model can then identify style factors by comparing the changes in styles between the original and augmented images.
However, the major challenge lies in the fact that we generally cannot design sufficient data aug- mentations to control all style factors in an image. For instance, in the CIFAR- 10 dataset [25], some pictures with the label "horse" contain a person. In this context, the "person" acts as a style factor. Existing data augmentations cannot control this, as they cannot remove the person from images easily, This simple example illustrates that generally impossible to control all style factors through data augmentations. Then the assumption required by existing methods usually is hard to satisfy.

---

The violation of the assumption leads to the learned representations for content containing style information.
Challenge in isolating content from style with noisy labels. We generally cannot design data augmentations that can control all style factors in an image. In this case, clean labels are essential to help isolate content from styles. Intuitively, by comparing the change of images with different clean labels that share the same styles, one can infer content factors used for classification. This achieves isolation of content and style. However, when dealing with noisy data, relying on labels becomes problematic as they contain label errors. Images with the same content factors can have different noisy labels, making it fail to infer content factors used for classification. This situation further complicates the task of isolating content from styles, highlighting the challenges posed by noisy labels.
3.2 CS-Isolate for Extracting Hard Example
In this paper, we find that key reason for some hard examples being close to the decision boundary is the entanglement of style factors with content factors. Intuitively, style factors render the learned representation of certain examples less discriminative, thereby making them close to the decision boundary. If content and style information can be isolated, many hard examples would be easily distinguished by ignoring the style information. This is because we have left only content information in the learned representation, making these representations far from the decision boundary.
Inspired by these findings, we propose CS-Isolate, which aims to isolate content from styles for extracting confident examples. To achieve isolation, we utilize self-supervised learning. Specifically, we ensure that original and augmented images maintain the same content factors despite having different style factors due to data augmentations. This achieves a preliminary level of isolating content from styles. As previously mentioned, is generally impossible to manipulate all style factors through data augmentation. Some style factors remain uncontrolled and stay contained in the learned representations for content. To further encourage the isolation, labels of confident examples are used. Specifically, as training progresses, we identify and select confident examples whose labels likely to be accurate. By harnessing the labels from these confident examples, we encourage the examples with the same label to have the consistent content factors, irrespective of stylistic variations. This further isolates content from uncontrolled styles. By leveraging different data augmentations and confident examples, our method can effectively separate the learned latent representations into two parts which exclusively contain either style or content information. Subsequently, classifier heads can be trained purely on the representations containing content information. This approach not only improves the identification of hard confident examples but also enhances classification performance.
Isolating content from styles using auxiliary variables. We follow existing work using the variational autoencoder with auxiliary variables [23] to isolate content from styles. Let U2 and Ue denote the auxiliary variables that control the style factor Z, and content factor Ze. respectively. Specifically, by reconstructing an image with the supervision of auxiliary variables U, and Ue. we encourage the images with the same content information but different style information have the same content factor Ze but the different style factor Z,. This allows us to isolate content from styles within an image. Then only employing Ze for selecting hard confident examples.
However, given only noisy data, we face the challenge of the unavailability of auxiliary variables U. and U.. In response, we devise surrogates for these auxiliary variables. To find a surrogate for the style auxiliary variable U.. we employ different data augmentation techniques, each of which generates a unique style domain with distinct style factors. We then assign the style domain ID as the style auxiliary variable U.. To find a surrogate for the content auxiliary variable Uc, we adopt the philosophy of self-supervised learning to assign a unique content ID as the content auxiliary variable Ue to each image. The images that have the same content ID are encouraged to have consistent content factors. Moreover, during the learning process, some confident examples can be extracted. To effectively leverage these examples in learning content factors, we reassign the content ID) of confident examples belonging to the same class to be the same. This allows our model learn consistent content factors for the images in the same class.
Constructing style auxiliary variables via data augmentations. For the learning of distinet content factors and style factors, it is essential to construct images from domains with different styles
5

---

by using data augmentations. Each image carries its corresponding domain ID as the style auxiliary
variable U.. Let A (An, A2....... .AM) represents the set of data augmentations, and M the number of data augmentations. Note that, for convenience, we use Ag to denote data augmentation tations that come from the i-th style domain, we can assign its style ID to be i. Specifically, for each image æ, each data augmentation A; E AA is applied, resulting in a set containing pairs of the augmented image and the corresponding style ID, i.e.,
such that applying the data augmentation Ag to the image x does not change it at all, Le., x = Ao(x).
When applying these diverse data augmentations to an 2, different augmented images are obtained with different style factors. For the augmented image æd obtained by using the i-th data augmen-
{( == (x)x) (((A)) := i) Vi [O,...,M}},
where U is defined be the style ID (which serves the style auxiliary variable) of the image by applying i-th data augmentation. In this manner, for each image H, we can generate set of augmented images ([x^:) fi=n). including the original images and the ones with different style factors controlled by different style IDs.
Constructing content auxiliary variables in a self-supervised manner. To guide the generative model in learning the consistent content factor across images sharing the same content information, we aim assign the same content ID (which serves the content auxiliary variable) to images with the same content information. However, clean labels cannot be obtained in learning with noisy labels, meaning we cannot know which images have the same content information. To construct content auxiliary variables without clean labels, we adopt self-supervised learning. Specifically, we consider that different data augmentations applied to an image typically do not alter its content information. Hence, we can assign a unique content ID to each original image and its augmented versions. This assignment can be mathematically expressed as follows:
Sall = ((1)) == Al(x),U, (17²) = i (1)
= C ViE to, MJ, Vj {T,,,
where T is the total number of distinct original images in the training data, Ue represents the content ID of the j-th image in the training data after applying the data augmentation A,, and C is the number of classes.
Moreover, this initial assignment of content IDs can be refined by using confident examples, denoted as S₁. As the training process progresses, we can extract these confident examples by leveraging the small-loss trick [13, 3, 28). The content IDs are then further refined based on the labels of the examples within the confident examples. The refinement process can be described as:
U. )
=
V(zj, v5) E Si, Vi E {O..., M),
where of is the label of the confident example (f). After this refinement process, confident examples in the same class will have the same content ID. Consequently, these refined content IDs could enable the model to learn consistent content factors for the images in the same class.
Encouraging style and content isolation for extracting hard example. After obtaining the auxiliary variable, we isolate content from styles by leveraging iVAE [20]. Specifically, the prior distribution of the content factor Ze is conditional on the auxiliary variable Uc, i.e. Po. (Z. Similarly, the prior distribution of the style factor Z, is conditional on the auxiliary variable U.. i.e. Po. (U.). The O. and 6, are the learnable parameters of the distribution. The objective is to maximize the data likelihood which is as follows.
Egg (XIU,, U.)] Enp [//
po(X 2.) )oo (=e U.)oo (=.
where we use (n to denote the empirical distribution of the training sample Sall. We use the variational inference method to approximate the underlying posterior distribution pa (Ze: X,IX, U,, U.) [23, 24). Specifically, two inference models (encoders) qo. (2.lX) and qo, (Z, X)) are introduced to infer latent variables Ze and Z, respectively and model the distribution q(Z,, XxIX) that is used to approximate the distribution pa (Ze, Z. Uc, U.). Therefore, the distribution qi Zo: Xx(X) can be decomposed as follows:
g (2, X) = 9o. (2.

---

We learn the parameters {8,, 0s, de. det by maximizing the evidence lower-bound ELBO for each example (z, Uhe U,). The corresponding loss is:
min CELBO min Bup - 0 0
0c.0,.0
- (Zel Ilpo, (2c) (Zc(we)) - (2.l2))lp,,, (zs(u,))]]>
where KL is the Kullback- divergence. Intuitively, when the style changes, the content remains constant. The model has infer the unchanged content factors for image reconstruction. Thus, the content factors can be isolated from style factors. The content factors can be identified up to the block-identifiable defined in [46].
Utilizing content information to extract hard examples. After minimizing the ELBO loss LELBO. we have learned representations 2. and 2, which exclusively contain either style or content information. Then we employ existing sample-selection-basse confident example selection, classification head fe Ze Ac-1 is introduced. The classification head maps the space of content factors to a C' probability simplex, where C' represents the number of classes.
methods solely focus on Z. for
In practice, we propose an end-to-end approach to learn the classification head fy and learn to infer content factor for extracting hard confident examples. This approach simultaneously minimizes the evidence lower bound loss LELBO and the loss of existing sample sample-selection cece--seecccoobbseeeee based methods by leveraging the Lagrangian method [22, 12]. The content IDs refined dynamically during the learning process. The overall loss function Call is thus given as:
min full min [Cest AEI.BOLELLOO Arefl 0c.0..0.v
6c.0..9.4
where Last denotes the loss of the sample selection method, which optimizes the parameters of the classification head & and the content encoder &z. The loss Lref is the cross-entropy loss on confident examples. The hyper-pape hyper-parameter λELBO and Aref are used to control strength of LELBO and Cref respectively. Here, we illustrate a concrete example of employing the inference model 93. in conjunction with DivideMix [28] in an end-to-end implementation. For detailed walkthrough, please refer to the podd provided in Appendix and the original paper [28].
DivideMix [28] applies a Gaussian mixture model to enhance MixMatch [43] for confident example selection and classifiers training. In DivideMix, during each epoch of training, the data is divided into a set of confident examples S) and a set of unconfident examples S,. The confident examples contain the sharpened soft labels mixed by their labels in datasets and predicted labels given by the classification head fa. The unconfident examples contain the sharpened soft labels predicted by the classification head fø. Then, the semi-supervised learning approach, MixMatch [5] employed by transforming confident (Si) and unconfident (S.)) samples into augmented confident (S)) and unconfident (S,) samples by a linearly mixing.
The overall loss function, composed of a confident sample loss, an unconfident sample loss, and a regularization term, ELBOO loss and classification loss, i.e.,
Call Ls. + AuCs. + Xr.ree +EELBOCELBO + Avej
DivideMix
Intuitively, Ls, is cross-entropy loss for the labeled examples; Ls, is the mean squared error for the unlabeled samples; Lreg is regularization term to prevent the model from predicting all samples belong single class. These three terms are defined follows specifically.
Ls, = (5>-1S))
Σ
Σ
oi log(fe o (x))
SS
2
ES/
Ls.
Σ
1lv"
So qs (x)112
2.y" E.y.
freg Σ
log(1
C
Σ
fo o d' (z))
S
2ES
where y" is the sharpened soft label and 9%. denotes the i-th coordinate of its output on input x. In Appendix B, illustrate method with other sample selection methods.
7

---

Table
Precision of confident examples on CIFAR-ION.
Worst
Aggregate
Random I
Random 2
Random
Me-Momentum
0,881 + 0.011
0.972 + 0.002
0.962 + 0.005
0.960 = 0.003
0.948 + 0.005 0.983 i 0.000
DivideMix
0,951 1 0.003
0.989 1 0.000
0.983 1 0.000
0.983 1 0.000
CS-Isolate-DM
0.956 0.001
0.989 0.000
0.984 + 0.000
0.985 + 0.000
0.983 0.000
Table 2: Recall of confident examples on CIFAR-ION.
Worst
Aggregate
Random 1
Random 2
Random
Me-Momentum 0.920 0.026
0.969 0.019
0.946 + 0.021
0.958 + 0.022
0.962 0.018
DivideMix
0.966 0.000
0.963 0.000
0.975 + 0.000
0.976 = 0.000
0.977 0.000
CS-Isolate-DM
0,980 0.000
0.975 0.002
0,980 + 0.001
0.982 + 0.001
0.982 0.001
Table 3: Means and standard deviations (percentage) of classification accuracy on MNSHi
CIFAR-10 and CIFAR-100.
Fashion- MNIST
CIFAR-10
CIFAR-100
IDN-0.2
IDN-0.4
IDN-0.2
IDN-0.4
IDN-0.4
CE
88.54 + 0.32
84.22 = 0.35
75.81 + 0.26
62.45 + 0.86
21.45 + 0.70
Co- Co-Teaching
91.21 + 0.31
89.10 = 0.29
80.96 + 0.31
73.41 0.78
28.04 + 1.43
Forward
90.05 + 0.43
86.27 0.48
74.64
0.26
60.21 0.75
26.75 + 0.93
T-Revision
91.58 0.31
89.46
76.15 0.37
65.09 0.37
27.23 + 1.13
BETM
91.20 0.27
82.42+
77.50 + 1.30
63.20
35.67
1.97
CausaINL
90.84 0.31
90.01 0.45
80.91 1.14
79.08
34.02 0.95
Me-Momentum
92.85 0.64
90.06 + 0.51
90.86
0.21
86.66 0.91
58.38
1.28
DivideMix
94.85 + 0.15
92.28 + 0.13
94.93 0.15
94.16 0.35
70.50 + 0.25
CS-Isolate-DM
95.16 0.07
94.40 + 0.09
95.90 0.10
95.54 + 0.06
73.11 + 0.36
Table 4: Means and standard deviations (percentage) of classification accuracy on CIFAR-
Worst
Aggregate
Random 1
Random 2
Random 3
CB
77.69 + 1.55
87.77 + 0.38
85.02 + 0.65
86.14 + 0.24
86.12 + 0.16
Co-Teaching
82.04 + 0.06
91.11 + 0.10
89.61 + 0.18
88.98 + 0.11
89.49 + 0.06
Forward
79.79 0.46
88.24 0.22
86.88 0.50
86.14 0.24
87.04 0.35
T-Revision
80.48 1.20
88.52
88.33 0.32
87.71 + 1.02
BLTM
68.21 1.67
79.41
78.09 1.03
76.99 + 1.23
76.26 0.71
CausaINL
82.41 0.24
9
89.06 0.05 89.03 0.02
89.21 0.13
Mc-Momentum
84.21 0.70
91.34
89.51
0.42
90.14
89.62 + 0.31
DivideMix
92.48 + 0.16
94.11:0.22
94.77 + 0.15
94.79
94.79 + 0.15
CS- Isolsstee DM
94.28 0.07
95.34 = 0.10
95.36 0.14
95.34 + 0.12
95.48 0.14
4
Experiments
In this section, we introduce the setting of our experiments and compare our experimental results with existing methods, Most of our experiments are left out in Appendix C due to the limited space.
4.1 Experiment Setting
Dataset and noise type. We evaluate our methods on three synthetic noise datasets FNSH
[54], CIFAR- 10 [25]. and CIFAR-100 [25], and two real-world label-noise datasets, CIN
[48] and Clothing MM [55]. FashionMNIST includes 70,000 images of size 24 x 24, categorized into 10 classes with 60,000 for training and 10,000 for testing. Both CIFAR- 10 and CIFAR-100 contain
50,000 training images and 10,000 test images; CIFAR-10 comprises 10 classes, while CIFAR-100 includes 100 classes. The image size in both CIFAR datasets is 32 32 x 3. To generate noisy labels for these clean datasets, we employ the instance-dependen noisy label generation methods proposed in [52]. CIFAR- a noisy version of CIFAR-10, 10, includes five types of label noise: "woorst", "aggregate", "random 1", "random 2", and "random 3", all annotated by humans. The noise rates are 40.21% %, 9.03%, 17.23%, 18. 12%, and 17. 17.64%, respectively. Clothing M contains million images with world noisy labels for training and 10,000 images with clean labels for testing.
8

---

Table 5: Means and standard deviations (percentage) of classification accuracy on Clothing IM.
CF
Decoupling [35]
MentorNet [21]
Co-teaching [13]
Forward [39]
68.88
54,53
56.79
60.15
69.91
T-Revision [53]
BLTM-V [56]
CausaINL [58]
DivideMix [28]
CS-Isolate-DM
70.97
73.39
72.24
74.76
74.92
Network structure and optimization. We implemented our method using PyTorch 1.12.1 and performed experiments on the NVIDIA Tesla V100. For fair comparisons, we chose PreAct ResNet-18 18 [15] as the backbone of the encoder Io. for Fashion-MNIST CIFAR-10, 10, CIFAR-100, and CIFAR-ION ION, a ResNet-50 [14] with ImageNet pre-trained weight for Clothing IM. We employed SGD with momentum of 0.9 and a weight decay of 0.0005 to optimize the encoder Je. and the classifier head fe. We used Adam with default parameters to optimize the encoder qs, and the decoder Pa. For experiments on synthesis datasets, the initial learning rate for SGD was set at 0.02 and for Adam at 0.001. The batch size is 64. For experiments on Reshion-MNIST our network was trained for 100 epochs. Both learning rates were reduced by factor of 10 after 80 epochs. For experiments on CIFAR-IO, C00,R-100, and CIFAR-ION our network was trained for 300 epochs. Both learning rates were reduced by factor of 10 after 1 150 epochs. For experiments on Clothing M, our network was trained for 80 epochs with a batch size of 32. The initial learning rate for SGD was set at 0.002 and for Adam at 0.001. Both learning rates were reduced by a factor of 10 after 40 epochs. The encoder qo. and the classifier head fe were warmed up on noisy data for 10 epochs for CIFAR-10 and CIFAR- warmed up for 30 epochs for CIFAR- 100, 5 epochs for Fashion-MNIST and 1 epoch for Clothing MM. The dimensions of Ze and Z, were set at 32. The data augmentation techniques include shift scale rotation, random crop and horizontal flip, random brightness contrast, color jitter, and random to gray. For all experiments, we set M 1000, X = 1 and λELBO = le 1e-3. 3. Aref was increased gradually then kept constant at le 3 after 140 epochs. For synthetic datasets, we set An as 0 and 15 for the noise rates of 0.2 and 0.4 in FashionMNIST and CIFAR-10 datasets and 100 in CIFAR-10 100. For CIFAR-1ON, we set X, 50 in the "worst" noise type and O in the rest of the noise types. For Clothing IM, we set Xa as O.
Baselines and measurements. We compare our method against several oort----rtt-ttttt techniques: (1) CE: the standard cross-entropy loss, with the model trained directly on noisy data; (2) Co- Co-Teaching [13]: involves training two networks and uses the small-loss trick to select confident examples for each other; (3) Forward [39]: applies a transition matrix to correct the loss function; (4) T-Revision
[53]: revises the transition matrix to enhance performance; (5) Me-momentum [2]: uses confident examples to refine the classifier, which is then used to update confident examples alternately: (6) BLTM [56]: uses Bayes optimal label to estimate the transition matrix; (7) CausaINL [58]: employs a causal model to assist the learning of the classifier; (8) DivideMix [28]: employs two networks to select confident examples and uses semi-supervised techniques to utilize unlabeled examples. We report the precision, recall and AUC for the confident examples selection. We also report the test accuracy on the test dataset. The value is the average over the last 10 epochs.
4.2 Confident Examples Quality
To evaluate the quality of the confident examples, examine the precision and recall of these examples. The precision quantifies the proportion of accurately labeled examples within the set of confident examples. On the other hand, recall measures the proportion of accurately labeled examples in the confident examples relative to the total number of correct labels. The experimental results are displayed in Tab. and Tab. 2. The results indicate that the precision performance of proposed method matches the state-of-the-arr levels. However, the recall of our method surpasses that of existing methods. This demonstrates that our method can identify more confident examples without compromising precision. This suggests that our approach effectively maintains high precision and improves recall, thus selecting more confident examples with correct labels.
4.3 Classification Accuracy
We conducted extensive experiments on three synthetic noise datasets (Fashion-MNIST, CIFAR-10 and CIFAR-100 100) and two real-world datasets (CIFAR-1ON and Clothing For the synthetic

---

Oriaced image
UnaICAM of DivisAii
Unisne ITARE
GraFAM Divakhin
CeaF of boiee DM
skip
Origndl image
Caa DivasAlis
GIGCAM CS
STaRE
Grad-CAM Divaktts
(a)
(b)
Origina Imagee
Origial suage
Socke
GnalICAM DiDimiAAG
(c)
(d)
Figure 5: Grad-CAM visualizations of hard confident examples. CS-Isoalte-D DM successfully iden- tifies these confident examples, but DivideMix does not. The activation map of predominantly highlights semantic objects, whereas DivideMix emphasizes non-object pixels.
datasets, employed instance-dependee noisy label generation methods. proposed by [52]. We experimented with noise rates of 0.2 and 0.4, denoted by IDN-0.2 and IDN-0.4 respectively, The experiment results presented in Tab. 3, Tab. 4 and Tab. 5. Our proposed method outperforms existing methods in terms test accuracy on both synthetic and real-world datasets containing label noise. Notably, the noise rate increases, the performance gap between method, CS-Isolate-DN and the existing methods becomes more pronounced. This highlights the robustness and effectiveness of our approach in scenarios with higher levels of label noise.
4.4 Hard Confident Examples Visualization
The proposed method is expected to select confident examples based on content factors rather than style factors. To analyze this, we use Grad-CAM [42] to visualize the regions used to select confident examples. The visualization results world dataset CIFAR-1ON, and the noise type is "worst". The experiment results demonstrate that our method can correctly focus the object in images. Specifically, when there exist uncontrolled style factors, e.g., the person near the horse, the activation maps for CS-Isolate-DN can successfully focus on the right object used for classifying the horse instead of uncontrolled style factors. In contrast, the baseline method, DivideMix, focuses on the style factors that are not related to the class "horse" and fails to select these confident examples.
shown in Fig. 5. The experiment is conducted
the real-
Conclusion
This paper is motivated by the fact that only focusing on content factors such semantic information makes examples more discriminative. We, therefore, proposed novel CS-Isolate approach infer and isolate the content information for classification. This is achieved by leveraging variational inference and constructing auxiliary variables via data augmentation techniques to modify style factors while regularizing content factors using confident examples. By training existing sample- selection-based methods with our inferred content factors, CS-Isolate improves their effectiveness in learning hard examples and classification accuracy on different image datasets.
10

---

Acknowledgments and Disclosure of Funding
Tongliang Liu is partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP22000057,, LP220200949, and IC190100031. Dong Xu is partially supported by the Hong Kong Jockey Club Charities Trust under Grant 2022-0174, the Hong Kong Research Grant Council under General Research Fund (17203023), the Startup Funding and the Seed Funding for Basic Research for New Staff from The University of Hong Kong, and the funding from UBTECH Robotics.
References
[1] Devansh Arpit, Stanislaw Jastrzehski, Nicolas Ballas, David Krueger, Emmanuel Bengin, Maxinder S Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, al. A closer look memorization in deep networks. In ICML, pages 233-242, 2017.
[2] Yingbin Bai and Tongliang Liu. Me-momentum: Extracting hard confident examples from noisily labeled data. In Proceedings of the IEEF/CVF international conference computer vision, pages 9312-9321, 2021.
[3] Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang. Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Un- derstanding and improving early stopping for learning with noisy labels. arXiv preprint ar 106.:3306.15853, 2021.
[4] Yingbin Bai, Erkun Yang. Zhaoqing Wang. Yuxuan Du, Bo Han, Cheng Deng, Dadong Wang, and Tongliang Liu. Rsa: Reducing semantic shift from aggressive augmentations for sulf-supervised learning. Advances in Neural Information Processing Systems, 35:21 128-2114111 141, 2022.
[5] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint arXiv: / 1905.02249, 2019.
[6] Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
[7] Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance and debenede label noise. In International Conference on Machine Learning, pages PMLR, 2020.
[8] Madalina Ciortan, Romain Dupuis, and Thomas Peel. framework using contrastive learning for classification with noisy labels. Data, 6(6)61, 2021.
[9] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[10] Erik Englesson and Hossein Azizpour. Consistency regularization can improve robustness label noise. arXiv preprint 1rX.v:2110.01222, 2021.
[11] Aritra Ghosh and Andrew Lan. Contrastive learning improves model robustness under label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2703-2708, 2021.
[12] Giorgio Giorgi and Tinne Hoff Kjeldsen. Traces and emergence of nonlinear programming. Springer Science & Business Media, 2013.
[13] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-tcaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS, pages 8527-8537, 2018.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer MisionEECCY 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 630-645. Springer, 2016.
[16] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International conference on learning representations, 2017.
11

---

[17] Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing out-of-distribution examples via augmenting content and style. arXiv preprint arXiv:2207.03162, 2022.
[18] Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. Advances in Neural Information Processing Systems, 34:26714 26725, 2021.
[19] Zhuo Huang, Miaoxi Zhu, Xiaobo Xia, Li Shen, Jun Yu, Chen Gong, Bo Han, Bo Du, and Tongliang Liu. Robust generalization against moooon-limited corruptions via worst-case sharpness minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16175- 16185, 2023.
[20] Aapo Hyvarinen. Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference Artificial Intelligence and Statistics, pages 859- 868. PMLR. 2019.
[21] Jiang, Zhengyuan Zhou, Thomas Leung. Li-Jia Li, and Li Fei-Fei. MentorNet: Learning data data-driven curriculum for very deep neural networks on corrupted labels. In ICML, pages
2018.
[22] William Karush. Minima functions several variables with inequalities as side constraints. M. Sc. Dissertation, Dept. of Mathematics, Univ. of Chicago, 1939.
[23] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages
PMLR, 2020.
[24] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv: 132.6143 2013.
[25] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[26] Harold W Kuhn and Alhert W Tucker. Nonlinear programming. In Traces and emergence of nonlinear programming, pages 247-258. Springer, 2014.
[27] Samuli Laine Timo Aila. Temporal ensembling for sepi-super learning. arXiv preprint arXiv:1610.02242. 2016.
[28] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In ICLR, 2019.
[29] Junnan Li, Caiming Xiong, and Steven CH Hoi. Mopro: Webly supervised learning with momentum prototypes. arXiv preprint 07X19:2009.07995, 2020.
[30] Shikun Li. Xiaobo Xia, Shiming Ge. and Tongliang Liu. Selective-supervinn contrastive learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patiern Recognition, pages 316-325, 2022.
[31] Shikun Li. Xiaobo Xia, Hansong Zhang, Yibing Zhan, Shiming Ge, and Tongliang Liu, Estimating noise transition matrix with label correlations for noisy multi-labo learning. Advances in Neural Information Processing Systems, 35:24184-24199 2022.
[32] Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. arXiv preprint 2102.02000 2021.
[33] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447 4611, 2016.
[34] Franeesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning, pages 6348-6359 PMLR, 2020.
[35] Fran Malach and Shai Shalev-Shwartz. Decoupling" when to update" from" how to update". In NeurlPS, pages 960-970, 2017.
[36] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018
[37] Due Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self-ensembling. In ICLR, 2019.
12

---

[38] Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411, 2021.
[39] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, pages 1944 1952, 2017.
[40] Luis Perez and Jason Wang. The effectiveness of data augmentation image classification using deep learning. arXiv preprint
2017.
[41] Mengye Ren, Wenyuan Zeng, Bin Yang. and Raquel Urtasun. Learnngg reweight examples for robust deep learning. In International Conference on Machine Learning, pages 4334 4343, PMLR, 2018.
[42] Ramprasaath R Selvaraju, Michacl Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- localization. In Proceedings of the IEEE international conference computer vision, pages 618-626, 2017.
[43] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang. and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.0760
2020.
[44] Hwanjun Song, Minseok Kim, Dongmin Park, and Jac- Gil Lec. How does early stopping help generalization against label noise? arXiv preprint 1cXiv:19/L.0805. 2019.
[45] Cheng Tan, Jun Xia, Lirong Wu, and Stan 7. Li. Co-leaming: Learning from noisy labels with self- supervision. In Proceedings of the 29th ACM International Conference on Multimedia, pages 1405-1413, 2021.
[46] Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervise learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34: 16451-1641164 2021.
[47] Hongxin Wei, Lei Feng. Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13726-13735, 2020.
[48] Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. arXiv preprint arXiv:21 10. /2088, 2021.
[49] Xiaobo Xia, Bo Han, Nannan Wang, Jiankang Deng. Jiatong Li, Yinian Mao, and Tongliang Liu. Extended t: Learning with mixed closed-set and open-set noisy labels. IEEE Trunsactions Pattern Analysis and Machine Intelligence, 45(3):3047-3058, 2022.
[50] Xiaobo Xia. Tongliang Liu, Bo Han, Chen Gong. Nannan Wang. Zongyuan Ge, and Yi Chang. Robust carly-lcaming: Hindering the memorization of noisy labels. In ICIR, 2021.
[51] Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. arXiv preprint arXiv:2106.0044: 2021.
[52] Xiaobo Xia. Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong. Haifeng Liu, Gang Niu. Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards gatance-deeendem label noise. NerurIPS, 2020.
[53] Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurlPS, pages 6835-6846,, 2019.
[54] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mate machine learning algorithms. arXiv preprint arXiv:1708.077477 2017.
novel image dataset for benchmarking
[55] Tong Xiao, Tian Xia. Yi Yang. Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. CVPR, pages 2691 12699, 2015.
[56] Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance- dependent bayes-label transition matrix using a deep neural network. International Conference on Machine Learning, pages 25302-25312. PMLR, 2022.
[57] Yazhou Yao, Zeren Sun, Chuanyi Zhang, Fumin Shen, Qi Wu, Jian Zhang, and Zhenmin Tang. Jo-src: A contrastive appreach for combating noisy labels. In Proceedings of the IEEE/CVF Conference on Computer 2021.
Vision and Pattern Recognition, pages
13

---

[58] Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label-noise learning under structural causal model. Advances in Neural Information Processing Systems, 34:4409 4420, 2021.
[59] LIN Yong. Renjie Pi, Weizhong Zhang. Xiaobo Xia, Jiahui Gao, Xiao Zhou, Tongliang Liu, and Bo Han. holistic view of label noise transition matrix deep learning and beyond. In The Eleventh International Conference on Learning Representations. 2022.
[60] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
[61] Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M Bronstein, and Litany. Contrast to divide: Self-supervised pre-training for learning with noisy labels. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1657-1667, 2022.
14

---

A A Theoretical View of Content and Style Isolation When Learning with Noisy Labels
Noisy data generative process. Recall the data generative process in our main paper, to learn latent factors by leveraging generative models. The generative model has to model the data generative process of noisy data. Firstly, we introduce the generative process of noisy data. We denote observed variables with gray color and latent variables with white color. Specifically, the content factor Ze is generated by the latent label Y. The different style domain Ug give rise to the different style factor Z.. Subsequently, the image X is generated by the combined influence of the style factor Z. and the content factor Ze. Noisy labels y are then generated based the image X. In general cases, Ze and Z, also have statistical or causal dependencies. We follow existing work that assumes the content factors are invariant across different styles [46].
Zo
Zs
Figure 6: The noise data gener- ative process.
Firstly, we introduce the concept of an uncontrolled style factor. This refers to a specific style factor, denoted Z that remains invariant when data augmentation is applied. To formalize this concept, consider an invertible function f 2 x X. Let A denote set of data augmentations, where each augmentation A is subset ranging from 1 to M. Additionally, let P(A) represent a probability distribution over the set of these augmentations A. Now, consider Z subset of style factors drawn from larger set 7g. We partition the latent factors of each instance x into three distinct parts: uncontrolled style factor Ze, content factor Ze, and style factors influenced by data augmentation Z such that
f (x) [20/, 2c, 2s/5/ z. The term 2s/s denotes the set of style factors with style factors 2g excluded.
Definition 1 (Uncontrolled Style Factors) We say that factor Zy as uncontrolled under the following conditions:
For any augmentation ~ P(A), and for any instance x, the first ny components of the inverse function f (x:) remain unchanged even when A(x) is applied, i.e., f- (z) = (A(z))
Here, f (x)1=m, is defined as the underlying partition that contains only and all the information related to the style factor Z of the instance
Why do confident examples encourage content-style isolation? Here, we explain the reason that confident examples encourage content-style isolation. Suppose that there exist some uncontrolled style factors that cannot be adjusted or manipulated through data augmentation. This implies that for an image these style factors remain unaffected, regardless of the data augmentation techniques employed. For instance, as discussed in our paper, in the CIFAR- dataset [25], some pictures with the label "horse" contain person. In this context, the "person" acts style factor. Existing data augmentations cannot control this, as they cannot remove the person from images easily.
It's essential to understand that although data augmentation cannot control all style factors, it still offers the benefit of "partial isolation". If we consider a situation where the uncontrolled style factors are not effects of other style factors, then we can isolate other style factors affected by data augmentation from content factors after matching the data likelihood [34]. This result is modified application of Mlack-identffabiliiy [46].
Specifically, let 2r.ar := Zimmta,, be partition of learned representations of a generative model, where Ze contain all and only information about Ze and Zg. Suppose that assumptions of Theorem 4.2 in [34] are satisfied, 2a.s' guaranteed to be learned, thereby allowing for the partial isolation of the remaining style factors, denoted as Za/s'-
Despite Ze,e/ guaranteed to be learned, the information from the uncontrolled style factor Z is entangled with the content factor Z. in the learned Ze.s'. To isolate this information further, the employment of confident examples is necessary.
Specifically, we can generate 2,..' using invertible function according to the label Ý of the confi- dent example, and subsequently reconstruct image x. Following the data likelihood matching [34], the information related to the uncontrolled style factors becomes apparent due to the establishment of
15

---

Algorithm 1 CS-Isolate-DM
Input: A noisy dataset S, a style ID list Us. data augmentation set A, confident threshold 7, total epoch Taxx-
1: fen: f 9az (/a (Ze ()), 90z XX WarmUP(S);
2: For T= 1,..,, mani
3:
X,U f- Co guessing(S fug far aot 9dg, 7);
4:
X' + MixUp(X,U))
5: For 2=
6: Sample (x. D. uh) ~ S, Us:
7: T * Au. (z);
8:
Feed I to encoders dot and dsy to content factors 2e and style factors 2s. respectively;
9:
Feed ue and Ug to decoders Pog and Poz to get the prior Pog (Zeluc) and Por (Z, lug);
10: Feed &e and 2, to decoders Por to get the reconstruct image d;
11: Feed 2g to classifier head frz to predicted label ÿ;
12: Calculate the loss using Eq. and update networks;
Output: The inference networks and classifier heads tot: 902 Sox Sa2.
Algorithm 2 CS-Isolate-Co
Input: noisy dataset Š, a style ID list Ug, a data augmentation set A, Total epoch Tmaz
1: For T= ,
2: Fetch mini-batch S from S;
3: Obtain S arg min ming. (forr (o!.
4: Obtain 5₂ = arg sf {(S22 20z. S*);
5:
For k=1, 2:
6: Sample (x. G. ue) ~ Sk, ~ U.;
7:
f au, (z);
8:
Feed T to encoders det and dsy to content factors 2, and style factors 2., respectively;
9:
Feed Me and u, to decoders Pot and Poz to get the prior Poy (Ze(ue) and Pos (7. lu,);
10:
Feed &e and 2g to decoders Pax to get the reconstruct image à; 11: Feed and 2, to classification model fer to predicted labels ÿ;
12: Calculate the loss using Eq. and update networks;
Output: The inference networks and classifier heads lo! 9og, fwa fV
one-to-one mapping between the label Ý and Zo.s'. This consequently forces examples with identical labels to share the same Zc.=', regardless of any alterations in the uncontrolled style factor Z This approach. therefore, ensures that styles changes don't 't affect the derived content representation for the same label.
It is worth mentioning that to fully isolate uncontrolled style factors and content factors, it requires that there exists confident examples with all possible uncontrolled style factors. This can be hard to achieve when learning with noisy labels. Therefore, in general, the selected confident examples can only encourage isolation but can not fully isolate uncontrolled style factors and content factors.
B Apply CS-Isolate to Existing Methods for Learning with Noisy Labels
Applying CS-Isolate to DivideMix. DivideMix [28] uses two classifiers to select confident examples for each other. To utilize the unlabeled data, they combine the semi-supervised technique MixMatch [5]. Specifically, the classifiers, after warmed up, are used to calculate the loss of examples. They use a Gaussian Mixture Model (GMM) to divide the examples into confident and unlabeled examples. Finally, confident and unlabeled examples are used to train the models based on the MixMatch algorithm. Our method can be plugged into DivideMix easily. Specifically, we use decoder 42.2. X, to obtain content factor Z. and style factor Z.. A classifier head fo is used to
16

---

Algorithm 3 CS-Isolate-Me
Input: noisy training dataset 8, noisy validation dataset S., a style ID list Us. a data
augmentation set A, iteration number T Timner, Touter-
1: Initialize encoders (gøg and (,0), decoders (pøg. Pog and Poo), classification model fvo by
using the noisy training data and early stopping;
2: For i == 1....... T Touter:
3: For jor Tinner:
4: Update the extracted confident examples S by using مي- and f
5: Train networks by using the loss in Eq. 4 on confident examples St;
6: Obtain døg and Jey through the highest noisy validation accuracy throughout the training
procedure;
7: Break and output 90g and f if the highest validation accuracy is non-increasing in the
loop:
8: Re-initialize encoders (9ep and (ag), decoders (Poe, Pao and pgo), classification model f
9:
Train networks by using the loss in Eq. on confident examples S;; Obtain dag and fwh through the highest noisy validation accuracy throughout the training
10:
procedure;
11:
Break and output مz-- and føy-1 if the highest validation accuracy is non-increasing in the
loop:
Output:
The inference network and the classification model q fø-
predict labels, and only the content factors Ze are used as input. The prior distribution of the content factors Ze is conditional on the auxiliary variable Uc, i.e., Po, (Z. (Z,lUe), where U. is the content ID. Similarly, the prior distribution of the style factor Z, is conditional on the auxiliary variable U.. i.e,, Po. (U,, where U, is the style ID. A decoder (x Z.) is used to reconstruct input images. The combination of CS-Isolate and DivideMix is called CSl-stett-- The loss function of CS-Isolate-DM is shown in Eq. 2. Algorithm delineates the full algorithm.
Ldm Ls, XuCs, + +XyFreg +EELBOLELEO + AresPref-
DivideMix loss
Applying CS-Isolate to Co- Co-Teaching. Co-Teaching [13] uses two classifiers to select confident examples for each other. Proposed CS-Isolate be embedded in Co- Co-Tecchng easily. Similar to CS-Isolate- DM, we use a decoder 90 Ze. Zg X) to obtain content factors Z. and style factors Z, A classifier head fu is used to predict labels, and only the content factors Ze are used input. The prior distribution of the content factors Z. is conditional on the auxiliary variable Uc. i.e., Po_ Po_(Ze U2). where Ue is the content ID. Similarly, the prior distribution of the style factor Z, is conditional on the auxiliary variable U., i.e., Pa, (Z,(U.), where U, is the style ID. decoder Po(X Z,) is used to reconstruct input images. We call the combined method as CS-Isolate Co. Let S be the confident examples selected by another classifier head. For each network, the loss is defined as:
Lro
E (a.g))ss (a.i))s leolSe 90 (x), + +XELBOCELBO.
Co-Teaching loss
where tee is the cross-entropy loss, 1 is the indicator function. The algorithm of CS-Isolate-CC Teaching is summarized Algorithm 2.
We use a PreAct ResNet- 18 as the backbone. We used Adam with default parameters to optimize the encoder T classifier head fg. the encoder go, and the decoder po- The initial learning rate is 0.001, divided by 10 after 80 epochs.
Applying CS-Isolate to Me-Momentum. Me-Momentum proposes to use one classifier to select confident examples. Then the parameters of the classifier will be reinitialized, and the classifier will be trained on the confident examples. The confident examples and the parameters of the classifier are updated alternately. We combine the Mc-Momentum with our method and call it CS-Isolate-Me. Let
17

---

Table 6: Means and standard deviations (percentage) of classification accuracy on CIFAR-10.
Sym-20%
Sym-40%
Sym-60%
Sym-80%
Pair-45%
CE
84.12 + 0.20
78.82 + 0.38
64.80 + 0.47
47.39
0.69
66.09 + 0.96
Co-Teaching
88.74 + 0.19
84.23 0.81
77.93 + 0.75
29.57
1.39
76.65 = 2.97
Forward
88.31 0.23
82.73
76.58 + 1.59
47.18 4.63
76.76 = 4.94
T-Revision
87.83 + 0.63
0.73
77.39 + 0.83
57.61 3.93
72.81 = 7.01
BLTM
76.54 + 1.37
73.50 1.38
58.94 + 1.84
39.28 + 4.00
67.97 + 1.45
CausaINT.
89.68 + 0.09
86.37 + 0.09
79.54 + 0.09
29.72 0.03
71.53 + 0.37
Me-Momentum
91.44 + 0.33
88.39 0.34
82.58 0.30
62.70 0.59
66.41 + 0.17
DivideMix
95.93 + 0.04
94.51 + 0.12
94.55 + 0.07
92.43 - 0.13
70.86 = 0.87
CS-Isolate-DNN
96.05 + 0.13
95.57 + 0.12
94.65 + 0.10
92.57 0.10
87.54 + 0.83
Table Precision ratio (percentage) of confident examples on CIFAR-ION.
Worst
Aggregate
Random
Random 2
Random 3
Co-Teaching
89.49 + 0.11
96.61 + 0.04
96.06 + 0.03
95.65 + 0.02
96.01 + 0.03
Me-Momentun
88.14 + 1.10
97.16 0.19
96.18 + 0.47
95.96 0.27
94.78 1 0.47
CS CS-Isolate-Co
90.88 1 0.08
97.25 1 0.04
96.95 1 0.03
96.60 i 0.06
96.82 1 0.05
CS
90.74 + 1.16
97.72 0.07
96.64 + 0.21
96.92 0.17
96.14 + 0.22
Table 8: Recall ratio (percentage) of confident examples CIFAR- ION.
Worst
Aggregate
Random
Random 2
Random 3
Co-Teaching
89.11 + 0.07
95.43 + 0.04
92.59 + 0.04
93.19 + 0.02
93.00 + 0.04
Mc-Momentum
92.02 + 2.63
96.90 1.91
94.58 + 2.07
95.80 2.20
96.22 = 1.82
CS-Isolate-Co
90.53 + 0.06
96.10 0.02
93.44 + 0.04
94.13
0.06
93.79 + 0.03
CS
95.76 + 1.55
97.76 0.79
97.10 + 0.45
96.34 0.81
96.70 + 0.89
St be the confident examples selected by the classifier of the last iteration. For each network, the loss is defined as:
Lyme = - (x. (a. o 9a. (x), 0) + λELBOLELBO-
(4)
where tee is the cross-entropy loss, 1 is the indicator function. The algorithm of CS- CS-Isolate-Mo Momentum is summarized in Algorithm 3.
C Experiments
In this section, we first introduce the experiment results the proposed methods, including the classification of CS-Isolate-DM Isolate-Co and CS-Isolate-Me on the real-world dataset CIFAR-1ON, and classification performance of CS-Isolate-Co and CS-Isolate-Me on real-world the dataset CIFAR-1ON, Second, we provide the details of data augmentation used in methods. Then, we conduct the ablation study for CS-Isolate-DMM Finally, we visualize the easy confident examples, content factors, and style factors.
class-dependent label noise, sample selection quality of CS-
C.1 Experiments on Class-dependent label noise
We report the classification performance of CS-Isolate-DM on class-dependent label noise, including commetry-ffliping noise and pair-flipping noise. The dataset used in the experiment is CIFAR-10 10. The experiment results are shown in Tab. 6. The experiment results demonstrate that CS-Isolate-DM can also perform well under class-dependent label noise.
C.2 Applying CS-Isolate to Existing Sample-Selection Methods
We combine our method with existing methods including DivideMix, Co- Co-Teaching, and Me- Momentum.
18

---

Table 9: Means and standard deviations (percentage) of classification accuracy on CIFAR- ION.
Worst
Aggregate
Random
Random 2
Random 3
Co-Teaching
82.04 + 0.06
91.11 + 0.10
89.61 + 0.18
88.98 0.11
89.49 + 0.06
Mc-Momentum
84.21 + 0.70
91.34 0.16
89.51 + 0.42
90.14 0.28
89.62 = 0.31
CS-Isolate-Co
83.93 + 0.17
91.30 + 0.10
90.57 + 0.14
90.14 + 0.03
90.76 + 0.15
CS-Isolate-Me
86.95 + 0.13
91.49 0.17
90.30 + 0.12
90.18 0.07
90.22 + 0.21
Table 10: Data Augmentation Techniques Description
Data augmentation
Shift scale rotation
Randomly shifts, scales, and rotates the image. Randomly crops the image to specified height and width.
Random crop
Horizontal flip
Horizontally flips the image randomly.
Random brightness contrast
Randomly changes the brightness and contrast of the image.
Color jitter
Randomly adjusts image color properties.
Random to gray
Converts the image to grayscale with a specified probability,
In the experiments for Co- To-Tecching and Me-Momentum, we use a PreAct ResNet-118 18 the backbone. We use SGD with momentum 0.9 and weight decay 10-4 to optimize the encoder 9ø, and classifier head fu- We used Adam with default parameters to optimize the encoder qe. and the decoder po- The network is trained for 100 epochs. The initial learning rate for SGD is 0.01, and for Adam is 0.001. The learning rate divided by 10 after 40 epochs and 80 epochs.
C.3 Improves Sample Selection Quality with CS-Isolate
We conducted experiments on CIFAR-ION a dataset reflecting real-world label noise. We illustrate the precision and recall ratios of our confident examples in Tab. and Tab. 8. By employing our method, existing methods achieve improvements in terms of precision and recall. The experiment results indicate that our approach can efficiently improve the quality and number of confident examples.
C.4 Comparison of Classification Performance
The accuracy of the baseline methods, as well as the combination of our proposed methods and the baselines, is shown in Tab. 9. The results demonstrate that improving the quality of the confident examples by using our method boosts the classification performance of the existing methods.
C.5 Data Augmentation Details
The detailed description of data augmentation techniques used in our method is shown in Tab 10. When generating data augmentation A, E A, the probabilities apply shift scale rotation, random crop and horizontal flip, random brightness contrast, color jitter, and random to gray are 0.5, 1, 0.5, 0.5, 0.8, and 0.2, respectively, then the implementation details of the data augmentations will be recorded for the replaying. For instance, if data augmentation A, flips the image horizontally, its behavior will be recorded. When the data augmentation A, is used during the training process, the images used to train the network will be applied a horizontal flip.
C.6 Ablation Study
In this subsection, we present the results of the ablation study on the hyper-parameters λELBO. Xref and the dimensions of Ze and Z,. The experiments are conducted on the real-world dataset CIFAR-ION with the noise type "worst". The experiment results are shown in Fig. The experiment results show that λELBO and Aref are not sensitive in the range from 0.0005 to 0.005. In our experiments, we set the value of both XELBO and Xref as 0.001, which is the middle value between 0.0005 and 0.005. For the ablation study on the dimension of Zr and Z., the test accuracy increases gradually until the dimension 32. After the dimension larger than 64, the test accuracy decreases. We set the dimension as 32 in experiments.
19

---

94.0
83.5
93.5
92.5
93.0
93.0
92.0
Tell accurny of CS C5-bcbli-CM
92.5
Telt accuncy of Diesalo
91.0
10001
0. 0.0005
0.301
0,005
0.01
0.0001
0.0035
0.001
0.005
0.01
Aus
(a) λELBO
(b) Avef
(c) The dimensions Ze and Z,
Figure 7: Ablation study the hyper-parameters ÆELBO, Aref and the dimensions of Ze and Z
Rid
C5-5abas-IM
ski
Orizzal
Geal-CAM Divakiis
GiaB-CM oc CS-bolac-DM
Orgind Gra--AM DiakMa
of C--olzze
(a)
(b)
Sxp
Originl Image
Ofd DiDiMMM
Originl image
Grd D'DiskMMs
Originl Image
GAd Diviots
Origiall Vaage
Grd-EAM Bisasii
Cr holore
(c)
Figure 8: Grad-CAM visualizations of easy confident examples. Both CS-Isoalte --MMllltte-MM and DivideMix successfully identify these confident examples. The activation map of CS-Isolate-DN, predominantly highlights semantic objects.
C.7 Visualization of Grad-CAM on Easy Confident Examples
We visualize the Grad-CAM for CS-Isolate-DM and DivideMix easy confident examples. The easy confident examples are the examples identified successfully by both CS-Isolate-DDe and DivideMix. The dataset is CIFAR- 1ON, and the noisy type is "worst". Fig. shows the visualizations of easy confident examples. When the confident example is easy, the Grad-CAM visualizations for CS- Isolate-DM and DivideMix do not differ significantly. The activation maps for both CS-Isolate- DM and DivideMix can focus on the objects. However, when the confident example is hard, only the activation maps for CS-Isolate-Dee DM focus the objects, which has already been shown in the main paper.
C.8 Visualization of Grad-CAM for Content Factors and Style Factors
We visualize the Grad-CAM of Style and Content Factors for CS-Isolate-DM. The visualization results are shown in Fig. 9. Grad-CAM of content factors mainly concentrates on the objects, while Grad-CAM of style factors mainly concentrates other pixels in the images. The visualization results demonstrate that CS-Isolate- can isolate content and style factors successfully.
20

---

Bid
Orizzed
GusACMM ofobie
GIS cuneel
Orgine Gral-AM of faston
QaaM-AM coslor faston
Ortanel image
QcaFCAM sta0 Bavo
Gnd-UMM conned lano
Onaind sTaR
Gral-CAM st Eaton
OcaFCAM of coclor faton
(b)
Origiel Image
GodCCMM ofstk Salos
God-CMMooomne feoos
Origina inage
GnA-CAM faten
Go fatun
Origiel Trage
Origial inage
&Cyye
(c)
(d)
Figure 9: Grad-CAr Visualizations of Style and Content Factors for CS-Isolate DM. The activation map corresponding to the content factor prominently highlights semantic objects, indicating the model's 's emphasis on capturing meaningful context. Conversely, the activation maps associated with the style factor predominantly focus non-object pixels across the images. The results show that DS-Isolate-DMM can isolate the content factors from styles.
21


Reviews:
[{"review_id": "ozSGX3MUfy", "reviewer": "Reviewer_fCED", "summary": "The paper proposed a novel method to extract the hard confidence examples under the noisy labels. By splitting the images into content factors and style factors, the hard confidence examples can be more easily extracted by solely considering the content factors. By training with the hard confidence examples, the models can achieve better generalization ability.", "questions": "1. Can authors provide the visualization of the hard confidence examples as well as the content/style factors? It is good for readers to understand the effectiveness of the proposed method.\n\n2. The paper just gives a rough description of the data augmentation techniques. Can author specify how many types of data augmentation used in the paper? (The value of $M$)? What is the difference between them? How the number of data augmentation techniques (value $M$) affects the disentanglement of content factors and style factors?\n\n3. In eq. 3 authors mentioned about the refinement of content ID. How the authors deal with the conflicting of labels and content ID? e.g., sample whose label is not $p$ but assigned with content ID $p$. ", "limitations": "Not applicable.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a deep insight of the hard confidence example and why hard confidence examples are hard to extract.\n\n2. A novel method is proposed in the paper to extract the hard confidence examples by splitting the examples into content factors and style factors. The two independent factors are produced by generative model with variational inference method. The experiments show that the proposed method achieves better performance than previous methods in the extraction of the hard confidence examples.\n\n3. The proposed method shows better classification results than baselines.", "weaknesses": "1. Ablation study on hyper-parameters is missing.\n\n2. The experiment datasets shown in the paper are relatively small dataset. Authors should provide experimental results on larger datasets such as Clothing1M. Since both baseline (DivideMix and Me-Momentum) mentioned in the paper report the results on this dataset. It is better to compare the results of the proposed methods against the baselines on this dataset.\n\n3. Some errors in the writing. For instance. on line 276, the equation of conditional probability of style factors is incorrect. In eq.7, the minus sign should be plus sign.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1. Visualize the hard confident examples and the content/style factors.**\n\n**A1**. We visualize some examples that failed to be identified as confident examples by DivideMix, but are successfully identified by our method in Fig. 1 of Rebuttal PDF.\nWe also visualize some examples that are successfully identified by both DivideMix and our method in Fig. 2 of Rebuttal PDF. It shows that objects in Fig. 1 are hard examples compared with objects in Fig. 2. Specifically, the objects in Fig. 1 are mixed with other objects, small, or located in the corners. The visualization implies that hard examples exist and our method can successfully identify these hard confident examples.\n\nIn Fig. 3 of Rebuttal PDF, we also visualize Grad-CAM for content factors and style factors extracted by our proposed method, respectively. Grad-CAM of content factors mainly concentrates on the objects, in contrast, Grad-CAM of style factors mainly concentrates on other pixels on the images.\n\nAdditionally, In Fig. 1 and 2 of Rebuttal PDF, we visualize Grad-CAM of the representation learned by DivideMix, which can be viewed as the mixture of content factors and style factors. It shows that for easy confident examples, DivideMix can identify the confident example successfully. The corresponding heatmap highlights semantic parts of the images. However, DivideMix fails to identify the hard confident examples and the corresponding heatmap highlights non-semantic parts of the images.\n\nThe visualizations demonstrate that our method can disentangle content and style factors successfully and can help the existing method extract hard examples. \n\n>**Q2. Ablation study on hyper-parameters.**\n\n**A2**. We have conducted ablation studies on the dimension of $Z_c$ and $Z_s$, as well as the hyper-parameter $\\lambda_{ELBO}$ on CIFAR-10N, and the noise type is ”Worst”. The experiment settings are as same as the description in Sec. 4.1 of our paper. \n\n**The ablation study on $\\lambda_{ELBO}$**. We first fix the dimension of $Z_c$ and $Z_s$ on 32. Then change the value of $\\lambda_{ELBO}$ to 0, 0.0001, 0.0005, 0.001, 0.002, 0.005 and 0.01, respectively. The experiment results are shown in Tab. 1 of Rebuttal PDF. The test accuracy increases gradually until $\\lambda_{ELBO}$=0.001, then starts to decrease. The experiment results show that the $\\lambda_{ELBO}$ should not be too large or too small. A preferred value of is near 0.001.\n\n**The ablation study on the dimension of $Z_c$ and $Z_s$**. with $\\lambda_{ELBO}$ fixed on 0.001. The value of the dimension of $Z_c$ and $Z_s$ is 4, 8, 16, 32, 64, 128 and 256 respectively. The experiment results are shown in Tab. 2 of Rebuttal PDF. The test accuracy increases gradually until the dimension is 64. After the dimension is larger than 64, the test accuracy decreases slightly.\n\n>**Q3. Experimental results on Clothing1M.**\n\n**A3**. The results on the Clothing1M dataset are in Tab. 4 of our Appendix B.2. We will move the results from Clothing1M to the main paper to emphasize the results on the real word large noisy dataset. Additionally, we have also added the additional experiment of Cs-Isolate-Me and Cs-Isolate-Co. We follow the settings from their original paper. The results are shown in Tab. 4 of Rebuttal PDF.\n\n>**Q4. How many types of data augmentation were used?**\n\n**A4**. In our paper, we utilize 6 types of data augmentation techniques that follow [1]. \n\nThese techniques include shift scale rotation, random crop and horizontal flip, random brightness contrast, color jitter, and random to gray.  We generate different styles via the combination of different types of data augmentation. Each type of data augmentation has a probability to decide whether this augmentation will be applied. The probability of the random crop is 1; The probability of shift scale rotation, horizontal fip, random brightness contrast and random to gray is 0.5; The probability of color jitter is 0.8. Each combination acts as a style and will be assigned a unique style ID.\nWe set the number of different data augmentation $M$ to be 200 in experiments. \n\n>**Q5. How does the number of data augmentation techniques affect the disentanglement?**\n\n**A5**. Existing theoretical works [1] show that disentanglement can be achieved if each style variable has a data augmentation to control its change. \nAlthough it posits that data augmentations can fully disentangle style, we've found that this assumption may be overly optimistic. Even with a rich set of augmentations, some style factors are typically uncontrolled. For example, in the CIFAR-10 dataset, horses often appear with people—an uncontrolled style factor that cannot be removed through data augmentation (Fig. 1 of Rebuttal PDF).\n\nTherefore, we emphasize using the labels of confident examples as content auxiliary variables to encourage the isolation of content factors from uncontrolled style factors. Intuitively, by ensuring that confident examples with the same label share the same content factor, we can disentangle uncontrolled style factors (see formal analysis in Appendix A).\n\n>**Q6. In the refinement process, how to deal with the conflicting labels and content ID?**\n\n**A6**. The content ID is initially randomly assigned. In the training process, we can obtain some high-quality confident examples with a precision exceeding 95% (as shown in Tab. 1 of our main paper). For these confident examples, we align their content IDs with the noisy labels, thereby refining the content IDs to accurately represent and be consistent with their clean labels. For unconfident examples, we let them have different content IDs and do not encourage their content factors to be the same.\n\n### References\n\n[1] Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style, NeurIPS21.\n\n"}, {"review_id": "anSWM4ktiV", "reviewer": "Reviewer_NWTZ", "summary": "The problem of training deep neural networks with label noise is considered. Unlike traditional methods proposing to extract confident examples far from the classification boundary, this work proposes an approach named CS-Isolate to extract hard examples close to the classification boundary by disentangling content factor $Z_c$ and style factor $Z_s$. Different data augmentation techniques were employed to change the style factor $Z_c$ of images while the content factor remains unchanged $Z_s$. Two inference models were proposed to model the conditional distribution $Z_c |X$ and $Z_s |X$, respectively. Evaluation on synthetic noisy version of FashionMNIST, CIFAR-10, CIFAR-100 demonstrated the effectiveness of the proposed approach.", "questions": "1.\tIn subsection section 2.2 (Sample selection methods for learning with noisy labels), it was worth discussing the idea to collect hard examples by active learning (e.g. [6]).", "limitations": "I didn't find a section dedicated to the discussion of limitations. One limitation in my mind is the lack of evaluation on real-world and large-scale label noise.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tTo the best of my knowledge, this is the first work introducing fashion factor for extracting confident examples under label noise.\n2.\tThe effectiveness of CS-Isolate was validated on the synthetic instance-dependent label noise.\n", "weaknesses": "1.\tAll evaluations were made on toy-level dataset CIFAR-10/100 and FashionMNIST and synthetic label noise. The effectiveness of CS-Isolate on large-scale dataset and real-world label noise (e.g., Clothing1M) is worth investigating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and effort to review our manuscript and please find the response to your comments below.\n\n>**Q1. The effectiveness of CS-Isolate on large-scale datasets and real-world label noise (e.g., Clothing1M) is worth investigating.**\n\n**A1**. The results on the Clothing1M dataset are in Tab. 4 of our Appendix B.3. We will move the results from Clothing1M to the main paper to emphasize the results on the real word noisy dataset. Additionally, we have also added the additional experiment of Cs-Isolate-Me and Cs-Isolate-Co. We follow the settings directly from their original paper. The experiment results are shown in Tab. 4 of Rebuttal PDF.\n\n>**Q2. In subsection 2.2 (Sample selection methods for learning with noisy labels), it was worth discussing the idea of collecting hard examples through active learning.**\n\n**A2**. Thanks for your insightful question. Firstly we would like to briefly introduce their method. Then we will analysis the connection between their methods and our methods. \n\nIntuitively, Jiacheng et al. [1] proposed using an estimated noise class posterior to select confident examples. Their method select the example with the high confidence of noise class posterior as confident examples. The reason they provide is that high confidence in the noisy class posterior likely corresponds to a small noise rate. Therefore, if we further assumes that easy-to-distinguish examples generally have a low noise rate, then Jiacheng et al.'s method can effectively select easy confident examples without using active learning.\n\nHowever, to select hard confident examples, they have to engage human annotators (active learning). Specifically, they randomly select some unconfident examples and manually label them. By contrast, we propose a method that can automatically learn some hard confident examples by leveraging disentangled representations. Our approach can be seen as an approximation of human labeling, and it does not require the intervention of human annotators.\n\nWe would like to carefully sort this out in the final version.\n\n>**Q3. Discuss the limitations of the proposed method.**\n\n**A3.** Thank you for your advice. We will include limitations in the final version as follows.\n\nManipulating all style factors through data augmentations is generally considered unrealistic, leading to a situation where content and styles are not completely disentangled but only partially disentangled. Achieving complete separation between uncontrolled style factors and content factors can be particularly challenging when learning with noisy labels. As a result, the confident examples selected in our method can encourage disentanglement but hard to guarantee the complete disentanglement of uncontrolled style factors from content factors. Additionally, our approach requires additional computational resources to train extra encoder models.\n\n### Reference\n\n[1] Learning with bounded instance and label-dependent label noise, ICML20.\n\n"}, {"review_id": "PY4e629bil", "reviewer": "Reviewer_koUE", "summary": "This paper focuses on improving the ability to extract hard confident examples (close to the decision boundary), a critical aspect of noisy label learning.  The authors find that a key reason for some hard examples being close to the decision boundary is due to the entanglement of style factors with content factors. In response to this finding, they strive to maintain the consistency of content factors across all examples within the same underlying clean class, even as their style information undergoes changes. This approach is grounded in the belief that the content factors should remain invariant, thereby allowing for a more accurate extraction of hard confident examples. Their proposed method, CS-Isolate, embodies this concept and leverages various data augmentation to alter the styles while regularizing content factors based on some confident examples. CS-Isolate demonstrates robust performance when tested on synthetic noise datasets.", "questions": "* \"If extracted examples have high quality, the performance of a classifier on them is enhanced.” This sentence is a bit confusing. Is the model trained on these examples?\n* \"However, During the training process, it is possible to identify and select some confident examples.\" It seems that \"However\" should be removed.\n* \"Some variation has also been proposed for example, some methods opt to reweight examples, thereby decreasing the contribution of mislabeled samples to the overall loss\" This sentence could benefit from improved punctuation to enhance readability. Breaking it into smaller sentences or using punctuation to separate ideas could make the authors' point clearer and easier to understand.", "limitations": "The authors did not discuss the limitations of the proposed method. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper exhibits a commendable level of organization, facilitating a smooth and intuitive reading experience. Its logical structure aids in the comprehension of complex concepts and methodologies.\n* The inclusion of visual aids for the toy example, significantly enhances the reader's understanding. These illustrations provide tangible insights into the underlying intuition of the proposed method, making the abstract concepts more accessible.\n* The proposed method achieves new SOTA performance on various benchmark datasets. ", "weaknesses": "* The authors base their proposed method on the premise that hard examples being close to the decision boundary is due to the entanglement of unhelpful style factors with useful content factors. However, this key finding appears to lack empirical support from experimental results, which weakens the foundation of their argument.\n\n* I still can’t wrap my head around how the proposed method solves this challenge of identifying or designing data augmentations that can manipulate all style factors in an image.\n\n* The proposed method seems to focus on learning style factors based on artificial styles introduced through data augmentations, rather than modelling the natural style of an image. If the authors' assertion that the proximity of hard examples to the decision boundary is due to the entanglement of unhelpful style factors with useful content factors holds true, it would be logical to base the style factor on the natural style of images. This discrepancy needs to be addressed.\n\n* It's unclear how this method compares to other existing methods in terms of computational efficiency. If CS-Isolate requires significantly more computational resources, it may not be practical for use in real-world applications.\n\n* The authors did not discuss the limitations of the proposed method. \n\n* Several sentences in the paper require further clarification, which will be discussed in the Questions section. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1. Support that hard examples are close to the decision boundary is due to the entanglement of style and content factors.**\n\n**A1**.\n - **Visualization of Examples.** In Fig. 1 of Rebuttal PDF, We demonstrate examples that failed to be identified as confident by DivideMix but were successfully identified by our method. These examples often involve objects that are mixed with others, small, or located in corners.\n\n- **Grad-CAM Analysis of DivideMix.** We employ Grad-CAM to analyze the representation learned by DivideMix, and the results (also shown in Fig. 1 Rebuttal PDF) reveal that the corresponding heatmap highlights style parts of the images. This implies that DivideMix's learned representation contains style information that doesn't contribute usefully to the classification, causing the examples to lie closer to the decision boundary and then can not be selected as confident examples.\n\n- **Comparison with Our Proposed Method.**  We employ Grad-CAM to analyze content factors learned by CS-Isolate-DM (shown in Fig. 1 Rebuttal PDF). It shows that the content factors concentrate on the objects, effectively isolating content from unhelpful style factors that concentrate on other parts of the images. As a result, these examples are not hard examples anymore and are selected as confident examples by our method.\n\n>**Q2. How to design data augmentations that can manipulate all style factors in images.**\n\n**A2**.  As mentioned in Sec. 3 of our paper, in general, manipulating all style factors through data augmentations is considered unrealistic. Specifically, existing theoretical works [1] show that the disentanglement can be achieved if each style variable has a data augmentation to control its change. Although it claims that data augmentations can fully disentangle style, we've found that this assumption may be overly optimistic. Even with a rich set of augmentations, some style factors are typically uncontrolled. Therefore, we need additional auxiliary variables to further disentangle these uncontrolled style factors from content factors.\n\nSpecifically, to further encourage disentanglement, we propose to leverage the labels of confident examples. Intuitively, when content labels are available, we can utilize them to control content factors, encouraging confident examples with the same label to share the same content factor. This process can further disentangle some uncontrolled style factors from learned content factors.\n\nTo provide more insights, we have also formally analyzed this approach in Appendix A. If data augmentation is unable to control all style factors, it can only result in partial isolation, allowing for the separation of controlled style factors from content. To extend this separation to uncontrolled style factors, we can employ the labels of confident examples. For complete isolation of uncontrolled style factors and content factors, the existence of confident examples for all possible uncontrolled style factors is required.\n\n>**Q3. CS-Isolate focuses on learning style factors introduced through data augmentations, rather than modeling the natural style of an image.**\n\n**A3**. For most label-noise datasets, we do not have access to different style domain labels, making it hard to completely control the natural style of an image. Therefore, we propose using data augmentation as a surrogate to approximate style domains, achieving only partial isolation. To further encourage the disentanglement of content and style, we leverage labels from confident examples (See details in A2).\n\nIt's also important to highlight that if datasets contain domain labels (whether labeled by humans or predicted by large models), they can be directly employed by our method to further enhance performance without any changes.\n\n>**Q4. Whether CS-Isolate requires significantly more computational resources.**\n\n**A4**. The additional computational resources are not a lot. For example, training 300 epochs on CIFAR-10 with batch size 64,  the training time of CS-Isolate-DM is approximately 21415s, and the training time of DivideMix is approximately 17667s; the GPU memory usage of DivideMix is 3457MiB, and the GPU memory usage of CS-Isolate-DM is 5007MiB; the number of parameters of DivideMix is 11171274, and the number of parameters of CS-Isolate-DM is 13250829.\n\n>**Q5. limitations of the proposed method.**\n\n**A5**. We will include limitations in the final version as follows. \n\nManipulating all style factors through data augmentations is generally considered unrealistic, leading to a situation where content and styles are not completely disentangled but only partially disentangled. Achieving complete separation between uncontrolled style factors and content factors can be particularly challenging when learning with noisy labels. As a result, the confident examples selected in our method can encourage disentanglement but hard to guarantee the complete disentanglement of uncontrolled style factors from content factors.  Additionally, our approach requires additional computational resources to train extra encoder models.\n\n>**Q6. Is the model trained on extracted confident examples?**\n\n**A6**. All existing methods treat extracted confident examples as clean examples. Their noisy labels are used to train classifiers. Therefore, if the extracted confident examples have higher quality, the classifier trained on these examples can have higher classification accuracy.\nAs a concrete example, Me-momentum [2] uses early stopping to select confident examples and train classifiers on the selected confident examples. \n\n### References\n\n[1] Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style, NeurIPS21.\n\n[2] Me-momentum: Extracting hard confident examples from noisily labeled data, ICCV21.\n"}, {"review_id": "a7SXXeaV9s", "reviewer": "Reviewer_MtkC", "summary": "This paper provides insights into the extraction of hard and confident examples from noisy datasets, which is an important problem when learning with noisy labels. The authors discover that by removing irrelevant style information from the data, hard examples can become more discernible. Building on this finding, they design an innovative method that leverages existing identifiable results of latent variables. With no side information available with learning with noisy labels, this method constructs the necessary information by using data augmentations and confident examples for the identification of latent variables. They support their solution with theoretical analysis, demonstrating its empirical performance on different datasets.", "questions": "1) If the pseudo-code for CS-Isolate-Me is missing or omitted, it would indeed be helpful to understand its differences compared to CS-Isolate-Co. Both methods appear to use a very similar loss function. \n2) How the evidence Lower Bound (ELBO) are derived, it is neither the main paper nor the appendix. \n3) What will be the results of the proposed method on CIFAR10 and CIFAR100 with symmetric noise and pairflip noise. \n4) Does the proposed method sensitive to the dimensionality of $z$? \n5) Could the authors provide more concrete details about the application of various augmentations on each dataset?\n", "limitations": "The authors have mentioned a limitation in appendix that the assumption to achieve the fully isolation of style and content variables is strong. Specifically, achieving the fully isolation of uncontrolled style factors and content factors demands the existence of confident examples of all possible uncontrolled style factors. Consequently, the authors' methods typically encourage, but do not guarantee, the full isolation of these uncontrolled style and content factors.\nNevertheless, I believe it's crucial to include a section dedicated to limitations in the main paper and move this point to that section. More broadly, a method's limitations can encompass a range of aspects, such as computational requirements, sensitivity to hyperparameters, and generalizability across different datasets or noise types, among other factors. I recommend the authors carefully review the proposed method to identify if any of these limitations.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper studies an important problem in machine learning. Specifically, the problem that how to extract hard confident examples when dealing with datasets containing label noise. The ``easy’’ confident examples have been well studied and can be easily identified by using the different techniques such as small-loss trick and memorization effect. However, how to identify the hard and confident examples have only been discussed by a few work. How to effectively identify the hard and confident examples still remains unclear. This problem is important as the hard confident examples usually plays an important role for shaping the decision boundary and improve the classification performance, which mentioned in the previous paper. \n\nThe authors present new insight about hard confident examples. They claim that some examples become difficult to discriminate due to the entanglement of non-useful style information in the learned representation. This finding is interesting and easy to understand, has been surprisingly overlooked in existing work. This finding may pave a new way on discover hard examples in noisy datasets.\n\nThe provided solution is innovative for me. To enjoy the identifiable of latent factors by using variational inference, the availability of side information is necessary. Nonetheless, when learning in the presence of label noise, this side information is not readily accessible. To navigate this challenge, the authors ingeniously devise supervised information for style and content factors by utilizing different data augmentations and confident examples. They also provide intuitive reasoning and a theoretical analysis to demonstrate how identifiability can be achieved, reinforcing the originality and rigor of their approach. \n", "weaknesses": "The reproducibility of the proposed method could be enhanced. Specifically, the authors mentioned employing various data augmentation methods such as shift scale rotate, random crop and horizontal flip, random brightness contrast, color jitter, and random grayscale. However, it remains unclear how concretely the different augmentations are sampled for each dataset, which could affect replication efforts.\n\nThe paper's experimental setup could benefit from further expansion. For synthetic noise datasets, only instance-dependent label noise is used. The performance under widely used noise types like symmetric noise and pairflip noise remains untested. Additionally, an analysis of the method's sensitivity in response to different dimensions of latent variables Z could provide a more comprehensive understanding of the model's robustness.\n\nThe last two paragraphs of introduction are redundant and could be effectively condensed into one concise paragraph. Additionally, it might be beneficial to move the identifiable analysis from the appendix to the main paper after introducing the method for a more seamless reader experience.\n\nThe paragraph ``Disentangled latent factors via auxiliary variable’’ feels somewhat abstract. As these techniques are leveraged to encourage isolation, further explanation could help reduce the reading difficulty for readers. Providing more in-depth discussion in appendix might make this critical point more accessible.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1. It remains unclear how concretely the different augmentations are sampled for each dataset.**\n\n**A1**. In our paper, we utilize 6 types of data augmentation techniques that follow [1]. \n\n**Detailed types of data augmentation.** These techniques include shift scale rotation, random crop and horizontal flip, random brightness contrast, color jitter, and random to gray. \n**How to generate style ID.** We generate different styles via the combination of different types of data augmentation. Each type of data augmentation has a probability to decide whether this augmentation will be applied. The probability of the random crop is 1; The probability of shift scale rotation, horizontal flip, random brightness contrast and random to gray is 0.5; The probability of color jitter is 0.8. Each combination acts as a style and will be assigned a unique style ID.\nWe set the number of different data augmentation $M$ to be 200 in experiments.\n\n>**Q2. Noise types like symmetric noise and pairflip noise remain untested.**\n\n**A2**. We have conducted experiments on the symmetric noise and pairflip noise on CIFAR-10. The noise rate in symmetric noise is 0.2, 0.4 0.6 and 0.8. The noise rate in pairflip is 0.45. The experiment settings are the same as the experiment in the paper. The results are shown in the Tab. 3 of Rebuttal PDF. CS-Isolate-DM performs better than DivideMix.\n\n>**Q3. Sensitivity in response to different dimensions of latent variables Z.**\n\n**A3**. We have conducted the ablation study on the dimension of $Z_c$ and $Z_s$ on CIFAR-10N, and the noise type is ”Worst”. The experiment settings are as same as the description in Sec. 4.1 of our paper. \nThe value of the dimension of $Z_c$ and $Z_s$ is 4, 8, 16, 32, 64, 128 and 256 respectively. The experiment results are shown in Tab. 2 of Rebuttal PDF. The test accuracy increases gradually until the dimension is 64. After the dimension is larger than 64, the test accuracy decreases slightly.\n\n\n>**Q4. The paragraph “Disentangled latent factors via auxiliary variable” feels somewhat abstract.**\n\n**A4**. Thanks for your advice. A brief explanation is shown as follows. We will include more details in the final version.\n\n**Understanding Latent Factors.**\nLatent factors represent underlying attributes or features in the data. For example, in an image of a car, latent factors might include the color, shape, and size of the car. These factors might be mixed or entangled with each other, making it challenging to manipulate or understand them individually.\n\n**Introducing Auxiliary Variables.**\nAuxiliary variables can be thought of as additional information or constraints that assist in the disentanglement process. These could be external labels, time indexes in a time series, or other side information that correlates with the latent factors.\n\n**Model Design.**\n\nTypically, a model such as a Variational Autoencoder (VAE) may be employed to learn a representation of the data. The model would have specific components that are designed to capture the content, style, or other latent factors.\n\n**Incorporating Auxiliary Variables.**\nIf the auxiliary variables are given, these can be used as conditional inputs to the model, guiding the learning process; or can be used to define additional loss functions or regularization terms that encourage the model to learn disentangled representations; or to ensure that specific neurons are responsive to particular factors, thus forcing a separation of these factors in the learned representation.\n\n>**Q5. Combine two paragraphs of introduction. Pseudo-code for CS-Isolate-Me**\n\n**A5**. Thanks. We will carefully merge two paragraphs and move the theoretical analysis into the main paper as suggested. We will also add the pseudo-code of CS-Isolate-Me in the final version.\n\n>**Q6. How the evidence Lower Bound (ELBO) is derived.**\n\n**A6**. We apologize for any confusion surrounding the derivation of the Evidence Lower Bound (ELBO) in our work. Our ELBO is derived by directly applying Bayesian inferences similar to ELBO in iVAE framework [2]. The idea is Instead of using a combined latent variable $z$, we partition it into two components: content factors $z_c$ and style factors $z_s$. This leads to the formulation of two separate KL-divergence terms, $KL(q(z_c|x)||p(z_c|u_c))$ and $KL(q(z_s|x)||p(z_s|u_s))$, rather than the aggregated term $KL(q(z|x)||p(z|u))$ found in Eq. (53) of paper [2]. This separation allows us to disentangle content and style factors and aligns with our problem setting. \n\n\n### References\n\n[1] Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style, NeurIPS21.\n\n[2] Variational Autoencoders and Nonlinear ICA: A Unifying Framework, AISTATS20.\n\n"}, {"review_id": "JVKjGqP8ix", "reviewer": "Reviewer_25vx", "summary": "Summary: This paper presents a solution to infer content factors for classification when learning with the noisy labels especially with hard samples that are close to decision boundary. In order to achieve the goal, they tend to separate the style information from the semantic information with some techniques including data augmentations. Some experimental evaluations and comparisons are shown to prove the effectiveness of the paper.", "questions": "(1) As mentioned before, the overall novelty for this work is small given the fact that disentangled representations can boost the performance including denoising.\n\n(2) The experimental evaluations are not sufficient and questionable due to lack of details. Typically, the level of noisy label percentage is not reported, making the comparison not convincing. Instead, the author simply used some vague statements like \"random 1\", \"random 2\" \"worst\" which leaves many unanswered questions. For instance, how large is the \"random 1\" noise level ? Is the worst case representing a more than 50% corrupted noisy label space ? What does \"Aggregate\" mean ? (Is it implies voting from multiple people for labeling).\nI tried very hard to find the answers but fail to see the detailed meanings of them.\n\n", "limitations": "The paper is proposed with limited novelty, questionable experimental design and unclear writing.\n\n", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The idea of disentangled representation is widely applied for many deep generative models. In this paper, the solution to separate style factors from semantic information also falls into this category which helps to reduce the effect from noisy labels.", "weaknesses": "(1)\tFirst of all, it is not clear for the label noise level, what does random 1, 2, 3 indicate in the experimental evaluations for all the Tables 1-4. Because in the regime of robust learning, usually the percentage of noisy labels is an important parameter in the experimental design. Without clearly stating the noise level, it is very difficult for readers to compare if the approach can work well in the high noise ratio level. Nor does this comparison show fairness as random 1 can be a very low level label noise for method A and a high level label noise for method B.\n(2)\tAs the authors mentioned, if the style and semantic information are important to their idea and the design of the method, it is imperative to visualize the style and semantic information extracted from their images and show it is processed by their method. Lacking of visualization make the readers confusing and difficult to understand. Namely, the method is not well justified and stays on the safe ground without analyzing the semantic information (with visualization). Specifically, it is important to show say for MNIST datasets, semantic information implies the numbers and style information indicates the writing style in this case.\n(3)\tMoreover, there are quite a lot of related work on applying the embedding (dimensionality reduction) to disentangle the latent space for improving the denoising performance (e.g. generative models). After reading the paper, I feel the method claimed is merely a small tweak of the existing methods along this line where the author called it “style and semantic spaces”, while it can actually be more general for all subspaces.\n(4)\tThe papers with similar ideas can be found:\na.\tLearning disentangled representations via product manifold projection, ICML 2021.\nb.\tLearning disentangled representations in the imaging domain, Medical Image Analysis, 2022\nJust to name a few, especially in the reference [b], the solution to introduce the disentangled representations is very similar to the paper. Moreover, it is well known that using disentangled representations can boost performance including denoising with noisy labels. \nNamely, from the high level, the result shown in this paper is only a subset or a specific case of the previous work, which is not novel enough. While the specific application is somewhat interesting, I donot see it is as a Neurips paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1. The authors used confusing words \"random 1, 2, 3, worst\" to call noise types on CIFAR-10N.**\n\n**A1**. The terms \"Random 1\", \"worst,\" and \"aggregate\" are directly from the original paper [1]. They correspond to specific noise types on the real-world noise dataset CIFAR-10N. For example, “Worst\" is a real-world noise type with a noise rate of  40.21%.\n\n>**Q2. Visualize style and content factors.**\n\n**A2**. **Visualize style and content factors.** In Fig. 3 of Rebuttal PDF, we visualize Grad-CAM for content factors and style factors extracted by our method, respectively. Grad-CAM of content factors mainly concentrates on the objects, in contrast, Grad-CAM of style factors mainly concentrates on other pixels in the images.\n\n **Grad-CAM Analysis of DivideMix.** We also employ Grad-CAM to analyze the representation learned by DivideMix, and the results (also shown in Fig. 1 Rebuttal PDF) reveal that the corresponding heatmap highlights non-semantic parts of the images. This implies that DivideMix's learned representation contains style information that doesn't contribute usefully to the classification, causing the examples to lie closer to the decision boundary and then can not be selected as confident examples.\n\n**Comparison with CS-Isolate-DM.**  Moreover, we employ Grad-CAM to analyze content factors learned by CS-Isolate-DM (shown in Fig. 1 Rebuttal PDF). It shows that the content factors concentrate on objects, effectively isolating content from style factors that concentrate on other parts of the images. As a result, these examples are not hard examples anymore and are selected as confident examples by our method.\n\n>**Q3. Noise type Random 1 leads to different noise levels for different methods, then the comparison is unfair.**\n\n**A3**.The comparison is fair. It's important to note that for the same noise type on CIFAR-10N (such as Random 1), the exact same noisy examples are used across all methods being compared. Therefore, the noise level is fixed for all methods.\n\n>**Q4. Applying disentangled representations for learning with noisy labels is a special case and is not novel.**\n\n**A4**. Existing disentangled representation methods can not be directly applied to learning with noise labels. We would like to discuss our method in detail and highlight our novelty. Hopefully, it can address your concern well.\n\n **Theoretical Alignment.** Our method aligns with the framework that employs auxiliary variables to disentangle latent representations and nonlinear ICA, a proven approach when auxiliary variables are available. However, applying it to noisy data is a novel challenge, as auxiliary variables are not accessible for label-noise datasets.\n\n**New Challenge with Noisy Data.** Given a noisy dataset without access to clean labels, the disentanglement frameworks cannot be employed due to missing auxiliary variable information. Our paper addresses this gap by targeting two new questions: 1) what auxiliary variables need to be constructed, and 2) how to effectively construct them.\n\n**Answers to these questions.** We identify that both style and content auxiliary variables are required to achieve disentanglement through intuitive examples and theoretical analyses in our main paper and Appendix A. Specifically, although existing theoretical work [1] suggests that data augmentation can fully disentangle style, we found that their assumption is too strong. We emphasize that in practice, it's impossible to find sufficient data augmentations to control all style factors. Some factors remain uncontrolled, and their information will be mixed into content factors. We propose that leveraging confident examples is key to further disentangling this information. \nAfter having identified what auxiliary variables are required, we carefully design CS-Isolate which effectively disentangles content and style factors by properly constructing both style and content auxiliary variables.\n\n**Peer Agreement on Novelty or Insight.** Reviewers NWTZ, MtkC, and fCED have acknowledged the newness, innovation, and novelty of our method. Additionally, as agreed by Reviewers fCED, koUE, and MtkC, our work provides new insight into how to extract hard confidence examples in labeling with noisy labels.\n>**Q5. The solution to introduce the disentangled representations is very similar to the paper [3].**\n\n**A5**. Paper [3] is a survey paper on methods for disentangling representations in the image domain. It of course contains some applications that leverage the framework of using auxiliary variables to disentangle representations when auxiliary variables are available. However, our work goes beyond this by addressing missing auxiliary variables in learning with label noise. \n\nAs aforementioned in A4, unlike scenarios where auxiliary variables are available, given only noisy labels, we have to focus on determining what auxiliary variables need to be constructed, and how to effectively construct them. Moreover, we provide theoretical explanations to support the necessity of using confident examples; we also offer new insight into why certain correct examples are hard to be extracted as confident examples. \n\n>**Q6. This paper shares a similar idea with the papers [4].**\n\n**A6**. The ideas must not be similar, as the assumptions for achieving disentanglement are completely different. Specifically, the method in paper [4] relies on having non-i.i.d. examples (data pairs) to learn the disentangled representations. However, our method is in the settings of i.i.d. examples and employs auxiliary variables to disentangle latent representations. \n\n### References\n\n[1] Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations, ICLR22.\n\n[2] Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style, NeurIPS21.\n\n[3] Learning disentangled representations in the imaging domain, Medical Image Analysis 22.\n\n[4] Learning disentangled representations via product manifold projection, ICML21.\n"}]

Author rebuttal:
Dear Reviewers,

**We sincerely thank you all for reviewing our paper and providing insightful feedback to help us improve our work.** 

We are glad that the reviewers collectively acknowledge the paper's deep **insight into the extraction of hard confident examples** (Reviewer fCED, Reviewer NWTZ, Reviewer MtkC), addressing an **important and overlooked problem** in the field (Reviewer MtkC). The **newness, innovation, and novelty** of our methods have been appreciated, particularly those proposed for separating style factors and handling noisy labels (Reviewer fCED, Reviewer NWTZ, Reviewer MtkC). Our paper's method has been praised for its **effectiveness** in achieving better performance and new state-of-the-art results (Reviewer fCED, Reviewer NWTZ, Reviewer koUE, Reviewer 25vx). The writing quality was also commended, with specific compliments for the organization and **inclusion of visual aids** (Reviewer koUE).

We have tried our best to address all concerns and have uploaded Rebuttal PDF containing some exciting experimental results. The main contents of Rebuttal PDF are as follows:
- Visualization results of hard/easy confident examples, as well as the content/style factors;
- Ablation study on hyper-parameters;
- Experiment results on symmetric and pairflip noise;
- Experiment on Clothing1M.

**Once again, we thank all the reviewers for their time and effort in reviewing our paper. Please let us know if there are any further concerns.**

Sincerely,

Authors



Meta-review: I have read all the materials of this paper including the manuscript, appendix, comments, and response. Based on collected information from all reviewers and my personal judgment, I can make the recommendation on this paper, acceptance. 

**Research Problem**

This paper considers a very specific research question on hard confident samples. Specifically, the existing literature regards the confident samples are far from the decision boundary, while the authors believe there exist some confident samples that are close to the decision boundary but hard to distinguish. In this paper, the authors aimed to extract these samples that are called them “hard confident samples.”

PS: 1. The authors use “examples” along with the manuscript, while I would like to use “samples.” 2. For some well-defined research question, I usually use one or two sentences for summarization. But for this paper, which targets a very specific research question, I have to restate some background. 


**Motivation**

In Line 171-198, the authors clearly illustrated the research challenges. So, I omitted it here.

**Philosophy**

In Line 199-221, the authors demonstrated the philosophy to tackle the above challenges via data augmentation and self-learning. So, I omitted it again. 

**Technique**

In general, the techniques in Section 3 make sense to me. A pseudo code is suggested in the Appendix.


**Experiments**

1. I agree with one reviewer that the current experimental results are reported in tables. It would be better to have some figures as well. For example, Figure 1 should be moved into this part.
2. Significant tests are suggested.
3. The authors need to report how many hard confident samples their algorithm detects and verify whether they are hard samples.

**Related Work**

I noticed a reference [1], which is not related to this paper in terms of research question but considers the content and style for a different task. I am not sure whether the authors are inspired by this paper. But it is better to discuss it in the manuscript.

[1] Harnessing Out-Of-Distribution Examples via Augmenting Content and Style, ICLR’23.

**Presentation**

Although the current version is readable and understood, the presentation can be further improved.

1. CS-Isolate in Line 16 is used without an introduction.
2. “During” in Line 87 -> “during.”
3. Figure 1 is the experimental result, rather than some evidence for illustrating the motivation. In a CS-style paper, this part should move to the experimental part.
4. Figure 2 is an illustrative example of motivation. However, more descriptions are needed; for example, what does the big red circle mean? What does blue and yellow mean? 
5. “Challenging” in Line 171 should be “Challenges.”
6. “7”. In Line 184 should be “7.”
7. The layout of Figure 4 should not occupy Line 171. It is better not to occupy Line 158-159, either. 
8. The roadmap before Section 3.1 is good.
9. It is better to provide informal definitions of hard/confident/noisy samples.
10. The intuition in Line 199-221 can be more concise. 
11. $U_s$ and $U_c$ in Line 227 are used without an introduction.
12. There are many notations. If some notations are only used for 1-2 times, there is no necessary to give them a notation. Instead using their descriptions should be better.
Decision: Accept (poster)