Title: CS-Isolate: Extracting Hard Confident Examples by Content and Style Isolation
Venue: NeurIPS 2023
Accepted: yes
Keywords: learning with label errors
Abstract: Label noise widely exists in large-scale image datasets. To mitigate the side effects of label noise, state-of-the-art methods focus on selecting confident examples by leveraging semi-supervised learning. Existing research shows that the ability to extract hard confident examples, which are close to the decision boundary, significantly influences the generalization ability of the learned classifier.
In this paper, we find that a key reason for some hard examples being close to the decision boundary is due to the entanglement of style factors with content factors. The hard examples become more discriminative when we focus solely on content factors, such as semantic information, while ignoring style factors. Nonetheless, given only noisy data, content factors are not directly observed and have to be inferred.
To tackle the problem of inferring content factors for classification when learning with noisy labels, our objective is to ensure that the content factors of all examples in the same underlying clean class remain unchanged as their style information changes.
To achieve this, we utilize different data augmentation techniques to alter the styles while regularizing content factors based on some confident examples. By training existing methods with our inferred content factors, CS-Isolate proves their effectiveness in learning hard examples on benchmark datasets. The implementation is available at https://github.com/tmllab/2023_NeurIPS_CS-isolate.

CS-Isolate: Extracting Hard Confident Examples by Content and Style Isolation

Abstract

Label noise widely exists in large-scale image datasets. To mitigate the side effects of label noise, state-of-the-art methods focus on selecting confident examples by leveraging semi-supervised learning. Existing research shows that the ability to extract hard confident examples, which are close to the decision boundary, significantly influences the generalization ability of the learned classifier. In this paper, we find that a key reason for some hard examples being close to the decision boundary is due to the entanglement of style factors with content factors. The hard examples become more discriminative when we focus solely on content factors, such as semantic information, while ignoring style factors. Nonetheless, given only noisy data, content factors are not directly observed and have to be inferred. To tackle the problem of inferring content factors for classification when learning with noisy labels, our objective is to ensure that the content factors of all examples in the same underlying clean class remain unchanged as their style information changes. To achieve this, we utilize different data augmentation techniques to alter the styles while regularizing content factors based on some confident examples. By training existing methods with our inferred content factors, CS-Isolate proves their effectiveness in learning hard examples on benchmark datasets. The implementation is available at https://github.com/tmllab/ 2023_NeurIPS_CS-isolate .

1 Introduction

Large-scale machine learning datasets frequently contain noisy labels, as seen in datasets like ImageNet [9] and Clothing1M [55]. Training deep neural networks with such noisy data would result in poor generalization ability, as these networks can memorize incorrect labels [13, 3].

To mitigate the side effects of label noise, different methods have been proposed [33, 13, 37, 61, 30, 31, 49]. Major existing state-of-the-art methods are based on confident examples selection [61, 2]. Intuitively, those methods first exploit the memorization effect , enabling deep neural networks to learn simple patterns shared by the majority of training examples [60, 1, 19]. Since clean labels typically constitute the majority in each noisy class [7, 39], deep neural networks initially fit the training data with correct labels and then progressively fit examples with incorrect labels [13]. To prevent the model from fitting incorrect labels, early stopping is usually employed [44, 3, 50, 4]. Then the small-loss trick is used to extract the confident examples with high certainty [21, 35, 28, 38]. If extracted examples have high quality, the performance of a classifier can be enhanced.

To improve the performance of a classifier, it is crucial to ensure that the confident examples have a distribution similar to that of clean data [36, 32]. Specifically, it is necessary to extract not only confident examples that are far from the decision boundary but those that are close to it. The former examples are easy to be identified and extracted. However, the latter examples, which are close to the decision boundary, often become entangled with mislabeled examples, making them challenging to be identified or extracted. In this paper, we discover that on image datasets, the entanglement of unhelpful style factors with useful content factors is a key reason that leads to the hard examples becoming hard to be classified, and thus these examples are close to the decision boundary.

(a)

(b)

In Fig. 1, we provide an intuitive understanding of the relationship between hard examples and the entanglement of style factor Z s and content factor Z c . Let us first assume the underlying content factor Z c and style factor Z s are given. In Fig. 1a, by visualizing Z s and Z c , we can observe a separation between the examples belonging to underlying clean classes 0 and 1 . In real-world scenarios, however, Z c and Z s are not directly available. Instead, existing methods [13, 28] could learn representations Z 1 and Z 2 from noisy data to extract confident examples. These learned representations, Z 1 and Z 2 , are nonlinear transformations of style factor Z s and content factor Z c . They do not ensure the disentanglement of Z s and Z c [24, 20, 59]. As the example illustrated in Fig. 1b, both Z 1 and Z 2 contain the information of the style factor Z s and the content factor Z c . In the illustrated representation space, examples within the region encircled by red dashed lines are close together. The confident examples in this region are entangled with mislabeled examples, making them challenging to be extracted by existing methods without isolating (disentangling) style and content information in learned representations.

We have discussed that the entanglement of style factor Z s and content factor Z c in the representation space can create hard examples. However, if we can isolate content from styles, then by focusing solely on the content factor Z c , these hard examples become more discriminative. For example, let's consider projecting the examples onto the Z c -axis by ignoring style factor Z s in Fig. 1a. The data points on the Z c -axis in Fig. 1b illustrate that even after the nonlinear transformation, the examples remain distinguishable. This motivates us to tackle the problem of inferring content factors for extracting hard examples when learning with noisy labels.

Precision: 0.957

Recall: 0.978

Precision: 0.954

Recall: 0.966

Precision: 0.896

Recall: 0.940

To isolate style and content information in learned representations, it is crucial to learn invariant representations of all examples as their styles change. However, when data contains label noise, we can not access clean classes, making it challenging to achieve this isolation. To address this issue, we introduce a method built upon identifiable Variational Autoencoder (iVAE) [23]. The core intuition behind our approach is to construct different styles for each example through various data augmentation techniques. We then encourage both the original and the augmented examples to possess different style factors but maintain

identical content factors, which is a self-supervised learning manner. This initial step could achieve a preliminary level of isolation between style and content factors. As the training process progresses, we can identify and select certain confident examples based on the content factors. Leveraging these confident examples, we further enhance the isolation process by encouraging all examples sharing the same label to share identical content factors. By isolating content and style factors, our method, CS-Isolate, effectively helps existing sample selection methods to extract hard examples typically overlooked by existing sample selection methods. In Fig 2. we evaluate the performance of confident examples selection on CIFAR10N with the noise type 'worst' [48] by using two metrics: precision and recall. When comparing these two metrics, CS-Isolate consistently outperforms DivideMix [28] and Me-Momentum [2]. This result demonstrates that our method can not only select confident examples more accurately, as shown by the higher precision, but also capture a larger portion of the correctly labeled examples from the entire dataset, as evidenced by the higher recall. By improving the quality of confident examples with the isolation of the content and style factors, our method effectively helps improve the test accuracy of existing methods. We have also theoretically analyzed the identifiability of content and style factors in Appendix A.

2 Background and Related Work

Problem setup. Let's denote ˜ D as the distribution of a noisy example ( X, ˜ Y ) from the set X × { 1 , 2 , . . . , C } , where X denotes the variable of instances, ˜ Y represents the variable of noisy labels, X is the feature space, { 1 , 2 , . . . , C } is the label space, and C is the total number of classes. In the learning scenario with noisy labels, clean labels are not observed. Given a noisy training sample ˜ S = { x i , ˜ y i } N i =1 , independently drawn from ˜ D , the objective is to leverage this sample ˜ S to learn a classifier robust against label noise.

Sample selection methods for learning with noisy labels. In learning with noisy labels, major current state-of-the-art (SOTA) methodologies predominantly involve sample selection strategies. These strategies seek to divide the dataset into confident and unconfident examples. The basis of these strategies is the exploitation of the memorization effect inherent in deep neural networks. The memorization effect enables the networks to initially grasp and learn the simple patterns, and then learn the complex patterns gradually [60, 1]. Given that clean labels usually form the majority within each noisy class [7, 39], these networks would initially fit the examples with accurate labels, and subsequently fit the examples with incorrect labels over time [13].

Preventing the learning model from fitting incorrect labels is crucial to ensure sample selection quality. To achieve this, strategies such as early stopping are often employed [44, 3]. Additionally, the smallloss trick is utilized to identify and extract confident examples with high certainty [28, 38, 51, 18]. Some variation has also been proposed, e.g. , some methods opt to reweight examples, thereby decreasing the contribution of mislabeled samples to the overall loss [41, 13]. To guarantee the statistical consistency of algorithms, Jiacheng et al. [7] introduces active learning to acquire the labels of randomly chosen examples from unconfident examples to mitigate the bias introduced by sample selection.

Moreover, the set of confident examples often undergoes dynamic changes during the training stage. This is achieved by leveraging semi-supervised learning methods to relabel training instances and reselect confident and unconfident examples using the small loss trick. Various techniques for this purpose have been proposed and empirically demonstrated superior performance, including consistency regularization [27] adopted by [10], MixMatch [5] used by [28], co-regularization by [47], and contrastive learning by [30, 29, 45, 8, 11, 57, 61]. Self-training and co-training have been used by [2] and [35, 13, 21], respectively.

Representation learning by generative model. Consider a data generation in Fig. 3a, X is the observed data, and Z is the unknown underlying representation to generate X . Variational autoencoder framework [24]

(a)

(b)

can be used to learn the latent representation. In this process, a standard normal distribution is utilized as a prior for the latent variables, and a variational posterior q ( Z | X ) is employed to approximate the unknown underlying posterior p ( Z | X ) . Disentangled representation is very important, which

can allow a rich class of properties to be imposed on the learned representation, such as sparsity and clustering. To disentangle the representation, the variational autoencoder framework has been further expanded by modifying the original loss function, resulting in various algorithms. β -VAE [16] suggests an adaptation framework that adjusts the weight of the KL term to balance the independence of disentangled factors and reconstruction performance. β -TCVAE [6] further analyzes the KL term of β -VAE [16] and only adjusts the total correlation term to achieve disentanglement. HOOD [17] uses the clean labels and domain labels to disentangle content and style factors, but the clean labels and domain labels are unknown in the setting of learning with noisy labels. These methods are toward the goal of disentanglement but do not have theoretical guarantees of identifiability of their inferred latent representations.

Disentangled latent factors via auxiliary variable. Recent studies [20, 23] show that theoretical guarantees of identifiability can be achieved if an auxiliary variable related to the representation can be obtained. Khemakhem et al. [23] provide identifiability of iV AE with additional inputs by employing the theory of nonlinear Independent Component Analysis (nonlinear ICA) [20]. Intuitively, given a factorized prior distribution over the latent variables that are conditional on an additional auxiliary variable U , i.e. , the class label and the time index in a time series, the latent factors are identifiable up to a certain degree. The data generation processing is changed to Fig. 3b. To disentangle the representation, the method assumes that each factor of the representation is independent.

3 Content and Style Isolation When Learning with Noisy Labels

In this section, we present CS-Isolate, a method designed to help select hard confident examples by content and style isolation.

3.1 Preliminaries

Noisy data generative process. To learn latent factors by leveraging generative models, the generative model has to model the noisy data generative process. Firstly, we introduce the noisy data generative process. We denote observed variables with gray circles and latent variables with white circles. Specifically, the content factor Z c is generated by the latent label Y . The different style domains U s give rise to the different style factor Z s . Then, the image X is generated by the combined influence of the style factor Z s and the content factor Z c . Noisy labels ˜ Y are then generated based on the image X . In general cases, Z c and Z s can also have statistical or causal dependencies. We follow existing work that assumes the content factors are unchanged across different styles [46].

Challenge in isolating content from styles without labels. Isolating content from style without labels is a challenging task [46]. We first introduce the assumptions made by existing methods, as well as difficulties that may be encountered in practice.

Existing methods [26, 23, 46] for isolating content from styles without labels assume that the data augmentation for controlling each style factor can be designed, which means that we can intervene (control) all style factors. For instance, the data augmentations for controlling rotation angle and scaling of images can be designed by using affine transformation [40]. When we apply a data augmentation that rotates an image, the style factor for the rotation angle changes in the augmented image compared to the original. Similarly, when we scale an image, the style factor for scaling becomes different in the augmented image. If we have the data augmentation to control each style factor, then by training a generative model with the augmented images, the model can then identify style factors by comparing the changes in styles between the original and augmented images.

However, the major challenge lies in the fact that we generally cannot design sufficient data augmentations to control all style factors in an image. For instance, in the CIFAR-10 dataset [25], some pictures with the label 'horse' contain a person. In this context, the "person" acts as a style factor. Existing data augmentations cannot control this, as they cannot remove the person from images easily. This simple example illustrates that it is generally impossible to control all style factors through data augmentations. Then the assumption required by existing methods usually is hard to satisfy.

The violation of the assumption leads to the learned representations for content containing style information.

Challenge in isolating content from style with noisy labels. We generally cannot design data augmentations that can control all style factors in an image. In this case, clean labels are essential to help isolate content from styles. Intuitively, by comparing the change of images with different clean labels that share the same styles, one can infer content factors used for classification. This achieves isolation of content and style. However, when dealing with noisy data, relying on labels becomes problematic as they contain label errors. Images with the same content factors can have different noisy labels, making it fail to infer content factors used for classification. This situation further complicates the task of isolating content from styles, highlighting the challenges posed by noisy labels.

3.2 CS-Isolate for Extracting Hard Example

In this paper, we find that a key reason for some hard examples being close to the decision boundary is the entanglement of style factors with content factors. Intuitively, style factors render the learned representation of certain examples less discriminative, thereby making them close to the decision boundary. If content and style information can be isolated, many hard examples would be easily distinguished by ignoring the style information. This is because we have left only content information in the learned representation, making these representations far from the decision boundary.

Inspired by these findings, we propose CS-Isolate, which aims to isolate content from styles for extracting confident examples. To achieve isolation, we utilize self-supervised learning. Specifically, we ensure that original and augmented images maintain the same content factors despite having different style factors due to data augmentations. This achieves a preliminary level of isolating content from styles. As previously mentioned, it is generally impossible to manipulate all style factors through data augmentation. Some style factors remain uncontrolled and stay contained in the learned representations for content. To further encourage the isolation, labels of confident examples are used. Specifically, as training progresses, we identify and select confident examples whose labels are likely to be accurate. By harnessing the labels from these confident examples, we encourage the examples with the same label to have the consistent content factors, irrespective of stylistic variations. This further isolates content from uncontrolled styles. By leveraging different data augmentations and confident examples, our method can effectively separate the learned latent representations into two parts which exclusively contain either style or content information. Subsequently, classifier heads can be trained purely on the representations containing content information. This approach not only improves the identification of hard confident examples but also enhances classification performance.

Isolating content from styles using auxiliary variables. We follow existing work using the variational autoencoder with auxiliary variables [23] to isolate content from styles. Let U s and U c denote the auxiliary variables that control the style factor Z s and content factor Z c , respectively. Specifically, by reconstructing an image with the supervision of auxiliary variables U s and U c , we encourage the images with the same content information but different style information to have the same content factor Z c but the different style factor Z s . This allows us to isolate content from styles within an image. Then only employing Z c for selecting hard confident examples.

However, given only noisy data, we face the challenge of the unavailability of auxiliary variables U c and U s . In response, we devise surrogates for these auxiliary variables. To find a surrogate for the style auxiliary variable U s , we employ different data augmentation techniques, each of which generates a unique style domain with distinct style factors. We then assign the style domain ID as the style auxiliary variable U s . To find a surrogate for the content auxiliary variable U c , we adopt the philosophy of self-supervised learning to assign a unique content ID as the content auxiliary variable U c to each image. The images that have the same content ID are encouraged to have consistent content factors. Moreover, during the learning process, some confident examples can be extracted. To effectively leverage these examples in learning content factors, we reassign the content ID of confident examples belonging to the same class to be the same. This allows our model to learn consistent content factors for the images in the same class.

Constructing style auxiliary variables via data augmentations. For the learning of distinct content factors and style factors, it is essential to construct images from domains with different styles

by using data augmentations. Each image carries its corresponding domain ID as the style auxiliary variable U s . Let A = { A 0 , A 1 , A 2 , . . . , A M } represents the set of data augmentations, and M is the number of data augmentations. Note that, for convenience, we use A 0 to denote a data augmentation such that applying the data augmentation A 0 to the image x does not change it at all, i.e. , x = A 0 ( x ) .

When applying these diverse data augmentations to an x , different augmented images are obtained with different style factors. For the augmented image x A i obtained by using the i -th data augmentations that come from the i -th style domain, we can assign its style ID to be i . Specifically, for each image x , each data augmentation A i ∈ A is applied, resulting in a set containing pairs of the augmented image and the corresponding style ID, i.e. ,

where U x Ai s is defined to be the style ID (which serves the style auxiliary variable) of the image x by applying i -th data augmentation. In this manner, for each image x , we can generate a set of augmented images ( { x A i } i M =0 ), including the original images and the ones with different style factors controlled by different style IDs.

Constructing content auxiliary variables in a self-supervised manner. To guide the generative model in learning the consistent content factor across images sharing the same content information, we aim to assign the same content ID (which serves as the content auxiliary variable) to images with the same content information. However, clean labels cannot be obtained in learning with noisy labels, meaning we cannot know which images have the same content information. To construct content auxiliary variables without clean labels, we adopt self-supervised learning. Specifically, we consider that different data augmentations applied to an image typically do not alter its content information. Hence, we can assign a unique content ID to each original image and its augmented versions. This assignment can be mathematically expressed as follows:

where T is the total number of distinct original images in the training data, U ( x Ai j ) c represents the content ID of the j -th image in the training data after applying the data augmentation A i , and C is the number of classes.

Moreover, this initial assignment of content IDs can be refined by using confident examples, denoted as S l . As the training process progresses, we can extract these confident examples by leveraging the small-loss trick [13, 3, 28]. The content IDs are then further refined based on the labels of the examples within the confident examples. The refinement process can be described as:

where y c j is the label of the confident example ( x j , y c j ) . After this refinement process, confident examples in the same class will have the same content ID. Consequently, these refined content IDs could enable the model to learn consistent content factors for the images in the same class.

Encouraging style and content isolation for extracting hard example. After obtaining the auxiliary variable, we isolate content from styles by leveraging iV AE [20]. Specifically, the prior distribution of the content factor Z c is conditional on the auxiliary variable U c , i.e. P θ c ( Z c | U c ) . Similarly, the prior distribution of the style factor Z s is conditional on the auxiliary variable U s , i.e. P θ s ( Z s | U s ) . The θ c and θ s are the learnable parameters of the distribution. The objective is to maximize the data likelihood which is as follows.

where we use q D to denote the empirical distribution of the training sample S all . Weuse the variational inference method to approximate the underlying posterior distribution p θ ( Z c , Z s | X,U c , U s ) [23, 24]. Specifically, two inference models (encoders) q ϕ c ( Z c | X ) and q ϕ s ( Z s | X ) are introduced to infer latent variables Z c and Z s respectively and model the distribution q ( Z c , Z s | X ) that is used to approximate the distribution p θ ( Z c , Z s | X,U c , U s ) . Therefore, the distribution q ( Z c , Z s | X ) can be decomposed as follows:

We learn the parameters { θ c , θ s , ϕ c , ϕ c } by maximizing the evidence lower-bound ELBO for each example ( x, u c , u s ) . The corresponding loss is:

where KL is the Kullback-Leibler divergence. Intuitively, when the style changes, the content remains constant. The model has to infer the unchanged content factors for image reconstruction. Thus, the content factors can be isolated from style factors. The content factors can be identified up to the block-identifiable defined in [46].

Utilizing content information to extract hard examples. After minimizing the ELBO loss L ELBO , we have learned representations ˆ Z c and ˆ Z s which exclusively contain either style or content information. Then we employ existing sample-selection-based methods solely focus on ˆ Z c for confident example selection, a classification head f ψ : Z c → ∆ C -1 is introduced. The classification head maps the space of content factors to a C -1 probability simplex, where C represents the number of classes.

In practice, we propose an end-to-end approach to learn the classification head f ψ and learn to infer content factor for extracting hard confident examples. This approach simultaneously minimizes the evidence lower bound loss L ELBO and the loss of existing sample-selection-based methods by leveraging the Lagrangian method [22, 12]. The content IDs are refined dynamically during the learning process. The overall loss function L all is thus given as:

where L ssl denotes the loss of the sample selection method, which optimizes the parameters of the classification head ψ and the content encoder ˆ ϕ c . The loss L ref is the cross-entropy loss on confident examples. The hyper-parameter λ ELBO and λ ref are used to control strength of L ELBO and L ref , respectively. Here, we illustrate a concrete example of employing the inference model q ˆ ϕ c in conjunction with DivideMix [28] in an end-to-end implementation. For a detailed walkthrough, please refer to the pseudo-code provided in Appendix B and the original paper [28].

DivideMix [28] applies a Gaussian mixture model to enhance MixMatch [43] for confident example selection and classifiers training. In DivideMix, during each epoch of training, the data is divided into a set of confident examples S l and a set of unconfident examples S u . The confident examples contain the sharpened soft labels mixed by their labels in datasets and predicted labels given by the classification head f ψ . The unconfident examples contain the sharpened soft labels predicted by the classification head f ψ . Then, the semi-supervised learning approach, MixMatch [5] is employed by transforming confident ( S l ) and unconfident ( S u ) samples into augmented confident ( S ′ l ) and unconfident ( S ′ u ) samples by a linearly mixing.

The overall loss function, composed of a confident sample loss, an unconfident sample loss, and a regularization term, ELBO loss and a classification loss, i.e. ,

Intuitively, L S l is a cross-entropy loss for the labeled examples; L S u is the mean squared error for the unlabeled samples; L reg is a regularization term to prevent the model from predicting all samples to belong to a single class. These three terms are defined as follows specifically.

where y s is the sharpened soft label and q i ϕ c denotes the i -th coordinate of its output on input x . In Appendix B, we illustrate our method with other sample selection methods.

4 Experiments

In this section, we introduce the setting of our experiments and compare our experimental results with existing methods. Most of our experiments are left out in Appendix C due to the limited space.

4.1 Experiment Setting

Dataset and noise type. We evaluate our methods on three synthetic noise datasets FashionMNIST [54], CIFAR-10 [25], and CIFAR-100 [25], and two real-world label-noise datasets, CIFAR-10N [48] and Clothing1M [55]. FashionMNIST includes 70,000 images of size 24 × 24 , categorized into 10 classes with 60,000 for training and 10,000 for testing. Both CIFAR-10 and CIFAR-100 contain 50,000 training images and 10,000 test images; CIFAR-10 comprises 10 classes, while CIFAR-100 includes 100 classes. The image size in both CIFAR datasets is 32 × 32 × 3 . To generate noisy labels for these clean datasets, we employ the instance-dependent noisy label generation methods proposed in [52]. CIFAR-10N, a noisy version of CIFAR-10, includes five types of label noise: 'worst', 'aggregate', 'random 1', 'random 2', and 'random 3', all annotated by humans. The noise rates are 40.21%, 9.03%, 17.23%, 18.12%, and 17.64%, respectively. Clothing1M contains 1 million images with real-world noisy labels for training and 10,000 images with clean labels for testing.

Network structure and optimization. We implemented our method using PyTorch 1.12.1 and performed experiments on the NVIDIA Tesla V100. For fair comparisons, we chose PreAct ResNet-18 [15] as the backbone of the encoder q ϕ c for Fashion-MNIST, CIFAR-10, CIFAR-100, and CIFAR-10N, a ResNet-50 [14] with ImageNet pre-trained weight for Clothing1M. We employed SGD with a momentum of 0.9 and a weight decay of 0.0005 to optimize the encoder q ϕ c and the classifier head f ψ . We used Adam with default parameters to optimize the encoder q ϕ s and the decoder p θ . For experiments on synthesis datasets, the initial learning rate for SGD was set at 0.02 and for Adam at 0.001. The batch size is 64. For experiments on Fashion-MNIST, our network was trained for 100 epochs. Both learning rates were reduced by a factor of 10 after 80 epochs. For experiments on CIFAR-10, CIFAR-100, and CIFAR-10N, our network was trained for 300 epochs. Both learning rates were reduced by a factor of 10 after 150 epochs. For experiments on Clothing1M, our network was trained for 80 epochs with a batch size of 32. The initial learning rate for SGD was set at 0.002 and for Adam at 0.001. Both learning rates were reduced by a factor of 10 after 40 epochs. The encoder q ϕ c and the classifier head f ψ were warmed up on noisy data for 10 epochs for CIFAR-10 and CIFAR-10N, warmed up for 30 epochs for CIFAR-100, 5 epochs for Fashion-MNIST, and 1 epoch for Clothing1M. The dimensions of Z c and Z s were set at 32. The data augmentation techniques include shift scale rotation, random crop and horizontal flip, random brightness contrast, color jitter, and random to gray. For all experiments, we set M = 1000 , λ r = 1 and λ ELBO = 1 e -3 . λ ref was increased gradually then kept constant at 1 e -3 after 140 epochs. For synthetic datasets, we set λ u as 0 and 15 for the noise rates of 0.2 and 0.4 in FashionMNIST and CIFAR-10 datasets and as 100 in CIFAR-100. For CIFAR-10N, we set λ u as 50 in the "worst" noise type and 0 in the rest of the noise types. For Clothing1M, we set λ u as 0.

Baselines and measurements. We compare our method against several state-of-the-art techniques: (1) CE: the standard cross-entropy loss, with the model trained directly on noisy data; (2) Co-Teaching [13]: involves training two networks and uses the small-loss trick to select confident examples for each other; (3) Forward [39]: applies a transition matrix to correct the loss function; (4) T-Revision [53]: revises the transition matrix to enhance performance; (5) Me-momentum [2]: uses confident examples to refine the classifier, which is then used to update confident examples alternately; (6) BLTM [56]: uses a Bayes optimal label to estimate the transition matrix; (7) CausalNL [58]: employs a causal model to assist the learning of the classifier; (8) DivideMix [28]: employs two networks to select confident examples and uses semi-supervised techniques to utilize unlabeled examples. We report the precision, recall and AUC for the confident examples selection. We also report the test accuracy on the test dataset. The value is the average over the last 10 epochs.

4.2 Confident Examples Quality

To evaluate the quality of the confident examples, we examine the precision and recall of these examples. The precision quantifies the proportion of accurately labeled examples within the set of confident examples. On the other hand, recall measures the proportion of accurately labeled examples in the confident examples relative to the total number of correct labels. The experimental results are displayed in Tab. 1 and Tab. 2. The results indicate that the precision performance of our proposed method matches the state-of-the-art levels. However, the recall of our method surpasses that of existing methods. This demonstrates that our method can identify more confident examples without compromising precision. This suggests that our approach effectively maintains high precision and improves recall, thus selecting more confident examples with correct labels.

4.3 Classification Accuracy

We conducted extensive experiments on three synthetic noise datasets (Fashion-MNIST, CIFAR-10 and CIFAR-100) and two real-world datasets (CIFAR-10N and Clothing1M). For the synthetic

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Cat

Horse

(a)

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Bird

Ship

(b)

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Bird

Horse

(c)

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Ship

Cat

(d)

datasets, we employed instance-dependent noisy label generation methods, as proposed by [52]. We experimented with noise rates of 0.2 and 0.4, denoted by IDN-0.2 and IDN-0.4 respectively. The experiment results are presented in Tab. 3, Tab. 4 and Tab. 5. Our proposed method outperforms existing methods in terms of test accuracy on both synthetic and real-world datasets containing label noise. Notably, as the noise rate increases, the performance gap between our method, CS-Isolate-DM, and the existing methods becomes more pronounced. This highlights the robustness and effectiveness of our approach in scenarios with higher levels of label noise.

4.4 Hard Confident Examples Visualization

The proposed method is expected to select confident examples based on content factors rather than style factors. To analyze this, we use Grad-CAM [42] to visualize the regions used to select confident examples. The visualization results are shown in Fig. 5. The experiment is conducted on the realworld dataset CIFAR-10N, and the noise type is 'worst'. The experiment results demonstrate that our method can correctly focus on the object in images. Specifically, when there exist uncontrolled style factors, e.g. , the person near the horse, the activation maps for CS-Isolate-DM can successfully focus on the right object used for classifying the horse instead of uncontrolled style factors. In contrast, the baseline method, DivideMix, focuses on the style factors that are not related to the class 'horse' and fails to select these confident examples.

5 Conclusion

This paper is motivated by the fact that only focusing on content factors such as semantic information makes examples more discriminative. We, therefore, proposed a novel CS-Isolate approach to infer and isolate the content information for classification. This is achieved by leveraging variational inference and constructing auxiliary variables via data augmentation techniques to modify style factors while regularizing content factors using confident examples. By training existing sampleselection-based methods with our inferred content factors, CS-Isolate improves their effectiveness in learning hard examples and classification accuracy on different image datasets.

Acknowledgments and Disclosure of Funding

Tongliang Liu is partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031. Dong Xu is partially supported by the Hong Kong Jockey Club Charities Trust under Grant 2022-0174, the Hong Kong Research Grant Council under General Research Fund (17203023), the Startup Funding and the Seed Funding for Basic Research for New Staff from The University of Hong Kong, and the funding from UBTECH Robotics.

References

Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In ICML , pages 233-242, 2017.

Yingbin Bai and Tongliang Liu. Me-momentum: Extracting hard confident examples from noisily labeled data. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9312-9321, 2021.

Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. arXiv preprint arXiv:2106.15853 , 2021.

Yingbin Bai, Erkun Yang, Zhaoqing Wang, Yuxuan Du, Bo Han, Cheng Deng, Dadong Wang, and Tongliang Liu. Rsa: Reducing semantic shift from aggressive augmentations for self-supervised learning. Advances in Neural Information Processing Systems , 35:21128-21141, 2022.

David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint arXiv:1905.02249 , 2019.

Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems , 31, 2018.

Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance and label-dependent label noise. In International Conference on Machine Learning , pages 1789-1799. PMLR, 2020.

Madalina Ciortan, Romain Dupuis, and Thomas Peel. A framework using contrastive learning for classification with noisy labels. Data , 6(6):61, 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Erik Englesson and Hossein Azizpour. Consistency regularization can improve robustness to label noise. arXiv preprint arXiv:2110.01242 , 2021.

Aritra Ghosh and Andrew Lan. Contrastive learning improves model robustness under label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2703-2708, 2021.

Giorgio Giorgi and Tinne Hoff Kjeldsen. Traces and emergence of nonlinear programming . Springer Science & Business Media, 2013.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS , pages 8527-8537, 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14 , pages 630-645. Springer, 2016.

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations , 2017.

Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing out-of-distribution examples via augmenting content and style. arXiv preprint arXiv:2207.03162 , 2022.

Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. Advances in Neural Information Processing Systems , 34:26714-26725, 2021.

Zhuo Huang, Miaoxi Zhu, Xiaobo Xia, Li Shen, Jun Yu, Chen Gong, Bo Han, Bo Du, and Tongliang Liu. Robust generalization against photon-limited corruptions via worst-case sharpness minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1617516185, 2023.

Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 859-868. PMLR, 2019.

Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML , pages 2309-2318, 2018.

William Karush. Minima of functions of several variables with inequalities as side constraints. M. Sc. Dissertation. Dept. of Mathematics, Univ. of Chicago , 1939.

Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics , pages 2207-2217. PMLR, 2020.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Harold W Kuhn and Albert W Tucker. Nonlinear programming. In Traces and emergence of nonlinear programming , pages 247-258. Springer, 2014.

Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242 , 2016.

Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In ICLR , 2019.

Junnan Li, Caiming Xiong, and Steven CH Hoi. Mopro: Webly supervised learning with momentum prototypes. arXiv preprint arXiv:2009.07995 , 2020.

Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu. Selective-supervised contrastive learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 316-325, 2022.

Shikun Li, Xiaobo Xia, Hansong Zhang, Yibing Zhan, Shiming Ge, and Tongliang Liu. Estimating noise transition matrix with label correlations for noisy multi-label learning. Advances in Neural Information Processing Systems , 35:24184-24198, 2022.

Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. arXiv preprint arXiv:2102.02400 , 2021.

Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence , 38(3):447-461, 2016.

Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning , pages 6348-6359. PMLR, 2020.

Eran Malach and Shai Shalev-Shwartz. Decoupling" when to update" from" how to update". In NeurIPS , pages 960-970, 2017.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.

Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self-ensembling. In ICLR , 2019.

Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research , 70:1373-1411, 2021.

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR , pages 1944-1952, 2017.

Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv:1712.04621 , 2017.

Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning , pages 4334-4343. PMLR, 2018.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618-626, 2017.

Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07685 , 2020.

Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. How does early stopping help generalization against label noise? arXiv preprint arXiv:1911.08059 , 2019.

Cheng Tan, Jun Xia, Lirong Wu, and Stan Z Li. Co-learning: Learning from noisy labels with selfsupervision. In Proceedings of the 29th ACM International Conference on Multimedia , pages 1405-1413, 2021.

Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems , 34:16451-16467, 2021.

Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13726-13735, 2020.

Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. arXiv preprint arXiv:2110.12088 , 2021.

Xiaobo Xia, Bo Han, Nannan Wang, Jiankang Deng, Jiatong Li, Yinian Mao, and Tongliang Liu. Extended t t: Learning with mixed closed-set and open-set noisy labels. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(3):3047-3058, 2022.

Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust early-learning: Hindering the memorization of noisy labels. In ICLR , 2021.

Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. arXiv preprint arXiv:2106.00445 , 2021.

Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. NerurIPS , 2020.

Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurIPS , pages 6835-6846, 2019.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR , pages 2691-2699, 2015.

Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instancedependent bayes-label transition matrix using a deep neural network. In International Conference on Machine Learning , pages 25302-25312. PMLR, 2022.

Yazhou Yao, Zeren Sun, Chuanyi Zhang, Fumin Shen, Qi Wu, Jian Zhang, and Zhenmin Tang. Jo-src: A contrastive approach for combating noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5192-5201, 2021.

Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label-noise learning under a structural causal model. Advances in Neural Information Processing Systems , 34:4409-4420, 2021.

LIN Yong, Renjie Pi, Weizhong Zhang, Xiaobo Xia, Jiahui Gao, Xiao Zhou, Tongliang Liu, and Bo Han. A holistic view of label noise transition matrix in deep learning and beyond. In The Eleventh International Conference on Learning Representations , 2022.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR , 2017.

Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M Bronstein, and Or Litany. Contrast to divide: Self-supervised pre-training for learning with noisy labels. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1657-1667, 2022.

A A Theoretical View of Content and Style Isolation When Learning with Noisy Labels

Noisy data generative process. Recall the data generative process in our main paper, to learn latent factors by leveraging generative models. The generative model has to model the data generative process of noisy data. Firstly, we introduce the generative process of noisy data. We denote observed variables with gray color and latent variables with white color. Specifically, the content factor Z c is generated by the latent label Y . The different style domain U s give rise to the different style factor Z s . Subsequently, the image X is generated by the combined influence of the style factor Z s and the content factor Z c . Noisy labels ˜ Y are then generated based on the image X . In general cases, Z c and Z s can also have statistical or causal dependencies. We follow existing work that assumes the content factors are invariant across different styles [46].

Firstly, we introduce the concept of an uncontrolled style factor . This refers to a specific style factor, denoted Z s ′ , that remains invariant when a data augmentation A is applied. To formalize this concept, consider an invertible function f : Z×X . Let A denote a set of data augmentations, where each augmentation A is a subset

ranging from 1 to M . Additionally, let P ( A ) represent a probability distribution over the set of these augmentations A . Now, consider Z s ′ as a subset of style factors drawn from a larger set Z s . We partition the latent factors z of each instance x into three distinct parts: uncontrolled style factor Z s ′ , content factor Z c , and style factors influenced by data augmentation Z s/s ′ , such that f -1 ( x ) = [ z s ′ , z c , z s/s ′ ] = z . The term z s/s ′ denotes the set of style factors with style factors z s ′ excluded.

Definition 1 (Uncontrolled Style Factors) We say that a style factor Z s ′ as uncontrolled under the following conditions:

For any augmentation A ∼ P ( A ) , and for any instance x , the first n s ′ components of the inverse function f -1 ( x ) remain unchanged even when A ( x ) is applied, i.e., f -1 ( x ) 1: n s ′ = f -1 ( A ( x )) 1: n s ′ .

Here, f -1 ( x ) 1: n s ′ is defined as the underlying partition that contains only and all the information related to the style factor Z s ′ of the instance x .

Why do confident examples encourage content-style isolation? Here, we explain the reason that confident examples encourage content-style isolation. Suppose that there exist some uncontrolled style factors that cannot be adjusted or manipulated through data augmentation. This implies that for an image x , these style factors remain unaffected, regardless of the data augmentation techniques employed. For instance, as discussed in our paper, in the CIFAR-10 dataset [25], some pictures with the label 'horse' contain a person. In this context, the "person" acts as a style factor. Existing data augmentations cannot control this, as they cannot remove the person from images easily.

It's essential to understand that although data augmentation cannot control all style factors, it still offers the benefit of 'partial isolation'. If we consider a situation where the uncontrolled style factors are not effects of other style factors, then we can isolate other style factors affected by data augmentation from content factors after matching the data likelihood [34]. This result is a modified application of block-identifiability [46].

Specifically, let ˆ Z c,s ′ := ˆ Z 1: n c + n s ′ be a partition of learned representations of a generative model, where ˆ Z c,s ′ contain all and only information about Z c and Z s ′ . Suppose that assumptions of Theorem 4.2 in [34] are satisfied, ˆ Z c,s ′ is guaranteed to be learned, thereby allowing for the partial isolation of the remaining style factors, denoted as Z s/s ′ .

Despite ˆ Z c,s ′ is guaranteed to be learned, the information from the uncontrolled style factor Z s ′ is entangled with the content factor Z c in the learned ˆ Z c,s ′ . To isolate this information further, the employment of confident examples is necessary.

Specifically, we can generate ˆ Z c,s ′ using an invertible function according to the label ˆ Y of the confident example, and subsequently reconstruct image x . Following the data likelihood matching [34], the information related to the uncontrolled style factors becomes apparent due to the establishment of a

Algorithm 1 CS-Isolate-DM

Input: A noisy dataset ˜ S , a style ID list U s , a data augmentation set A , confident threshold τ , total epoch T max .

1: f ψ 1 , f ψ 2 , q ϕ 1 c ( Z c | X ) , q ϕ 2 c ( Z c | X ) ← WarmUP(S) ;

3: X , U ← Co -guessing( ˜ S , f ψ 1 , f ψ 2 , q ϕ 1 c , q ϕ 2 c , τ ) ;

2: For T = 1 , . . . , T max :

4: X ′ , U ′ ← MixUp( X , U ) ;

5: For k= 1 , 2 :

6: Sample ( x, ˜ y, u c ) ∼ ˜ S , u s ∼ U s ;

7: ˜ x ← A u s ( x ) ;

8: Feed ˜ x to encoders ˆ q ϕ k c and ˆ q ϕ k s to content factors z c and style factors z s , respectively;

9: Feed u c and u s to decoders ˆ p θ k c and ˆ p θ k s to get the prior ˆ p θ k c ( Z c | u c ) and ˆ p θ k s ( Z s | u s ) ;

10: Feed z c and z s to decoders ˆ p θ k to get the reconstruct image ˆ ˜ x ;

11: Feed z c to classifier head f ψ k to predicted label ˆ y ;

12: Calculate the loss using Eq. 2 and update networks;

Output: The inference networks and classifier heads q ϕ 1 c , q ϕ 2 c , f ψ 1 , f ψ 2 .

Algorithm 2 CS-Isolate-Co

Input: A noisy dataset ˜ S , a style ID list U s , a data augmentation set A , Total epoch T max .

1: For T = 1 , . . . , T max :

3: Obtain ¯ S 1 = arg min S ′ : | S ′ |≥ R ( T ) | ¯ S | ℓ ( f ψ 1 , q ϕ 1 c , S ′ ) ;

2: Fetch mini-batch ¯ S from ˜ S ;

4: Obtain ¯ S 2 = arg min S ′ : | S ′ |≥ R ( T ) | ¯ S | ℓ ( f ψ 2 , q ϕ 2 c , S ′ ) ;

5: For k= 1 , 2 :

6: Sample ( x, ˜ y, u c ) ∼ ¯ S k , u s ∼ U s ;

7: ˜ x ← a u s ( x ) ;

8: Feed ˜ x to encoders ˆ q ϕ k c and ˆ q ϕ k s to content factors z c and style factors z s , respectively;

9: Feed u c and u s to decoders ˆ p θ k c and ˆ p θ k s to get the prior ˆ p θ k c ( Z c | u c ) and ˆ p θ k s ( Z s | u s ) ;

10: Feed z c and z s to decoders ˆ p θ k to get the reconstruct image ˆ ˜ x ;

11: Feed z c and z s to classification model f ψ k to predicted labels ˆ y ;

12: Calculate the loss using Eq. 3 and update networks;

Output: The inference networks and classifier heads q ϕ 1 c , q ϕ 2 c , f ψ 1 , f ψ 2 .

one-to-one mapping between the label ˆ Y and ˆ Z c,s ′ . This consequently forces examples with identical labels to share the same ˆ Z c,s ′ , regardless of any alterations in the uncontrolled style factor Z ′ s . This approach, therefore, ensures that styles changes don't affect the derived content representation for the same label.

It is worth mentioning that to fully isolate uncontrolled style factors and content factors, it requires that there exists confident examples with all possible uncontrolled style factors. This can be hard to achieve when learning with noisy labels. Therefore, in general, the selected confident examples can only encourage isolation but can not fully isolate uncontrolled style factors and content factors.

B Apply CS-Isolate to Existing Methods for Learning with Noisy Labels

Applying CS-Isolate to DivideMix. DivideMix [28] uses two classifiers to select confident examples for each other. To utilize the unlabeled data, they combine the semi-supervised technique MixMatch [5]. Specifically, the classifiers, after warmed up, are used to calculate the loss of examples. They use a Gaussian Mixture Model (GMM) to divide the examples into confident and unlabeled examples. Finally, confident and unlabeled examples are used to train the models based on the MixMatch algorithm. Our method can be plugged into DivideMix easily. Specifically, we use a decoder q ϕ ( Z c , Z s | X ) to obtain content factor Z c and style factor Z s . A classifier head f ψ is used to

Algorithm 3 CS-Isolate-Me

Input: A noisy training dataset ˜ S , a noisy validation dataset ˜ S v , a style ID list U s , a data augmentation set A , iteration number T inner , T outer .

1: Initialize encoders ( ˆ q ϕ 0 c and ˆ q ϕ 0 s ), decoders ( ˆ p θ 0 c , ˆ p θ 0 s and ˆ p θ 0 ), a classification model f ψ 0 by using the noisy training data and early stopping;

2: For i = 1 , . . . , T outer :

3: For j = 1 , . . . , T inner :

4: Update the extracted confident examples S l by using ˆ q ϕ j -1 c and f ψ j -1 ;

5: Train networks by using the loss in Eq. 4 on confident examples S l ;

6: Obtain ˆ q ϕ j c and f ψ j through the highest noisy validation accuracy throughout the training procedure;

7: Break and output ˆ q ϕ j -1 c and f ψ j -1 if the highest validation accuracy is non-increasing in the loop;

8: Re-initialize encoders ( ˆ q ϕ 0 c and ˆ q ϕ 0 s ), decoders ( ˆ p θ 0 c , ˆ p θ 0 s and ˆ p θ 0 ), a classification model f ψ 0 ;

9: Train networks by using the loss in Eq. 4 on confident examples S l ;

10: Obtain ˆ q ϕ 0 c and f ψ 0 through the highest noisy validation accuracy throughout the training procedure;

11: Break and output ˆ q ϕ j -1 c and f ψ j -1 if the highest validation accuracy is non-increasing in the loop;

Output: The inference network and the classification model q ϕ c , f ψ .

predict labels, and only the content factors Z c are used as input. The prior distribution of the content factors Z c is conditional on the auxiliary variable U c , i.e. , P θ c ( Z c | U c ) , where U c is the content ID. Similarly, the prior distribution of the style factor Z s is conditional on the auxiliary variable U s , i.e. , P θ s ( Z s | U s ) , where U s is the style ID. A decoder P θ ( X | Z c , Z s ) is used to reconstruct input images. The combination of CS-Isolate and DivideMix is called CS-Isolate-DM. The loss function of CS-Isolate-DM is shown in Eq. 2. Algorithm 1 delineates the full algorithm.

Applying CS-Isolate to Co-Teaching. Co-Teaching [13] uses two classifiers to select confident examples for each other. Proposed CS-Isolate can be embedded in Co-Teaching easily. Similar to CS-Isolate-DM, we use a decoder q ϕ ( Z c , Z s | X ) to obtain content factors Z c and style factors Z s . A classifier head f ψ is used to predict labels, and only the content factors Z c are used as input. The prior distribution of the content factors Z c is conditional on the auxiliary variable U c , i.e. , P θ c ( Z c | U c ) , where U c is the content ID. Similarly, the prior distribution of the style factor Z s is conditional on the auxiliary variable U s , i.e. , P θ s ( Z s | U s ) , where U s is the style ID. A decoder P θ ( X | Z c , Z s ) is used to reconstruct input images. We call the combined method as CS-Isolate-Co. Let S l be the confident examples selected by another classifier head. For each network, the loss is defined as:

We use a PreAct ResNet-18 as the backbone. We used Adam with default parameters to optimize the encoder q ϕ c , classifier head f ψ , the encoder q ϕ s and the decoder p θ . The initial learning rate is 0.001, divided by 10 after 80 epochs.

where ℓ ce is the cross-entropy loss, ✶ is the indicator function. The algorithm of CS-Isolate-CoTeaching is summarized in Algorithm 2.

Applying CS-Isolate to Me-Momentum. Me-Momentum proposes to use one classifier to select confident examples. Then the parameters of the classifier will be reinitialized, and the classifier will be trained on the confident examples. The confident examples and the parameters of the classifier are updated alternately. We combine the Me-Momentum with our method and call it CS-Isolate-Me. Let

S l be the confident examples selected by the classifier of the last iteration . For each network, the loss is defined as:

L me = E ( x, ˜ y ) ∼ ˜ S [ ✶ ( x, ˜ y ) ∈S l ℓ ce ( f ψ ◦ q ϕ c ( x ) , ˜ y )] + λ ELBO L ELBO . (4) where ℓ ce is the cross-entropy loss, ✶ is the indicator function. The algorithm of CS-Isolate-MeMomentum is summarized in Algorithm 3.

C Experiments

In this section, we first introduce the experiment results of the proposed methods, including the classification of CS-Isolate-DM on class-dependent label noise, sample selection quality of CSIsolate-Co and CS-Isolate-Me on the real-world dataset CIFAR-10N, and classification performance of CS-Isolate-Co and CS-Isolate-Me on real-world the dataset CIFAR-10N. Second, we provide the details of data augmentation used in methods. Then, we conduct the ablation study for CS-Isolate-DM. Finally, we visualize the easy confident examples, content factors, and style factors.

C.1 Experiments on Class-dependent label noise

We report the classification performance of CS-Isolate-DM on class-dependent label noise, including symmetry-flipping noise and pair-flipping noise. The dataset used in the experiment is CIFAR-10. The experiment results are shown in Tab. 6. The experiment results demonstrate that CS-Isolate-DM can also perform well under class-dependent label noise.

C.2 Applying CS-Isolate to Existing Sample-Selection Methods

We combine our method with existing methods including DivideMix, Co-Teaching, and MeMomentum.

In the experiments for Co-Teaching and Me-Momentum, we use a PreAct ResNet-18 as the backbone. We use SGD with momentum 0.9 and weight decay 10 -4 to optimize the encoder q ϕ c and classifier head f ψ . We used Adam with default parameters to optimize the encoder q ϕ s and the decoder p θ . The network is trained for 100 epochs. The initial learning rate for SGD is 0.01, and for Adam is 0.001. The learning rate is divided by 10 after 40 epochs and 80 epochs.

C.3 Improves Sample Selection Quality with CS-Isolate

We conducted experiments on CIFAR-10N, a dataset reflecting real-world label noise. We illustrate the precision and recall ratios of our confident examples in Tab. 7 and Tab. 8. By employing our method, existing methods achieve improvements in terms of precision and recall. The experiment results indicate that our approach can efficiently improve the quality and number of confident examples.

C.4 Comparison of Classification Performance

The test accuracy of the baseline methods, as well as the combination of our proposed methods and the baselines, is shown in Tab. 9. The results demonstrate that improving the quality of the confident examples by using our method boosts the classification performance of the existing methods.

C.5 Data Augmentation Details

The detailed description of data augmentation techniques used in our method is shown in Tab 10. When generating a data augmentation A i ∈ A , the probabilities to apply shift scale rotation, random crop and horizontal flip, random brightness contrast, color jitter, and random to gray are 0.5, 1, 0.5, 0.5, 0.8, and 0.2, respectively, then the implementation details of the data augmentations will be recorded for the replaying. For instance, if a data augmentation A i flips the image horizontally, its behavior will be recorded. When the data augmentation A i is used during the training process, the images used to train the network will be applied a horizontal flip.

C.6 Ablation Study

In this subsection, we present the results of the ablation study on the hyper-parameters λ ELBO , λ ref and the dimensions of Z c and Z s . The experiments are conducted on the real-world dataset CIFAR-10N with the noise type 'worst'. The experiment results are shown in Fig. 7. The experiment results show that λ ELBO and λ ref are not sensitive in the range from 0.0005 to 0.005. In our experiments, we set the value of both λ ELBO and λ ref as 0.001, which is the middle value between 0.0005 and 0.005. For the ablation study on the dimension of Z c and Z s , the test accuracy increases gradually until the dimension is 32. After the dimension is larger than 64, the test accuracy decreases. We set the dimension as 32 in our experiments.











ELBO























(a)

λ

ELBO











ref













(b)

λ

ref



























(c) The dimensions of

Z

c

and

Z

s

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Cat

Horse

(a)

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Bird

Ship

(b)

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Bird

Horse

(c)

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Original image

Grad-CAM of DivideMix

Grad-CAM of CS-Isolate-DM

Ship

Cat

(d)

C.7 Visualization of Grad-CAM on Easy Confident Examples

We visualize the Grad-CAM for CS-Isolate-DM and DivideMix on easy confident examples. The easy confident examples are the examples identified successfully by both CS-Isolate-DM and DivideMix. The dataset is CIFAR-10N, and the noisy type is 'worst'. Fig. 8 shows the visualizations of easy confident examples. When the confident example is easy, the Grad-CAM visualizations for CSIsolate-DM and DivideMix do not differ significantly. The activation maps for both CS-Isolate-DM and DivideMix can focus on the objects. However, when the confident example is hard, only the activation maps for CS-Isolate-DM can focus on the objects, which has already been shown in the main paper.

C.8 Visualization of Grad-CAM for Content Factors and Style Factors

We visualize the Grad-CAM of Style and Content Factors for CS-Isolate-DM. The visualization results are shown in Fig. 9. Grad-CAM of content factors mainly concentrates on the objects, while Grad-CAM of style factors mainly concentrates on other pixels in the images. The visualization results demonstrate that CS-Isolate-DM can isolate content and style factors successfully.

Original image

Grad-CAM of style factors

Grad-CAM of content factors

Original image

Grad-CAM of style factors

Grad-CAM of content factors

Cat

Horse

(a)

Original image

Grad-CAM of style factors

Grad-CAM of content factors

Original image

Grad-CAM of style factors

Grad-CAM of content factors

Bird

Ship

(b)

Original image

Grad-CAM of style factors

Grad-CAM of content factors

Original image

Grad-CAM of style factors

Grad-CAM of content factors

Bird

Horse

(c)

Original image

Grad-CAM of style factors

Grad-CAM of content factors

Original image

Grad-CAM of style factors

Grad-CAM of content factors

Ship

Cat

(d)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary: This paper presents a solution to infer content factors for classification when learning with the noisy labels especially with hard samples that are close to decision boundary. In order to achieve the goal, they tend to separate the style information from the semantic information with some techniques including data augmentations. Some experimental evaluations and comparisons are shown to prove the effectiveness of the paper.\n\nSTRENGTHS:\nThe idea of disentangled representation is widely applied for many deep generative models. In this paper, the solution to separate style factors from semantic information also falls into this category which helps to reduce the effect from noisy labels.\n\nWEAKNESSES:\n(1)\tFirst of all, it is not clear for the label noise level, what does random 1, 2, 3 indicate in the experimental evaluations for all the Tables 1-4. Because in the regime of robust learning, usually the percentage of noisy labels is an important parameter in the experimental design. Without clearly stating the noise level, it is very difficult for readers to compare if the approach can work well in the high noise ratio level. Nor does this comparison show fairness as random 1 can be a very low level label noise for method A and a high level label noise for method B.\n(2)\tAs the authors mentioned, if the style and semantic information are important to their idea and the design of the method, it is imperative to visualize the style and semantic information extracted from their images and show it is processed by their method. Lacking of visualization make the readers confusing and difficult to understand. Namely, the method is not well justified and stays on the safe ground without analyzing the semantic information (with visualization). Specifically, it is important to show say for MNIST datasets, semantic information implies the numbers and style information indicates the writing style in this case.\n(3)\tMoreover, there are quite a lot of related work on applying the embedding (dimensionality reduction) to disentangle the latent space for improving the denoising performance (e.g. generative models). After reading the paper, I feel the method claimed is merely a small tweak of the existing methods along this line where the author called it “style and semantic spaces”, while it can actually be more general for all subspaces.\n(4)\tThe papers with similar ideas can be found:\na.\tLearning disentangled representations via product manifold projection, ICML 2021.\nb.\tLearning disentangled representations in the imaging domain, Medical Image Analysis, 2022\nJust to name a few, especially in the reference [b], the solution to introduce the disentangled representations is very similar to the paper. Moreover, it is well known that using disentangled representations can boost performance including denoising with noisy labels. \nNamely, from the high level, the result shown in this paper is only a subset or a specific case of the previous work, which is not novel enough. While the specific application is somewhat interesting, I donot see it is as a Neurips paper.\n\nQUESTIONS:\n(1) As mentioned before, the overall novelty for this work is small given the fact that disentangled representations can boost the performance including denoising.\n\n(2) The experimental evaluations are not sufficient and questionable due to lack of details. Typically, the level of noisy label percentage is not reported, making the comparison not convincing. Instead, the author simply used some vague statements like \"random 1\", \"random 2\" \"worst\" which leaves many unanswered questions. For instance, how large is the \"random 1\" noise level ? Is the worst case representing a more than 50% corrupted noisy label space ? What does \"Aggregate\" mean ? (Is it implies voting from multiple people for labeling).\nI tried very hard to find the answers but fail to see the detailed meanings of them.\n\nLIMITATIONS:\nThe paper is proposed with limited novelty, questionable experimental design and unclear writing.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides insights into the extraction of hard and confident examples from noisy datasets, which is an important problem when learning with noisy labels. The authors discover that by removing irrelevant style information from the data, hard examples can become more discernible. Building on this finding, they design an innovative method that leverages existing identifiable results of latent variables. With no side information available with learning with noisy labels, this method constructs the necessary information by using data augmentations and confident examples for the identification of latent variables. They support their solution with theoretical analysis, demonstrating its empirical performance on different datasets.\n\nSTRENGTHS:\nThis paper studies an important problem in machine learning. Specifically, the problem that how to extract hard confident examples when dealing with datasets containing label noise. The ``easy’’ confident examples have been well studied and can be easily identified by using the different techniques such as small-loss trick and memorization effect. However, how to identify the hard and confident examples have only been discussed by a few work. How to effectively identify the hard and confident examples still remains unclear. This problem is important as the hard confident examples usually plays an important role for shaping the decision boundary and improve the classification performance, which mentioned in the previous paper. \n\nThe authors present new insight about hard confident examples. They claim that some examples become difficult to discriminate due to the entanglement of non-useful style information in the learned representation. This finding is interesting and easy to understand, has been surprisingly overlooked in existing work. This finding may pave a new way on discover hard examples in noisy datasets.\n\nThe provided solution is innovative for me. To enjoy the identifiable of latent factors by using variational inference, the availability of side information is necessary. Nonetheless, when learning in the presence of label noise, this side information is not readily accessible. To navigate this challenge, the authors ingeniously devise supervised information for style and content factors by utilizing different data augmentations and confident examples. They also provide intuitive reasoning and a theoretical analysis to demonstrate how identifiability can be achieved, reinforcing the originality and rigor of their approach.\n\nWEAKNESSES:\nThe reproducibility of the proposed method could be enhanced. Specifically, the authors mentioned employing various data augmentation methods such as shift scale rotate, random crop and horizontal flip, random brightness contrast, color jitter, and random grayscale. However, it remains unclear how concretely the different augmentations are sampled for each dataset, which could affect replication efforts.\n\nThe paper's experimental setup could benefit from further expansion. For synthetic noise datasets, only instance-dependent label noise is used. The performance under widely used noise types like symmetric noise and pairflip noise remains untested. Additionally, an analysis of the method's sensitivity in response to different dimensions of latent variables Z could provide a more comprehensive understanding of the model's robustness.\n\nThe last two paragraphs of introduction are redundant and could be effectively condensed into one concise paragraph. Additionally, it might be beneficial to move the identifiable analysis from the appendix to the main paper after introducing the method for a more seamless reader experience.\n\nThe paragraph ``Disentangled latent factors via auxiliary variable’’ feels somewhat abstract. As these techniques are leveraged to encourage isolation, further explanation could help reduce the reading difficulty for readers. Providing more in-depth discussion in appendix might make this critical point more accessible.\n\nQUESTIONS:\n1) If the pseudo-code for CS-Isolate-Me is missing or omitted, it would indeed be helpful to understand its differences compared to CS-Isolate-Co. Both methods appear to use a very similar loss function. \n2) How the evidence Lower Bound (ELBO) are derived, it is neither the main paper nor the appendix. \n3) What will be the results of the proposed method on CIFAR10 and CIFAR100 with symmetric noise and pairflip noise. \n4) Does the proposed method sensitive to the dimensionality of $z$? \n5) Could the authors provide more concrete details about the application of various augmentations on each dataset?\n\nLIMITATIONS:\nThe authors have mentioned a limitation in appendix that the assumption to achieve the fully isolation of style and content variables is strong. Specifically, achieving the fully isolation of uncontrolled style factors and content factors demands the existence of confident examples of all possible uncontrolled style factors. Consequently, the authors' methods typically encourage, but do not guarantee, the full isolation of these uncontrolled style and content factors.\nNevertheless, I believe it's crucial to include a section dedicated to limitations in the main paper and move this point to that section. More broadly, a method's limitations can encompass a range of aspects, such as computational requirements, sensitivity to hyperparameters, and generalizability across different datasets or noise types, among other factors. I recommend the authors carefully review the proposed method to identify if any of these limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on improving the ability to extract hard confident examples (close to the decision boundary), a critical aspect of noisy label learning.  The authors find that a key reason for some hard examples being close to the decision boundary is due to the entanglement of style factors with content factors. In response to this finding, they strive to maintain the consistency of content factors across all examples within the same underlying clean class, even as their style information undergoes changes. This approach is grounded in the belief that the content factors should remain invariant, thereby allowing for a more accurate extraction of hard confident examples. Their proposed method, CS-Isolate, embodies this concept and leverages various data augmentation to alter the styles while regularizing content factors based on some confident examples. CS-Isolate demonstrates robust performance when tested on synthetic noise datasets.\n\nSTRENGTHS:\n* The paper exhibits a commendable level of organization, facilitating a smooth and intuitive reading experience. Its logical structure aids in the comprehension of complex concepts and methodologies.\n* The inclusion of visual aids for the toy example, significantly enhances the reader's understanding. These illustrations provide tangible insights into the underlying intuition of the proposed method, making the abstract concepts more accessible.\n* The proposed method achieves new SOTA performance on various benchmark datasets.\n\nWEAKNESSES:\n* The authors base their proposed method on the premise that hard examples being close to the decision boundary is due to the entanglement of unhelpful style factors with useful content factors. However, this key finding appears to lack empirical support from experimental results, which weakens the foundation of their argument.\n\n* I still can’t wrap my head around how the proposed method solves this challenge of identifying or designing data augmentations that can manipulate all style factors in an image.\n\n* The proposed method seems to focus on learning style factors based on artificial styles introduced through data augmentations, rather than modelling the natural style of an image. If the authors' assertion that the proximity of hard examples to the decision boundary is due to the entanglement of unhelpful style factors with useful content factors holds true, it would be logical to base the style factor on the natural style of images. This discrepancy needs to be addressed.\n\n* It's unclear how this method compares to other existing methods in terms of computational efficiency. If CS-Isolate requires significantly more computational resources, it may not be practical for use in real-world applications.\n\n* The authors did not discuss the limitations of the proposed method. \n\n* Several sentences in the paper require further clarification, which will be discussed in the Questions section.\n\nQUESTIONS:\n* \"If extracted examples have high quality, the performance of a classifier on them is enhanced.” This sentence is a bit confusing. Is the model trained on these examples?\n* \"However, During the training process, it is possible to identify and select some confident examples.\" It seems that \"However\" should be removed.\n* \"Some variation has also been proposed for example, some methods opt to reweight examples, thereby decreasing the contribution of mislabeled samples to the overall loss\" This sentence could benefit from improved punctuation to enhance readability. Breaking it into smaller sentences or using punctuation to separate ideas could make the authors' point clearer and easier to understand.\n\nLIMITATIONS:\nThe authors did not discuss the limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The problem of training deep neural networks with label noise is considered. Unlike traditional methods proposing to extract confident examples far from the classification boundary, this work proposes an approach named CS-Isolate to extract hard examples close to the classification boundary by disentangling content factor $Z_c$ and style factor $Z_s$. Different data augmentation techniques were employed to change the style factor $Z_c$ of images while the content factor remains unchanged $Z_s$. Two inference models were proposed to model the conditional distribution $Z_c |X$ and $Z_s |X$, respectively. Evaluation on synthetic noisy version of FashionMNIST, CIFAR-10, CIFAR-100 demonstrated the effectiveness of the proposed approach.

STRENGTHS:
1.	To the best of my knowledge, this is the first work introducing fashion factor for extracting confident examples under label noise.
2.	The effectiveness of CS-Isolate was validated on the synthetic instance-dependent label noise.

WEAKNESSES:
1.	All evaluations were made on toy-level dataset CIFAR-10/100 and FashionMNIST and synthetic label noise. The effectiveness of CS-Isolate on large-scale dataset and real-world label noise (e.g., Clothing1M) is worth investigating.

QUESTIONS:
1.	In subsection section 2.2 (Sample selection methods for learning with noisy labels), it was worth discussing the idea to collect hard examples by active learning (e.g. [6]).

LIMITATIONS:
I didn't find a section dedicated to the discussion of limitations. One limitation in my mind is the lack of evaluation on real-world and large-scale label noise.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a novel method to extract the hard confidence examples under the noisy labels. By splitting the images into content factors and style factors, the hard confidence examples can be more easily extracted by solely considering the content factors. By training with the hard confidence examples, the models can achieve better generalization ability.\n\nSTRENGTHS:\n1. The paper presents a deep insight of the hard confidence example and why hard confidence examples are hard to extract.\n\n2. A novel method is proposed in the paper to extract the hard confidence examples by splitting the examples into content factors and style factors. The two independent factors are produced by generative model with variational inference method. The experiments show that the proposed method achieves better performance than previous methods in the extraction of the hard confidence examples.\n\n3. The proposed method shows better classification results than baselines.\n\nWEAKNESSES:\n1. Ablation study on hyper-parameters is missing.\n\n2. The experiment datasets shown in the paper are relatively small dataset. Authors should provide experimental results on larger datasets such as Clothing1M. Since both baseline (DivideMix and Me-Momentum) mentioned in the paper report the results on this dataset. It is better to compare the results of the proposed methods against the baselines on this dataset.\n\n3. Some errors in the writing. For instance. on line 276, the equation of conditional probability of style factors is incorrect. In eq.7, the minus sign should be plus sign.\n\nQUESTIONS:\n1. Can authors provide the visualization of the hard confidence examples as well as the content/style factors? It is good for readers to understand the effectiveness of the proposed method.\n\n2. The paper just gives a rough description of the data augmentation techniques. Can author specify how many types of data augmentation used in the paper? (The value of $M$)? What is the difference between them? How the number of data augmentation techniques (value $M$) affects the disentanglement of content factors and style factors?\n\n3. In eq. 3 authors mentioned about the refinement of content ID. How the authors deal with the conflicting of labels and content ID? e.g., sample whose label is not $p$ but assigned with content ID $p$.\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}