SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yexiong Lin<sup>1</sup> Yu Yao<sup>2,3</sup> Xiaolong Shi<sup>1</sup>
Mingming Gong $^{4}$  Xu Shen $^{5}$  Dong Xu $^{6}$  Tongliang Liu $^{1*}$
<sup>1</sup>The University of Sydney; <sup>2</sup>Mohamed bin Zayed University of Artificial Intelligence;
$^{3}$ Carnegie Mellon University;  $^{4}$ The University of Melbourne;
<sup>5</sup>Alibaba DAMO Academy; <sup>6</sup>The University of Hong Kong.
Label noise widely exists in large-scale image datasets. To mitigate the side effects of label noise, state-of-the-art methods focus on selecting confident examples by leveraging semi-supervised learning. Existing research shows that the ability to extract hard confident examples, which are close to the decision boundary, significantly influences the generalization ability of the learned classifier. In this paper, we find that a key reason for some hard examples being close to the decision boundary is due to the entanglement of style factors with content factors. The hard examples become more discriminative when we focus solely on content factors, such as semantic information, while ignoring style factors. Nonetheless, given only noisy data, content factors are not directly observed and have to be inferred. To tackle the problem of inferring content factors for classification when learning with noisy labels, our objective is to ensure that the content factors of all examples in the same underlying clean class remain unchanged as their style information changes. To achieve this, we utilize different data augmentation techniques to alter the styles while regularizing content factors based on some confident examples. By training existing methods with our inferred content factors, CS-Isolate proves their effectiveness in learning hard examples on benchmark datasets. The implementation is available at https://github.com/tmllab/2023_NeurIPS_CS-isolate.
Large-scale machine learning datasets frequently contain noisy labels, as seen in datasets like ImageNet [9] and Clothing1M [55]. Training deep neural networks with such noisy data would result in poor generalization ability, as these networks can memorize incorrect labels [13, 3].
To mitigate the side effects of label noise, different methods have been proposed [33, 13, 37, 61, 30, 31, 49]. Major existing state-of-the-art methods are based on confident examples selection [61, 2]. Intuitively, those methods first exploit the memorization effect, enabling deep neural networks to learn simple patterns shared by the majority of training examples [60, 1, 19]. Since clean labels typically constitute the majority in each noisy class [7, 39], deep neural networks initially fit the training data with correct labels and then progressively fit examples with incorrect labels [13]. To prevent the model from fitting incorrect labels, early stopping is usually employed [44, 3, 50, 4]. Then the small-loss trick is used to extract the confident examples with high certainty [21, 35, 28, 38]. If extracted examples have high quality, the performance of a classifier can be enhanced.
*Corresponding author: Tongliang Liu (tongliang.liu@sydney.edu.au).
To improve the performance of a classifier, it is crucial to ensure that the confident examples have a distribution similar to that of clean data [36, 32]. Specifically, it is necessary to extract not only confident examples that are far from the decision boundary but those that are close to it. The former examples are easy to be identified and extracted. However, the latter examples, which are close to the decision boundary, often become entangled with mislabeled examples, making them challenging to be identified or extracted. In this paper, we discover that on image datasets, the entanglement of unhelpful style factors with useful content factors is a key reason that leads to the hard examples becoming hard to be classified, and thus these examples are close to the decision boundary.
We have discussed that the entanglement of style factor  $Z_{s}$  and content factor  $Z_{c}$  in the representation space can create hard examples. However, if we can isolate content from styles, then by focusing solely on the content factor  $Z_{c}$ , these hard examples become more discriminative. For example, let's consider projecting the examples onto the  $Z_{c}$ -axis by ignoring style factor  $Z_{s}$  in Fig. 1a. The data points on the  $Z_{c}$ -axis in Fig. 1b illustrate that even after the nonlinear transformation, the examples remain distinguishable. This motivates us to tackle the problem of inferring content factors for extracting hard examples when learning with noisy labels.
To isolate style and content information in learned representations, it is crucial to learn invariant representations of all examples as their styles change. However, when data contains label noise, we can not access clean classes, making it challenging to achieve this isolation. To address this issue, we introduce a method built upon identifiable Variational Autoencoder (iVAE) [23]. The core intuition behind our approach is to construct different styles for each example through various data augmentation techniques. We then encourage both the original and the augmented examples to possess different style factors but maintain
In Fig. 1, we provide an intuitive understanding of the relationship between hard examples and the entanglement of style factor  $Z_{s}$  and content factor  $Z_{c}$ . Let us first assume the underlying content factor  $Z_{c}$  and style factor  $Z_{s}$  are given. In Fig. 1a, by visualizing  $Z_{s}$  and  $Z_{c}$ , we can observe a separation between the examples belonging to underlying clean classes 0 and 1. In real-world scenarios, however,  $Z_{c}$  and  $Z_{s}$  are not directly available. Instead, existing methods [13, 28] could learn representations  $Z_{1}$  and  $Z_{2}$  from noisy data to extract confident examples. These learned representations,  $Z_{1}$  and  $Z_{2}$ , are nonlinear transformations of style factor  $Z_{s}$  and content factor  $Z_{c}$ . They do not ensure the disentanglement of  $Z_{s}$  and  $Z_{c}$  [24, 20, 59]. As the example illustrated in Fig. 1b, both  $Z_{1}$  and  $Z_{2}$  contain the information of the style factor  $Z_{s}$  and the content factor  $Z_{c}$ . In the illustrated representation space, examples within the region encircled by red dashed lines are close together. The confident examples in this region are entangled with mislabeled examples, making them challenging to be extracted by existing methods without isolating (disentangling) style and content information in learned representations.
Figure 1: Illustrating the entanglement of content and style factors. The circles on the left side of the dotted line represent examples with the underlying clean label  $Y = 0$ , while the ones on the right side represent examples with clean label  $Y = 1$ . The blue filling corresponds to the noisy label  $\tilde{Y} = 0$ , and the yellow filling corresponds to the noisy label  $\tilde{Y} = 1$ . Black outlines indicate that the labels of examples are correct, whereas red outlines indicate that the labels of examples are incorrect. In (a), we visualize the underlying style factor  $Z_{s}$  and the underlying content factor  $Z_{c}$ . (b) shows the impact of a nonlinear transformation on  $Z_{s}$  and  $Z_{c}$ , leading to their entanglement in the new space defined by representations  $Z_{1}$  and  $Z_{2}$ .
Figure 2: Illustrate the comparison of the performance of confident examples selection of our method CS-Isolate with DivideMix and Me-Momentum on CIFAR10N Worst [48]. The highest points on AUCs are indicated by arrows. High precision indicates that most of the confident examples are correctly labeled, while recall indicates the fraction of correctly labeled examples out of all examples that are correctly labeled.
identical content factors, which is a self-supervised learning manner. This initial step could achieve a preliminary level of isolation between style and content factors. As the training process progresses, we can identify and select certain confident examples based on the content factors. Leveraging these confident examples, we further enhance the isolation process by encouraging all examples sharing the same label to share identical content factors. By isolating content and style factors, our method, CS-Isolate, effectively helps existing sample selection methods to extract hard examples typically overlooked by existing sample selection methods. In Fig 2. we evaluate the performance of confident examples selection on CIFAR10N with the noise type "worst" [48] by using two metrics: precision and recall. When comparing these two metrics, CS-Isolate consistently outperforms DivideMix [28] and Me-Momentum [2]. This result demonstrates that our method can not only select confident examples more accurately, as shown by the higher precision, but also capture a larger portion of the correctly labeled examples from the entire dataset, as evidenced by the higher recall. By improving the quality of confident examples with the isolation of the content and style factors, our method effectively helps improve the test accuracy of existing methods. We have also theoretically analyzed the identifiability of content and style factors in Appendix A.
Problem setup. Let's denote  $\tilde{D}$  as the distribution of a noisy example  $(X, \tilde{Y})$  from the set  $\mathcal{X} \times \{1, 2, \ldots, C\}$ , where  $X$  denotes the variable of instances,  $\tilde{Y}$  represents the variable of noisy labels,  $\mathcal{X}$  is the feature space,  $\{1, 2, \ldots, C\}$  is the label space, and  $C$  is the total number of classes. In the learning scenario with noisy labels, clean labels are not observed. Given a noisy training sample  $\tilde{S} = \{x_i, \tilde{y}_i\}_{i=1}^N$ , independently drawn from  $\tilde{D}$ , the objective is to leverage this sample  $\tilde{S}$  to learn a classifier robust against label noise.
Sample selection methods for learning with noisy labels. In learning with noisy labels, major current state-of-the-art (SOTA) methodologies predominantly involve sample selection strategies. These strategies seek to divide the dataset into confident and unconfident examples. The basis of these strategies is the exploitation of the memorization effect inherent in deep neural networks. The memorization effect enables the networks to initially grasp and learn the simple patterns, and then learn the complex patterns gradually [60, 1]. Given that clean labels usually form the majority within each noisy class [7, 39], these networks would initially fit the examples with accurate labels, and subsequently fit the examples with incorrect labels over time [13].
Preventing the learning model from fitting incorrect labels is crucial to ensure sample selection quality. To achieve this, strategies such as early stopping are often employed [44, 3]. Additionally, the small-loss trick is utilized to identify and extract confident examples with high certainty [28, 38, 51, 18]. Some variation has also been proposed, e.g., some methods opt to reweight examples, thereby decreasing the contribution of mislabeled samples to the overall loss [41, 13]. To guarantee the statistical consistency of algorithms, Jiacheng et al. [7] introduces active learning to acquire the labels of randomly chosen examples from unconfident examples to mitigate the bias introduced by
sample selection.
Moreover, the set of confident examples often undergoes dynamic changes during the training stage. This is achieved by leveraging semi-supervised learning methods to relabel training instances and reselect confident and unconfident examples using the small loss trick. Various techniques for this purpose have been proposed and empirically demonstrated superior performance, including consistency regularization [27] adopted by [10], MixMatch [5] used by [28], co-regularization by [47], and contrastive learning by [30, 29, 45, 8, 11, 57, 61]. Self-training and co-training have been used by [2] and [35, 13, 21], respectively.
Representation learning by generative model. Consider a data generation in Fig. 3a,  $X$  is the observed data, and  $Z$  is the unknown underlying representation to generate  $X$ . Variational autoencoder framework [24]
can be used to learn the latent representation. In this process, a standard normal distribution is utilized as a prior for the latent variables, and a variational posterior  $q(Z|X)$  is employed to approximate the unknown underlying posterior  $p(Z|X)$ . Disentangled representation is very important, which
Figure 3: The different data generative processes with or without auxiliary variables.
can allow a rich class of properties to be imposed on the learned representation, such as sparsity and clustering. To disentangle the representation, the variational autoencoder framework has been further expanded by modifying the original loss function, resulting in various algorithms.  $\beta$ -VAE [16] suggests an adaptation framework that adjusts the weight of the KL term to balance the independence of disentangled factors and reconstruction performance.  $\beta$ -TCVAE [6] further analyzes the KL term of  $\beta$ -VAE [16] and only adjusts the total correlation term to achieve disentanglement. HOOD [17] uses the clean labels and domain labels to disentangle content and style factors, but the clean labels and domain labels are unknown in the setting of learning with noisy labels. These methods are toward the goal of disentanglement but do not have theoretical guarantees of identifiability of their inferred latent representations.
Disentangled latent factors via auxiliary variable. Recent studies [20, 23] show that theoretical guarantees of identifiability can be achieved if an auxiliary variable related to the representation can be obtained. Khemakhem et al. [23] provide identifiability of iVAE with additional inputs by employing the theory of nonlinear Independent Component Analysis (nonlinear ICA) [20]. Intuitively, given a factorized prior distribution over the latent variables that are conditional on an additional auxiliary variable  $U$ , i.e., the class label and the time index in a time series, the latent factors are identifiable up to a certain degree. The data generation processing is changed to Fig. 3b. To disentangle the representation, the method assumes that each factor of the representation is independent.
3 Content and Style Isolation When Learning with Noisy Labels
In this section, we present CS-Isolate, a method designed to help select hard confident examples by content and style isolation.
Noisy data generative process. To learn latent factors by leveraging generative models, the generative model has to model the noisy data generative process. Firstly, we introduce the noisy data generative process. We denote observed variables with gray circles and latent variables with white circles. Specifically, the content factor  $Z_{c}$  is generated by the latent label  $Y$ . The different style domains  $U_{s}$  give rise to the different style factor  $Z_{s}$ . Then, the image  $X$  is generated by the combined influence of the style factor  $Z_{s}$  and the content factor  $Z_{c}$ . Noisy labels  $\tilde{Y}$  are then generated based on the image  $X$ . In general cases,  $Z_{c}$  and  $Z_{s}$  can also have statistical or causal dependencies. We follow existing work that assumes the content factors are unchanged across different styles [46].
Figure 4: The noisy data generative process.
Challenge in isolating content from styles without labels. Isolating content from style without labels is a challenging task [46]. We first introduce the assumptions made by existing methods, as well as difficulties that may be encountered in practice.
Existing methods [26, 23, 46] for isolating content from styles without labels assume that the data augmentation for controlling each style factor can be designed, which means that we can intervene (control) all style factors. For instance, the data augmentations for controlling rotation angle and scaling of images can be designed by using affine transformation [40]. When we apply a data augmentation that rotates an image, the style factor for the rotation angle changes in the augmented image compared to the original. Similarly, when we scale an image, the style factor for scaling becomes different in the augmented image. If we have the data augmentation to control each style factor, then by training a generative model with the augmented images, the model can then identify style factors by comparing the changes in styles between the original and augmented images.
However, the major challenge lies in the fact that we generally cannot design sufficient data augmentations to control all style factors in an image. For instance, in the CIFAR-10 dataset [25], some pictures with the label "horse" contain a person. In this context, the "person" acts as a style factor. Existing data augmentations cannot control this, as they cannot remove the person from images easily. This simple example illustrates that it is generally impossible to control all style factors through data augmentations. Then the assumption required by existing methods usually is hard to satisfy.
The violation of the assumption leads to the learned representations for content containing style information.
Challenge in isolating content from style with noisy labels. We generally cannot design data augmentations that can control all style factors in an image. In this case, clean labels are essential to help isolate content from styles. Intuitively, by comparing the change of images with different clean labels that share the same styles, one can infer content factors used for classification. This achieves isolation of content and style. However, when dealing with noisy data, relying on labels becomes problematic as they contain label errors. Images with the same content factors can have different noisy labels, making it fail to infer content factors used for classification. This situation further complicates the task of isolating content from styles, highlighting the challenges posed by noisy labels.
3.2 CS-Isolate for Extracting Hard Example
In this paper, we find that a key reason for some hard examples being close to the decision boundary is the entanglement of style factors with content factors. Intuitively, style factors render the learned representation of certain examples less discriminative, thereby making them close to the decision boundary. If content and style information can be isolated, many hard examples would be easily distinguished by ignoring the style information. This is because we have left only content information in the learned representation, making these representations far from the decision boundary.
Inspired by these findings, we propose CS-Isolate, which aims to isolate content from styles for extracting confident examples. To achieve isolation, we utilize self-supervised learning. Specifically, we ensure that original and augmented images maintain the same content factors despite having different style factors due to data augmentations. This achieves a preliminary level of isolating content from styles. As previously mentioned, it is generally impossible to manipulate all style factors through data augmentation. Some style factors remain uncontrolled and stay contained in the learned representations for content. To further encourage the isolation, labels of confident examples are used. Specifically, as training progresses, we identify and select confident examples whose labels are likely to be accurate. By harnessing the labels from these confident examples, we encourage the examples with the same label to have the consistent content factors, irrespective of stylistic variations. This further isolates content from uncontrolled styles. By leveraging different data augmentations and confident examples, our method can effectively separate the learned latent representations into two parts which exclusively contain either style or content information. Subsequently, classifier heads can be trained purely on the representations containing content information. This approach not only improves the identification of hard confident examples but also enhances classification performance.
Isolating content from styles using auxiliary variables. We follow existing work using the variational autoencoder with auxiliary variables [23] to isolate content from styles. Let  $U_{s}$  and  $U_{c}$  denote the auxiliary variables that control the style factor  $Z_{s}$  and content factor  $Z_{c}$ , respectively. Specifically, by reconstructing an image with the supervision of auxiliary variables  $U_{s}$  and  $U_{c}$ , we encourage the images with the same content information but different style information to have the same content factor  $Z_{c}$  but the different style factor  $Z_{s}$ . This allows us to isolate content from styles within an image. Then only employing  $Z_{c}$  for selecting hard confident examples.
However, given only noisy data, we face the challenge of the unavailability of auxiliary variables  $U_{c}$  and  $U_{s}$ . In response, we devise surrogates for these auxiliary variables. To find a surrogate for the style auxiliary variable  $U_{s}$ , we employ different data augmentation techniques, each of which generates a unique style domain with distinct style factors. We then assign the style domain ID as the style auxiliary variable  $U_{s}$ . To find a surrogate for the content auxiliary variable  $U_{c}$ , we adopt the philosophy of self-supervised learning to assign a unique content ID as the content auxiliary variable  $U_{c}$  to each image. The images that have the same content ID are encouraged to have consistent content factors. Moreover, during the learning process, some confident examples can be extracted. To effectively leverage these examples in learning content factors, we reassign the content ID of confident examples belonging to the same class to be the same. This allows our model to learn consistent content factors for the images in the same class.
Constructing style auxiliary variables via data augmentations. For the learning of distinct content factors and style factors, it is essential to construct images from domains with different styles
by using data augmentations. Each image carries its corresponding domain ID as the style auxiliary variable  $U_{s}$ . Let  $\mathcal{A} = \{A_0, A_1, A_2, \ldots, A_M\}$  represents the set of data augmentations, and  $M$  is the number of data augmentations. Note that, for convenience, we use  $A_0$  to denote a data augmentation such that applying the data augmentation  $A_0$  to the image  $x$  does not change it at all, i.e.,  $x = A_0(x)$ .
When applying these diverse data augmentations to an  $x$ , different augmented images are obtained with different style factors. For the augmented image  $x^{A_i}$  obtained by using the  $i$ -th data augmentations that come from the  $i$ -th style domain, we can assign its style ID to be  $i$ . Specifically, for each image  $x$ , each data augmentation  $A_i \in \mathcal{A}$  is applied, resulting in a set containing pairs of the augmented image and the corresponding style ID, i.e.,
$$
\left\{\left(x ^ {A _ {i}} := A _ {i} (x), U _ {s} ^ {\left(x ^ {A _ {i}}\right)} := i\right) \mid \forall i \in \{0, \dots , M \} \right\},
$$
where  $U_{s}^{x^{A_{i}}}$  is defined to be the style ID (which serves the style auxiliary variable) of the image  $x$  by applying  $i$ -th data augmentation. In this manner, for each image  $x$ , we can generate a set of augmented images  $(\{x^{A_{i}}\}_{i=0}^{M})$ , including the original images and the ones with different style factors controlled by different style IDs.