SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Constructing content auxiliary variables in a self-supervised manner. To guide the generative model in learning the consistent content factor across images sharing the same content information, we aim to assign the same content ID (which serves as the content auxiliary variable) to images with the same content information. However, clean labels cannot be obtained in learning with noisy labels, meaning we cannot know which images have the same content information. To construct content auxiliary variables without clean labels, we adopt self-supervised learning. Specifically, we consider that different data augmentations applied to an image typically do not alter its content information. Hence, we can assign a unique content ID to each original image and its augmented versions. This assignment can be mathematically expressed as follows:
$$
\mathcal {S} _ {a l l} = \{(x _ {j} ^ {A _ {i}} := A _ {i} (x), U _ {s} ^ {(x _ {j} ^ {A _ {i}})} := i, U _ {c} ^ {(x _ {j} ^ {A _ {i}})} := C + j) | \forall i \in \{0, \dots , M \}, \forall j \in \{1, \dots , T \} \},
$$
where  $T$  is the total number of distinct original images in the training data,  $U_{c}^{(x_{j}^{A_{i}})}$  represents the content ID of the  $j$ -th image in the training data after applying the data augmentation  $A_{i}$ , and  $C$  is the number of classes.
Moreover, this initial assignment of content IDs can be refined by using confident examples, denoted as  $S_{l}$ . As the training process progresses, we can extract these confident examples by leveraging the small-loss trick [13, 3, 28]. The content IDs are then further refined based on the labels of the examples within the confident examples. The refinement process can be described as:
$$
U _ {c} ^ {\left(x _ {j} ^ {A _ {i}}\right)} := y _ {j} ^ {c} \mid \forall (x _ {j}, y _ {j} ^ {c}) \in \mathcal {S} _ {l}, \forall i \in \{0, \dots , M \},
$$
where  $y_{j}^{c}$  is the label of the confident example  $(x_{j},y_{j}^{c})$ . After this refinement process, confident examples in the same class will have the same content ID. Consequently, these refined content IDs could enable the model to learn consistent content factors for the images in the same class.
Encouraging style and content isolation for extracting hard example. After obtaining the auxiliary variable, we isolate content from styles by leveraging iVAE [20]. Specifically, the prior distribution of the content factor  $Z_{c}$  is conditional on the auxiliary variable  $U_{c}$ , i.e.  $P_{\theta_c}(Z_c|U_c)$ . Similarly, the prior distribution of the style factor  $Z_{s}$  is conditional on the auxiliary variable  $U_{s}$ , i.e.  $P_{\theta_s}(Z_s|U_s)$ . The  $\theta_c$  and  $\theta_{s}$  are the learnable parameters of the distribution. The objective is to maximize the data likelihood which is as follows.
$$
\mathbb {E} _ {q _ {D}} \left[ p _ {\theta} \left(X \mid U _ {c}, U _ {s}\right) \right] = \mathbb {E} _ {q _ {D}} \left[ \int_ {z _ {c}, z _ {s}} p _ {\theta} \left(X \mid z _ {c}, z _ {s}\right) p _ {\theta_ {c}} \left(z _ {c} \mid U _ {c}\right) p _ {\theta_ {s}} \left(z _ {s} \mid U _ {s}\right) \mathrm {d} z _ {s} \mathrm {d} z _ {c} \right], \tag {1}
$$
where we use  $q_{D}$  to denote the empirical distribution of the training sample  $\mathcal{S}_{all}$ . We use the variational inference method to approximate the underlying posterior distribution  $p_{\theta}(Z_c,Z_s|X,U_c,U_s)$  [23, 24]. Specifically, two inference models (encoders)  $q_{\phi_c}(Z_c|X)$  and  $q_{\phi_s}(Z_s|X)$  are introduced to infer latent variables  $Z_{c}$  and  $Z_{s}$  respectively and model the distribution  $q(Z_{c},Z_{s}|X)$  that is used to approximate the distribution  $p_{\theta}(Z_c,Z_s|X,U_c,U_s)$ . Therefore, the distribution  $q(Z_{c},Z_{s}|X)$  can be decomposed as follows:
$$
q _ {\phi} \left(Z _ {c}, Z _ {s} | X\right) = q _ {\phi_ {c}} \left(Z _ {c} | X\right) q _ {\phi_ {s}} \left(Z _ {s} | X\right).
$$
We learn the parameters  $\{\theta_c, \theta_s, \phi_c, \phi_c\}$  by maximizing the evidence lower-bound ELBO for each example  $(x, u_c, u_s)$ . The corresponding loss is:
$$
\begin{array}{l} \min  _ {\phi_ {c}, \phi_ {s}, \theta} \mathcal {L} _ {E L B O} := \min  _ {\phi_ {c}, \phi_ {s}, \theta} \mathbb {E} _ {q _ {D}} \left[ - \mathbb {E} _ {\left(z _ {c}, z _ {s}\right) \sim q _ {\phi_ {c}, \phi_ {s}} \left(Z _ {c}, Z _ {s} \mid x\right)} \left[ \log p _ {\theta} \left(x \mid z _ {c}, z _ {s}\right) \right. \right. \\ \left. - K L \left(q _ {\phi_ {c}} \left(z _ {c} | x\right) \| p _ {\theta_ {c}} \left(z _ {c} \mid u _ {c}\right)\right) - K L \left(q _ {\phi_ {s}} \left(z _ {s} | x\right) \| p _ {\theta_ {s}} \left(z _ {s} \mid u _ {s}\right)\right) \right], \\ \end{array}
$$
where  $KL$  is the Kullback-Leibler divergence. Intuitively, when the style changes, the content remains constant. The model has to infer the unchanged content factors for image reconstruction. Thus, the content factors can be isolated from style factors. The content factors can be identified up to the block-identifiable defined in [46].
Utilizing content information to extract hard examples. After minimizing the ELBO loss  $\mathcal{L}_{ELBO}$ , we have learned representations  $\hat{Z}_c$  and  $\hat{Z}_s$  which exclusively contain either style or content information. Then we employ existing sample-selection-based methods solely focus on  $\hat{Z}_c$  for confident example selection, a classification head  $f_{\psi}:\mathcal{Z}_c\to \Delta_{C - 1}$  is introduced. The classification head maps the space of content factors to a  $C - 1$  probability simplex, where  $C$  represents the number of classes.
In practice, we propose an end-to-end approach to learn the classification head  $f_{\psi}$  and learn to infer content factor for extracting hard confident examples. This approach simultaneously minimizes the evidence lower bound loss  $\mathcal{L}_{ELBO}$  and the loss of existing sample-selection-based methods by leveraging the Lagrangian method [22, 12]. The content IDs are refined dynamically during the learning process. The overall loss function  $\mathcal{L}_{all}$  is thus given as:
$$
\min  _ {\phi_ {c}, \phi_ {s}, \theta , \psi} \mathcal {L} _ {a l l} = \min  _ {\phi_ {c}, \phi_ {s}, \theta , \psi} [ \mathcal {L} _ {s s l} + \lambda_ {E L B O} \mathcal {L} _ {E L B O} + \lambda_ {r e f} \mathcal {L} _ {r e f} ],
$$
where  $\mathcal{L}_{ssl}$  denotes the loss of the sample selection method, which optimizes the parameters of the classification head  $\psi$  and the content encoder  $\hat{\phi}_c$ . The loss  $\mathcal{L}_{ref}$  is the cross-entropy loss on confident examples. The hyper-parameter  $\lambda_{ELBO}$  and  $\lambda_{ref}$  are used to control strength of  $\mathcal{L}_{ELBO}$  and  $\mathcal{L}_{ref}$ , respectively. Here, we illustrate a concrete example of employing the inference model  $q_{\hat{\phi}_c}$  in conjunction with DivideMix [28] in an end-to-end implementation. For a detailed walkthrough, please refer to the pseudo-code provided in Appendix B and the original paper [28].
DivideMix [28] applies a Gaussian mixture model to enhance MixMatch [43] for confident example selection and classifiers training. In DivideMix, during each epoch of training, the data is divided into a set of confident examples  $S_{l}$  and a set of unconfident examples  $S_{u}$ . The confident examples contain the sharpened soft labels mixed by their labels in datasets and predicted labels given by the classification head  $f_{\psi}$ . The unconfident examples contain the sharpened soft labels predicted by the classification head  $f_{\psi}$ . Then, the semi-supervised learning approach, MixMatch [5] is employed by transforming confident  $(S_{l})$  and unconfident  $(S_{u})$  samples into augmented confident  $(S_{l}')$  and unconfident  $(S_{u}')$  samples by a linearly mixing.
The overall loss function, composed of a confident sample loss, an unconfident sample loss, and a regularization term, ELBO loss and a classification loss, i.e.,
$$
\mathcal {L} _ {a l l} = \underbrace {\mathcal {L} _ {\mathcal {S} _ {l}} + \lambda_ {u} \mathcal {L} _ {\mathcal {S} _ {u}} + \lambda_ {r} \mathcal {L} _ {\mathrm {r e g}}} _ {\text {D i v i d e M i x l o s s}} + \lambda_ {E L B O} \mathcal {L} _ {E L B O} + \lambda_ {r e f} \mathcal {L} _ {r e f}.
$$
Intuitively,  $\mathcal{L}_{\mathcal{S}_l}$  is a cross-entropy loss for the labeled examples;  $\mathcal{L}_{\mathcal{S}_u}$  is the mean squared error for the unlabeled samples;  $\mathcal{L}_{\mathrm{reg}}$  is a regularization term to prevent the model from predicting all samples to belong to a single class. These three terms are defined as follows specifically.
$$
\mathcal {L} _ {\mathcal {S} _ {l}} = - \frac {1}{| \mathcal {S} _ {l} ^ {\prime} |} \sum_ {x, y ^ {s} \in \mathcal {S} _ {l} ^ {\prime}} \sum_ {i} y _ {i} ^ {s} \log (f _ {\psi} \circ q _ {\phi_ {c}} (x)),
$$
$$
\mathcal {L} _ {\mathcal {S} _ {u}} = \frac {1}{| \mathcal {S} _ {u} ^ {\prime} |} \sum_ {x, y ^ {s} \in \mathcal {S} _ {u} ^ {\prime}} \| y ^ {s} - f _ {\psi} \circ q _ {\phi_ {c}} (x) \| _ {2} ^ {2},
$$
$$
\mathcal {L} _ {\mathrm {r e g}} = \sum_ {i} \frac {1}{C} \log (1 / \frac {C}{| \mathcal {S} _ {l} ^ {\prime} | + | \mathcal {S} _ {u} ^ {\prime} |} \sum_ {x \in \mathcal {S} _ {l} ^ {\prime} + \mathcal {S} _ {u} ^ {\prime}} f _ {\psi} \circ q _ {\phi_ {c}} ^ {i} (x)),
$$
where  $y^{s}$  is the sharpened soft label and  $q_{\phi_c}^i$  denotes the  $i$ -th coordinate of its output on input  $x$ . In Appendix B, we illustrate our method with other sample selection methods.
Table 1: Precision of confident examples on CIFAR-10N.
Worst	Aggregate	Random 1	Random 2	Random 3
Me-Momentum	0.881 ± 0.011	0.972 ± 0.002	0.962 ± 0.005	0.960 ± 0.003	0.948 ± 0.005
DivideMix	0.951 ± 0.003	0.989 ± 0.000	0.983 ± 0.000	0.983 ± 0.000	0.983 ± 0.000
CS-Isolate-DM	0.956 ± 0.001	0.989 ± 0.000	0.984 ± 0.000	0.985 ± 0.000	0.983 ± 0.000
Table 2: Recall of confident examples on CIFAR-10N.
Worst	Aggregate	Random 1	Random 2	Random 3
Me-Momentum	0.920 ± 0.026	0.969 ± 0.019	0.946 ± 0.021	0.958 ± 0.022	0.962 ± 0.018
DivideMix	0.966 ± 0.000	0.963 ± 0.000	0.975 ± 0.000	0.976 ± 0.000	0.977 ± 0.000
CS-Isolate-DM	0.980 ± 0.000	0.975 ± 0.002	0.980 ± 0.001	0.982 ± 0.001	0.982 ± 0.001
Table 3: Means and standard deviations (percentage) of classification accuracy on FashionMNIST, CIFAR-10 and CIFAR-100.
Fashion-MNIST	CIFAR-10	CIFAR-100
IDN-0.2	IDN-0.4	IDN-0.2	IDN-0.4	IDN-0.4
CE	88.54 ± 0.32	84.22 ± 0.35	75.81 ± 0.26	62.45 ± 0.86	21.45 ± 0.70
Co-Teaching	91.21 ± 0.31	89.10 ± 0.29	80.96 ± 0.31	73.41 ± 0.78	28.04 ± 1.43
Forward	90.05 ± 0.43	86.27 ± 0.48	74.64 ± 0.26	60.21 ± 0.75	26.75 ± 0.93
T-Revision	91.58 ± 0.31	89.46 ± 0.42	76.15 ± 0.37	65.09 ± 0.37	27.23 ± 1.13
BLTM	91.20 ± 0.27	82.42 ± 1.51	77.50 ± 1.30	63.20 ± 4.52	35.67 ± 1.97
CausalNL	90.84 ± 0.31	90.01 ± 0.45	80.91 ± 1.14	79.08 ± 0.50	34.02 ± 0.95
Me-Momentum	92.85 ± 0.64	90.06 ± 0.51	90.86 ± 0.21	86.66 ± 0.91	58.38 ± 1.28
DivideMix	94.85 ± 0.15	92.28 ± 0.13	94.93 ± 0.15	94.16 ± 0.35	70.50 ± 0.25
CS-Isolate-DM	95.16 ± 0.07	94.40 ± 0.09	95.90 ± 0.10	95.54 ± 0.06	73.11 ± 0.36
Table 4: Means and standard deviations (percentage) of classification accuracy on CIFAR-10N.
Worst	Aggregate	Random 1	Random 2	Random 3
CE	77.69 ± 1.55	87.77 ± 0.38	85.02 ± 0.65	86.14 ± 0.24	86.12 ± 0.16
Co-Teaching	82.04 ± 0.06	91.11 ± 0.10	89.61 ± 0.18	88.98 ± 0.11	89.49 ± 0.06
Forward	79.79 ± 0.46	88.24 ± 0.22	86.88 ± 0.50	86.14 ± 0.24	87.04 ± 0.35
T-Revision	80.48 ± 1.20	88.52 ± 0.17	88.33 ± 0.32	87.71 ± 1.02	87.79 ± 0.67
BLTM	68.21 ± 1.67	79.41 ± 1.00	78.09 ± 1.03	76.99 ± 1.23	76.26 ± 0.71
CausalNL	82.41 ± 0.24	90.43 ± 0.14	89.03 ± 0.02	89.06 ± 0.05	89.21 ± 0.13
Me-Momentum	84.21 ± 0.70	91.34 ± 0.16	89.51 ± 0.42	90.14 ± 0.28	89.62 ± 0.31
DivideMix	92.48 ± 0.16	94.11 ± 0.21	94.77 ± 0.15	94.79 ± 0.14	94.79 ± 0.15
CS-Isolate-DM	94.28 ± 0.07	95.34 ± 0.10	95.36 ± 0.14	95.34 ± 0.12	95.48 ± 0.14
In this section, we introduce the setting of our experiments and compare our experimental results with existing methods. Most of our experiments are left out in Appendix C due to the limited space.
Dataset and noise type. We evaluate our methods on three synthetic noise datasets FashionMNIST [54], CIFAR-10 [25], and CIFAR-100 [25], and two real-world label-noise datasets, CIFAR-10N [48] and Clothing1M [55]. FashionMNIST includes 70,000 images of size  $24 \times 24$ , categorized into 10 classes with 60,000 for training and 10,000 for testing. Both CIFAR-10 and CIFAR-100 contain 50,000 training images and 10,000 test images; CIFAR-10 comprises 10 classes, while CIFAR-100 includes 100 classes. The image size in both CIFAR datasets is  $32 \times 32 \times 3$ . To generate noisy labels for these clean datasets, we employ the instance-dependent noisy label generation methods proposed in [52]. CIFAR-10N, a noisy version of CIFAR-10, includes five types of label noise: "worst", "aggregate", "random 1", "random 2", and "random 3", all annotated by humans. The noise rates are  $40.21\%$ ,  $9.03\%$ ,  $17.23\%$ ,  $18.12\%$ , and  $17.64\%$ , respectively. Clothing1M contains 1 million images with real-world noisy labels for training and 10,000 images with clean labels for testing.
Table 5: Means and standard deviations (percentage) of classification accuracy on Clothing1M.
CE	Decoupling [35]	MentorNet [21]	Co-teaching [13]	Forward [39]
68.88	54.53	56.79	60.15	69.91
T-Revision [53]	BLTM-V [56]	CausalNL [58]	DivideMix [28]	CS-Isolate-DM
70.97	73.39	72.24	74.76	74.92
Network structure and optimization. We implemented our method using PyTorch 1.12.1 and performed experiments on the NVIDIA Tesla V100. For fair comparisons, we chose PreAct ResNet-18 [15] as the backbone of the encoder  $q_{\phi_c}$  for Fashion-MNIST, CIFAR-10, CIFAR-100, and CIFAR-10N, a ResNet-50 [14] with ImageNet pre-trained weight for Clothing1M. We employed SGD with a momentum of 0.9 and a weight decay of 0.0005 to optimize the encoder  $q_{\phi_c}$  and the classifier head  $f_{\psi}$ . We used Adam with default parameters to optimize the encoder  $q_{\phi_s}$  and the decoder  $p_{\theta}$ . For experiments on synthesis datasets, the initial learning rate for SGD was set at 0.02 and for Adam at 0.001. The batch size is 64. For experiments on Fashion-MNIST, our network was trained for 100 epochs. Both learning rates were reduced by a factor of 10 after 80 epochs. For experiments on CIFAR-10, CIFAR-100, and CIFAR-10N, our network was trained for 300 epochs. Both learning rates were reduced by a factor of 10 after 150 epochs. For experiments on Clothing1M, our network was trained for 80 epochs with a batch size of 32. The initial learning rate for SGD was set at 0.002 and for Adam at 0.001. Both learning rates were reduced by a factor of 10 after 40 epochs. The encoder  $q_{\phi_c}$  and the classifier head  $f_{\psi}$  were warmed up on noisy data for 10 epochs for CIFAR-10 and CIFAR-10N, warmed up for 30 epochs for CIFAR-100, 5 epochs for Fashion-MNIST, and 1 epoch for Clothing1M. The dimensions of  $Z_c$  and  $Z_s$  were set at 32. The data augmentation techniques include shift scale rotation, random crop and horizontal flip, random brightness contrast, color jitter, and random to gray. For all experiments, we set  $M = 1000$ ,  $\lambda_r = 1$  and  $\lambda_{ELBO} = 1e - 3$ .  $\lambda_{ref}$  was increased gradually then kept constant at  $1e - 3$  after 140 epochs. For synthetic datasets, we set  $\lambda_u$  as 0 and 15 for the noise rates of 0.2 and 0.4 in FashionMNIST and CIFAR-10 datasets and as 100 in CIFAR-100. For CIFAR-10N, we set  $\lambda_u$  as 50 in the "worst" noise type and 0 in the rest of the noise types. For Clothing1M, we set  $\lambda_u$  as 0.
Baselines and measurements. We compare our method against several state-of-the-art techniques: (1) CE: the standard cross-entropy loss, with the model trained directly on noisy data; (2) Co-Teaching [13]: involves training two networks and uses the small-loss trick to select confident examples for each other; (3) Forward [39]: applies a transition matrix to correct the loss function; (4) T-Revision [53]: revises the transition matrix to enhance performance; (5) Me-momentum [2]: uses confident examples to refine the classifier, which is then used to update confident examples alternately; (6) BLTM [56]: uses a Bayes optimal label to estimate the transition matrix; (7) CausalNL [58]: employs a causal model to assist the learning of the classifier; (8) DivideMix [28]: employs two networks to select confident examples and uses semi-supervised techniques to utilize unlabeled examples. We report the precision, recall and AUC for the confident examples selection. We also report the test accuracy on the test dataset. The value is the average over the last 10 epochs.
4.2 Confident Examples Quality
To evaluate the quality of the confident examples, we examine the precision and recall of these examples. The precision quantifies the proportion of accurately labeled examples within the set of confident examples. On the other hand, recall measures the proportion of accurately labeled examples in the confident examples relative to the total number of correct labels. The experimental results are displayed in Tab. 1 and Tab. 2. The results indicate that the precision performance of our proposed method matches the state-of-the-art levels. However, the recall of our method surpasses that of existing methods. This demonstrates that our method can identify more confident examples without compromising precision. This suggests that our approach effectively maintains high precision and improves recall, thus selecting more confident examples with correct labels.
4.3 Classification Accuracy
We conducted extensive experiments on three synthetic noise datasets (Fashion-MNIST, CIFAR-10 and CIFAR-100) and two real-world datasets (CIFAR-10N and Clothing1M). For the synthetic
(c)
(d)
Figure 5: Grad-CAM visualizations of hard confident examples. CS-Isoalte-DM successfully identifies these confident examples, but DivideMix does not. The activation map of CS-Isolate-DM predominantly highlights semantic objects, whereas DivideMix emphasizes non-object pixels.
datasets, we employed instance-dependent noisy label generation methods, as proposed by [52]. We experimented with noise rates of 0.2 and 0.4, denoted by IDN-0.2 and IDN-0.4 respectively. The experiment results are presented in Tab. 3, Tab. 4 and Tab. 5. Our proposed method outperforms existing methods in terms of test accuracy on both synthetic and real-world datasets containing label noise. Notably, as the noise rate increases, the performance gap between our method, CS-Isolate-DM, and the existing methods becomes more pronounced. This highlights the robustness and effectiveness of our approach in scenarios with higher levels of label noise.
4.4 Hard Confident Examples Visualization
The proposed method is expected to select confident examples based on content factors rather than style factors. To analyze this, we use Grad-CAM [42] to visualize the regions used to select confident examples. The visualization results are shown in Fig. 5. The experiment is conducted on the real-world dataset CIFAR-10N, and the noise type is "worst". The experiment results demonstrate that our method can correctly focus on the object in images. Specifically, when there exist uncontrolled style factors, e.g., the person near the horse, the activation maps for CS-Isolate-DM can successfully focus on the right object used for classifying the horse instead of uncontrolled style factors. In contrast, the baseline method, DivideMix, focuses on the style factors that are not related to the class "horse" and fails to select these confident examples.
This paper is motivated by the fact that only focusing on content factors such as semantic information makes examples more discriminative. We, therefore, proposed a novel CS-Isolate approach to infer and isolate the content information for classification. This is achieved by leveraging variational inference and constructing auxiliary variables via data augmentation techniques to modify style factors while regularizing content factors using confident examples. By training existing sample-selection-based methods with our inferred content factors, CS-Isolate improves their effectiveness in learning hard examples and classification accuracy on different image datasets.
Tongliang Liu is partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031. Dong Xu is partially supported by the Hong Kong Jockey Club Charities Trust under Grant 2022-0174, the Hong Kong Research Grant Council under General Research Fund (17203023), the Startup Funding and the Seed Funding for Basic Research for New Staff from The University of Hong Kong, and the funding from UBTECH Robotics.
[58] Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label-noise learning under a structural causal model. Advances in Neural Information Processing Systems, 34:4409-4420, 2021.
[59] LIN Yong, Renjie Pi, Weizhong Zhang, Xiaobo Xia, Jiahui Gao, Xiao Zhou, Tongliang Liu, and Bo Han. A holistic view of label noise transition matrix in deep learning and beyond. In The Eleventh International Conference on Learning Representations, 2022.
[60] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
[61] Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M Bronstein, and Or Litany. Contrast to divide: Self-supervised pre-training for learning with noisy labels. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1657-1667, 2022.
A A Theoretical View of Content and Style Isolation When Learning with Noisy Labels
Noisy data generative process. Recall the data generative process in our main paper, to learn latent factors by leveraging generative models. The generative model has to model the data generative process of noisy data. Firstly, we introduce the generative process of noisy data. We denote observed variables with gray color and latent variables with white color. Specifically, the content factor  $Z_{c}$  is generated by the latent label  $Y$ . The different style domain  $U_{s}$  give rise to the different style factor  $Z_{s}$ . Subsequently, the image  $X$  is generated by the combined influence of the style factor  $Z_{s}$  and the content factor  $Z_{c}$ . Noisy labels  $\tilde{Y}$  are then generated based on the image  $X$ . In general cases,  $Z_{c}$  and  $Z_{s}$  can also have statistical or causal dependencies. We follow existing work that assumes the content factors are invariant across different styles [46].
Firstly, we introduce the concept of an uncontrolled style factor. This refers to a specific style factor, denoted  $Z_{s'}$ , that remains invariant when a data augmentation  $A$  is applied. To formalize this concept, consider an invertible function  $f: \mathcal{Z} \times \mathcal{X}$ . Let  $\mathcal{A}$  denote a set of data augmentations, where each augmentation  $A$  is a subset
ranging from 1 to  $M$ . Additionally, let  $P(A)$  represent a probability distribution over the set of these augmentations  $\mathcal{A}$ . Now, consider  $Z_{s'}$  as a subset of style factors drawn from a larger set  $Z_s$ . We partition the latent factors  $z$  of each instance  $x$  into three distinct parts: uncontrolled style factor  $Z_{s'}$ , content factor  $Z_c$ , and style factors influenced by data augmentation  $Z_{s/s'}$ , such that  $f^{-1}(x) = [z_{s'}, z_c, z_{s/s'}] = z$ . The term  $z_{s/s'}$  denotes the set of style factors with style factors  $z_{s'}$  excluded.
Figure 6: The noise data generative process.
Definition 1 (Uncontrolled Style Factors) We say that a style factor  $Z_{s'}$  as uncontrolled under the following conditions:
For any augmentation  $A \sim P(A)$ , and for any instance  $x$ , the first  $n_{s'}$  components of the inverse function  $f^{-1}(x)$  remain unchanged even when  $A(x)$  is applied, i.e.,  $f^{-1}(x)_{1:n_{s'}} = f^{-1}(A(x))_{1:n_{s'}}$ .
Here,  $f^{-1}(x)_{1:n_{s'}}$  is defined as the underlying partition that contains only and all the information related to the style factor  $Z_{s'}$  of the instance  $x$ .