Learning the Latent Causal Structure for Modeling Label Noise
Authors: Yexiong Lin
Yu Yao
Tongliang Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Label noise
transition matrices
Primary area: interpretability_and_explainability

Abstract: In label-noise learning, the noise transition matrix reveals how an instance transitions from its clean label to its noisy label. Accurately estimating an instance's noise transition matrix is crucial for estimating its clean label. However, when only a noisy dataset is available, noise transition matrices can be estimated only for some "special" instances. To leverage these estimated transition matrices to help estimate the transition matrices of other instances, it is essential to explore relations between the matrices of these "special" instances and those of others. Existing studies typically build the relation by explicitly defining the similarity between the estimated noise transition matrices of "special" instances and those of other instances. However, these similarity-based assumptions are hard to validate and may not align with real-world data. If these assumptions fail, both noise transition matrices and clean labels cannot be accurately estimated. In this paper, we found that by learning the latent causal structure governing the generating process of noisy data, we can estimate noise transition matrices without the need for similarity-based assumptions. Unlike previous generative label-noise learning methods, we consider causal relations between latent causal variables and model them with a learnable graphical model. Utilizing only noisy data, our method can effectively learn the latent causal structure. Experimental results on various noisy datasets demonstrate that our method achieves state-of-the-art performance in estimating noise transition matrices, which leads to improved classification accuracy. The code is available at: https://github.com/tmllab/2024_NeurIPS_CSGN.

Learning the Latent Causal Structure for Modeling Label Noise
Yexiong Lin Yu Yao
Tongliang Liut
Sydney Al Centre, The University of Sydney
Abstract
In label-noise learning, the noise transition matrix reveals how an instance transi- tions from its clean label to its noisy label. Accurately estimating an instance's s noise transition matrix is crucial for estimating its clean label. However, when only noisy dataset is available, noise transition matrices can be estimated only for some "special" instances. To leverage these estimated transition matrices to help estimate the transition matrices of other instances, it is essential to explore relations between the matrices of these "special" instances and those of others. Existing studics typically build the relation by explicitly defining the similarity between the estimated noise transition matrices of "special" instances and those of other instances. However, these similarity-basee may not align with real-world data. If these assumptions fail, both noise transition matrices and clean labels cannot be accurately estimated. In this paper, found that by learning the latent causal structure governing the generating process of noisy data, estimate noise transition matrices without the need for similarity-based assumptions. Unlike previous generative label-noise learning methods, we consider causal relations between latent causal variables and model them with learnable graphical model. Utilizing only noisy data, our method can effectively learn the latent causal structure. Experimental results on various noisy datasets demonstrate that our method achieves state-of of-the-art performance in estimating noise transition matrices, which leads improved classification accuracy. The code is available sttps://gil rrtbbub.coo cww com/mmllab/2024 Mtw.aaaaa2222e2erPP....
assumptions are hard to validate and
Introduction
Supervised learning relies on annotated large-scale datasets, which can be both time-consuming and costly to create. Although several existing annotation methods offer cost-effective alternatives, such as online queries (5). crowdsourcing [52). and image engines (31), the datasets obtained by these
methods are imperfect. The labels of these datasets usually contain errors. These noisy labels would
be harmful to deep neural networks because the network can memorize noisy labels easily 59 13 and lead to the degradation classification accuracy.
Modeling label noise using noise transition matrices plays an important role in label-noise learning 36, 40 30 [5]. Let Y, X, and Y denote the variables for the clean label, the instance, and the noisy label, respectively. The noise transition matrix for an instance æ can be represented æ), which reveals the probability that, given an instance, its latent clean label is transited to the observed noisy label. Given these transition matrices, a classifier trained on noisy data by leveraging these matrices can be consistent with the optimal classifier trained on clean data 36 40 50
by ,Y,
1 CCoreeponneence Tongliang (tongliang liu @iw.glldy..dddd......
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, noise transition matrices are generally unknown and need to be estimated. When only a noisy dataset is available, only the noise transition matrices of some "special" instances can be estimated. For example, if instances belong to clean class with a probability of one (known as anchor points), their noise transition matrices can be estimated [50]. To utilize these learned transitions to help in estimating others, it is crucial to establish relations between the transition matrices of these "special" instances and those of the other instances. Existing studies usually build this relation by proposing similarity assumptions for noise transition matrices across different instances. For example: I). the noise transition matrices of instances within the same class are identical [36 40 [0]; 2). the noise transition matrices of instances lying in the same manifold are identical [8] 3). the noise
transition matrices are the same for the closest three instances within the same clean class [61 37].
However, these predefined assumptions may not hold in real-world cases. Moreover, verifying these assumptions with only noisy data is challenging. If these assumptions are violated, it can lead to estimation errors in both transition matrices and clean labels.
In this paper, to establish relations between the transition matrices of these "special" instances and those of the others without relying on predefined similarities, we propose exploring the latent causal structure that generates noisy data. By understanding this structure, the relations among noise tran- sition matrices across instances would be implicitly captured. Subsequently, the noise transition matrices of other instances can be estimated by making use of the estimated noise transition matrices of the "special" instances. To offer some intuition, in Fig. D we illustrate two noisy examples in the benchmark noisy dataset CIFAR- Suppose there is a latent variable for the "presence of fur", and an annotator is biased by the "presence of fur". If the first "cat" image containing "fur" is mislabeled as "dog" by this annotator, then a second "cat" image also containing "fur" will likely be mislabeled as "dog" with high probability by the same annotator. This leads us to understand that the noise transition matrices for the two images are similar when their causal variables have the same value. In essence, the relation between noise transition matrices across these two instances is estab- lished through the causal structure. Such insights have driven us to develop an algorithm designed to learn the latent causal structure. thereby enabling the accurate estimation of noise transition matrices for new instances based on existing ones.
Nakey labe: Dog Clean label. Cat
Notay Dog
Clamlabd: Cat
(a)
(b)
Figure I: The pictures
contain the same noisy labels.
There are a few studies (55 10 by modeling the data-generating process with generative model and lever- aging causality. However, they may fail to capture the relation among noise transition matrices as the data-generating process proposed by these methods is restrained. Specifically, as shown in Fig. 2a current methods assume that the instance X is direct cause of the noisy label Ý in the data-generating process. However, this assumption does not hold for many machine learning datasets consisting of perceptual data such as images, videos, and natural language. In these datasets, the latent causal variables such as shape, color, and semantic concepts are the direct causes of the noisy label Y rather than the perceptual data X itself. As previously mentioned, the mislabeling of an image of a "cat" as a "dog" could be due to the latent causal variable "presence of fur". Two images generated under similar latent variable values should have similar transition matrices. Therefore, it important to con- sider and learn these latent causal variables in the data- Sata-generating process. Motivated by this insight, we propose to incorporate latent causal variables in the data-generating process.
that tackle the label-noise learning problem
S
(a) CausalNL and In- stanceGM
(b) Our model
Figure 2: Noisy data- generating processes assumed by different methods. S is the vari- able unrelated to the clean label Y.
As shown in Fig. 2b the observed instance X is generated by two sets of latent variables: S which are not influenced by the clean label Y and Z which are caused by the clean label Y. Furthermore, we allow for causal dependencies among the variables in Z. We assume that the cause of the noisy label Y also depends on the clean label Y. For example, the latent causal variable "presence of fur" influences the noisy label Y and depends on the clean label (c.g.. cat or dog), which satisfies this assumption. To learn the data- generaeenr process, we employ learnable graphical model that models all latent causal variables and their causal relations. Once the model is trained, it can infer the values of latent causal variables for
2

---

different images, thereby implicitly capturing the relations among noise transition matrices across instances. Theoretically, we explain the sufficient conditions for identifying the latent causal variables in Appendix B Empirically, our method achieves state-of-the-art performance in estimating the noise transition matrices and predicting clean labels.
2 Related Work
Modeling the Noise Transition Matrices Modeling label noise by noise transition matrices plays an important role for label-noise learning. Some methods 36
30 assume the noise transition matrix is instance-indee indeeedeenn,,,ndnnt, i... p(Y i|Y j. = = p(Y =|\ =)). The instance-independee transition matrix is identifiable under some mild conditions. For example, the inneeeeeende transition matrix can be estimated using anchor points, where the clean class posterior probability of a class is one [36] 40, 550 [56.. Li et al. [30] assume that the clean class-posterior distribution is sufficiently scattered and learn the instance- indeeeedende transition matrix by minimizing the volume of the transition matrix.
40
50
56
To learn instance-dependent transition matrices, existing methods often rely on predefined statistical dependencies to explain how noisy labels are generated. Xia el al. [49] propose to linearly decompose an instance into features (or "parts"), assuming that the noisy label depends Based on this assumption, they estimate instance-dependent transition matrices by combining the transition matrices associated with these features. Cheng er al. [8 assume that noisy labels are dependent on the data manifold, which leads to a manifold -rggullriere method for learning transition matrices. Unlike these methods, our proposed method automatically discovers dependencies by identifying latent variables that causally influence noisy labels. Furthermore, our method can be viewed as a generalization of Xia et al. work [49]. Specifically, rather than assuming that noisy labels depend on linearly decomposed features, our method employs nonlinear model and leverages causal representation learning [38] to infer the features responsible for generating the noisy labels. CausalNL (55) and InstanceGM [10] also leverage causality to learn transition matrices. However, the lata-generating processes assumed by these methods can be constrained, as they do not account for the latent causal relationships and variables considered by our method. Other methods for learning with noisy labels are discussed in Appendix
all these features.
Causal Representation Learning Causal representation learning 45 aims to identify latent causal variables from observations. The generating process for observations from the latent variables typically non-linear. Previous research has established that identifying latent variables in an unsupervised manner is generally infeasible [20] 39]. Recently, the focus has shifted to weakly- supervised or self-supervised methods that incorporate additional supervised information. For example, methods utilizing temporal information 44 18 [19]. auxiliary variables that cause the latent variables 21 23), and multiview information have been employed to learn causal representations.
3 Learning the Latent Causal Structure for Generation of Noisy Data
Data-Generating Process Let X and Y denote the observed variables of instances and noisy labels, respectively. The observed variable X is generated by a subset of causal variables, Z. along with other variables S. Similarly, the noisy label Y is also generated by a subset of the causal variables Z. The causal variables in Z can exhibit causal dependence, meaning some variables are effects of others. The influence of other variables on causal variable Z; can be represented Z, fz (pa( N,,, where fz is the causal mechanism, po(Z.) denotes all the causes of Zr and N; is the corresponding latent noise variable. The latent causal structure, which includes the latent causal variables Z and the causal dependencies among them, can be represented by Directed Acyelic Graph (DAG). Given that this latent causal structure is unknown, our aim is to learn the causal mechanisms that generate both the observed instances and the noisy labels.
W
An illus- tration of the noisy data-genen nata-generating pro- cess with latent vari- ables.
Figure 3:
The data- setergeneraiing process illustrated in Fig. 3 The latent causal variable Z; is generated by its parent variables, denoted pa(Z.), along
3

---

with a corresponding latent noise variable N₁. The instance X and the noisy label Y are generated by different subsets of latent variables. The black arrow in the graph indicates the causal direction in the structural causal model, while the blue arrow indicates the weights of the edges vary with the clean label Y. The instance X and the noisy label Y are dependent because they are generated by the common cause variables Z 42 which implies that the transition matrices in our model are instance-deeendent.
Let ) and pa( denote the sets of causal variables for generating X and Y, respectively. Let fx and fy denote nonlinear mixing functions. Let ex and Ey denote independent noise variables with probability density functions Pex ( and Peş (ey)). ). The generating process of the noisy data can be modeled as a structural causal model (SCM) as follows:
N: N(HN, (x),x(())))
Z: (Y)Z N.,
(1)
X = (pa (X), S) + Ex,
Y : (pa + Eq,
where N(NN (Y), N. (Y)) represents the Gaussian distribution with mean JuN, (Y) and variance 2, (Y). The matrix W (Y) represents the causal association among causal variables. The conditional distribution for the causal variables Z given the clean label Y can be denoted as PW (Z
Unlike previous methods [55] I0], we do not assume the latent causal variables are independent. We allow causal relations between different causal variables in Z, which is more general. For example, in an image, the causal variable associated with the "presence of the sun" can influence the causal variable related to "brightness". Moreover, we allow both the latent noise variable N and the weights of causal association W,, (Y)) be different across different clean labels. This variability enables latent causal variables to exert varying degrees of influence on noisy labels in different classes. For example, mislabeling a "cat" as a "dog" might be influenced by the causal variable "presence of fur", whereas mislabeling a "house" as a "car" is likely less influenced by "presence of fur" and more by other causal variable, such as the causal variable related to "shape". The data generating process in model is nonlinear. Specifically, the instance X and the noisy label Y are generated through nonlinear functions. However, to establish sufficient conditions for identifiability. the causal relations within Z are assumed to be linear. This assumption is further elaborated in Appendix B
Existing theories in causal represen- suggest that the data-genene gata-generating process can be efficiently learned with the aid of additional supervised information. In label-noise learning, this additional supervised information typically requires some clean examples. We follow the previous methods [55] [0] that assume a subset of clean examples be selected from the noisy training data using current techniques [28). With some supervised information derived from these selected clean examples, learning the
Intuition about Learning Latent Data DentaGGnn Process
tation learning 53 3
data-generating process becomes feasible.
Here, we provide some intuition about the core idea behind the existing identifiability result [38]. explaining why the data data-generating process can be effectively learned. Central to this understanding is the realization that the parameters governing this data- genta-gnngating process are not unique individual examples but are shared across them. Specifically, examples from the same class share the same data-genent geta-generating process. The essence of learning the data- settragnnnraingg process lies in learning these shared parameters. When these parameters are shared across different examples, the total number of parameters does not increase as more examples are provided. Instead, these parameters are selected to fit the examples as well as possible. For example, in a linear model, more data are provided, the system accumulates more linear equations but maintains a fixed number of parameters. This setup leads to identifiability when the number of equations exceeds the number of parameters, which enables precise estimation of the parameters. With a set of selected clean examples along with their noisy labels, one can calculate the most probable parameter value for generating these examples. This approach reduces the uncertainty of the parameter value. In Appendix [ we discuss the specific assumption to make the latent generating process of noisy data identifiable.
3.1 Methodology
We propose model-based method which learns the latent Causal Structure for the Generation of Noisy data (CSGN). The workflow of our method is illustrated Fig. 4 A elassification network gr is used to model the distribution gu (X). An encoder gz.s is employed to model the distribution

---

Pfx (XX -
Waz
X ()(()))
X 9 (Z, Y)
Was
Pfp (M)
y
Estimate clean labels
Inference
Generation
Figure 4: A workflow of our method. In the inference stage, classification network is used to learn the clean labels of instances; An encoder is used to learn the causal variables. In the generation stage, different subsets of causal variables used to generate instances and noisy labels are selected by masking. Two decoders are used to generate the instances and noisy labels.
(Z, Y). Two vectors Mx and My are employed to mask some causal variables Z. These vectors select different causal variables for the generation of the instance X and the noisy label Y. The distribution Psx MX| @ Z, S) is modeled by a decoder fx for generating instances. The distribution pyy (Y| M o Z) is modeled by decoder fy for generating noisy labels. These encoders and decoders can be effectively learned within the Variational Autoencoder (VAE) framework [24].
that employ generative models require some clean examples. Similarly, our method also requires clean examples, where their clean labels provide additional supervised information learn latent causal variables and their causal relations. In settings in- volving label-noise learning, the clean label is typically unknown. We only have noisy dataset
Warmup Existing methods 55 10
D D= (xx(), )/= where n is the number of examples. To obtain the additional supervision information, follow the existing methods [55] TO] which adopt the small-loss trick K to select some clean examples. This method is based on the observation that a classification network trained on noisy data tends first memorize examples with correct labels before those with incorrect ones. Consequently, during the early stages of training, the losses for examples with correct labels are usually smaller than those for examples with incorrect labels. By analyzing these loss values, we can distinguish clean examples from noisy ones. We adopt semi-supervised learning approach, as detailed in 4 train the classification network for selecting clean examples. Further specifics of this approach are provided in Appendix C After this initial warmup phase, once some clean examples have been identified, we compile a new dataset consisting of instances, noisy labels, and clean labels as D (xf ,g()) (())) where m the size of the dataset.
Modeling Latent Causal Variables and their Causal Relations We first model the generating process of the causal variables Z using the SCM in Eq. (T). Given that the relationships between the latent variables are linear functions of their causes, plus some independent Gaussian noise, this linearity implies that the distribution of the latent causal variables Z, conditioned on the true label Y, also follows Gaussian distribution. Specifically, the conditional distribution given by:
Pw o (2|Y) = WN E),
(2)
where μz is the mean vector, and E is the covariance matrix of the latent variables Z. The mean μz and the covariance Σ are determined by the weight matrix W and the parameters of the latent noise variables, specifically ,NN and 0. Note that the weight matrix W encodes the causal relationships among the latent variables Z, with each entry representing the strength of the causal effect from one variable to another.
Let 0z 22 denote the element in the i-th row and j-th column of the covariance matrix Σ. Given that the generating process of causal variables is modeled by a linear SCM, we can compute the parameters Hz., OZ 22 and 2, for a causal variable Z, by the following recursion relations [25]:
pz, = E W,,(Y)zz, + (Y).
JEPa;
02 Σ
(Y)o2, +0x, (Y):
jEpa;
02
Σ
(Y) for i i ji
kEPa;
5

---

where pa; represents the set of indices for the parents of the causal variable Z₁. We let W, AN and o be influenced by Y. To provide identifiable results, W, HN and oN have to be determined by Exploring ways to relax this assumption will be a focus of our future work.
38
Modeling the Generation of Observed Variables In real-world scenarios, instances and noisy labels may be generated by different subsets of latent causal variables. To accommodate this, two learnable masks are employed selectively activate different subsets of causal variables for generating instances and noisy labels, respectively. We also utilize distinct decoders to model the generation processes of instances and noisy labels.
Let X and Zy denote the subsets causal variables used to generate the instance X and the noisy label Y. To select the subsets of causal variables x and Zy. we emmply two masks. Specifically, let Mx and My denote two vectors for sparsity. They contain learnable parameters that dictate which causal variables are active during the generation process. The masking process is follows:
Zx Mx Z, Z My Z:
where @ is the element-wise multiplication. To let vectors Mx and My act masks, these vectors are designed to be sparse. This sparsity is encouraged through an I.1 regularization loss:
LM
Mx ||I
|Mpll1
Note that since parameters in the mask learnable, our masking method can select either different subsets or the same subsets variables for generating the instance X and the noisy label Y by optimizing the loss defined in Eq. 5
Instances and noisy labels different decoders: fx for the instances and fy for the noisy labels. The generating process of these variables is defined as follows:
generated through different mechanisms. Thus, employ two
fx S) Ex, fy +ey.
where x and ey represent independent noise variables, each with respective probability density functions Pex (ex) and PEş (. The above equations establish the framework for our generative models for instances and noisy labels, which are probabilistic models defined as follows:
Pfx (X Zx 5) Pex (X fx( S)),
Pfy [ Zyy Pay (Ý (Zx))
As shown in Fig. 3 we consider the mechanisms generating instances and noisy labels to be independent. Therefore, the joint generating process of X and Y conditioned on the latent variables can be modeled by the product of their individual probabilities, i.e.,
Py (X, Y Z, S) Psx (x S)p,, (Y 1g),
where f - (fx. encompasses both decoders, and = -Z U }.
We further assume that p(S), which is the distribution of the latent variable S. follows standard multivariate normal distribution. Integrating all components, the overall generative model is defined as probabilistic model parameterized by 0 (f, W, HNNONN x defined as:
ps X, ł, Z, S|Y) - pr (X, Ý łIz, S)PW uN oy (Z Yp()).
(4)
Here, Pw "WNN..o 3. Y, parameterized by the weight matrix W and the parameters of the independent noise, ,NN, o
Y specifies the distribution of the latent variables conditioned on the true label
Optimization After the warmup phase, can construct new dataset D = (+) (((), (1)
where each instance includes the data (0(), the noisy label g(0) and the corresponding true label g We train our model's parameters using this dataset. Let p(X, Y, Y) denote the underlying joint distribution of the variables X, Y. and Y. In line with prior work (55], approximate this distribution with qD
p X, Y, YY using the product of the empirical data distribution and the model's predictive distribution yo (X, Y, Y) RJ 9 (X, Y yg (Y)X) where ( Ý) is the empirical distri- bution derived from the noisy dataset D, and go (Y the estimated clean class posterior provided by the classification network gy. To optimize the encoder and parameters in 0 for the generative

---

model, we maximize the Evidence Lower Bound (ELBO) on the marginal likelihood of the observed data D. The ELBO is formulated as:
ELBO -B(a.b.))=pp [E(a.))) log ū ūlz,
-KL(gp(z)z,)) o (=ly)) KL will())))
(5)
where KT. denotes the Kullback-Leiblee divergence, (z(2) v) and Qg( xt y) are approximate posterior distributions which can be derived by with the encoder 9z.s- The derivation of ELBO is detailed in Appendix A
The final loss function used to train the networks is
L Lsems AELBOELBO + Xa LM.
(6)
where XELBO and aas are hyperparameters. The algorithm of CSGN is given in Alg.
4 Experiments
In this section, we report the experiment results of our method. We first compare the effectiveness of the proposed data-generating process with existing methods. We then compare the estimation error of noise transition matrices with other methods and the classification performance of the proposed method with that of stateoof-the-a methods on synthetic and real-world noisy datasets. The sensitivity tests of the hyper-parameters are in Appendix H The results of the ablation study are
[]] The number and the accuracy of the selected clean samples are available in Appendix M We report the performance of classification network trained on these clean samples in Appendix K To verify the effectiveness of our method in recovering the causal graph, we conduct experiments on the synthetic dataset in Appendix L The visualization of the transition matrices is in Appendix M
in Appendix
4.1 Experiment Setup
Datasets We empirically verify the performance of our method on three synthesis datasets, i.e.,
Fashion-MNIST [51], CIFAR-IO (26), CIFAR-100 [26). and two real-world datasets, i.e., CIFAR-N
48] and Webvision (2)]. MaNSS contains 70,000 28x28 grayscale images with 10 classes total, 60,000 images for training, and 10,000 images for testing. Both CIFAR-10 and CIFAR- 100 contain 50,000 training images and 10,000 testing images. The image size is 32x32. CIFAR-10 has 10 classes of images, and CIFAR- 100 has 100 classes of images. The three datasets contain clean labels. We corrupted the training data manually according to the instance-dependent noisy label generation method proposed in [4.. CIFAR-N contains CIFAR-ION and CIFAR-1OON. CIFAR- is a real-world label-noise version of CIFAR-10. It contains human-annotatee noisy labels with five different types of noise (Worst, Aggregate, Random 1, Random 2, and Random 3). The corresponding noise rates are 40.21 11, 9.03%. % 17.23%, 18. 12%, and 17.64%. CIFAR-IOON is real-world label- noise version of CIFAR-I00. It contains the type of noise (Fine). The corresponding noise rates are 40.20%. Webvision dataset (29 is a large-scale real-world dataset. We follow the previous work 7 train the model on the first 50 classes of the Google image subset and test the model on the Web Wisvision validation set and the ImageNet ILSVRC12 validation set.
Implementation We implement our algorithm using PyTorch and conduct experiments on eight RTX-3090 GPUs. We use a PreAct ResNet-18 (14] as the classification network for Fashion-MNIST
[T], CIFAR-IO 226) CIFAR-100 26)], and CIFAR-N 48). an inception-resnet v2 [45] as the classification network for Web More details about the structure of decoders and encoders are in Appendix D To prevent the accumulation of errors from biased selection, we adopt the approach of using two neural networks to select clean examples for each other, following the approach in previous work [3. The number of variables in Z is set to in all our experiments. For experiments on Fashion-MNIE
CIFAR-10, 10, CIFAR- 100 and CIFAR-N, NN, we employed SGD with a momentum of 0.9 and a weight decay of 0.0005 to optimize the classification network gy. We used Adam with default parameters to optimize the encoder gz.s. the decoder fx, and the decoder fy and other parameters { {W, UN. o }. The initial learning rate for SGD was set at 0.02 and for Adam at 0.001. Our networks were trained for 200 epochs with batch size of 64. Both learning rates were reduced by factor of 10 after 100 epochs. For experiments on Web we changed the weight decay of SGD to 0.001. The initial learning rate for SGD was set at 0.04 and for Adam at 0.004. Other parameters of optimizers remain unchanged. Our networks were trained for 80 epochs with a batch
7

---

Estimation Error
1.00
Estimation Error
1.00
1.5
0.8
Estimation
Estimation
0.6
1.0
0.50
0.50
0.4
0.5
0.4
Rate
(a) MNIST
(b) Fashion-MNIST
(c) CIFAR-10
(d) CIFAR-100
Figure 5: The estimation error of noise transition matrices. The datasets are MNIST, Fashion-MNIST,
CIFAR-10 and CIFAR-100 with the instance-dependent label noise. The error bar for standard deviation in each figure has been shaded.
Table I: Replace the generative model of CausalNL and InstanceGM with ours. Experiments on CIFAR-10.
CIFAR-10
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN 50%
CausalNL.
90.31 + 0.09
89.76 = 0.08
88.06 = 0.53
80.97 + 1.99
57.93 1: 7.53
CausalNL'
91.59 0.26
91.29 0.07
90.44 - 0.05
89.49 + 0.31
88.83
InstanceGM
96.07 0.14
96.00
0.09
95.95
0.11
95.99 + 0.10
95.81 0.07
InstanceGMi
96.20 0,07
96.59 = 0.07
96.45 = 0.11
96,52 + 0.11
96.32 0.07
Table 2: Replace the generative model of CausaINI. and InstanceGM with ours. Experiments are on CIFAR-100.
CIFAR-100 100
IDN-10S:
IDN-200
IDNN-30
IDN-40%
IDN-50%
CausaINI.
65.33 0.64
63.40
0.49
55.37 + 1.09
48.73 + 1.49
38.69 + 1.72
CausalNL'
72.24 1: 0.14
71.54 = 0.09
70.36 = 0.12
67.60 + 0.25
59.64 : 0,99
InstanceGM
79.30 0.11
78.03 = 0.15
77.89
: 0.18
77.48 + 0.16
76.64 1: 0.38 79.23
Instance(MM
79.87 0.09
79.53 0.17
79.43
0.16
size of 16. Both learning rates were reduced by factor of 10 after 40 epochs. Due to limited space, more details about the hyperparameters are in Appendix D
Baselines The baselines used in our experiments for comparison are: 1). CE, training the clas- sification network using standard cross-entropy loss noisy data directly; 2). MentorNet [22).
pretraining a classification network to select reliable examples for the main classification network; 3). Co-Teaching [13), which uses two classification networks select reliable examples for each other; 4). Reweight (36,, using the importance reweighting method to estimate the unbiased risk defined on clean data; 5). Forward [40], which assumes the noise transition matrix is class dlas-deeenn,nt,, then corrects the loss function; 6). PTD 49 estimating instance- -dennndeeddedednt noisy transition through the parts of instances; 7). CausaINL (55), which explores the information in the instances help the
learning of classification network; 8). CCR [9] uses forward-backwant cycle-consistency regulariza- tion learn noise transition matrtices; 9). MEIDTM (8), which uses Lipschitz continuity constrain the noise transition matrix in the manifold; 10). BLTM (54), which learn the noise transition matrix on a part of dataset with Bayes optimal label; 11). NPC (L. which boosts the pre-trained classifier performances by calibrating the noisy predictions: 12). RENT (2], which utilizes the transition matrix for resampling. 13). DivideMix (28), which divides the noisy examples into labeled examples and unlabeled examples and trains the classification network using the semi- supervised technique MixMatch 4 14). SOP [35], which employs the sparse property of the label noise to prevent models from overfitting to label noise.
4.2 Effectiveness the Proposed Data- Data-Generating Process
To demonstrate our data- seeeeeennerrnngg process (shown in Fig. 2b) is more effective than the existing one (shown in Fig. 2a we replace the generative model in CausalNL and InstanceGM with our proposed model while other experiment settings remain the same. We refer to the modified versions of CausaINL and InstanceGM as CausalNI.' and InstanceGM', respectively. Experiments are conducted on CIFAR-10 and CIFAR- datasets with desendeede label noise. The experiment results presented in Tab. and which show that CausaINL and InstanceGM' outperform their original

---

Table Means and standard deviations (percentage) of classification accuracy on
Fishion-MNIST
IDN-10%
IDN 20%
IDN-30%
IDN-40%
IDN-50%
CE
94.23 0.08
93.94 +: 0.13
93.14 + 0.03
89.54 + 0.32
63.82 3.87
MentorNet
93.16 + 0.01
91.57 + 0.29
90.52 - 0.41
88.14 + 0.76
61.62 + 1.42
Co-Teaching
94.67 + 0.08
94,23
0.01
93.79 + 0.07
92.83 + 0.10
83,99 + 4.57
Reweight
93.42 0.16
0
92.19 + 0.18
88.51
1.52
75.00 + 5.28
Forward
93.48 = 0.11
92.82 0.12
91,05 = 1.44
87.82 + 1.81
78.34 +: 2.98
FTD
92.01 + 0.35
91.08 +: 0.46
89.66 + 0.43
85.69 +: 0.77
75.96 +: 1.38
CausaINI.
93.97 0.334
93.83
90.13
CCR
88.48 0.16
75,40 = 0.19
64.39 + 0.30
50.17 + 0.29
MEIDTM
86.00 -
80.99
73.12 + 2.34
63.81 + 3.02
58.60 + 3.32
BETM
91.28 1,93
91.20
67.65
NPC
=
88.05 0.02
84,99 = 1.20
82.59 + 1.22
70.58 + 4.43
RENT
85,50
79.82 0.40
71.81 + 0.86
61.70 1 0,50
48.98 1: 1.57
DivideMix
95.24 0.11
94.95 0.10
94.36
0.10
89.95
83.35
SOP
95.72 # 0.08
95.40 + 0.03
94.88 + 0.03
92.54
+ 0.09
81.75
0.30
CSGN
95.71 = 0.10
95.46 + 0.06
95.41 + 0.07
95.25 + 0.03
94.59 + 0.05
Table 4: Means and standard deviations (percentage) of classification accuracy on CIFAR-10.
CIFAR-10
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
CE
88.22 + 0.22
86.04 + 0.38
82.40 + 0,50
77.89 +: 1.46
53,97 +: 6.54
MentorNet
89.56 1 0.19
88.80
0.14
86.10
0.12
81.13
58.57 0.71
Co-Teaching
90.32 0.199
89.45 + 0.33
87.08 + 0.36
81.11 + 0.62
52.09
+ 2.00
Reweight
89.63 - 0.27
87.85
+: 0.97
81.29 + 6.49
80.33 + 3.75
75.14 1: 2.40
Forward
88.89 0.18
87.83
79.49
71.11 8.78
PTD
79.01 = 0.20
0.53 76.05
72.28 + 0.49
58.62 + 0.88
53,98
+ 2.34
CausalNL
90.31 0.09
0.08
88.06 + 0,53
80.97 + 1.99
57.93
+: 7.53
CCR
91.43
90.93
90.15 + 0.11
89.01 + 0.15
86.05 0.18
MEIDTM
86.52
77.35 + 0.21
68.21 + 2.09
57.84
3,51
BLTM
80.16 0.37
77.50
71.47 + 2.33
63.20 + 4.52
48.12 +: 9.03
NPC
84.83 = 0.22
83.13
79.48 + 0.31
73.85 + 0,41
67.04 0.06
RENT
80,90 1.15
75.91
74.06 1,30
66.95
2.82
52.73 4.92
DivideMix
96.11 + 0.07
95.75 + 0.04
95.55 + 0.23
95.30
+ 0.11
86.30 + 0.14
SOP
96.08 0.06
95.61 + 0.05
94.14 + 0.07
85.43
- 0.24
63.43 0.55
CSGN
96.46 0.06
96.32 : 0.05
96.27 1 0.11
96.30 1 0.11
95.88 + 0.08
counterparts. This improvement indicates that our data-generating process is more effective than their data-generating process.
4.3 Noise Transition Matrices Estimation Error
We compare the noise transition matrices estimation error of the proposed method with other methods on the four datasets MNIST [77. Fashion-MNIST [11, CIFAR-10 (26] and CIFAR-100 [26]. The labels are corrupted manually by using the instance-dependent noisy label generation method
proposed in [49. The noise rates are from 0.1 to 0.5. The baselines used to compare the noise
transition matrices estimation error are BLTM 54 and MEIDTM S
To calculate the noise transition matrix, we first estimate the clean class posterior p(Y using classification network trained the clean data. Then, the trained encoder and decoders are used to learn the causal variables, reconstruct images and noisy labels based on the instance X and the sampling from the estimated p(Y XX) The class posterior outputted by the noisy label decoder fy is the estimated noise transition matrices. To quantify the estimation error, our methodology aligns with the precedent set by previous research (50), employing the relative error as the evaluative metric. The results of these experiments are shown in Fig. 5 which illustrates that our proposed method surpasses the other approaches in the estimation of noise transition matrices.
4.4 Classification Accuracy
We conduct experiments on both synthetic datasets [51 26 and real-world datasets [48 29
To synthesize noisy labels, we employed instance-dependent noisy label generation methods for the synthetic datasets, as proposed by [49]. We experimented with noise rates of 0.1, 0.2, 0.3, 0.4, and 0.5, denoted by IDN-0.1, IDN-0.2, IDN-0.3. IDN-0.4, and IDN-0.5 respectively.

---

Table 5: Means and standard deviations (percentage) of classification accuracy on and CIFAR-100.
CIFAR-100
1D%-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
CE
63.46 0.56
58.92
0.40
52.55
0.18
45.28 = 0.50
37.48 :: 0.62
MentorNet
67.13 0.03
65.83 4 0.18
61.79 +
55.57 = 0.37
47.78 : 0.18
Co-Veaching
66.44 0.4
0.56 63.76
58.37 +: 0.27
49.72 + 0.71
37.77
+ 1.31
Reweight
59.38 0.33
55.14
1.26 46.91
37.80 1011
28.45
2.57
Forward
59,58 0.42
56.59 0.25
52.75
46.03 = 0.65
35.07
= 0.91
PTD
67.33 6:33
65.33 4 0.59
64.56 1.55
59.73 = 0.76
56.80
+ 1.32
CausaINI.
65.33
63.40
1.09 55.37
48.73 1.49
38.69
1.72
CCR
0.07
68.84 0.09
67.65 0.08
66.54 0.09
64.66 = 0.11
MEIDTM
69.88 0.45
69.16 0.16
66.76
59.18 0.16
1.10 46.61
41.35
1.97 35.67
29.28 0.74
NPC
57.84 0.82
50.05
46.53 0.06
32.60 = 6.47
RENT
38.88 2.24
35.64 0.60
31.53
21.11 = 0.60
DivideMix
77.26
0.24
76.81
0.14
76.54
0.144
73.12 + 0.32
61.54 0.34
SOP
78.68
0.06
77.26
0.17
74.94
+ 0.06
68.94 = 0.18
60.03 = 0.12
CSGN
80.59 + 0.07
79.35 0.12
78,91 + 0.11
78.58 = 0.15
74.60 = 0.17
Table 6: Means and standard deviations (percentage) of classification accuracy on CIFAR-N.
CIFAR- ION
CIFAR-10ON
Worst
Aggregate
Random
Random 2
Random 3
Noisy
CE
77.17 0.56
88.28 0.20
85.79 0.32
85.35
0.44
85.22
0.03
52.45 0.38
MentorNet
81.82 = 0.21
89.92 = 0.10
88,80 = 0,18
88.68
0.15
88.44
0.35
57.70 + 0.09
Co-Teaching
79.51 = 0.72
90.59 =
88.77
0.19
89.00
0.38
88.55
0.33
55.11
1.18
Reweight
77.68 2.46
89.34 = 0.09
88.44 = 0,10
88.16 0.10
88.03
0.100
53.08
0.61
Forward
79.27 1.18
89.22
0.21
86.84
0.97
86.99
0.10
87.53
0.34
53,02
0.52
PTD
65.62 5.28
84.66
3.28 3.28
82.11
3.17
74.76
9.98
84.29
0.64
23,68
2.65
CausalNI.
78.48 2.35
90.24 0.12
0.38 88.08
87.54
0.34
0.16 87.67
53.38
1.42
CCR
80.43 0.24
00.10
88.53 0.08
88.21
0.11
88 88.46
0.08
57.21
0.25
MEIDTM
79.59 =
90.15=0027
87.81 = 0.52
88.07 = 0.18
87,86 - 0.21
38.90 + 0.91
BLTM
1.67
79.41
78.09
76.99 1.23
0.46
NPC
7
0.47
83.07 0.30
83.34 0.18
83.27 = 0.36
46.02 + 0.36
RENT
70.01 2.23
80.94
77.46
77.34 1.06
32.79 1.39
DivideMix
93.41 0.19
95.12
0.15
95.32 0.13
95.15 0.09
95.23 0.16
69.45 0.19
SOP
93.24
95.61
95.28
95.31
95.39
67.81 0.23
CSGN
94.01 0.12 95.87
0.08
95.47
0.12
95.45
0.09
95.54
0.06 71.99
0.08
Table 7: Test accuracy of CSGN on the WibVision validation set and the Ima- geNet ILSVRC12 validation set.
The experimental results for synthetic datasets are pre- sented in Tab. 3 Tab. and Tab. 5 The real-world dataset experiment results presented in Tab. Gand Tab. 7 Gen- erally, our proposed method outperforms existing methods in terms of test accuracy on both synthetic and real-world datasets containing label noise. In particular, when the noise rate is equal 50%, the accuracy of the model is about ten percentage points higher than the best baseline method. The results demonstrate that the proposed method can capture the noise transition matrix under different set- tings and improve the classification performance.
Vebiis
ILSVRC12
top-1
top-5
top-1
top-5
Forward
61.12
82.68
$7.36
82.36
Decoupling
62.54
84.74
58.26
82.26
MentorNet
63.00
81.40
57.80
79.92
Co- To-aacching
63.58
85,20
61.48
84.70
DivideMix
77.32
91.64
75.20
90.84
ELR+
77.78
91.68
70.29
89.76
CSGN
79.84
93.52
76.56
93.76
5 Conclusion
Noise transition matrices important for many label-noise learning algorithms. However, current label-noise learning methods often can only estimate the noise transition matrices of some instances. To leverage these estimated transition matrices to help estimate transition matrices of other instances. It is crucial to establish the connection among the noise transition matrices for different instances. Prior work tackled this issue by manually defining the similarities of noise transition matrices across different instances. Given only noisy data, the introduced similarity based assumptions are hard to verify. If similarities are not truthful, the estimation error of the noise transition matrix could be large, leading to performance degeneration for label-noise learning algorithms. We propose an effective method to capture the connection among the noise transition matrices implicitly by modeling the latent causal structures in the generation of noisy data. Experimental results on different noisy datasets show that our method achieves state-of-the performance in estimating noise transition matrices, which leads to improved classification accuracy.
10

---

Acknowledgments and Disclosure of Funding
Tongliang Liu is partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100577, LP220200949, and IC190100031.
References
[1] HeeSun Bae, Seungjae Shin, Byeonghu Na, JoonHoo Jang, Kyungwoo Song, and I1-Chul Moon. From noisy prediction to true label: Noisy prediction calibration via generative model. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning. ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 1277-1297. PMLR, 2022.
[2] HeeSun Bae, Seungjae Shin, Byeonghu Na, and II-Chul Moon. Dirichlet-based per-sample weighting by transition matrix for noisy label learning, arXiv preprint arXiv:2433.02699 2024.
[3] Yingbin Bai, Erkun Yang. Bo Han. Yanhua Yang. Jiatong Li. Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021. December 6- 14, 2021, virtual, pages 24492,2403, 2021.
[4] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach semi-supervised learning. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché- duce-Buc, Emily B. Fox, and Roman Gamett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 5050-5000000 (960- 2019.
[5] Avrim Blum, Adam Kalai, and Hal Wasserman. Noise-tolerant learning, the parity problem, and the statistical query model. J. ACM, 50(4):506 -599,, 2003.
[6] Johann Brehmer, Pim de Haan, Phillip Lippe, and Taco S. Cohen. Weakly supervised causal representation learning. In NeurIPS, 2022.
[7] Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang, Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learning, pages 1062-1070. PMIR, 2019.
[8] De Cheng, Tongliang Liu, Yixiong Ning, Nannan Wang, Bo Han, Gang Niu, Xinbo Gao, and Masashi Sugiyama. Instance-dependent label-noise learning with manifold-regullriized transition matrix estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 16609 16618. IEEE, 2022.
[9] Cheng, Yixiong Ning, Nannan Wang, Xinbo Gao, Heng Yang. Yuxuan Du Bo Han, and Tongliang Liu. Class-dependent label-noise learning with cycle-consistency regularization. In NeurlPS, 2022.
[10] Arpit Garg, Cuong Nguyen, Rafael Felix, Thanh-Toan Do Gustavo Carneiro. Instance dependen noisy label learning via graphical modelling. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages
January 2023.
[11] Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[12] Aritra Ghosh, Naresh Manwani, and PS Sastry. Making risk minimization tolerant to label noise. Newro- computing, 160:93-107, 2015.
[13] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicold Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 8536-8546. 2018.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings deep residual networks. In Bastian Leibe, Jini Matas, Nicu Sebe, and Max Welling, editors, Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands. October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pages 645. Springer, 2016.
11

---

[15] Zhuo Huang, Chang Liu, Yinpeng Dong, Hang Su, Shibao Zheng, and Tongliang Liu. Machine vision therapy: Multimodal large language models enhance visual robustness via denoising in-context learning. In Forty-first International Conference on Machine Learning, 2023.
[16] Zhuo Huang, Li Shen, Jun Yu, Bo Han, and Tongliang Liu. Flatmatch: Bridging labeled data and unlabeled data with cross-sharpness for semi- supervisd learning. Advances in Neural Information Processing Systems, 0el118844418499 2023.
[17] Zhuo Huang. Xiaobo Xia, Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing out-of-distribution examples via augmenting content and style. In The Eleventh International Conference on Learning Representations, 2023.
[18] Aapo Hyvärinen and Hiroshi Morioka. Unsupervised fcature extraction by time-contrastive learning and nonlincar ICA, In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Procexsing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3765-3773, 2016.
[19] Aapo Hyvärinen and Hiroshi Morioka. Nonlinear ICA of temporally dependent stationary sources. In Aarti Singh and Xiaojin (Jerry) Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FI,, USA, volume 54 of Procee Proceedings of Machine Learning Research, pages 460-469. PMLR, 2017.
[20] Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent componett analysis: Existence and uniqueness results. Neural Networks, 12(3):429-339, 1999.
[21] Aapo Hyvärinen, Hiroaki Sasaki, and Richard E. Turner. Nonlincar ICA using auxiliary variables and generalized contrastive learning. In Kamalika Chaudhuri and Masashi Sugiyama, editors, The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-1 18 April 2019, Naha, Okinawa, Japan, volume of Proceedings of Machine Learning Research, pages 859-868, PMLR, 2019.
[22] Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven cur- riculum for very deep neural networks on corrupted labels. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML, 2018, Stockholmsmüisson Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2309-2318. PMLR, 2018
[23] llyes Khemakhem, Diederik Kingma, Ricardo Pio Monti, and Aapo Hyvärinen. Variational autoencoders and nonlinear ICA: A unifying framework. In Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 26-28August 2020, Online [Palermo, Sicily, Itll], volume 108 of Proceedings of Machine Learning Research, pages 2207-2217.
PMLR, 2020.
[24] Diederik Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
[25] Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
[26] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[27] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: htip://yann. lecun. com/exdb/nmist, 2, 2010.
[28] Junnan Li, Richard Socher, and Steven C. H. Hoi. Dividemix: Learning with noisy lahels as semi- supervised learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[29] Wen Li, Limin Wang. Wei Li, Birikur Agustsson, and Lue Van Gool. Webvision database: Visual learning and understanding from web data, arXiv preprint / 02862, 2017.
[30] Xuefeng Li. Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 6403-6413. PMLR, 2021.
[31] Yuncheng Li. Jianchao Yang. Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 1928- 1936. IEEE Computer Society, 2017.
12

---

[32] Yexiong Lin, Yu Yao. Yuxuan Du, Jun Yu, Bo Han, Mingming Gong, and Tongliang Liu. Do need to penalize variance of losses for learning with label noise? arXiv preprint arXiv:2201.12739) 2022.
[33] Yexiong Lin, Yu Yao, Xiaolong Shi, Mingming Gong. Xu Shen, Dong Xu, and Tongliang Liu. Cs-isolate: Extracting hard confident examples by content and style isolation. Advances in Neural Information Processing Systems, 36, 2024.
[34] Sheng Liu, Jonathan Niles Weed, Narges Razavian, and Carlos Gernandez-Grandee Early-learning regular- ization prevents memorization of noisy labels. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[35] Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over- parameterization, In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022. Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 14153 14172. PMLR, 2022.
[36] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Trans. Pattern Anal. Mach. Intell., 38(3)4447-411, 2016.
[37] Yang Liu, Hao Cheng, and Kun Zhang. Identifiability of label noise transition matrix. In International Conference on Machine Learning, pages 21475-21496. PMLR, 2023.
[38] Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong. Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi, Identifying weight-variant latent causal models. arXiv preprint arXiv:2208. 14153, 2022.
[39] Francesco Locatello, Stefan Bauer. Mario Lucic, Gunnar Rätsch. Sylvain Gelly, Bernhard Schölkopf. and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 41 4114-4122 4124, PMLR, 2019.
[40] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honoluda, HI, USA, July 21-26, 2017, pages 2233-2241. TEFE Computer Society, 2017.
[41] Hans Reichenbach. The Direction of Time. Dover Publications, Mineola, N.Y., 1956.
[42] Bernhard Schölkopf. Causality for machine learning. In Probabilistic and causal inference: The works of Judea Pearl, pages 765-804. 2022.
[43] Bernhard Schölkopf. Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proc. IEEE, 109(5):612-65 634, 2021.
[44] Henning Sprekeler. Tiziano Zito. and Laurenz Wiskott. An extension of slow feature analysis for nonlinear blind source separation. Mach. Learn. Res., 15(1):921-947, 2014.
[45] Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander Alemi. Inception v4, inception ressnct and the impact of residual connections on learning. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[46] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEFF conference on computer vision and pattern recognition, pages 5552-5560, 2018
[47] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision, pages 322-330, 2019.
[48] Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25.29, 2022. OpenReview.net, 2022.
13

---

[49] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards unstance-dependett label noise. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[50] Xiaobo Xia. Tongliang Liu, Nannan Wang. Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise lcarning? In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Brrt
Emily B. Fox, and Roman Gamett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 6835-6846, 2019.
[51] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for henchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017.
[52] Yan Yan, Rómer Rosales, Glenn Fung, Subramanian Ramanathan, and Jennifer G. Dy. Learning from multiple annotators with varying expertise. Mach. Learn., 95(3):291-327, 2014.
[53] Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvac: Disentan- gled representation learning via neural structural causal models. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 9593-9602. Computer Vision Foundation/ IEEE, 2021.
[54] Shuo Yang. Erkun Yang. Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance- dependent bayes-label transition matrix using deep neural network. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 2233222... PMLR, 2022.
[55] Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label- noise learning under a structural causal model. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 4409-4420, 2021.
[56] Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual Reducing estimation error for transition matrix label- moise leamiin. Advances in neural information processing systems, 33:7260-7271, 2020.
[57] Suqin Yuan, Lei Feng, and Tongliang Liu. Late stopping: Avoiding confidently learning from mislabeled examples. In Proceedings of the IEÉE/CVF International Conference on Computer Vision, pages 16079- 16088, 2023.
[58] Suqin Yuan, Lei Feng, and Tongliang Liu. Farly stopping against label noise without validation data. In The Twelfth International Conference on Learning Representations, 2024.
[59] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[60] Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In International Conference on Machine Learning, pages 11447-11457. PMLR, 2020.
[61] Zhaowei 7hu, Yiwen Song, and Yang Liu. Clusterability with noisy labels. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICMI, 2021, 18-24 July 2021, Virtual Event, volume 139 Proceedings of Machine Learning Research, pages 12912-12923, PMLR, 2021.
alteraativ to anchor points when learning
14

---

Appendix
Table of Contents
Derivation of ELBO
15
B Identifiability Analysis
16
More Details the Warmup Strategy
D More Implementation Details
18
E Pseudocode
18
F Limitations
19
G Other Methods in Learning with Noisy Labels
19
II Sensitivity Tests for Hyper-Paramettrs
19
Ablation Study
20
The Number and the Accuracy of the Selected Clean Samples
21
The Performance on the Selected Clean Samples
21
Experiments on the Synthetic Datasel
22
M Comparison of the Noise Transition Matrix
22
Impact Statements
23
A
Derivation of ELBO
The derivation of ELBO shown as follows:
[log pa (x, Øly]]
pa(a v. s|y) (z,
y)
log
(7)
(2, s| v) po (z, a|w,
=B(m.5))>pp // a (2, s|x,) log
po v. z, s/v) Uue (z. 8 v) dzda
(8)
(2 a|x,
po (z, s|m.
/// 2 (2, s|x, log
(0 7. s(v)
=B(n.5.)>>p
deds
(= y)
s/x, 8) v/llpo s|a, v1))
(9)
2 Bi ///
po (z. v. z, a/y) dzds (2, y)
qu (z, s|x, v) log
(10)
15

---

[f 9p (2 y)
po Us 2, sly) dzds
ELBO = (a. (a....) (a..8)))~
(11)
4r (2 8 4)
|/ 4r (z, (/e, log Ps(a, ūl 8, y Pw JN uN (zlu)p(s)
dzds
(12)
9o 8 0. y)
_a(a. [f g0(2, sll, logpy ū| ydzzs
|| (s)p y) log Pw Pw (z)))((s) dzds
(13)
(p(2|x,)) y)
-a [B log (a. 0| s)
KL(a (z w) lPw www.p//ww.................. NN (=lw)) ( w)1 p()))
(14)
-Baa.s.)> ( log (z. ū1
KKL www.w oN (zlw)) K L (sl U) p(s))
|Pw
(15)
where Eq. 13 holds because we the approximate the distribution Uo (Z, X, Y) by the posterior
distribution 44 (Z|XY) Y) and (S) Y), i.e., 9, SI, 9p Z X, y) (S)
B
Identifiability Analysis
In this section, we discuss the required conditions for identifying latent causal structures. The eetta-eneraaing process can be defined as:
P Z. Pfx X Mx Z, Z,S) (Y My G ZZ) Pax Pfx (Mx @ Z, 5))P) (Ý -Pyy (My о Z)),
(16)
which means that the value of X and Y can be decomposed as X fx @ +6x,+
fy (My @ Z) + Ey. where ex and my are independent noise variables with probability density functions Pex (Ex) and Pey (E The variables S represent other variables unrelated the clean label in the instances, such as the rotation and brightness [17]
Intuitively, the instances are generated by subset of the causal variables and the variables S. while the noisy labels are generated by another subset of the causal variables. Let f represent the combination of the functions fx and fy given style s.
Let denote the number of the causal variables, which is also the number of the latent noise variables; Let & denote the dimension of the sufficient statistics for the causal variables Z. We have the following theorem:
Theorem B.1. 138) Suppose latent causal variables and the observed variables Y, y follow the generative model defined in Eq. 44 with parameters (f WWWNNN ,N,, o x. Given style assume the following holds:
1. The set fx E Xl(pe(()) has measure zero, where is the characteristic function of the density Pe-
2. The function fa is bijective.
3. There exist 2m + distinct points yN UN 2m
2m: such that the matrix
LN (Y yN1) (N yNo).... ...., Tre (Y MN TN uNu)) (17)
of size 2m x 2m is invertible.
4. There exist l + distinct points yzu yZ .Z such that the matrix
Lz (T yz1) 7z (Y Uzn).... Mz Vzk) n-(Y yzn)) (18)
of size k x is invertible.
16

---

5. The function class of Wa, can be expressed by Taylor series: for each 5 W k. = 0.
then the true latent causal variables Z are related to the estimated latent causal variables 2 by the following relationship: Z = P7 + c, where P denotes the permutation matrix with scaling, c denotes a constant vector.
The theorem guarantees that the causal variables be identified up to simple linear transformations, i.c., permutation and scaling, under these assumptions. Intuitively, it shows that in the worst case, given fixed style (fixed rotations, lighting conditions, positions, etc.), requires k + clean examples from distinct classes to recover the causal variables and their relations, where k is the dimension of the sufficient statistics for the causal variables Z. For the identifiability of the causal structure among latent variables Z, the Corollary 2 of the existing paper [8) demonstrates that the causal structure can be identified up to the Markov equivalence class.
In our method, we assume the causal association among causal variables is a fully-connected directed acyclic graph, and we assume the distribution of latent noise variables is Gaussian. We can obtain k = m + (m(m- + 12)/2, where m is the number of causal variables. If n, is the number of different style combinations, then (k + 1) clean examples from distinct classes are required. This is because. for each style combination, the parameters of the generative models can generally differ, necessitating the selection of clean examples for each style combination identify the different parameters. Given that the number of latent causal variables in our causal graph is four, n, x 15 clean examples from distinct classes are required to identify the causal model.
Note that this is the theoretical worst-case scenario under the nonlinear ICA framework. If we assume that changes in style combinations do not affect the parameters (fe, W, , then the data-generating process is invariant across different styles. Under this assumption, a best-case scenario can be derived such that we only need k + clean examples to identify the parameters. For our causal graph, which includes four latent causal variables, only 15 clean examples from distinct classes are necessary identify the causal model.
By demonstrating both the worst-case and best-case scenarios, and showing the assumptions required to achieve the best cases, we believe the theorem not only provides valuable insight into our methods but also enhances the understanding of previous methods for learning the generating process (55)
C More Details of the Warmup Strategy
To improve the performance of the classification network, the information of the remaining exam- ples is exploited to train the classification network by using the semi-supervised learning method MixMatch [4]. Specifically, let the selected clean examples be the labeled examples Sx and the remaining examples be the unlabeled examples St. The labels in the labeled examples Sx are refined through the output of the classification network gy. The outputs of the classification network gy for unlabeled examples are used to generate guessed labels. Then, the temperature sharpening is applied to the refined labels and guessed labels on the labeled examples and unlabeled examples, respectively, After that, the labeled examples Sx and the unlabeled examples Sy are transformed into augmented labeled examples S'x and augmented unlabeled examples Sy by using linear mixing. The loss function used to train the classification network
Lsemi = LSx + XuLSs + XpT.rog
where L.Sx is the cross-entropy loss for the labeled examples; LSy is the mean squared error the unlabeled examples; Lreg is a regularization term to prevent the model from predicting all examples to belong to single class. These three terms are defined follows specifically.
Lsx
Σ
Σ
pa log(as = i[z)),
SSx
P.P'S'
Lse
Σ
lp
(Y
ISO
2.PE
Lieg
Σ
log(1
C
Σ
go = iix)),
|8x1 + +1861
wE8x +St
where p is the label, go (Y = (g class.
1 (r))
46 (Y = C²=))² and C denote number of
17

---

D More Implementation Details
We use 6-hidden-layee convolutional network as the encoder 92,s, and the channel sizes of corresponding feature maps are 32, 64, 128, 256, 512, and 512 for Fashion-MNIST, CIFAR-10. CIFAR-100, and CIFAR-N. We 5-hidden-layer convolutional network the encoder 9z.s. and the channel sizes corresponding feature maps are 32, 64, 128, 256, and 512 for WebVision. A 6-hidden-layer waasposed-connolltiie of corresponding feature maps are 512, 512, 256, 128, 64, and 32 for Fashion-MNIST, CIFAR- 10, CIFAR-100, and CIFAR-N. S-hidden-layer rranspsed-coovolutioon network is used as the instance decoder, and the channel sizes of the corresponding feature maps are 512, 256, 128, 64, and 32 for Web We use a three-layer MLP with the Leak ReLU activation function as the weight model to learn the weight of causal relations among the causal variables. To generate noisy labels, three-layer MLP with the Leak ReLU activation function is used the noisy label decoder.
network as the instance decoder and the channel size
The settings of hyperparameters for semi-supervised loss follow previous work [28]. The hyperpa- rameter X, is set to 1. For INSS and CIFAR-10 10 dataset, the hyperparameter X, is set to 5, 10, 15, 20 and 25 for noise rates 10%, 20%, 30%, 40% and 50%. For CIFAR-100 dataset, the hyperparameter A,, is set 10, 25, 50, 100 and 150 for noise rates 10%, 20%, 30%, 40% and 50%. For the CIFAR-N dataset, An is set to 20 for the noise type "Worst", 100 for the noise type "Noisy" and 0 for other noise types. For WebVision dataset, A, is set to 0. The hyperparameters XELBO and AM are set to 0.5 and 0.001 for synthetic datasets. For the real-world datasets, λELBO and AM are to 0.1 and 0.001.
Algorithm CSGN
Input: noisy dataset D, Warm Up epoch Two Total epoch Tmaz.
1: gr.ge f- WarmUP(D):
2: For T= Tw:
3: For k=1,
4: Sample (x,5) D:
5: t+ gf
6:
Sample (2 df S| = =x, =
7:
Calculate the prior distribution p/ NN. ( ÿ) based Ø;
8:
Sample & p (X Z Mx 2 2 = 8):
9:
Sample 1 p Z Z My ④ z);
10:
Calculate the loss using Eq. 55 and update networks (-), p uN.o ()) O.ptx (.) and p (.);
11:
For T= 1....
..... maax: Sx. Sy + Selection(2 D. D.gh.9f);
12:
Sx.Sy St t MixUp(Sx, Sy);
14:
For 2=
15:
Sample (x, ū) ~ D;
16:
i g g+g (x);
17:
Sample (z, 8) ~ of S = -0);
18:
Calculate the prior distribution 1 PW., (Z o) based on Ø;
19:
Sample & f (X Z Mx 2 =);
20:
Sample i ~ r ( Z = My ④ z);
Calculate the loss using Eq. 5 and update networks (+), (,())) P
uN uN P.. (), x (.) and
p (-);
Output: The classification networks gr.97.
E Pseudocode
The algorithm of the proposed method, Causal Structure for the Generation of Noisy data (CSGN), is shown in Alg. [ To prevent the accumulation of errors from biased selection, we adopt the
18

---

approach of using two neural networks, gy and gÿ. to select clean examples for each other, following the approach in previous work [13]. These neural networks model the distributions q (Y) and L ),, respectively.
F
Limitations
In our paper, to provide the theoretical analysis, we assume that the causal relations among the latent causal variables are linear, i.e., the causal variables are influenced by other causal variables linearly. How identify nonlinear causal relations among the latent causal variables is still an open problem. We will continually target this problem in our future work. Moreover, to identify causal variables, similar to many existing work [33 28 55 10, we also require to selected clean examples from noisy data. In future work, we will study how to reduce the required number of clean examples to achieve better accuracy with realistic assumptions. Last, our method is a generative- based method, and requires an additional generative network, leading to more computation costs.
G Other Methods in Learning with Noisy Labels
Label-noise learning is a subset of learning with noisy labels. Methods in label-noise learning refer specifically the methods that model the transition matrices. In this section, we briefly introduce
other approaches in learning with noisy labels. Some algorithms [13 28 57 33 select examples which likely be correct for training. These selections are based on the memorization effeet 159 34 3 32 58], which suggests deep neural networks initially memorize major patterns before progressively memorizing minor ones. In datasets containing noisy labels, correctly labeled examples often form the majority. This enables the networks to prioritize learning from these examples at the early stage of training. As a result, these examples can typically identified by their low loss values early on. Co- Co-Teaching [3 employs this principle to identify small-loss examples as probably clean examples. DivideMix
28 uses a Gaussian Mixture Model to separate training examples into labeled and unlabeled sets based on their training loss, with the labeled set assumed to contain correct labels and trains networks in a semi-supervised manner 4 [6]. Some methods design robust loss functions for learning with noisy labels. If a loss function is symmetric, i.e., >, (Xg)(()))))
where
robust to label noise [ 12] By combating Cross Entropy and Reverse Cross Entropy, Symmetric Cross Entropy Learning is robust to label noise [47]. Some methods correct noisy labels in datasets [46] 60]. Tanaka er al. proposed a method that updates network parameters and class labels alternatively. Zheng al. provided theoretical guarantees for data-re-calibrating methods and proposed a lahel-correction algorithm with a guaranteed success rate 66) NPC (1 is post-proceed post-processing method that models the generation process of clean labels and then uses the model to calibrate the prediction of classifiers.
e is a constant, the loss function
Sensitivity Tests for Hyper-Parameters
$6.0
55.6
15.8
35.00
N
94.5
43.5
94.10
15.4
25.55
93.0
94.00
92.5
25.0
45. 80
$5.75
93
95.70
15
0.7
033
0.1
0.05
0.81
0.035
0.001
8,0005 8.000)
0.7 0.5
0.3
3.1
0.05 0.010.000
5.005
0.03
3,005
3,003 0.0305 O 020)
(a) İELBO on IDN-50%
(b) AM on IDN-509
(c) ÀELBO on Worst
(d) Aas on Worst
Figure 6: Illustration of the test accuracy on CIFAR-10 10 under instance instance-dependent noise with noise rate 0.5 and Worst human label set, respectively. The error bar for standard deviation has been shaded.
We conduct sensitivity tests on hyperparameters XFLRO and AM on CIFAR-10 dataset under the iistance-dependen label noise with the noise rate of 0.5 and CIFAR-ION dataset under the real-world label noise, the Worst human label set. The experiment results are shown Fig. ( The experiment results demonstrate that the hyperparameters λELBO and AM are not sensitive. For the datasets under
19

---

96.0
95.8
95.6
Acc
95.4
95.2
5
8 12 14
Number of
Figure 7: The test accuracy on CIFAR-10 different numbers causal variables under instance-
dependent noise with noise rate of 0.5. The error bar for standard deviation has been shaded.
the synthesis noise, the hyperparameters λELBO and Am are set as 0.5 and 0.001. For the datasets under the real-world noise, the hyperparameters λELBO and AM are set as 0.1 and 0.001.
We also conduct a sensitivity test for the number of causal variables CIFAR- 10 dataset under instance-dependent noise with a noise rate of 0.5. Experiment results are shown in Fig. 7 The results demonstrate that the proposed method is not sensitive to the number of causal variables ranging from 1 to 12. We set the number of causal variables for all experiments.
I Ablation Study
Table 8: Ablation study for warmup strategies. CSGN-WOSM indicates the version of CSGN without warmup using the semi-supervised learning technique. Experiments on CIFAR-10.
CIFAR-10
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
ESGN-WOSM
96.19 + 0.12
96.05 + 0.07
95.94 + 0.07
95.88 + 0.15
94.43 + 0.07
CSGN (ours)
96.46 + 0.06
96.32
0.05
96.27 + 0.11
96.30
0.11
95.88 + 0.08
Warmup Strategies We conduct the ablation study assess the impact of removing the semi- supervised learning warmup phase from the CSGN method. In the study, we replaced the semi- supervised warmup with regular early-stopping approach, where the neural networks were trained for 10 epochs on the training data. The variant of CSGN without the semi-supervised learning warmup denoted as that CSGN retains its effectiveness even without the semi-supervised learning warmup.
The experiment results are shown in Tab The results indicate
Table 9; Ablation study for ELBO loss and LM- Experiments are on CIFAR-10
CIFAR-10
IDN-10%
IDN-20%
IDN-30%
IDN-409
IDN-50%
Car
CSGN without ELBO
96.04 :: 0.05
95.75 =: 0.08
95.
0.09
95.29 = 0.05
88.47 : 0.13
CSGN without Lat
96.26 0.08
96.17 0.05
96.24 0.04
96.29
0.07
95.74 + 0.05
CSGN (ours)
96,46 0.06
96.32 0.05
96.27 : 0.11
96.30 = 0.11
95.88
Loss Functions To evaluate the impact of the loss functions, ELBO and LM, conduct ablation study by removing these loss functions and training classification networks to observe their classification performances. The experiment results shown in Tab. The results demonstrate the effectiveness of each loss function.
Semi-supervised Learning and Sample Selection We remove the semi-supervised learning tech- nique and clean sample selection in our method. Instead, we use another simple algorithm PES 3 which trains label-noise- robuust classifier via early stopping. We leverage the prediction of this classifier for estimating the clean label. We refer this variant of CSGN CSGN-PES. The loss of
20

---

Table 10: CSGN works with PES. Experiments are on CIFAR- 10.
CIFAR-I0
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
PES CSGN-PES
92.95 0.10
92.24 1 0.18
91.08 1 0.11
87.82 = 1.11
82,70 = 1.25
93.05 = 0.06
92.61
= 0.11
91.88
+ 0.07
89.42 = 0.05
85.45 = 0.05
CSGN (ours)
96.46 0.06
96.32
0.05
96.27
1 0.11
96.30 1 0.11
95.88
0.08
CSGN-PES is
Pps = B () (lc AELBOELBO + AMLM
(19)
where Lee is the eross-entropy loss.
The experiment results are shown in Tab. TO The experiment settings follow the settings PES. The experiment results show that our method can still work well when without using the semi-supervised learning technique and clean sample selection.
J The Number and the Accuracy of the Selected Clean Samples
Table 11: The number and the accuracy the selected clean samples on CIFAR-10.
CIFAR-I0
IDN-10%
IDN-20%
IDN-308
IDN-40%
IDN-50%
43707.00
39911.20
35040.2 98.86
25016.40
Aecuracy
99.41
0.00
99.05 1 0.03
0.01
98.77 1 0.01
97.93 1 0.02
Table 12: The number and the accuracy of the selected clean samples on CIFAR-100.
CIFAR-100
IDN-10%
IDN-20%
IDN-30%
IDN-
IDN-50%
Number
41354.40 49.
37391.40
90
32974.80 80 66
28210.00
44.47
26962.40 37.79
Accuracy
99.74
0.01
99.62 1 0.03
99.40
0.01
98.93 + 0.01
85.58 = 0.12
We report the number and the accuracy of the selected clean samples on CIFAR-10 1-10 and CIFAR- 100 datasets. The results shown in Tab. [ and [2] The results indicate that our method select large number of clean samples while maintaining high accuracy.
K The Performance on the Selected Clean Samples
Table 13: Performance of the classifier trained on selected clean samples. Experiments are on CIFAR-10.
CIFAR-10
IDN-109
IDN-205
IDN-30%
IDN-409:
IDN-50%
CE-olean
94.19 = 0.09
93.69 + 0.09
93.46 - 0.07
92.16 = 0.08
91.46 = 0.05
CSGN
96.46 0.06
96.32 :: 0.05
96.27 0.11
96.30
0.11
95.88 0.08
Table 14: Performance of the classifier trained on selected clean samples. Experiments are on CIFAR-100
CIFAR-100
IDN-10%
IDN- 20%
IDN-30%
IDN-40%
IDN-50%
CB-clean CSGN
75.28 = 0.16
74.66 + 0.15
73.44 0.122
71.89
0.09
66.03 + 0.25
0.07
79.35 + 0.12
78.91
0.11
78.58
0.15
74.60 + 0.17
We also conduct experiments on the selected clean samples. Specifically, we trained a classification network using standard cross-entropy loss these samples, employing the same architecture as our
model, CSGN. The results of these experiments are presented in Tab. [3] and 14 These experiment results show that the performance of the network trained on the selected clean samples is lower than
21

---

that of our method. The primary reasons are the reduced sample size of the selected clean samples and a distribution shift from the overall dataset, i.e., the distribution of the selected clean samples is different from the whole dataset. Consequently, despite lower noise rate in the selected clean samples, the performance of the classification network remains below that of our method.
L Experiments on the Synthetic Dataset
Visustizaiion of Noise for allall Somples
100
00
-30-
Festure
Figure 8:
The visualization of the training data. The noise rate is dependent on feature 2. The average noise rate is 0.3.
To verify the proposed method can learn the causal graph and the mask, we conduct an experiment using a synthetic dataset known as the "moon dataset". The data points have two dimensions and are categorized into two distinct categories. To create noisy labels caused by single factor, the noise rate for each data point is dependent on the value of its second dimension. The visualization of noise rates for all training samples shown in Fig. 8 Note that the causal variables of the moon dataset are independent. We train our model on this synthetic dataset with the dimension of the latent factor Z set 2.
After training, the causal weight between two causal variables is -0.0008. The influence is small enough, which means that they are independent. The values of the mask variable My for noisy labels are [0.0000, 0.0232], which shows that our mask mechanism effectively identifies and selects the critical latent factor responsible for generating noisy labels.
We also compare the performance of our method with that of MEIDTM and CausalNL. MEIDTM does not model any causal mechanism. Na roughly models the data-generating process, but the direct cause of noisy labels is the image, which is not aligned with the generating process of the moon dataset. Our model is the closest to the real-world data-generating process compared with these methods. Empirically, our method can achieve a test accuracy of 98.07 0.69% and an estimation error for the transition matrix of 0.10 + 0.07. In comparison, the test accuracy for CausaINL, which does not model the latent causal structure, is 97.88 + 0.75% and transition matrix estimation error of 0.12 + 0.06. At the same time, the test accuracy MEIDTM is 91.06 + 0.75%, and the estimation error of the transition matrix is 0.45 = 0.16. The results show that good causal structure can lead to good transition matrix estimation.
M Comparison of the Noise Transition Matrix
We (-SNE visualization compare the noise transition matrix learned by our method with those derived from the MEIDTM 88. We also select 30 pairs of data points with the same predicted clean labels. The dataset is CIFAR-10 with instance-dependent label noise, and the noise rate is 50%. The experiment results are shown in Fig. [ These data points are the same in two figures. We can see that the distance between the same pair different in the two images. For example, the pairs with number 25 are close to each other in the first figure but are further apart in the second figure. This can verify that the similarity learned by our method is different from the instance-dependent transition matrix- matrix-based method MEIDTM.
22

---

CSGN
(b) MEIDTM
Figure 9: The t-SNE visualization of the similarity of the learned noise transition. The pairs of data points with the same predicted clean label marked with the same number. The distances between two data points represent the difference between the two noise transition matrices of these data points. The distance between the pair with the same number is different in the two images.
N
Impact Statements
This paper presents work whose goal is to advance the field of learning with noisy labels. This work can reduce the cost (annotation costs) of training artificial intelligence models. Therefore, the threshold for using artificial intelligence has been lowered, and artificial intelligence technology is more widely used in society. There are no foreseeable negative effects.
NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We have included the motivation, research problem, and insight in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
23

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specinf well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tes system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification:
We have provided the theory assumptions and proofs the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide complete implementation details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
24

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code is our intellectual property and we will not release it before the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https)
swi.inosioopcccyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
25

---

Answer: [Yes]
Justification; All the details are in the Experiments Setup Section and the Appendix. We also include the sensitive test regard the hyperparameters in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the error bars in the main experiments, except the experiment on the Web Vision dataset, since previous work also does not report the error bars on Web Vision.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the computation devices in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
26

---

9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the
NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms to the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
The societal impacts is discussed in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our research does have such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
27

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [No]
Justification:
We can not find the licences for some assets, but we cite them all.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, papprssiiptheeeoooc thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This research does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This research does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "3An9uLY9lZ", "reviewer": "Reviewer_S6tG", "summary": "The author addresses the problem of instance-dependent label noise. While previous research has used models that generate images from true labels and then predict noisy labels from the images and true labels, the author takes a different approach. The proposed model generates images from some latent factors derived from the true labels and predicts noisy labels from other latent factors derived from the same true labels. This process is implemented using a sparse constraint on the latent factors that contribute to the image and noisy label generation, along with a VAE (Variational Autoencoder) method. This generative approach necessitates true labels, and the author employs a semi-supervised model (MixMatch) to predict the true labels.", "questions": "To better demonstrate the effectiveness of the proposed model, I recommend either reporting the performance of InstanceGM or describing the results of applying various classifier training methods instead of relying solely on MixMatch.", "limitations": "-", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The proposed method by the author is both intuitive and implemented in a logical manner, demonstrating remarkable performance improvements across various benchmarks. Although the proposed method involves multiple loss functions for different objectives, which introduces several hyperparameters, experimental results show that the sensitivity to these hyperparameters is not significant.", "weaknesses": "Although the proposed method is constructed with a sound idea and logical progression, it seems highly dependent on MixMatch. In other words, there is a lack of analysis on whether the model in Figure 2(b) is more effective than the model in Figure 2(a). By referencing representative papers of Figure 2(a), such as CausalNL and InstanceGM, a comparison with InstanceGM using MixMatch could provide a better explanation of the effectiveness of the model in Figure 2(b). However, in the experiments, only CausalNL was compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer S6tG\n \n**Q1**: There is a lack of analysis on whether the model in Figure 2(b) is more effective than the model in Figure 2(a). To better demonstrate the effectiveness of the proposed model, I recommend either reporting the performance of InstanceGM or describing the results of applying various classifier training methods instead of relying solely on MixMatch.\n \n**A1**: Thank you for your suggestion. We have conducted experiments to assess whether the model in Figure 2(b) outperforms the model in Figure 2(a). Specifically, we replaced the generative models in CausalNL and InstanceGM (as shown in Figure 2(a)) with our proposed model (Figure 2(b)), while maintaining identical settings for the other experimental parameters. These experiments were conducted on the CIFAR-10 and CIFAR-100 datasets, which feature instance-dependent label noise. We refer to the modified versions of CausalNL and InstanceGM as \"CausalNL'\" and \"InstanceGM'\", respectively. The results, presented in Tables 3 and 4 of the PDF, indicate that the model in Figure 2(b) is indeed more effective than the model in Figure 2(a). We will include these results in the final version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "15xDdi67Gw", "reviewer": "Reviewer_afzG", "summary": "The work tackles noisy labels learning under the context of classification problem. The proposal leverages a casual model to embed the relationship amongst features, labels and noisy labels. The authors show that the proposed causal model can be identified even under noisy data, hence enables learning the noise transition matrices and ultimately improves classification accuracy.", "questions": "- What is dimension of $\\textbf{Z}$, i.e., number of latent factors $Z_i$?\n- What are $g_Y^1, g_Y^2$ in the Algorithm 1?\n- How is $q_\\psi $ updated after the warm-up step?\n- How the mask variables $M_X, M_{\\widetilde{Y}}$ are updated? Or are they considered as variables?\n- How many clean samples have been selected in the experiment? How accurate are they?\n- Under the proposed casual model, can we have $P(\\widetilde{Y} | Y, X) = P(\\widetilde{Y} |Y) P(Y|X)$? I suggest to have such discussion to clearly distinguish the proposed model vs the instance-independent confusion matrix.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Overall, the paper is well-organized. The generative model is stated clearly. The proposed casual model enables identifying the transition matrix, which is a good-step toward instance-dependent noise. The proposed learning algorithms is discussed in details and demonstrated its effectiveness via several experiments on synthetic and real datasets.", "weaknesses": "- The latent factors generating $\\textbf{X}$ and latent factors generating noisy label $\\widetilde{Y}$ are different, as stated in several places, e.g, page 6, line 260. I agree with the intuition that only a subset of latent factors should affect the generation of $\\textbf{X}$ (or $\\widetilde{Y}$), hence the use of L1 norm in (3). But I don't get why do they have to be different? And how to enforce it?\n\n- Performance on real dataset, i.e, CIFAR-10N is not very convincing as the gaps between the best and second best method are less than 1% in several cases.\n\n- Performance wise, it is unclear what is the contributing factor to the performance of the algorithm 1. Particularly, how significant is the step of selecting clean samples? Can we have another baseline of training classifier only over these clean samples?\n\n- The optimization problem is complicated as it serves as a criterion to learn both casual model structure and parameters.\n\nIn addition, I would suggest to have more discussion on the identifiability result. Given that there are several missing details (articulated in the Questions), I'll consider changing score when those questions are answered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer afzG\n\n**Q1**: I don't get why latent factors generating $\\mathbf{X}$ and latent factors generating noisy label $\\tilde{Y}$ have to be different? And how to enforce it?\n\n**A1**: Thank you for your question. The latent factors for generating instance $X$ and noisy label $\\tilde{Y}$ can be the same, as our method is flexible enough to allow the masks to select any subset of latent factors, including potentially all factors, for generating $X$ and $\\tilde{Y}$.\n\nWe introduce the masks to select factors because not all latent features $X$ cause $\\tilde{Y}$. For instance, in Figure 1(a), the feature \"fur\" causes the noisy label \"dog,\" whereas other features like \"wall\" and \"floor\" are irrelevant to the noisy label. To effectively isolate the minimal necessary latent factors for generating noisy labels, we employ an L1 norm constraint on the mask variables.\n\nTo empirically validate our method, we conducted an experiment on \"moon dataset\". More settings about the \"moon dataset\" can be found in the global response. After training, the values of the mask variable $M_{\\tilde{Y}}$ for noisy labels are [6.0166e-05, 2.3246e-02], which means that the second factor is selected. These results demonstrate that our mask mechanism can effectively select the latent factor for generating noisy labels.\n\n**Q2**: Performance between the best and second best method are less than 1% in several cases on CIFAR-10N.\n\n**A2**: The CIFAR-10 is a relatively easy dataset where models can achieve high accuracy. For instance, the PreAct-ResNet-18 achieves an accuracy of 93.17 ± 0.04% when trained on clean data. However, our model demonstrates significant effectiveness on more challenging datasets with limited samples, such as CIFAR100N. Here, our method achieves an accuracy of 74.60 ± 0.17%, outperforming DivideMix, whose accuracy is 61.54 ± 0.34%. Moreover, our model also shows robust performance on the large-scale real-world dataset WebVision, achieving a top-1 accuracy of 79.84%, compared to 77.32% for DivideMix. \n\n**Q3**: How significant is the step of selecting clean samples? Can we have another baseline of training classifier only over these clean samples?\n\n**A3**: Thank you for your question. The selection of clean samples is a crucial step in many noise-robust algorithms, including CausalNL, InstanceGM, and DivideMix, which use a heuristic approach of alternatively training classifiers and selecting clean samples. The accuracy of the selected clean samples improves as the improvement of the classifier. Our method follows the heuristic process.\n\nWe train a baseline using cross-entropy loss on the selected clean samples. We use \"CE-clean\" to denote this baseline. The experiment results are in Table 5 and 6 of the PDF.\n\n**Q4**: The optimization problem is complicated as it serves as a criterion to learn both casual model structure and parameters.\n\n**A4**: Though the optimization problem is complicated, the optimization process of our model is stable using modern optimizers, such as SGD and Adam. \n\n**Q5**: I would suggest to have more discussion on the identifiability result.\n\n**A5**: Thanks for your advice. Here, we provide more discussion on the identifiability result:\nThe theoretical results indicate that when the number of causal factors is 4, $n_s*15$ confident examples from distinct classes are required to identify the causal model, where $n_s$ is the number of different style combinations. If the changes in style combinations do not affect the parameters of our causal model, only 15 confident examples from distinct classes are required. This discussion will be added to the final version of our paper.\n\n**Q6**: What is the number of latent factors $Z_i$?\n\n**A6**: Thank you for your suggestion. The number of latent factors is 4, as mentioned in Line 632 of our paper. To clarify this and prevent any confusion, we will add a statement in Section 4 of the final version of the paper to emphasize that the number of latent factors is 4.\n\n**Q7**: What are $g_Y^1,g_Y^2$ in the Algorithm 1?\n\n**A7**: $g_Y^1$ and $g_Y^2$ are two classification networks used to model the distribution of $q_\\psi(Y|X)$, which are defined in line 197 of the paper. Specifically, we follow previous work (DivideMix) to adopt a Co-Teaching learning paradigm. We will clarify this in Appendix D.\n\n**Q8**: How is $q_{\\psi}$ updated after the warm-up step?\n\n**A8**: After the warm-up step, the entire model, including $q_{\\psi}$, is optimized end-to-end by minimizing the loss defined in Equation 9.\n\n**Q9**: How the mask variables $M_X,M_{\\tilde{Y}}$ are updated? Or are they considered as variables?\n\n**A9**: The mask variables $M_X$ and $M_{\\tilde{Y}}$ are learnable parameters that are updated through the optimization of Equation 9. During this optimization, two key constraints apply to these variables: First, the causal factors selected by $M_X$ and $M_{\\tilde{Y}}$ can be used to generate $X$ and $\\tilde{Y}$; Second, the mask variables are required to be sparse. These constraints are integral to the loss defined in Equation 9. We will clarify the update of mask variables in the final version.\n\n**Q10**: How many clean samples have been selected in the experiment? How accurate are they?\n\n**A10**: Thanks for your insightful question. We report the number and the accuracy of the selected clean samples on the CIFAR-10 and CIFAR-100 datasets, as shown in Table 7 and 8 of the PDF. We will report the number and the accuracy of the selected clean samples in the Appendix of our paper.\n\n**Q11**: Under the proposed casual model, can we have $P(\\tilde{Y}|Y,X)=P(\\tilde{Y}|Y)P(Y|X)$ ? I suggest to have such discussion to clearly distinguish the proposed model vs the instance-independent confusion matrix.\n\n**A11**: Thanks for your insightful question. We believe you mean to $P(\\tilde{Y}|Y,X) = P(\\tilde{Y}|Y)$. Our model is instance-dependent (see details on the global response). We will clarify this distinction in the introduction of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y3MPmA5NVY", "reviewer": "Reviewer_pfn2", "summary": "In learning with noisy labels, estimating an instance's noise transition matrix is crucial for inferring its clean label. Current studies assume previous relations between transition matrices and it may not hold in real world scenarios. Motivated by the relation between noise transition matrices are established through the causal structure, this paper suggests to learn the latent causal structure through learnable graphical model.", "questions": "- How can I know the learned latent causal structure is good? Also, how can I measure good latent structure will lead to good transition matrix estimation?\n- According to Figure 5, the transition matrix estimation error of MEIDTM is smaller CSGN for CIFAR100 dataset. However, considering Table 3, the accuracy of MEIDTM is not better than CSGN. It means the proposed method does not always surpass other approaches in the estimation of noise transition matrics. What it means?\n- Any examples of similarity-based assumption failure cases? Can modeling latent causal structure solve those failures?", "limitations": ".", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "Using DAG for flexible latent causal structure.", "weaknesses": "- This paper points out that other studies estimate the transition matrix based on some similarity assumptions, e.g. class identity, manifold assumption. However, there is instance-dependent transition matrix modeling, which means that the transition matrix is different sample by sample. It means there is no similarity assumption for instance-dependent transition matrix as modeling part.\n- Furthermore, in the modeling of the paper, it is assumed that the relation structure is established from the causal structure. It means there exists some arbitrary similarity assumption of similar latent leads to similar transition matrix, even when the latent variable is not interpretable.\n- After all, the similarity in manifold and the similarity of latent structure could be very similar, since the samples generated from the same latent variable would have similar features, so they could exist in similar places in manifold space in implementation. If samples with similar latent variable exist in very different space in manifold, considering the latent can be a must.\n- In one sentence, the motivation to learn the latent causal structure is not convincing enough.\n\n- For implementation, it will require much time and memory computation cost since it should generate X (=input), which is a limitation of this method. \n- For experiments, the baselines are outdated. The most recent baseline seems to be published at 2022. There are more recent researches for learning with noisy label and they should be compared.\n- Why NPC and SOP are not included as baselines, although those are the methods of using generative process for noisy label problem? Since this paper utilizes the generative model for classification task, NPC and SOP must be compared as important baselines.\n- The position of Figure 5 is inadequate. It should be located at the experiment part.\n- No ablation studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer pfn2\n\n**Q1**: The transition matrix is different sample by sample in instance-dependent transition matrix modeling. It means there is no similarity assumption for instance-dependent transition matrix.\n\n**A1**: We believe that there is a misunderstanding. The transition matrix varying from sample to sample does not imply the absence of similarity assumption. For example, part-dependent label noise [3] represents an instance-dependent label noise scenario where it is assumed that transition matrices for certain parts of instances are similar. Additionally, without some form of similarity between samples, it is impossible to estimate transition matrices for the entire dataset, as only the noisy label is observable, and the clean label remains hidden. Therefore, existing methods employ predefined similarities to estimate transition matrices across a dataset.\n\n**Q2**: Any examples of similarity-based assumption failure cases? Can modeling latent causal structure solve those failures?\n\n**A2**: Yes, our model performs well when the similarity-based assumption fails. We conducted an experiment on the moon dataset. We synthesize the noisy labels where the transition matrices on the same manifold are not the same. The test accuracy of our method is 98.07±0.69%, and the estimation error of the transition matrix is 0.10±0.07. In contrast, the test accuracy of MEIDTM is 91.06±0.75%, and the estimation error of the transition matrix is 0.45±0.16. The results show that our method surpasses the MEIDTM in terms of test accuracy and estimation errors in the transition matrix.\n\nFurthermore, we tested our model in a scenario where the similarity-based assumption is valid, i.e., the transition matrix is the same across the same manifold. The test accuracy of our method is 98.35 ± 0.19%, and the estimation error of the transition matrix is 0.08±0.05. In contrast, the test accuracy of MEIDTM is 96.23±0.25%, and the estimation error of the transition matrix is 0.42±0.11. The experiment results show the generalizability and robustness of the proposed method.\n\n**Q3**: The paper assumes that the relation structure is established from the causal structure. It means there exists some arbitrary similarity assumption of similar latent leads to similar transition matrix. The similarity in manifold and the similarity of latent structure could be very similar, since the samples generated from the same latent variable would have similar features.\n\n**A3**: We do not manually define some similarities like the existing methods. Instead, our method aims to recover the causal factors that cause noisy labels, and the relations among different transition matrices are established based on the causal factors. For example, an annotator is likely to annotate the pictures with the feature “furs” as “dog”. When this annotator annotates the pictures containing the feature “furs,” their noisy labels are probably “dog.”\n\nThe similarity of our model is not similar to the similarity in the manifold since our method can still perform well when the similarity assumption in the manifold is not met.\n\n**Q4**: The motivation to learn the latent causal structure is not convincing enough.\n\n**A4**: Learning the latent causal structure can recover the causes of noisy labels and then establish the relations among different transition matrices without predefined similarity. The empirical results in A2 also demonstrate the effectiveness of our method.\n\n**Q5**: The proposed method will require much time and memory computation cost since it should generate X, which is a limitation of this method. \n\n**A5**: Our method is a generative-based method, and it requires an additional generative network, leading to more computation cost. We acknowledge this limitation and will include it in Appendix C. However, the existing work, CausalNL and InstanceGM, also have this limitation.\n\n**Q6**: The baselines are outdated. Since this paper utilizes the generative model for classification task, NPC and SOP are important baselines.\n\n**A6**: We have updated our set of baselines. RENT [4], which was published at ICLR 2024, is included. Additionally, NPC and SOP have been included as baselines as well, as shown in Table 1 and 2 of the PDF.\n\n**Q7**: The position of Figure 5 is inadequate. It should be located at the experiment part.\n\n**A7**: We will relocate the position of Figure 5 in the final version.\n\n**Q8**: No ablation studies.\n\n**A8**: We have conducted ablation studies, shown in Table 9 and 10 of the PDF.\n\n**Q9**: How can I know the learned latent causal structure is good? Also, how can I measure that a good latent structure will lead to good transition matrix estimation?\n\n**A9**: The efficacy of the learned latent causal structure can be empirically validated by comparing the performance outcomes of our model against those of other models. Specifically, we conducted experiments using the moon dataset, where our method achieved a test accuracy of 98.07±0.69% and an estimation error for the transition matrix of 0.10±0.07. In comparison, the CausalNL model, which does not model the latent causal structure, recorded a test accuracy of 97.88±0.75% and a transition matrix estimation error of 0.12±0.06. The results show that learning latent causal structure is good, and it can lead to good transition matrix estimation.\n\n**Q10**: The transition matrix estimation error of MEIDTM is smaller than the CSGN on the CIFAR100 dataset. But the accuracy of MEIDTM is not better than CSGN. \n\n**A10**: Thanks for your comment. There are other factors that can affect the test accuracy. For example, MEIDTM only employs one neural network to select clean data and trains the classifier. Moreover, MEIDTM does not employ the semi-supervised learning technique. CausalNL is the closest one to our setting. When the settings of the two methods are similar, the one with a smaller transition matrix estimation error has better accuracy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GvypID2cL2", "reviewer": "Reviewer_22VP", "summary": "This paper proposes that traditional noise-labeled learning methods based on noise transition matrices have limitations. Specifically, only the noise transition matrices of certain special examples can be effectively estimated, while the transition matrices of other examples need to be estimated based on similarity. However, these similarity assumptions are difficult to establish in many real-world applications. Therefore, this paper suggests that learning the latent causal structure governing the generative process of noisy data can help estimate noise transition matrices. This paper constructs a novel structural causal model to simulate the generative process of noisy data and integrates semi-supervised learning techniques, ultimately achieving state-of-the-art results.", "questions": "1.\tCan the authors explain why the clean label is the cause of the latent noise variables?\n2.\tCan the authors show the model performance without using semi-supervised learning?", "limitations": "The authors have discussed some limitation and it seems reasonable.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tIn the structural causal model, the assumption of dependency relationships among the latent variables z is novel, and the analysis of its reasonableness is convincing.\n2.\tThe theoretical analysis of the identifiability of latent variables in the appendix is reasonable.\n3.\tThe experimental design of this paper is comprehensive.", "weaknesses": "1. The structural causal model of the labeling process is not reasonable. According to the labeling process, the example feature x should be the cause of the noisy label y ̃. If only the clean label y is the cause of the noisy label, then this structural causal model is still modeling class-dependent noise rather than instance-dependent noise.\n2. In line 150, the assumption that latent noise variables are generated by the clean label is not reasonable.\n3. The CSGN method, like other noise-labeled learning methods based on semi-supervised learning, relies on the small-loss criterion and clean samples.\n4. The CSGN method introduces semi-supervised learning techniques for warmup; however, the experimental section does not include ablation studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer 22VP\n\n**Q1**: The structural causal model of the labeling process is not reasonable. The example feature $X$ should be the cause of the noisy label $\\tilde{Y}$. If only the clean label y is the cause of the noisy label, then this structural causal model is still modeling class-dependent noise rather than instance-dependent noise.\n\n**A1**: We believe that it is a misunderstanding. Our model indeed addresses instance-dependent noise. Specifically, the generation of instance $X$ and noisy label $\\tilde{Y}$ involves common latent causes $\\mathbf{Z}$, such as $Z_3$ depicted in Figure 2(b). In causality theory, the presence of common causes between two variables, such as $X$ and $\\tilde{Y}$, establishes dependence between them [1,2]. This indicates that our model is inherently instance-dependent, not instance-independent.\n\nMoreover, when there exists label noise, the real cause of the noise is some latent causal factors rather than the image itself, which is very intuitive. For example, we usually say that light and angle cause noise, but we do not say that the image causes noise. This implies that the direct causes of label noise are light and angle rather than image. However, previous work [6] ignores this point. In their generation model, instance $X$ is the direct cause of the noisy label $\\tilde{Y}$. This is one of the insights of our paper. We have also provided experiments to verify the effectiveness of our structural causal model, as shown in Table 3 and 4 of the PDF. The experiment details are in the response to Reviewer S6tG.\n\n**Q2**: In line 150, the assumption that latent noise variables are generated by the label is not reasonable.\n\n**A2**: Thank you for highlighting this concern. It is a challenge to identify the latent factors where these factors are not independent but causally related. The assumption that latent noise variables are generated by the label can provide a theoretical guarantee of identifiability. Though this assumption introduces certain constraints, compared to other models like CausalNL, our generative framework offers a more realistic representation of the underlying data generation processes.\n\nTo empirically validate our model, we conducted experiments by integrating our generative framework into CausalNL and InstanceGM. These experiments were carried out on the CIFAR-10 dataset, which features instance-dependent label noise. We refer to these adaptations as \"CausalNL'\" and \"InstanceGM'\", respectively. The experiment results are shown in the Table 3 and 4 in the PDF. The experiment results demonstrate the effectiveness of our model.\n\n**Q3**: The CSGN method, like other noise-labeled learning methods based on semi-supervised learning, relies on the small-loss criterion and clean samples. Can the authors show the model performance without using semi-supervised learning?\n\n**A3**: Thanks for your advice. The CSGN can also work with the early stopping method, PES [5]. We follow the setting in PES and conduct experiments on the CIFAR10 dataset. We refer to the version of CSGN that works with PES as “CSGN-PES”. Notably, CSGN-PES does not rely on semi-supervised learning, the small-loss criterion, or the selection of clean samples. The experiment results are shown in Table 11 of the PDF. The experiment results demonstrate that CSGN-PES maintains robust performance even in the absence of semi-supervised learning techniques.\n \n**Q4**: The CSGN method introduces semi-supervised learning techniques for warmup; however, the experimental section does not include ablation studies.\n\n**A4**: Thank you for pointing this out. We have conducted ablation studies to assess the impact of removing the semi-supervised learning warmup phase from the CSGN method. In these studies, we replaced the semi-supervised warmup with a regular early-stopping approach, where the neural networks were trained for 10 epochs on the training data. The variant of CSGN without the semi-supervised learning warmup is denoted as CSGN-WOSM. The experiment results are shown in Table 9 of the PDF. The results indicate that CSGN retains its effectiveness even without the semi-supervised learning warmup.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Global response

We sincerely appreciate the time and effort the reviewers invested in reviewing our manuscript. Your insightful comments and constructive advice have been instrumental in enhancing the quality and clarity of our work. We are grateful for your detailed feedback and guidance. In response to several frequently asked questions, we present a clarification and introduce a new experimental setting. The reference list below includes all papers cited in our responses.

### Distinguishing our model from instance-independent approaches
In our model, the generation of instance $X$ and noisy label $\tilde{Y}$ involves common latent causes $\mathbf{Z}$, such as $Z_3$ depicted in Figure 2(b). In causality theory, the presence of common causes between two variables, such as $X$ and $\tilde{Y}$, establishes dependence between them [1,2]. This indicates that our model is instance-dependent, not instance-independent ($P(\tilde{Y}|Y,X) = P(\tilde{Y}|Y)$).

### The setting on the moon dataset.

We conducted an experiment using a synthetic dataset known as the "moon dataset". The data points have two dimensions and are categorized into two distinct categories. To create noisy labels caused by a single factor, the noise rate for each data point is dependent on the value of its second dimension. The training data is shown in Figure 1 of the PDF. We trained our model on this synthetic dataset with the dimension of the latent factor $Z$ set to 2. 

### Reference list

[1] Schölkopf, Bernhard. "Causality for machine learning." *Probabilistic and causal inference: The works of Judea Pearl*. 2022. 765-804.

[2] Reichenbach, Hans. "The Direction Time". Berkeley, CA, USA: Univ. of California Press, 1956.

[3] Xia, Xiaobo, et al. "Part-dependent label noise: Towards instance-dependent label noise." Advances in Neural Information Processing Systems 33 (2020): 7597-7610.

[4] Bae, HeeSun, et al. "Dirichlet-based Per-Sample Weighting by Transition Matrix for Noisy Label Learning." *The Twelfth International Conference on Learning Representations*.

[5] Bai, Yingbin, et al. "Understanding and improving early stopping for learning with noisy labels." Advances in Neural Information Processing Systems 34 (2021): 24392-24403.

[6] Yao, Yu, et al. "Instance-dependent label-noise learning under a structural causal model." Advances in Neural Information Processing Systems 34 (2021): 4409-4420.

Meta-review: Two reviewers had highly critical reviews. However, the authors engaged substantially with both reviewers during the discussion period. It is my opinion that the concerns of both reviewers were sufficiently addressed by the authors, even if the reviewers did not acknowledge this. 

In one case, after a very fruitful back-and-forth, a reviewer decided that the independent contributions of the different modeling components was unclear and thus that the paper did not represent a meaningful methodological innovation. However, the authors countered this with results from a compelling experiment where they replace the causal models of two baseline methods (CausalNL and InstanceGM) with their proposed causal model, and showed that the performance of both baselines improved substantially. The authors further provided results from ablation studies in their rebuttal which suggested that the impressive performance of their proposed approach was not owed simply to a "bag of tricks". I found these supplementary experiments very convincing and commend the authors for supplying them on short notice.

In another case, a reviewer expressed skepticism that the paper had correctly implemented the myriad of baselines to which they compare the proposed method. In response, the authors very carefully walked through each baseline method, showing how the numbers obtained by their implementation closely matched the numbers reported in each of the relevant papers. I found this convincing. It is of course fair for the reviewer to ask this question, particularly since the paper reports SOTA results in every single experiment, along every metric, as compared to over 10 recent baselines. Such sweeping advantage is uncommon in my experience, and skepticism is warranted. However, some level of trust is inherent to the conference reviewing process, and the authors have done enough to earn it.

Beyond these two critical reviewers (whose concerns were appropriately addressed by the authors) the remaining reviewers were excited by the paper. They appreciated the elegance and conceptual clarity of the proposed causal model, found the motivation to be timely and important, and were very impressed by proposed approach's empirical performance. I think this is a very solid paper overall.

If accepted, the authors should edit the camera-ready version to incorporate all of the supplemental experiments they reported during the discussion period.
Decision: Accept (poster)