Instance-Dependent Label-Noise Learning under Structural Causal Models
Year: 2022

Abstract:
Label noise generally degenerates the performance of deep learning algorithms because deep neural networks easily overfit label errors. Let X and Y denote the instance and clean label, respectively. When Y is a cause of X , according to which many datasets have been constructed, e.g., SVHN and CIFAR, the distributions of P (X) and P (Y |X) are generally entangled. This means that the unsupervised instances are helpful to learn the classifier and thus reduce the side effect of label noise. However, it remains elusive on how to exploit the causal information to handle the label-noise problem. We propose to model and make use of the causal process in order to correct the label-noise effect. Empirically, the proposed method outperforms all state-of-the-art methods on both synthetic and real-world labelnoise datasets.

heading: 1 Introduction
text: Learning with noisy labels can be dated back to [1] and has recently drawn a lot of attention [15, 19, 11, 10, 28]. In real life, large-scale datasets are likely to contain label noise. It is partly because that many cheap but imperfect data collection methods such as crowd-sourcing and web crawling are widely used to build large-scale datasets. Training with such data usually lead to poor generalization abilities of deep neural networks because they can memorize noisy labels [2, 33].
To improve the generalization ability of training with noisy labels, one family of existing label-noise learning methods is to model how the label noise was generated [17, 19, 27, 34, 14]. Specifically, these methods try to reveal the transition relationship from clean labels to noisy labels of instances, i.e., the distribution P (Ỹ |Y,X), where Ỹ , Y and X are the random variables for the noisy label, latent clean label, and instance, respectively. The advantage of modelling label noise is that given only the noisy data, when the transition relationship is identifiable, classifiers can be learned to converge to the optimal ones defined by the clean data, with theoretical guarantees. However, the transition relationship is not identifiable in general. To make it identifiable, various assumptions have been made on the transition relationship. For example, Natarajan et al. [17] assume that the transition relationship is instance independent, i.e., P (Ỹ |Y,X) = P (Ỹ |Y ); Xia et al. [30] assume that the P (Ỹ |Y,X) is dependent on different parts of an instance. Cheng et al. [7] assume that the label noise rates are upper bounded. In practice, these assumptions may not be satisfied and are generally hard to be verified given noisy data alone.
Inspired by causal learning [20, 25, 21, 23], we provide a causal perspective of label-noise learning method named CausalNL. We exploit the causal information to help identifiability of the transition matrix P (Ỹ |Y,X) other than making assumptions directly on the transition relationship.
†Correspondence to Tongliang Liu (tongliang.liu@sydney.edu.au).
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Specifically, we assume that the data containing instancedependent label noise is generated according to the causal graph in Fig. 1. For example, for the Street View House Numbers (SVHN) dataset [18], X represents the image containing the digit; Y represents the clean label of the digit shown on the plate; Z represents the latent variable that captures the information affecting the generation of the images, e.g., orientation, lighting, and font style. Here Y is naturally a cause of X and the causal generative process can be described in the following way. First, the house plate is generated according to the street number and attached to the front door. Then, the house plate is captured by a camera (installed in a Google street view car) to form X , taking into account of other factors such as illumination and viewpoint. Finally, the images containing house numbers are collected and relabeled to form the dataset. Let us
denote the annotated label by the noisy label Ỹ as the annotator may not be always reliable, especially when the dataset is very large but the budget is limited. During the annotation process, the noisy labels were generated according to both the images and the range of predefined digit numbers. Hence, both X and Y are causes of Ỹ . Note that most existing image datasets are collected with the causal relationship that Y causes X . For example, see the widely used FashionMNIST and CIFAR. When we synthesize instance-dependent label noise based on them, we will have the causal graph illustrated in Fig. 1. Note also that some datasets are generated with the causal relationship that X causes Y . Other than using domain knowledge, the different causal relationships can be verified by employing causal discovery [26, 25, 21, 37].
When the latent clean label Y is a cause of X , P (X) will generally contain some information about P (Y |X). This is because, under such a generative process, the distributions of P (X) and P (Y |X) are entangled [22, 39]. To help estimate P (Y |X) with P (X), we make use of the causal generative process to estimate P (X|Y ), which directly benefits from P (X) by generative modeling. The modeling of P (X|Y ) in turn encourages the identifiability of the transition relationship and helps learn P (Y |X). For example, in Fig. 2(a), we have added instance-dependent label-noise with a rate 45% (i.e., IDLN-45%) to the MOON dataset and employed different methods [10, 36] to solve the label-noise learning problem. As illustrated in Fig. 2(b) and Fig. 2(c), previous methods fail to infer clean labels. In contrast, by constraining the conditional distribution of the instances, i.e., restricting the data of each class to be on a manifold by setting the dimension of the latent variable Z to be 1-dimensional, the label transition as well as the clean labels can be successfully recovered (by the proposed method), which is showed in Fig. 2(d). It is worth noting that the idea of finding P (Y |X) by modeling P (Y ) and P (X|Y ) instead has been exploited in the context of domain adaptation; for instance, it inspires target shift, (generalized) conditional shift and other settings for domain adaptation [38, 9]
Specifically, to make use of the causal graph to contribute to the identifiability of the transition matrix, we propose a causally inspired deep generative method, which models the causal structure with all the observable and latent variables, i.e., the instance X , the noisy label Ỹ , the latent feature Z, and the latent clean label Y . The proposed generative model captures the variables’ relationship indicated by the causal graph. Furthermore, built on the variational autoencoder (VAE) framework [12], we build an inference network which could efficiently infer the latent variables Z and Y when maximising the marginal likelihood p(X, Ỹ ) on the given noisy data. In the decoder phase, the data will be reconstructed by exploiting the conditional distribution of instances P (X|Y,Z) and the transition relationship P (Ỹ |Y,X), i.e.,
pθ(X, Ỹ ) = ∫ z,y P (Z = z)P (Y = y)pθ1(X|Y = y, Z = z)pθ2(Ỹ |Y = y,X)dzdy
will be exploited, where θ := (θ1, θ2) are the parameters of the causal generative model (more details can be found in Section 3). Ay a high level, according to the equation, given the noisy data and the distributions of Z and Y , constraining pθ1(X|Y,Z) will also greatly reduce the uncertainty of pθ2(Ỹ |Y,X) and thus contribute to the identifiability of the transition matrix. Note that adding a constraint on pθ1(X|Y, Z) is natural; for example, images often have a low-dimensional manifold [3]. We can restrict P (Z) to fulfill the constraint on pθ1(X|Y,Z). By exploiting the causal structure and
the constraint on instances to better model label noise, the proposed method significantly outperforms the baselines. When the label noise rate is large, the superiority is evidenced by a large gain in the classification performance.
The rest of the paper is organized as follows. In Section 2, we briefly review the background knowledge of label-noise learning and causality. In Section 3, we formulate our method, named CausalNL, and discuss how it helps to learn a clean classifier, followed by the implementation details. Experimental validations are provided in Section 4. Section 5 concludes the paper.

heading: 2 Noisy Labels and Causality
text: In this section, firstly, we introduce how to model label noise. Then, we introduce the structural causal model and discuss how to exploit the model to encourage the identifiability of the transition relationship and help learn the classifier.
Transition Relationship To build a statistically consistent classifier that converges to the optimal classifier defined on clean data by only employing noisy data, the transition relationship P (Ỹ |Y,X) has to be identified. Given an instance, the conditional distribution can be written in an C × C matrix which is called the transition matrix [19, 29, 30], where C represents the number of classes. Specifically, for each instance x, there is a transition matrix T (x). The ij-th entry of the transition matrix is Tij(x) = P (Ỹ = i|Y = j,X = x), which represents the probability that the instance x with the clean label Y = j will have a noisy label Ỹ = i.
The transition matrix has been widely studied to build statistically consistent classifiers, because the clean class posterior distribution P (Y |x) = [P (Y = 1|X = x), . . . , P (Y = C|X = x)]> can be inferred by using the transition matrix and the noisy class posterior P (Ỹ |x) = [P (Ỹ = 1|X = x), . . . , P (Ỹ = C|X = x)]>, i.e., we have P (Ỹ |x) = T (x)P (Y |x). Specifically, the transition matrix has been used to modify loss functions to build risk-consistent estimators; see, e.g., [8, 19, 35, 28], and has been used to correct hypotheses to build classifier-consistent algorithms; see, e.g., [17, 24, 19]. Moreover, the state-of-the-art statically inconsistent algorithms [11, 10] also use diagonal entries of the transition matrix to help select reliable examples used for training.
However, the distribution P (Ỹ |Y,X) is not generally identifiable [28]. To make it identifiable, one has to resort to additional assumptions. The most widely used assumption is that given clean label Y , the noisy label Ỹ is conditionally independent of instance X , i.e., P (Ỹ |Y,X) = P (Ỹ |Y ). Under such an assumption, the transition relationship P (Ỹ |Y ) can be successfully identified with the anchor point assumption [15, 34, 14]. However, in the real-world scenarios, this assumption may be hard to satisfied. Although P (Ỹ |Y ) can be used to approximate P (Ỹ |Y,X), the approximation error can be large in many cases. As for the efforts to model the instance-dependent transition matrix directly, existing methods rely on rather strong assumptions, e.g., the bounded noise rate assumption [7], the part-dependent label noise assumption [30], and the requirement of additional information about the transition matrix [4]. Although the assumptions help the methods achieve superior performance empirically, they are generally difficult to verify or fulfill, limiting their applications in practice.
Structural Causal Model Motivated by the limitation of the current methods, we provide a new causal perspective to learn the identifiable of instance-dependent label noise model. Here we briefly
introduce some background knowledge of causality [25] used in this paper. A structural causal model (SCM) consists of a set of variables connected by a set of functions. It represents a flow of information and reveals causal relationships among all the variables, providing a fine-grained description of the data generation process. The causal structure encoded by SCMs can be represented as a graphical casual model as shown in Fig. 1, where each node is a variable and each edge is a function involving noise. The SCM corresponding to the graph in Fig. 1 can be written as
Z = Z , Y = Y , X = f(Z, Y, X), Ỹ = f(X,Y, Ỹ ), (1) where Z , Y , X and Ỹ are independent exogenous variables, and they sometimes also called error variables. For example, X are an error variable for X , which is responsible for any difference between the actual value of X and the value predicted on the basis of Z and Y alone. Each equation specifies a distribution of a variable conditioned on its parents (which are an empty set for root cause variables in the graph).
By making use of the SCM, the benefit of the instances to learning the classifier can be clearly explained. Specifically, the instance X is a function of its label Y and latent feature Z, which means that the instance X is generated from Y and Z. Therefore X must contains information about its clean label Y and latent feature Z. That is the reason that P (X) can help identify P (Y |X) and also P (Z|X). However, since we do not have clean labels, it is hard to fully identify P (Y |X) from P (X) in the unsupervised setting. For example, on the MOON dataset shown in Fig. 2 , we can possibly discover the two clusters by enforcing the manifold constraint, but it is impossible to see which class each cluster belongs to. We show in the following that we can make use of the property of P (X|Y ) to help model label noise, i.e., encourage the identifiability of the transition relationship, thereby learning a better classifier.
Specifically, under the Markov condition [20], which intuitively means the independence of exogenous variables, the joint distribution P (Ỹ , X, Y, Z) specified by the SCM can be factorized as follows.
P (X, Ỹ , Y, Z) = P (Y )P (Z)P (X|Y,Z)P (Ỹ |Y,X). (2) This motivates us to extend VAE [12] to perform inference in our causal model to fit the noisy data in the next section. In the decoder phase, given the noisy data and the distributions of Z and Y , adding a constraint on P (X|Y,Z) will reduce the uncertainty in the distribution P (Ỹ |Y,X). In other words, modeling of P (X|Y,Z) will encourage the identifiability of the transition relationship and thus better model label noise. Since P (Ỹ |Y,X) functions as a bridge to connect the noisy labels to clean labels, we accordingly can better learn P (Y |X) or the classifier by only using the noisy data. There are normally two ways to add constraints on the instances, i.e., assuming a specific parametric generative model or introducing prior knowledge of the instances. In this paper, since we mainly study the image classification problem with noisy labels, we focus on the manifold property of images and apply the low-dimensional manifold constraint to the instances.

heading: 3 Causality Captured Instance-Dependent Label-Noise Learning
text: In this section, we propose a structural generative method which captures the causal relationship and utilizes P (X) to help identify the label-noise transition matrix, and therefore, our method leads to a better classifier that assigns more accurate labels.

heading: 3.1 Variational Inference under the Structural Causal Model
text: To model the generation process of noisy data and to approximate the distribution of the noisy data, our method is designed to follow the causal factorization (see Eq. 2). Specifically, our model contains two decoder networks which jointly model a distribution pθ(X, Ỹ |Y,Z) and two encoder (inference) networks which jointly model the posterior distribution qφ(Z, Y |X). Here we discuss each component of our model in detail.
Let the two decoder networks model the distributions pθ1(X|Y, Z) and pθ2(Ỹ |Y,X), respectively. Let θ1 and θ2 be learnable parameters of the distributions. Without loss of generality, we set p(Z) to a standard normal distribution and p(Y ) to a uniform distribution. Then, modeling the joint distribution in Eq. 2 boils down to modeling the distribution pθ(X, Ỹ |Y, Z), which is decomposed as follows:
pθ(X, Ỹ |Y, Z) = pθ1(X|Y,Z)pθ2(Ỹ |Y,X). (3)
To infer latent variables Z and Y with only observable variables X and Ỹ , we design an inference network which model the variational distribution qφ(Z, Y |Ỹ , X). Specifically, let qφ2(Z|Y,X) and qφ1(Y |Ỹ , X) be the distributions parameterized by learnable parameters φ1 and φ2, and then the posterior distribution can be decomposed as follows:
qφ(Z, Y |Ỹ , X) = qφ2(Z|Y,X)qφ1(Y |Ỹ , X), (4)
where we do not include Ỹ as a conditioning variable in qφ2(Z|Y,X) because the causal graph implies Z ⊥ Ỹ |X,Y . One problem with this posterior form is that we cannot directly employ qφ1(Y |Ỹ , X) to predict labels on the test data, on which Ỹ is absent. To reduce computational costs and to allow our method efficiently infer clean labels, we approximate qφ1(Y |Ỹ , X) by assuming that given the instance X , the clean label Y is conditionally independent from the noisy label Ỹ , i.e., qφ1(Y |Ỹ , X) = qφ1(Y |X). This approximation is expected not to have very large approximation error because the images contain sufficient information to predict the clean labels. Thus, we could simplify Eq. 4 as follows
qφ(Z, Y |X) = qφ2(Z|Y,X)qφ1(Y |X), (5)
such that our encoder networks model qφ2(Z|Y,X) and qφ1(Y |X), respectively. This way, qφ1(Y |X) can be used to infer clean labels efficiently.2 We also found that the encoder network modelling qφ1(Y |X) can directly act as a regularizer, which helps to identify pθ2(Ỹ |Y,X). Moreover, be benefited from this, our method can serve as a general framework which can be easily integrated with the current discriminative label-noise methods [28, 16, 10], and we will showcase it by collaborating co-teaching [10] with our method.
Optimization of Parameters Because the marginal distribution pθ(X, Ỹ ) is usually intractable, to learn the set of parameters {θ1, θ2, φ1, φ2} given only noisy data, we follow the variational inference framework [5] to maximize the negative evidence lower-bound ELBO(x, ỹ) of the marginal likelihood of each datapoint (x, ỹ) instead of maximizing the marginal likelihood itself. By ensembling our decoder and encoder networks, ELBO(x, ỹ) is derived as follows:
ELBO(x, ỹ) = E(z,y)∼qφ(Z,Y |x) [log pθ1(x|y, z)] + Ey∼qφ1 (Y |x) [log pθ2(ỹ|y, x)] − kl(qφ1(Y |x)‖p(Y ))− Ey∼qφ1 (Y |x) [kl(qφ(Z|y, x)‖p(Z))] , (6)
where kl(·) is the Kullback–Leibler divergence between two distributions. The derivation details are left out in Appendix A. Our model learns the class-conditional distribution P (X|Y ) by maximizing the first expectation in ELBO, which is equivalent to minimizing the reconstruction loss [12]. By learning P (X), the inference network qφ1(Y |X) has to select a suitable parameter φ∗ which samples the y and z to minimize the reconstruction loss E(z,y)∼qφ(Z,Y |x) [log pθ1(x|y, z)]. When the dimension of Z is chosen to be much smaller than that of X , to obtain a smaller reconstruction error, the decoder has to utilize the information provided by Y , and force the value of Y to be useful for
2Theoretically, this approximation is not necessary and can be easily removed. Specifically, we could let an encoder model qφ1(Y |Ỹ , X). Then q(Y |X) is obtained as q(Y |X) = ∑ i qφ1(Y |Ỹ = i,X)qφ3(Ỹ = i|X), where qφ3(Ỹ |X) is the noisy class-posterior distribution and can be learned by using noisy training data.
Algorithm 1 CausalNL Input: A noisy sample S, Average noise rate ρ, Total epoch Tmax, Batch size N .
1: For T = 1, . . . , Tmax: 2: For mini-batch S̄ = {xi}Ni=0, L̃ = {ỹi}Ni=0 in S: 3: Feed S̄ to encoders q̂φ11 and q̂φ21 to get clean label sets L1 and L2, respectively; 4: Feed (S̄, L1) to encoder q̂φ12 to get a representation set H1, feed (S̄, L2) to q̂φ22 to get H2; 5: Update q̂φ12 and q̂φ22 with co-teaching loss; 6: Feed (L1, H1) to decoder p̂θ11 to get reconstructed dataset S̄1, feed (L2, H2) to p̂θ21 to get S̄2; 7: Feed (S̄1, L1) to decoder p̂θ12 to get predicted noisy labels L̃1, feed (S̄2, L2) to p̂θ22 to get L̃2; 8: Update networks q̂φ11 , q̂φ12 , p̂θ11 and p̂θ12 by calculating ELBO on (S̄, S̄1, L̃, L̃1), update
networks q̂φ21 , q̂φ22 , p̂θ21 and p̂θ22 by calculating ELBO on (S̄, S̄2, L̃, L̃2); Output: The inference network q̂φ11 .
prediction. Furthermore, we constrain the Y to be a one-hot vector, and then Y could be a cluster ID of which the manifold of the X belongs.
So far, the latent variable Y can be inferred as a cluster ID instead of a clean class ID. To further link the clusters to clean labels, a naive approach is to select some reliable examples and keep the cluster numbers to be consistent with the noisy labels on these examples. In such a way, the latent representation Z and clean label Y can be effectively inferred, and therefore, it encourages the identifiability of the transition relationship pθ2(Ỹ |Y,X). To achieve this, instead of explicitly selecting the reliable example in advance, our method is trained end-to-end, i.e., reliable examples are selected dynamically during the update of parameters of our model by using the co-teaching technique [10]. The advantage of this approach is that the selection bias of the reliable example [6] can be greatly reduced. Intuitively, the accurately selected reliable examples can encourage the identifiability of pθ2(Ỹ |Y,X) and pθ1(X|Y, Z), and the accurately estimated pθ2(Ỹ |Y,X) and pθ1(X|Y,Z) will encourage the network to select more reliable examples.

heading: 3.2 Practical Implementation
text: Our method is summarized in Algorithm 1 and illustrated in Fig. 3. Here we introduce the structure of our model and loss functions.
Model Structure Because we incorporate co-teaching in our model training, we need to add a copy of the decoder and encoders in our method. As the two branches share the same architectures, we first present the details of the first branch and then briefly introduce the second branch.
For the first branch, we need a set of encoders and decoders to model the distributions in Eq. 3 and 5. Specifically, we have two encoder networks
Y1 = q̂φ11(X), Z1 ∼ q̂φ12(X,Y1)
for Eq. 5 and two decoder networks
X1 = p̂θ11 (Y1, Z1), Ỹ1 = p̂θ12 (X1, Y1)
for Eq. 3. The first encoder q̂φ11(X) takes an instance X as input q̂φ11(X) and output a predicted clean label Y1. The second encoder q̂φ12(X,Y1) takes both the instance X and the generated label Y1 as input and outputs a latent feature Z1. Then the generated Y1 and Z1 are passed through the decoder p̂θ11 (Y1, Z1) which will generate a reconstructed image X1. Finally, the generated X1 and Y1 are the input to another decoder p̂θ12 (X1, Y1) which returns predicted noisy labels Ỹ1. It is worth mentioning that the reparameterization trick [12] is used for sampling, so as to allow backpropagation in q̂φ12(X,Y1).
Similarly, the encoder and decoder networks in the second branch are defined as follows:
Y2 = q̂φ21(X), Z2 ∼ q̂φ22(X,Y2), X2 = p̂θ21 (Y2, Z2), Ỹ2 = p̂θ22 (X2, Y2).
During training, we let two encoders q̂φ11(X) and q̂φ21(X) teach each other given every mini-batch.
Loss Functions We divide the loss functions into two parts. The first part is the negative ELBO in Eq. 6, and the second part is a co-teaching loss. The detailed formulation is leaved in Appendix B.
For the negative ELBO, the first term−E(z,y)∼qφ(Z,Y |x) [log pθ1(x|y, z)] is a reconstruction loss, and we use the `1 loss for reconstruction. The second term is −Ey∼qφ1 (Y |x) [log pθ2(ỹ|y, x)], which aims to learn noisy labels given inference y and x, and can be simply replaced by the cross-entropy loss on outputs of both decoders p̂θ12 (X1, Y1) and p̂θ22 (X2, Y2) with the noisy labels contained in the training data. The additional two terms are two regularizers. To calculate kl(qφ1(Y |x)‖p(Y )), we assume that the prior P (Y ) is a uniform distribution. Then minimizing kl(qφ1(Y |x)‖p(Y )) is equivalent to maximizing the entropy of qφ1(Y |x) for each instance x, i.e., − ∑ y qφ1(y|x) log qφ1(y|x). The benefit for having this term is that it could reduce the overfiting problem of the inference network. For Ey∼qφ1 (Y |x) [kl(qφ(Z|y, x)‖p(Z))], we let p(Z) be a standard multivariate Gaussian distribution. Empirically, qφ(Z|y, x) is modeled by the encoders q̂φ11(X) and q̂φ21(X) which are designed to be deterministic mappings, therefore, the expectation can be removed, and only the kl term kl(qφ(Z|y, x)‖p(Z)) is left. When p(Z) is a Gaussian distribution, the kl term nicely has a closed form solution [12], i.e., − 12 ∑J j=1(1 + log((σj)
2)− (µj)2 − (σj)2), where J is the dimension of a latent representation z, and σj and µj are the encoder outputs.
For the co-teaching loss, we follow the work of Han et al. [10]. Intuitively, two encoders q̂φ11(X) and q̂φ21(X) feed all data forward and selects some data of possibly clean labels. Then, two networks communicate with each other to select possible clean data in this mini-batch and use them for training. Finally, each encoder backpropagates over the data selected by its peer network and updates itself by cross-entropy loss.

heading: 4 Experiments
text: In this section, we compare the classification accuracy of proposed method with popular label-noise learning algorithms [15, 19, 11, 10, 28, 36, 16] on both synthetic and real-world datasets.

heading: 4.1 Experimental Setup
text: Datasets We examine the efficacy of our approach on manually corrupted versions of four datasets, i.e., FashionMNIST [31], SVHN [18], CIFAR10, CIFAR100 [13], and one real-world noisy dataset, i.e., Clothing1M [32]. FashionMNIST contains 60,000 training images and 10,000 test images with 10 classes; SVHN contains 73,257 training images and 26,032 test images with 10 classes. CIFAR10 contains 50,000 training images and 10,000 test images. CIFAR10 and CIFAR100 both contain 50,000 training images and 10,000 test images but the former has 10 classes of images, and the latter has 10 classes of images. The four datasets contain clean data. We add instance-dependent label noise to the training sets manually according to Xia et al. [30]. Clothing1M has 1M images with real-world noisy labels and 10k images with clean labels for testing. For all the synthetic noisy datasets, the experiments are repeated 5 times.
Network structure and optimization For a fair comparison, all experiments are conducted on NVIDIA Tesla V100, and all methods are implemented by PyTorch. Dimension of the latent representation Z is set to 25 for all synthetic noisy datasets. For encoder networks q̂φ11(X) and q̂φ21(X), we use the same network structures with the baseline method. Specially, we use a ResNet-18 network for FashionMNIST, a ResNet-34 network for SVHN and CIFAR10, a ResNet-50 network for CIFAR100 without pretraining. For Clothing1M, we use ResNet-50 networks pre-trained on ImageNet. The data-augmentation methods random crop and horizontal flip are used for our method. For Clothing1M, we use a ResNet-50 network pre-trained on ImageNet, and the clean training data is not used. The dimensionality of the latent representation Z is set to 100. Due to limited space, we leave the detailed structure of other decoders and encoders in Appendix C.
Baselines and measurements We compare the proposed method with the following state-of-the-art approaches: (i). CE, which trains the standard deep network with the cross-entropy loss on noisy datasets. (ii). Decoupling [16], which trains two networks on samples whose predictions from the two networks are different. (iii). MentorNet [11] and Co-teaching [10], which mainly handle noisy labels
by training on instances with small loss values. (iv). Forward [19], Reweight [15], and T-Revision [28]. These approaches utilize a class-dependent transition matrix T to correct the loss function. For these baselines, we follow the experiments settings of the original papers. We report average test accuracy on over the last ten epochs of each model on the clean test set. Higher classification accuracy means that the algorithm is more robust to the label noise.

heading: 4.2 Classification Accuracy Evaluation
text: Results on synthetic noisy datasets Tables 1, 2, 3, and 4 report the classification accuracy on the datasets of F-MNIST, SVHN, CIFAR-10, and CIFAR100, respectively. The synthetic experiments reveal that our method is powerful in handling instance-dependent label noise particularly in the situation of high noise rates. Specifically, for all datasets, the classification accuracy of our method
decrease much slower than that of baseline methods. Additionally, the classification accuracies on these datasets are improved by using CausalNL, which implies that our method should capture the underlying data generation process, and then Y should be a cause of X for all these datasets.
For noisy F-MNIST, SVHN and CIFAR-10, in the easy case IDN-20%, almost all methods work well. When the noise rate is 30%, the advantages of causalNL begin to show. We surpassed all methods obviously. When the noise rate raises, all the baselines are gradually defeated. Finally, in the hardest case, i.e., IDN-50%, the superiority of causalNL widens the gap of performance. The classification accuracy of causalNL is at least over 10% higher than the best baseline method. For noisy CIFAR-100, none of the methods works well. However, causalNL still overtakes the other methods with clear gaps for all different levels of noise rate.
Results on the real-world noisy dataset On the real-world noisy dataset Clothing1M, our method causalNL outperforms all the baselines, as shown in Table 5. The experimental results also show that the noise type in Clothing1M is more likely to be instance-dependent label noise, suggesting that the instance-independent assumption on the transition matrix sometimes can be strong.

heading: 5 Conclusion
text: In this paper, we have investigated how to use P (X) to help learn instance-dependent label noise. Specifically, previous assumptions are made on the transition matrix, and the assumptions are hard to be verified and might be violated on real-world datasets. From inspired by a causal perspective, when Y is a cause of X , then P (X) should contain useful information to infer the clean label Y . We propose a novel generative approach called causalNL for instance-dependent label-noise learning. Our model makes use of the causal graph to contribute to the identifiability of the transition matrix, and therefore helps learn clean labels. In order to learn P (X), compared to the previous methods, our method contains more parameters. But experiments on both synthetic and real-world noisy datasets show that a bit sacrifice on computational efficiency is worth it, i.e., the classification accuracy of casualNL significantly outperforms all the state-of-the-art methods. Additionally, the results also indicates that in classification problems, Y can usually be considered as a cause of X , and suggests that the understanding and modeling of the data generation process can help leverage additional information that is useful in solving advanced machine learning problems concerning the relationship between different modules of the data joint distribution. In our future work, we will study the theoretical properties of our method and establish identifiability results under certain assumptions on the data-generative process.

heading: Acknowledgments
text: TL was partially supported by Australian Research Council Projects DP-180103424, DE-190101473, and IC-190100031. GM was supported by Australian Research Council Project DE210101624. BH was supported by supported by the RGC Early Career Scheme No. 22200720 and NSFC Young Scientists Fund No. 62006202. GN was supported by JST AIP Acceleration Research Grant Number JPMJCR20U3, Japan. KZ was supported in part by the National Institutes of Health (NIH) under Contract R01HL159805, by the United States Air Force under Contract No. FA8650-17-C7715, by the NSF-Convergence Accelerator Track-D award #2134901, and by a grant from Apple. The NIH or NSF is not responsible for the views reported in this article. The authors thank the reviewers and the meta-reviewer for their helpful and constructive comments on this work.

Review summaries:
In this paper, the authors handle the label noise problem by exploiting the causal information, which is a new perspective. The setting of instance-dependent label noise learning is more general and realistic in real life. The motivation of this paper is strong. The proposed method also has a good empirical performance and can be easily integrated into current methods.