Instance-dependent Label-noise Learning under a Structural Causal Model

1 Introduction . Learning with noisy labels can be dated back to [ 1 ] and has recently drawn a lot of attention [ 15 , 19 , 11 , 10 , 28 ] . In real life , large-scale datasets are likely to contain label noise . It is partly because that many cheap but imperfect data collection methods such as crowd-sourcing and web crawling are widely used to build large-scale datasets . Training with such data usually lead to poor generalization abilities of deep neural networks because they can memorize noisy labels [ 2 , 33 ] . To improve the generalization ability of training with noisy labels , one family of existing label-noise learning methods is to model how the label noise was generated [ 17 , 19 , 27 , 34 , 14 ] . Specifically , these methods try to reveal the transition relationship from clean labels to noisy labels of instances , i.e. , the distribution P ( Ỹ |Y , X ) , where Ỹ , Y and X are the random variables for the noisy label , latent clean label , and instance , respectively . The advantage of modelling label noise is that given only the noisy data , when the transition relationship is identifiable , classifiers can be learned to converge to the optimal ones defined by the clean data , with theoretical guarantees . However , the transition relationship is not identifiable in general . To make it identifiable , various assumptions have been made on the transition relationship . For example , Natarajan et al . [ 17 ] assume that the transition relationship is instance independent , i.e. , P ( Ỹ |Y , X ) = P ( Ỹ |Y ) ; Xia et al . [ 30 ] assume that the P ( Ỹ |Y , X ) is dependent on different parts of an instance . Cheng et al . [ 7 ] assume that the label noise rates are upper bounded . In practice , these assumptions may not be satisfied and are generally hard to be verified given noisy data alone . Inspired by causal learning [ 20 , 25 , 21 , 23 ] , we provide a causal perspective of label-noise learning method named CausalNL . We exploit the causal information to help identifiability of the transition matrix P ( Ỹ |Y , X ) other than making assumptions directly on the transition relationship . †Correspondence to Tongliang Liu ( tongliang.liu @ sydney.edu.au ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Specifically , we assume that the data containing instancedependent label noise is generated according to the causal graph in Fig . 1 . For example , for the Street View House Numbers ( SVHN ) dataset [ 18 ] , X represents the image containing the digit ; Y represents the clean label of the digit shown on the plate ; Z represents the latent variable that captures the information affecting the generation of the images , e.g. , orientation , lighting , and font style . Here Y is naturally a cause of X and the causal generative process can be described in the following way . First , the house plate is generated according to the street number and attached to the front door . Then , the house plate is captured by a camera ( installed in a Google street view car ) to form X , taking into account of other factors such as illumination and viewpoint . Finally , the images containing house numbers are collected and relabeled to form the dataset . Let us denote the annotated label by the noisy label Ỹ as the annotator may not be always reliable , especially when the dataset is very large but the budget is limited . During the annotation process , the noisy labels were generated according to both the images and the range of predefined digit numbers . Hence , both X and Y are causes of Ỹ . Note that most existing image datasets are collected with the causal relationship that Y causes X . For example , see the widely used FashionMNIST and CIFAR . When we synthesize instance-dependent label noise based on them , we will have the causal graph illustrated in Fig . 1 . Note also that some datasets are generated with the causal relationship that X causes Y . Other than using domain knowledge , the different causal relationships can be verified by employing causal discovery [ 26 , 25 , 21 , 37 ] . When the latent clean label Y is a cause of X , P ( X ) will generally contain some information about P ( Y |X ) . This is because , under such a generative process , the distributions of P ( X ) and P ( Y |X ) are entangled [ 22 , 39 ] . To help estimate P ( Y |X ) with P ( X ) , we make use of the causal generative process to estimate P ( X|Y ) , which directly benefits from P ( X ) by generative modeling . The modeling of P ( X|Y ) in turn encourages the identifiability of the transition relationship and helps learn P ( Y |X ) . For example , in Fig . 2 ( a ) , we have added instance-dependent label-noise with a rate 45 % ( i.e. , IDLN-45 % ) to the MOON dataset and employed different methods [ 10 , 36 ] to solve the label-noise learning problem . As illustrated in Fig . 2 ( b ) and Fig . 2 ( c ) , previous methods fail to infer clean labels . In contrast , by constraining the conditional distribution of the instances , i.e. , restricting the data of each class to be on a manifold by setting the dimension of the latent variable Z to be 1-dimensional , the label transition as well as the clean labels can be successfully recovered ( by the proposed method ) , which is showed in Fig . 2 ( d ) . It is worth noting that the idea of finding P ( Y |X ) by modeling P ( Y ) and P ( X|Y ) instead has been exploited in the context of domain adaptation ; for instance , it inspires target shift , ( generalized ) conditional shift and other settings for domain adaptation [ 38 , 9 ] Specifically , to make use of the causal graph to contribute to the identifiability of the transition matrix , we propose a causally inspired deep generative method , which models the causal structure with all the observable and latent variables , i.e. , the instance X , the noisy label Ỹ , the latent feature Z , and the latent clean label Y . The proposed generative model captures the variables ’ relationship indicated by the causal graph . Furthermore , built on the variational autoencoder ( VAE ) framework [ 12 ] , we build an inference network which could efficiently infer the latent variables Z and Y when maximising the marginal likelihood p ( X , Ỹ ) on the given noisy data . In the decoder phase , the data will be reconstructed by exploiting the conditional distribution of instances P ( X|Y , Z ) and the transition relationship P ( Ỹ |Y , X ) , i.e. , pθ ( X , Ỹ ) = ∫ z , y P ( Z = z ) P ( Y = y ) pθ1 ( X|Y = y , Z = z ) pθ2 ( Ỹ |Y = y , X ) dzdy will be exploited , where θ : = ( θ1 , θ2 ) are the parameters of the causal generative model ( more details can be found in Section 3 ) . Ay a high level , according to the equation , given the noisy data and the distributions of Z and Y , constraining pθ1 ( X|Y , Z ) will also greatly reduce the uncertainty of pθ2 ( Ỹ |Y , X ) and thus contribute to the identifiability of the transition matrix . Note that adding a constraint on pθ1 ( X|Y , Z ) is natural ; for example , images often have a low-dimensional manifold [ 3 ] . We can restrict P ( Z ) to fulfill the constraint on pθ1 ( X|Y , Z ) . By exploiting the causal structure and the constraint on instances to better model label noise , the proposed method significantly outperforms the baselines . When the label noise rate is large , the superiority is evidenced by a large gain in the classification performance . The rest of the paper is organized as follows . In Section 2 , we briefly review the background knowledge of label-noise learning and causality . In Section 3 , we formulate our method , named CausalNL , and discuss how it helps to learn a clean classifier , followed by the implementation details . Experimental validations are provided in Section 4 . Section 5 concludes the paper . 2 Noisy Labels and Causality . In this section , firstly , we introduce how to model label noise . Then , we introduce the structural causal model and discuss how to exploit the model to encourage the identifiability of the transition relationship and help learn the classifier . Transition Relationship To build a statistically consistent classifier that converges to the optimal classifier defined on clean data by only employing noisy data , the transition relationship P ( Ỹ |Y , X ) has to be identified . Given an instance , the conditional distribution can be written in an C × C matrix which is called the transition matrix [ 19 , 29 , 30 ] , where C represents the number of classes . Specifically , for each instance x , there is a transition matrix T ( x ) . The ij-th entry of the transition matrix is Tij ( x ) = P ( Ỹ = i|Y = j , X = x ) , which represents the probability that the instance x with the clean label Y = j will have a noisy label Ỹ = i . The transition matrix has been widely studied to build statistically consistent classifiers , because the clean class posterior distribution P ( Y |x ) = [ P ( Y = 1|X = x ) , . . . , P ( Y = C|X = x ) ] > can be inferred by using the transition matrix and the noisy class posterior P ( Ỹ |x ) = [ P ( Ỹ = 1|X = x ) , . . . , P ( Ỹ = C|X = x ) ] > , i.e. , we have P ( Ỹ |x ) = T ( x ) P ( Y |x ) . Specifically , the transition matrix has been used to modify loss functions to build risk-consistent estimators ; see , e.g. , [ 8 , 19 , 35 , 28 ] , and has been used to correct hypotheses to build classifier-consistent algorithms ; see , e.g. , [ 17 , 24 , 19 ] . Moreover , the state-of-the-art statically inconsistent algorithms [ 11 , 10 ] also use diagonal entries of the transition matrix to help select reliable examples used for training . However , the distribution P ( Ỹ |Y , X ) is not generally identifiable [ 28 ] . To make it identifiable , one has to resort to additional assumptions . The most widely used assumption is that given clean label Y , the noisy label Ỹ is conditionally independent of instance X , i.e. , P ( Ỹ |Y , X ) = P ( Ỹ |Y ) . Under such an assumption , the transition relationship P ( Ỹ |Y ) can be successfully identified with the anchor point assumption [ 15 , 34 , 14 ] . However , in the real-world scenarios , this assumption may be hard to satisfied . Although P ( Ỹ |Y ) can be used to approximate P ( Ỹ |Y , X ) , the approximation error can be large in many cases . As for the efforts to model the instance-dependent transition matrix directly , existing methods rely on rather strong assumptions , e.g. , the bounded noise rate assumption [ 7 ] , the part-dependent label noise assumption [ 30 ] , and the requirement of additional information about the transition matrix [ 4 ] . Although the assumptions help the methods achieve superior performance empirically , they are generally difficult to verify or fulfill , limiting their applications in practice . Structural Causal Model Motivated by the limitation of the current methods , we provide a new causal perspective to learn the identifiable of instance-dependent label noise model . Here we briefly introduce some background knowledge of causality [ 25 ] used in this paper . A structural causal model ( SCM ) consists of a set of variables connected by a set of functions . It represents a flow of information and reveals causal relationships among all the variables , providing a fine-grained description of the data generation process . The causal structure encoded by SCMs can be represented as a graphical casual model as shown in Fig . 1 , where each node is a variable and each edge is a function involving noise . The SCM corresponding to the graph in Fig . 1 can be written as Z = Z , Y = Y , X = f ( Z , Y , X ) , Ỹ = f ( X , Y , Ỹ ) , ( 1 ) where Z , Y , X and Ỹ are independent exogenous variables , and they sometimes also called error variables . For example , X are an error variable for X , which is responsible for any difference between the actual value of X and the value predicted on the basis of Z and Y alone . Each equation specifies a distribution of a variable conditioned on its parents ( which are an empty set for root cause variables in the graph ) . By making use of the SCM , the benefit of the instances to learning the classifier can be clearly explained . Specifically , the instance X is a function of its label Y and latent feature Z , which means that the instance X is generated from Y and Z . Therefore X must contains information about its clean label Y and latent feature Z . That is the reason that P ( X ) can help identify P ( Y |X ) and also P ( Z|X ) . However , since we do not have clean labels , it is hard to fully identify P ( Y |X ) from P ( X ) in the unsupervised setting . For example , on the MOON dataset shown in Fig . 2 , we can possibly discover the two clusters by enforcing the manifold constraint , but it is impossible to see which class each cluster belongs to . We show in the following that we can make use of the property of P ( X|Y ) to help model label noise , i.e. , encourage the identifiability of the transition relationship , thereby learning a better classifier . Specifically , under the Markov condition [ 20 ] , which intuitively means the independence of exogenous variables , the joint distribution P ( Ỹ , X , Y , Z ) specified by the SCM can be factorized as follows . P ( X , Ỹ , Y , Z ) = P ( Y ) P ( Z ) P ( X|Y , Z ) P ( Ỹ |Y , X ) . ( 2 ) This motivates us to extend VAE [ 12 ] to perform inference in our causal model to fit the noisy data in the next section . In the decoder phase , given the noisy data and the distributions of Z and Y , adding a constraint on P ( X|Y , Z ) will reduce the uncertainty in the distribution P ( Ỹ |Y , X ) . In other words , modeling of P ( X|Y , Z ) will encourage the identifiability of the transition relationship and thus better model label noise . Since P ( Ỹ |Y , X ) functions as a bridge to connect the noisy labels to clean labels , we accordingly can better learn P ( Y |X ) or the classifier by only using the noisy data . There are normally two ways to add constraints on the instances , i.e. , assuming a specific parametric generative model or introducing prior knowledge of the instances . In this paper , since we mainly study the image classification problem with noisy labels , we focus on the manifold property of images and apply the low-dimensional manifold constraint to the instances .

Summary: In this paper, the authors handle the label noise problem by exploiting the causal information, which is a new perspective. The setting of instance-dependent label noise learning is more general and realistic in real life. The motivation of this paper is strong. The proposed method also has a good empirical performance and can be easily integrated into current methods.