Approximating Instance-Dependent Noise via Instance-Confidence Embedding

1 INTRODUCTION . In modern machine learning , large-scale data has become indispensable ( Russakovsky et al. , 2015 ; Wang et al. , 2019a ) . A prevalent approach to collecting large-scale labeled datasets is to use imperfect sources such as crowdsourcing and web crawling ( Fergus et al. , 2005 ; Schroff et al. , 2010 ; Wang et al. , 2019a ) , which is usually less expensive and time-consuming than manual annotation by domain experts . However , such methods inevitably introduce label noise that may lead to overfitting and hurt the generalization of deep models ( Arpit et al. , 2017 ; Zhang et al. , 2017 ) . In such situations , it is often beneficial to ( i ) remove mislabeled data or abstain from using confusing instances ( Hara et al. , 2019 ; Thulasidasan et al. , 2019 ) ; ( ii ) increase robustness and reduce harmful influences of noisy labels ( Malach & Shalev-Shwartz , 2017 ; Mirzasoleiman et al. , 2020 ; Liu et al. , 2020 ) ; or ( iii ) explicitly model the transition from the unobservable true label to the noisy observation ( Goldberger & Ben-Reuven , 2017 ; Patrini et al. , 2017 ; Xia et al. , 2020 ) . In this work , we focus on explicit modeling of the label corruption process , which is model-agnostic and data-efficient . Most existing studies in this direction employ the class-conditional noise ( CCN ) assumption , i.e. , the noisy label is independent of the input feature given the true label ( Angluin & Laird , 1988 ; Natarajan et al. , 2013 ; Patrini et al. , 2017 ) . However , this assumption could be too strong to fit some real-world data well ( Xiao et al. , 2015 ; Chen et al. , 2021 ; Liu , 2021 ) . More importantly , CCN only captures the general label flipping patterns between classes for all instances . In applications such as data cleansing and human-in-the-loop interaction , instance-specific noise information itself could be of central interest . This urges us to consider not only the class-conditional noise pattern but also the instance-specific noise modeling . To handle this problem , in this work , we study the instance-dependent noise ( IDN ) model , where the noisy label also depends on the input . Several methods have been reported in the literature , but they either only focus on binary classification under strong assumptions ( Menon et al. , 2018 ; Cheng et al. , 2020 ) , are based on domain-specific knowledge ( Xia et al. , 2020 ) , or need extra supervision ( Berthon et al. , 2021 ) . In contrast , we propose a simple domain-agnostic approximation method for the multiclass IDN model , referred to as instance-confidence embedding ( ICE ) . Concretely , to avoid estimating the noise transition matrix for each instance , we propose a variational approximation that uses a scalar confidence parameter ( Section 3.2 ) . Then , we suggest using instance embedding that assigns a trainable parameter to each instance because the mapping from the instance to its confidence value could be non-smooth and is usually not required to generalize to unseen examples ( Section 3.3 ) . Lastly , we show the effectiveness of the proposed method and its ability to detect ambiguous or mislabeled instances through experiments on various image and text classification tasks ( Section 5 ) . 2 PROBLEM : INSTANCE-DEPENDENT NOISE . In this section , we give a brief overview of learning with instance-dependent noise ( IDN ) . 2.1 NOTATION . Consider a K-class classification problem , where X ∈ X is the input feature and Y ∈ { 1 , . . . , K } is the unobservable categorical true label . We assume that the clean class-posterior p ( Y |X ) comes from a parametric family of distributions : pφ ( Y |X ) : = Categorical ( Y |p = f ( X ; φ ) ) , ( 1 ) where p ∈ ∆K−1 is the probability parameter for Y in the ( K − 1 ) -dimensional probability simplex ∆K−1 , and f : X → ∆K−1 is a differentiable function parameterized by φ that maps the feature X to its corresponding probability parameter p. Then , let Ỹ ∈ { 1 , . . . , K } be the noisy label . The goal is to predict Y from X based on a finite i.i.d . sample of ( X , Ỹ ) -pairs . 2.2 DEPENDENCE . Next , we introduce the dependence structure between X , Y , and Ỹ , which characterize different noise models . The graphical representations of noise models are illustrated in Fig . 1 . In IDN , we assume that the joint distribution of X , Y , and Ỹ can be factorized as follows : p ( X , Y , Ỹ ) = p ( Ỹ |Y , X ) pφ ( Y |X ) p ( X ) . ( 2 ) That is , the noisy label Ỹ depends on both the instance X and the true label Y . Then , the noisy class-posterior p ( Ỹ |X ) can be obtained by marginalizing p ( Y , Ỹ |X ) over Y : pφ ( Ỹ |X ) : = Categorical ( Ỹ |q = ∑K Y=1 p ( Ỹ |Y , X ) pφ ( Y |X ) ) , ( 3 ) where q ∈ ∆K−1 denotes the probability parameter for Ỹ . Note that p ( Ỹ |Y , X ) plays a central role in IDN . Since both Y and Ỹ are categorical random variables , for a certain instance x , p ( Ỹ |Y , X = x ) can be seen as a K ×K stochastic matrix T ( x ) , whose elements are Tij ( x ) : = p ( Ỹ = j|Y = i , X = x ) for i , j ∈ { 1 , . . . , K } . Conventionally , T ( x ) is called a noise transition matrix ( Patrini et al. , 2017 ) . Then , p ( Ỹ |Y , X ) can be regarded as a matrix-valued function T : X → [ 0 , 1 ] K×K that maps each instance x to its corresponding IDN transition matrix T ( x ) . Without any restriction , we need K ×K parameters for each instance x . 2.3 APPROACH . Owing to its complexity , IDN has only been studied to a limited extent but is of great interest recently . A straightforward method is to jointly estimate the matrix-valued function T ( x ) as well as the clean class-posterior pφ ( Y |X ) using neural networks ( Goldberger & Ben-Reuven , 2017 ) . However , the estimation error of T ( x ) could be high , which deteriorates the classification performance . Another direction is to restrict the problem under certain conditions , so that we can provide theoretical guarantees ( Menon et al. , 2018 ; Cheng et al. , 2020 ) . However , existing work mainly focused on binary classification . A promising approach is to approximate IDN using a simpler dependence structure , such as a mixture of noises with different semantic meanings ( Xiao et al. , 2015 ) or a weighted combination of noises that depend on parts of the instance ( Xia et al. , 2020 ) . In this work , we also suggest that it might be unnecessary to obtain a K ×K matrix for each instance x : Note that pφ ( Ỹ |x ) can be seen as a linear combination of columns of T ( x ) weighted by pφ ( Y |x ) ; If the maximum value of pφ ( Y |x ) is close to 1 , i.e. , the label of the instance x is almost deterministic , the estimation of K − 1 columns of T ( x ) has only limited influence on the estimated noisy class-posterior p̂ ( Ỹ |x ) . This suggests the possibility of using a relatively simple model to approximate p ( Ỹ |X ) in real-world applications . In this work , we consider a single-parameter approximation for each instance , which is introduced in Section 3.2 and illustrated in Fig . 2 . Another issue is that existing methods still introduce some level of smoothness w.r.t . x into T ( x ) ( Goldberger & Ben-Reuven , 2017 ; Xiao et al. , 2015 ; Xia et al. , 2020 ) . In real-world problems , however , we can only access a finite sample of ( X , Ỹ ) -pairs that are possibly annotated by nonexperts or web crawlers ( Fergus et al. , 2005 ) . Thus , the label noise could be aleatory and T ( x ) could vary significantly for two adjacent instances . Also , the classifier pφ ( Y |X ) is desired but the generalization of T ( x ) to unseen examples is usually dispensable . This inspires us to use instance embedding instead of neural network approximation , which is discussed in Section 3.3 and demonstrated in Fig . 3 . 3 PROPOSED METHOD . In this section , we present our proposed method , instance-confidence embedding ( ICE ) . 3.1 VARIATIONAL LOWER BOUND . Note that T ( x ) serves as a linear mapping from p to q ( Eq . ( 3 ) ) . Due to the difficulty of estimating the matrix-valued function T ( x ) , we use a simpler function qθ , φ ( Ỹ |X ) parameterized by θ as a variational approximation to pφ ( Ỹ |X ) . The choice of the approximation family is discussed in Section 3.2 . Then , let us consider the expected log-likelihood as the learning objective , which can be rewritten as E Ỹ∼p ( Ỹ |X ) [ log p ( Ỹ |X ) ] = DKL ( pφ ( Ỹ |X ) ‖ qθ , φ ( Ỹ |X ) ) + L ( θ , φ ; X ) , ( 4 ) where DKL denotes the Kullback-Leibler ( KL ) divergence , and the second term is L ( θ , φ ; X ) : = E Ỹ∼p ( Ỹ |X ) [ log qθ , φ ( Ỹ |X ) ] . ( 5 ) Since the KL-divergence is always non-negative , this term gives a variational lower bound of the expected log-likelihood . Then , we have the following learning objective to maximize : L ( θ , φ ) : = E X∼p ( X ) [ L ( θ , φ ; X ) ] = E X , Ỹ∼p ( X , Ỹ ) [ log qθ , φ ( Ỹ |X ) ] . ( 6 ) In practice , the expectation can be approximated using the empirical distribution based on a finite i.i.d . sample of ( X , Ỹ ) -pairs . Clean Linear interpolation Power transformation Noisy 3.2 VARIATIONAL APPROXIMATION . Next , we discuss the choice of the variational approximation family of qθ , φ ( Ỹ |X ) . To approximate the effect of multiplying an IDN transition matrix T ( x ) that requires K × K parameters for each instance x , in this work , we use a simpler transformation from p to q , which is not necessarily linear . Compared with estimating a full matrix for each instance without any restriction ( Goldberger & Ben-Reuven , 2017 ) , obtaining only an approximation may cause higher approximation error , but on the other hand , it may reduce estimation error and thus improve the classification performance . The high estimation error of complex models might be more harmful , which is empirically validated in Section 5 . It is also the case when using CCN as an approximation of IDN to balance this trade-off . The difference is that CCN obtains a complete transition matrix common to all instances , but ICE aims to obtain an approximated trend for each instance , which gives useful instance-specific noise information . Then , we suggest to use a single-scalar parameter C ∈ [ 0 , 1 ] for each instance to control this approximation , which is useful for sorting and comparing training examples . This design might also be useful in data cleansing , learning with rejection , or active learning ( Hara et al. , 2019 ; Thulasidasan et al. , 2019 ; Charoenphakdee et al. , 2021 ) . This parameter is referred to as the confidence and is obtained via a function g : X → [ 0 , 1 ] parameterized by θ , i.e. , C = g ( X ; θ ) . Then , θ can be regarded as the collection of C for all instances . The confidence C plays a central role in our method , where C = 0 means that the instance is ambiguous or mislabeled and thus the classifier should not give a confident prediction . Finally , we need to design a transformation from p to q parameterized by the confidence C. We denote this function by h : ∆K−1 → ∆K−1 . In summary , qφ , θ ( Ỹ |X ) takes the following form : qφ , θ ( Ỹ |X ) : = Categorical ( Ỹ |q = h ( f ( X ; φ ) ; g ( X ; θ ) ) ) . ( 7 ) Next , we analyze what characteristics h needs to have . First , we suggest that h needs not necessarily to be a linear transformation because the transformation is instance-dependent and any function that maps p to q as close as possible for a certain instance x would suffice . Second , we require that arg max ( p ) = arg max ( q ) , i.e. , q = h ( p ; C ) should be an argmax-preserving function so that the top-1 index of the probability vector does not change . This is because h should only affect the confidence of the prediction , not the final decision . Otherwise , if h is too flexible and is able to map a confident prediction to a different confident prediction , then the output of f could be arbitrary , and consequently , no information of the true label can be learned from the noisy label supervision . Based on this motivation and the aforementioned semantics of C , we require that h ( p ; 1 ) = p and h ( p ; 0 ) = u , where u ∈ ∆K−1 is the uniform probability vector ( ui = 1K ) . Then , when the confidence C is high , the classifier gives a prediction closer to the original confident prediction p ; and when the confidence C is low , the classifier tends to give a random guess u . Here , we propose two functions for h that satisfy the above conditions : qi = Cpi + ( 1− C ) ui , ( linear interpolation ) ( 8 ) qi = pCi∑K j=1 p C j ∝ pCi , ( power transformation ) ( 9 ) for i = 1 , . . . , K. The visualization of these two transformations for K = 3 is given in Fig . 2 .

Summary: This paper proposes ICE (Instance-confident embedding) method for learning with IDN (instance-dependent label noise). ICE approximates IDN by using a single-scalar confident parameter for each sample based on the assumption that $P(Y|x)$ is close to a one-hot vector. ICE is incorporated into the loss function for efficient learning process under label noise. Authors conduct experiments on both image and text classification to verify the effectiveness  of ICE and perform gradient analysis for further understanding the method.