Approximating Instance-Dependent Noise via Instance-Confidence Embedding

1 INTRODUCTION . In modern machine learning , large-scale data has become indispensable ( Russakovsky et al. , 2015 ; Wang et al. , 2019a ) . A prevalent approach to collecting large-scale labeled datasets is to use imperfect sources such as crowdsourcing and web crawling ( Fergus et al. , 2005 ; Schroff et al. , 2010 ; Wang et al. , 2019a ) , which is usually less expensive and time-consuming than manual annotation by domain experts . However , such methods inevitably introduce label noise that may lead to overfitting and hurt the generalization of deep models ( Arpit et al. , 2017 ; Zhang et al. , 2017 ) . In such situations , it is often beneficial to ( i ) remove mislabeled data or abstain from using confusing instances ( Hara et al. , 2019 ; Thulasidasan et al. , 2019 ) ; ( ii ) increase robustness and reduce harmful influences of noisy labels ( Malach & Shalev-Shwartz , 2017 ; Mirzasoleiman et al. , 2020 ; Liu et al. , 2020 ) ; or ( iii ) explicitly model the transition from the unobservable true label to the noisy observation ( Goldberger & Ben-Reuven , 2017 ; Patrini et al. , 2017 ; Xia et al. , 2020 ) . In this work , we focus on explicit modeling of the label corruption process , which is model-agnostic and data-efficient . Most existing studies in this direction employ the class-conditional noise ( CCN ) assumption , i.e. , the noisy label is independent of the input feature given the true label ( Angluin & Laird , 1988 ; Natarajan et al. , 2013 ; Patrini et al. , 2017 ) . However , this assumption could be too strong to fit some real-world data well ( Xiao et al. , 2015 ; Chen et al. , 2021 ; Liu , 2021 ) . More importantly , CCN only captures the general label flipping patterns between classes for all instances . In applications such as data cleansing and human-in-the-loop interaction , instance-specific noise information itself could be of central interest . This urges us to consider not only the class-conditional noise pattern but also the instance-specific noise modeling . To handle this problem , in this work , we study the instance-dependent noise ( IDN ) model , where the noisy label also depends on the input . Several methods have been reported in the literature , but they either only focus on binary classification under strong assumptions ( Menon et al. , 2018 ; Cheng et al. , 2020 ) , are based on domain-specific knowledge ( Xia et al. , 2020 ) , or need extra supervision ( Berthon et al. , 2021 ) . In contrast , we propose a simple domain-agnostic approximation method for the multiclass IDN model , referred to as instance-confidence embedding ( ICE ) . Concretely , to avoid estimating the noise transition matrix for each instance , we propose a variational approximation that uses a scalar confidence parameter ( Section 3.2 ) . Then , we suggest using instance embedding that assigns a trainable parameter to each instance because the mapping from the instance to its confidence value could be non-smooth and is usually not required to generalize to unseen examples ( Section 3.3 ) . Lastly , we show the effectiveness of the proposed method and its ability to detect ambiguous or mislabeled instances through experiments on various image and text classification tasks ( Section 5 ) . 2 PROBLEM : INSTANCE-DEPENDENT NOISE . In this section , we give a brief overview of learning with instance-dependent noise ( IDN ) . 2.1 NOTATION . Consider a K-class classification problem , where X ∈ X is the input feature and Y ∈ { 1 , . . . , K } is the unobservable categorical true label . We assume that the clean class-posterior p ( Y |X ) comes from a parametric family of distributions : pφ ( Y |X ) : = Categorical ( Y |p = f ( X ; φ ) ) , ( 1 ) where p ∈ ∆K−1 is the probability parameter for Y in the ( K − 1 ) -dimensional probability simplex ∆K−1 , and f : X → ∆K−1 is a differentiable function parameterized by φ that maps the feature X to its corresponding probability parameter p. Then , let Ỹ ∈ { 1 , . . . , K } be the noisy label . The goal is to predict Y from X based on a finite i.i.d . sample of ( X , Ỹ ) -pairs . 2.2 DEPENDENCE . Next , we introduce the dependence structure between X , Y , and Ỹ , which characterize different noise models . The graphical representations of noise models are illustrated in Fig . 1 . In IDN , we assume that the joint distribution of X , Y , and Ỹ can be factorized as follows : p ( X , Y , Ỹ ) = p ( Ỹ |Y , X ) pφ ( Y |X ) p ( X ) . ( 2 ) That is , the noisy label Ỹ depends on both the instance X and the true label Y . Then , the noisy class-posterior p ( Ỹ |X ) can be obtained by marginalizing p ( Y , Ỹ |X ) over Y : pφ ( Ỹ |X ) : = Categorical ( Ỹ |q = ∑K Y=1 p ( Ỹ |Y , X ) pφ ( Y |X ) ) , ( 3 ) where q ∈ ∆K−1 denotes the probability parameter for Ỹ . Note that p ( Ỹ |Y , X ) plays a central role in IDN . Since both Y and Ỹ are categorical random variables , for a certain instance x , p ( Ỹ |Y , X = x ) can be seen as a K ×K stochastic matrix T ( x ) , whose elements are Tij ( x ) : = p ( Ỹ = j|Y = i , X = x ) for i , j ∈ { 1 , . . . , K } . Conventionally , T ( x ) is called a noise transition matrix ( Patrini et al. , 2017 ) . Then , p ( Ỹ |Y , X ) can be regarded as a matrix-valued function T : X → [ 0 , 1 ] K×K that maps each instance x to its corresponding IDN transition matrix T ( x ) . Without any restriction , we need K ×K parameters for each instance x . 2.3 APPROACH . Owing to its complexity , IDN has only been studied to a limited extent but is of great interest recently . A straightforward method is to jointly estimate the matrix-valued function T ( x ) as well as the clean class-posterior pφ ( Y |X ) using neural networks ( Goldberger & Ben-Reuven , 2017 ) . However , the estimation error of T ( x ) could be high , which deteriorates the classification performance . Another direction is to restrict the problem under certain conditions , so that we can provide theoretical guarantees ( Menon et al. , 2018 ; Cheng et al. , 2020 ) . However , existing work mainly focused on binary classification . A promising approach is to approximate IDN using a simpler dependence structure , such as a mixture of noises with different semantic meanings ( Xiao et al. , 2015 ) or a weighted combination of noises that depend on parts of the instance ( Xia et al. , 2020 ) . In this work , we also suggest that it might be unnecessary to obtain a K ×K matrix for each instance x : Note that pφ ( Ỹ |x ) can be seen as a linear combination of columns of T ( x ) weighted by pφ ( Y |x ) ; If the maximum value of pφ ( Y |x ) is close to 1 , i.e. , the label of the instance x is almost deterministic , the estimation of K − 1 columns of T ( x ) has only limited influence on the estimated noisy class-posterior p̂ ( Ỹ |x ) . This suggests the possibility of using a relatively simple model to approximate p ( Ỹ |X ) in real-world applications . In this work , we consider a single-parameter approximation for each instance , which is introduced in Section 3.2 and illustrated in Fig . 2 . Another issue is that existing methods still introduce some level of smoothness w.r.t . x into T ( x ) ( Goldberger & Ben-Reuven , 2017 ; Xiao et al. , 2015 ; Xia et al. , 2020 ) . In real-world problems , however , we can only access a finite sample of ( X , Ỹ ) -pairs that are possibly annotated by nonexperts or web crawlers ( Fergus et al. , 2005 ) . Thus , the label noise could be aleatory and T ( x ) could vary significantly for two adjacent instances . Also , the classifier pφ ( Y |X ) is desired but the generalization of T ( x ) to unseen examples is usually dispensable . This inspires us to use instance embedding instead of neural network approximation , which is discussed in Section 3.3 and demonstrated in Fig . 3 . 3 PROPOSED METHOD . In this section , we present our proposed method , instance-confidence embedding ( ICE ) . 3.1 VARIATIONAL LOWER BOUND . Note that T ( x ) serves as a linear mapping from p to q ( Eq . ( 3 ) ) . Due to the difficulty of estimating the matrix-valued function T ( x ) , we use a simpler function qθ , φ ( Ỹ |X ) parameterized by θ as a variational approximation to pφ ( Ỹ |X ) . The choice of the approximation family is discussed in Section 3.2 . Then , let us consider the expected log-likelihood as the learning objective , which can be rewritten as E Ỹ∼p ( Ỹ |X ) [ log p ( Ỹ |X ) ] = DKL ( pφ ( Ỹ |X ) ‖ qθ , φ ( Ỹ |X ) ) + L ( θ , φ ; X ) , ( 4 ) where DKL denotes the Kullback-Leibler ( KL ) divergence , and the second term is L ( θ , φ ; X ) : = E Ỹ∼p ( Ỹ |X ) [ log qθ , φ ( Ỹ |X ) ] . ( 5 ) Since the KL-divergence is always non-negative , this term gives a variational lower bound of the expected log-likelihood . Then , we have the following learning objective to maximize : L ( θ , φ ) : = E X∼p ( X ) [ L ( θ , φ ; X ) ] = E X , Ỹ∼p ( X , Ỹ ) [ log qθ , φ ( Ỹ |X ) ] . ( 6 ) In practice , the expectation can be approximated using the empirical distribution based on a finite i.i.d . sample of ( X , Ỹ ) -pairs . Clean Linear interpolation Power transformation Noisy 3.2 VARIATIONAL APPROXIMATION . Next , we discuss the choice of the variational approximation family of qθ , φ ( Ỹ |X ) . To approximate the effect of multiplying an IDN transition matrix T ( x ) that requires K × K parameters for each instance x , in this work , we use a simpler transformation from p to q , which is not necessarily linear . Compared with estimating a full matrix for each instance without any restriction ( Goldberger & Ben-Reuven , 2017 ) , obtaining only an approximation may cause higher approximation error , but on the other hand , it may reduce estimation error and thus improve the classification performance . The high estimation error of complex models might be more harmful , which is empirically validated in Section 5 . It is also the case when using CCN as an approximation of IDN to balance this trade-off . The difference is that CCN obtains a complete transition matrix common to all instances , but ICE aims to obtain an approximated trend for each instance , which gives useful instance-specific noise information . Then , we suggest to use a single-scalar parameter C ∈ [ 0 , 1 ] for each instance to control this approximation , which is useful for sorting and comparing training examples . This design might also be useful in data cleansing , learning with rejection , or active learning ( Hara et al. , 2019 ; Thulasidasan et al. , 2019 ; Charoenphakdee et al. , 2021 ) . This parameter is referred to as the confidence and is obtained via a function g : X → [ 0 , 1 ] parameterized by θ , i.e. , C = g ( X ; θ ) . Then , θ can be regarded as the collection of C for all instances . The confidence C plays a central role in our method , where C = 0 means that the instance is ambiguous or mislabeled and thus the classifier should not give a confident prediction . Finally , we need to design a transformation from p to q parameterized by the confidence C. We denote this function by h : ∆K−1 → ∆K−1 . In summary , qφ , θ ( Ỹ |X ) takes the following form : qφ , θ ( Ỹ |X ) : = Categorical ( Ỹ |q = h ( f ( X ; φ ) ; g ( X ; θ ) ) ) . ( 7 ) Next , we analyze what characteristics h needs to have . First , we suggest that h needs not necessarily to be a linear transformation because the transformation is instance-dependent and any function that maps p to q as close as possible for a certain instance x would suffice . Second , we require that arg max ( p ) = arg max ( q ) , i.e. , q = h ( p ; C ) should be an argmax-preserving function so that the top-1 index of the probability vector does not change . This is because h should only affect the confidence of the prediction , not the final decision . Otherwise , if h is too flexible and is able to map a confident prediction to a different confident prediction , then the output of f could be arbitrary , and consequently , no information of the true label can be learned from the noisy label supervision . Based on this motivation and the aforementioned semantics of C , we require that h ( p ; 1 ) = p and h ( p ; 0 ) = u , where u ∈ ∆K−1 is the uniform probability vector ( ui = 1K ) . Then , when the confidence C is high , the classifier gives a prediction closer to the original confident prediction p ; and when the confidence C is low , the classifier tends to give a random guess u . Here , we propose two functions for h that satisfy the above conditions : qi = Cpi + ( 1− C ) ui , ( linear interpolation ) ( 8 ) qi = pCi∑K j=1 p C j ∝ pCi , ( power transformation ) ( 9 ) for i = 1 , . . . , K. The visualization of these two transformations for K = 3 is given in Fig . 2 .

Summary: This paper studies instance-dependent noise (IDN) problems. It proposes a single-parameter (confidence) model for the noise generating process. Moreover, *instance-confidence embedding* (ICE) method is employed in the training process. The experimental results on various image and text classification tasks confirm the effectiveness of the proposed method.