Transfer and Marginalize: Explaining Away Label Noise with Privileged Information

1 INTRODUCTION . Supervised learning problems are typically formalized as learning a conditional distribution p ( y|x ) , y ∈ Y and x ∈ X from ( xi , yi ) , i = 1 , ... , N pairs . Yet we often have access to additional features a ∈ A at training time that will not be available at test time . These features are known as privileged information ( Vapnik & Vashist , 2009 ) , or PI for short . An example of PI are features describing the human annotator that provided a given label , such as the annotator ID , the length of time to provide the label , the experience of the annotator , etc . Annotators do not always agree on the correct label for a given x , some annotators may be more reliable than others and the reliability of annotators may depend on the location of x in the input domain X ( Snow et al. , 2008 ; Sheng et al. , 2008 ) . The expanded training dataset consists of ( xi , ai , yi ) triplets . Given that our test time predictive distribution can not be conditioned on a , what use is this PI ? As a thought experiment , suppose there exists a malicious ( or lazy ) annotator that provides random labels . It is known that random labels harm the performance of supervised learning models ( Frénay & Verleysen , 2013 ; Song et al. , 2020 ; Cordeiro & Carneiro , 2020 ) . If these random labels can be explained away via access to PI , such as the annotator ID , then this harm can be prevented . In particular we can use the PI to explain away noise in the labels which otherwise would be irreducible aleatoric uncertainty . More formally , suppose A , the PI random variable , is predictive of Y given X , in the sense that the conditional mutual information I ( Y ; A|X ) is non-zero . Then , the entropy of Y is reduced if we condition on both X and A rather than X alone , as summarised in Lemma 1.1 . Lemma 1.1 . I ( Y ; A|X ) > 0⇒H ( Y |X , A ) < H ( Y |X ) . In §2.1 we examine this lemma for a particular model , proving that under certain conditions , PI can be leveraged to lower the expected risk for linear regression problems . Additionally , prior work has proven that PI can lead to generalization bounds with better sample complexity ( Vapnik & Vashist , 2009 ; Lambert et al. , 2018 ) . Inspired by prior work and our theoretical analysis of a simple linear model , we focus on exploiting PI in supervised deep neural networks . The production deployment of such models often has tight latency and memory constraints . Hence a number of methods have been developed to utilize PI with the same test time memory and computation cost as networks trained without PI ( Yang et al. , 2017 ; Lambert et al. , 2018 ; Lopez-Paz et al. , 2015 ) . Yang et al . ( 2017 ) uses PI as a form of input-dependent regularizer . Lambert et al . ( 2018 ) train with heteroscedastic Gaussian dropout , with the training-time dropout variance a function of the PI . Lopez-Paz et al . ( 2015 ) distill a network trained with PI into a network without access to a . Below we develop a method , TRAM , which transfers knowledge via weight sharing from the part of the network trained using PI to the test time network which does not have access to PI . At test time , TRAM makes a simple , efficient approximation to the integral p ( y|x ) = ∫ p ( y|x , a ) p ( a|x ) da . Making predictions without PI is no more costly than that with a standard network trained without access to PI . Unlike prior work which requires specific techniques such as Gaussian Dropout , we need not constrain the form of the predictors to make the downstream marginalization possible . Implementation and training are simple . In summary the paper contributions are the following : • To better illustrate when PI is useful , we show analytically that , under certain conditions , PI reduces the expected risk for linear regression models . • We provide empirical evidence suggesting that the representations learned with access to PI are more robust against label noise . • We propose a novel efficient method , TRAM , which exploits PI in supervised deep neural networks and has zero computational overhead at prediction time . • Empirically , we show that our method performs better than a series of baselines on CIFAR- 10H , ImageNet and CivilComments benchmarks . 2 EXPLOITING PRIVILEGED INFORMATION . To build up intuition and to better illustrate situations where PI can be useful , we start with a simple linear model where a formal analysis can be carried out . Next , we look into non-linear models and provide a motivating experiment suggesting that useful PI can be leveraged in deep networks to improve representation learning . 2.1 WHEN CAN PI BE HELPFUL ? AN ANALYSIS IN A SIMPLE LINEAR MODEL . We consider the following regression generative model with target y y = x > w ? + a > v ? + ε where x ∈ Rd and a ∈ Rm correspond to standard and PI features respectively , while ε ∼ N ( 0 , σ2 ) stands for some additive noise . The two unknown parameters ( w ? , v ? ) establish the relationships between the target y and the features ( x , a ) . To model the fact that the PI features can themselves depend on the x—e.g. , raters having diverging assessments on ambiguous input samples—we assume that a ∼ p ( a|x ) = N ( µ ( x ) |Σ ( x ) ) for some mean and covariance dependent on x . Let us assume we have n observations from this generative model represented by y ∈ Rn , X ∈ Rn×d , A ∈ Rn×m and µ ( X ) ∈ Rn×m . We are interested in comparing different predictors τ ( X ) that can predict y only based on X , as required in the case of PI . To compare the predictors , we use the concept of risk Eε∼p ( ε ) , a∼p ( a|x ) [ R ( τ ( X ) ) ] , formally defined in Appendix E , to capture the expected error of τ in predicting y ; see Section 3.5 in Bach ( 2021 ) for more background about risk analysis . We defer to Appendix E a rigorous exposition of the results and convey instead here some intuitive messages . We first focus on the comparison between • ( NO-PI ) the least-square estimate ŵ0 = ( X > X ) −1X > y that ignores A , and • ( PI ) the joint least-square estimate [ ŵ1 ; v̂1 ] = ( Q > Q ) −1Q > y with Q = [ X , A ] in Rn× ( d+m ) . At prediction time , if we had access to ( xtest , atest ) for ( PI ) , we would predict with ŵ > 1 xtest + v̂ > 1 atest . However , since atest is not available in our context , we use instead its ( assumed known ) mean µ ( xtest ) , i.e. , mean imputation ( Little & Rubin , 2019 ) . Denoting by Πx the orthogonal projector associated with X , defined in Appendix E , our analysis shows that as long as • Variance of PI : The variance { ( v ? ) > Σ ( xi ) v ? } ni=1 due to PI is large enough and/or • Alignment of PI : The PI features A have a significant average component outside of the subspace spanned by the features X , i.e. , the term below is large enough 1 n ‖ ( I −Πx ) µ ( X ) v ? ‖2 ( 1 ) then the estimator ( PI ) has a lower risk compared to ( NO-PI ) . In other words , it is provably better to exploit the privileged information A at training time instead of ignoring it . Our analysis further covers the case of ( marg.NO-PI ) where we marginalize ŵ0 with respect to PI and we predict with XEa∼p ( a|x ) [ ŵ0 ] , which we compare with ( marg.PI ) the marginalized predictions Ea∼p ( a|x ) [ Xŵ1 + Av̂1 ] . In that case , we can show the same conclusion as previously , with the exception that the variance term { ( v ? ) > Σ ( xi ) v ? } ni=1 does not have influence anymore , only ( 1 ) drives the comparison . Indeed , the proof in Appendix E shows that marginalizing removes from the risk expressions the terms related to the variance of PI . 2.2 PI HELPS TO LEARN BETTER REPRESENTATIONS : A MOTIVATING EXPERIMENT . Our analysis of the effect of PI in a linear model has established key insights into the conditions under which PI is provably useful . We now look at a toy non-linear learning experiment using neural networks , which provides empirical evidence that PI can be helpful in deep networks . In particular , we show that representations learned with access to useful PI can explain away label noise and transfer better than representations learned without access to PI . This motivating experiment forms the basis of our TRAM method . We simulate a noisy annotator by assuming in this toy example that the PI is a binary indicator , a ∼ Bernoulli ( 0.3 ) , with a = 1 representing the case where the noisy annotator provides a label independent of x : y = ( 1− a ) · sin ( 2πx ) + a · v + , where x ∈ [ 0 , 1 ] , v ∼ U ( −1 , 1 ) and ∼ N ( 0 , 0.1 ) . We then fit two networks to N = 2 , 500 training examples generated according to this process . The first network does not have access PI and is a two layer MLP , both layers of dimension 64 , with tanh hidden activations and linear output activation ; see top left of Fig . 1a for an illustration . The second network has access to PI , and is defined as per the two-step TRAM approach . The part of the network which learns the x representation , φ , is defined exactly as the no-PI MLP . q ( y|x , a ) , the output head with access to PI , see Fig . 1a , is a single layer MLP with 64 units , tanh activation and linear output layer , with the concatenation of a and φ ( x ) as inputs ; see bottom left of Fig . 1a . Both networks are fit for 10 epochs by the Adam optimizer ( Kingma & Ba , 2014 ) with mean squared error loss function . We then extract the non-linear representations of x learned by both networks and fit a linear model on the dataset { φ ( xi ) , yi } , i = 1 , . . . , N . The linear model can be solved exactly using the L2 regularized ordinary least squares solution . We plot the results in Fig . 1 . We see that the representations learned by the model with access to PI in step # 1 enable a near perfect fit to the true expected marginal distribution , E ( a , y ) ∼p ( a , y|x ) [ y ] accross X space . However without access to PI the noise term a · v can not be explained away . As a result , the linear model fit on top of the representations learned without access to PI is substantially worse than the model fit using the two step procedure . We emphasize that both models have exactly the same capacity . In Appendix F we extend this representation learning procedure to a large-scale image classification case . We learn a representation with and without access to PI on a relabeled version of ImageNet ( details in §5.2 ) using a ResNet50 ( He et al. , 2016 ) . We then freeze the representation and evaluate using a linear model . Access to PI improves the representations learned .

Summary: This paper suggests a notion that using additional information called privilege information (PI) from annotators will help explain away the noise of the label they annotated and propose a way to implement that idea. The authors give the intuition from a simple linear model and a non-linear sine model that PI is useful. Then, an implementation of that idea called TRAM is proposed to make use of the information from PI through a stop gradient operator in training.  Since there are scant large-scale datasets with PI annotation, to evaluate the proposed approach, the authors have to create a synthetic one relabeled from ImageNet.