Transfer and Marginalize: Explaining Away Label Noise with Privileged Information

1 INTRODUCTION . Supervised learning problems are typically formalized as learning a conditional distribution p ( y|x ) , y ∈ Y and x ∈ X from ( xi , yi ) , i = 1 , ... , N pairs . Yet we often have access to additional features a ∈ A at training time that will not be available at test time . These features are known as privileged information ( Vapnik & Vashist , 2009 ) , or PI for short . An example of PI are features describing the human annotator that provided a given label , such as the annotator ID , the length of time to provide the label , the experience of the annotator , etc . Annotators do not always agree on the correct label for a given x , some annotators may be more reliable than others and the reliability of annotators may depend on the location of x in the input domain X ( Snow et al. , 2008 ; Sheng et al. , 2008 ) . The expanded training dataset consists of ( xi , ai , yi ) triplets . Given that our test time predictive distribution can not be conditioned on a , what use is this PI ? As a thought experiment , suppose there exists a malicious ( or lazy ) annotator that provides random labels . It is known that random labels harm the performance of supervised learning models ( Frénay & Verleysen , 2013 ; Song et al. , 2020 ; Cordeiro & Carneiro , 2020 ) . If these random labels can be explained away via access to PI , such as the annotator ID , then this harm can be prevented . In particular we can use the PI to explain away noise in the labels which otherwise would be irreducible aleatoric uncertainty . More formally , suppose A , the PI random variable , is predictive of Y given X , in the sense that the conditional mutual information I ( Y ; A|X ) is non-zero . Then , the entropy of Y is reduced if we condition on both X and A rather than X alone , as summarised in Lemma 1.1 . Lemma 1.1 . I ( Y ; A|X ) > 0⇒H ( Y |X , A ) < H ( Y |X ) . In §2.1 we examine this lemma for a particular model , proving that under certain conditions , PI can be leveraged to lower the expected risk for linear regression problems . Additionally , prior work has proven that PI can lead to generalization bounds with better sample complexity ( Vapnik & Vashist , 2009 ; Lambert et al. , 2018 ) . Inspired by prior work and our theoretical analysis of a simple linear model , we focus on exploiting PI in supervised deep neural networks . The production deployment of such models often has tight latency and memory constraints . Hence a number of methods have been developed to utilize PI with the same test time memory and computation cost as networks trained without PI ( Yang et al. , 2017 ; Lambert et al. , 2018 ; Lopez-Paz et al. , 2015 ) . Yang et al . ( 2017 ) uses PI as a form of input-dependent regularizer . Lambert et al . ( 2018 ) train with heteroscedastic Gaussian dropout , with the training-time dropout variance a function of the PI . Lopez-Paz et al . ( 2015 ) distill a network trained with PI into a network without access to a . Below we develop a method , TRAM , which transfers knowledge via weight sharing from the part of the network trained using PI to the test time network which does not have access to PI . At test time , TRAM makes a simple , efficient approximation to the integral p ( y|x ) = ∫ p ( y|x , a ) p ( a|x ) da . Making predictions without PI is no more costly than that with a standard network trained without access to PI . Unlike prior work which requires specific techniques such as Gaussian Dropout , we need not constrain the form of the predictors to make the downstream marginalization possible . Implementation and training are simple . In summary the paper contributions are the following : • To better illustrate when PI is useful , we show analytically that , under certain conditions , PI reduces the expected risk for linear regression models . • We provide empirical evidence suggesting that the representations learned with access to PI are more robust against label noise . • We propose a novel efficient method , TRAM , which exploits PI in supervised deep neural networks and has zero computational overhead at prediction time . • Empirically , we show that our method performs better than a series of baselines on CIFAR- 10H , ImageNet and CivilComments benchmarks . 2 EXPLOITING PRIVILEGED INFORMATION . To build up intuition and to better illustrate situations where PI can be useful , we start with a simple linear model where a formal analysis can be carried out . Next , we look into non-linear models and provide a motivating experiment suggesting that useful PI can be leveraged in deep networks to improve representation learning . 2.1 WHEN CAN PI BE HELPFUL ? AN ANALYSIS IN A SIMPLE LINEAR MODEL . We consider the following regression generative model with target y y = x > w ? + a > v ? + ε where x ∈ Rd and a ∈ Rm correspond to standard and PI features respectively , while ε ∼ N ( 0 , σ2 ) stands for some additive noise . The two unknown parameters ( w ? , v ? ) establish the relationships between the target y and the features ( x , a ) . To model the fact that the PI features can themselves depend on the x—e.g. , raters having diverging assessments on ambiguous input samples—we assume that a ∼ p ( a|x ) = N ( µ ( x ) |Σ ( x ) ) for some mean and covariance dependent on x . Let us assume we have n observations from this generative model represented by y ∈ Rn , X ∈ Rn×d , A ∈ Rn×m and µ ( X ) ∈ Rn×m . We are interested in comparing different predictors τ ( X ) that can predict y only based on X , as required in the case of PI . To compare the predictors , we use the concept of risk Eε∼p ( ε ) , a∼p ( a|x ) [ R ( τ ( X ) ) ] , formally defined in Appendix E , to capture the expected error of τ in predicting y ; see Section 3.5 in Bach ( 2021 ) for more background about risk analysis . We defer to Appendix E a rigorous exposition of the results and convey instead here some intuitive messages . We first focus on the comparison between • ( NO-PI ) the least-square estimate ŵ0 = ( X > X ) −1X > y that ignores A , and • ( PI ) the joint least-square estimate [ ŵ1 ; v̂1 ] = ( Q > Q ) −1Q > y with Q = [ X , A ] in Rn× ( d+m ) . At prediction time , if we had access to ( xtest , atest ) for ( PI ) , we would predict with ŵ > 1 xtest + v̂ > 1 atest . However , since atest is not available in our context , we use instead its ( assumed known ) mean µ ( xtest ) , i.e. , mean imputation ( Little & Rubin , 2019 ) . Denoting by Πx the orthogonal projector associated with X , defined in Appendix E , our analysis shows that as long as • Variance of PI : The variance { ( v ? ) > Σ ( xi ) v ? } ni=1 due to PI is large enough and/or • Alignment of PI : The PI features A have a significant average component outside of the subspace spanned by the features X , i.e. , the term below is large enough 1 n ‖ ( I −Πx ) µ ( X ) v ? ‖2 ( 1 ) then the estimator ( PI ) has a lower risk compared to ( NO-PI ) . In other words , it is provably better to exploit the privileged information A at training time instead of ignoring it . Our analysis further covers the case of ( marg.NO-PI ) where we marginalize ŵ0 with respect to PI and we predict with XEa∼p ( a|x ) [ ŵ0 ] , which we compare with ( marg.PI ) the marginalized predictions Ea∼p ( a|x ) [ Xŵ1 + Av̂1 ] . In that case , we can show the same conclusion as previously , with the exception that the variance term { ( v ? ) > Σ ( xi ) v ? } ni=1 does not have influence anymore , only ( 1 ) drives the comparison . Indeed , the proof in Appendix E shows that marginalizing removes from the risk expressions the terms related to the variance of PI . 2.2 PI HELPS TO LEARN BETTER REPRESENTATIONS : A MOTIVATING EXPERIMENT . Our analysis of the effect of PI in a linear model has established key insights into the conditions under which PI is provably useful . We now look at a toy non-linear learning experiment using neural networks , which provides empirical evidence that PI can be helpful in deep networks . In particular , we show that representations learned with access to useful PI can explain away label noise and transfer better than representations learned without access to PI . This motivating experiment forms the basis of our TRAM method . We simulate a noisy annotator by assuming in this toy example that the PI is a binary indicator , a ∼ Bernoulli ( 0.3 ) , with a = 1 representing the case where the noisy annotator provides a label independent of x : y = ( 1− a ) · sin ( 2πx ) + a · v + , where x ∈ [ 0 , 1 ] , v ∼ U ( −1 , 1 ) and ∼ N ( 0 , 0.1 ) . We then fit two networks to N = 2 , 500 training examples generated according to this process . The first network does not have access PI and is a two layer MLP , both layers of dimension 64 , with tanh hidden activations and linear output activation ; see top left of Fig . 1a for an illustration . The second network has access to PI , and is defined as per the two-step TRAM approach . The part of the network which learns the x representation , φ , is defined exactly as the no-PI MLP . q ( y|x , a ) , the output head with access to PI , see Fig . 1a , is a single layer MLP with 64 units , tanh activation and linear output layer , with the concatenation of a and φ ( x ) as inputs ; see bottom left of Fig . 1a . Both networks are fit for 10 epochs by the Adam optimizer ( Kingma & Ba , 2014 ) with mean squared error loss function . We then extract the non-linear representations of x learned by both networks and fit a linear model on the dataset { φ ( xi ) , yi } , i = 1 , . . . , N . The linear model can be solved exactly using the L2 regularized ordinary least squares solution . We plot the results in Fig . 1 . We see that the representations learned by the model with access to PI in step # 1 enable a near perfect fit to the true expected marginal distribution , E ( a , y ) ∼p ( a , y|x ) [ y ] accross X space . However without access to PI the noise term a · v can not be explained away . As a result , the linear model fit on top of the representations learned without access to PI is substantially worse than the model fit using the two step procedure . We emphasize that both models have exactly the same capacity . In Appendix F we extend this representation learning procedure to a large-scale image classification case . We learn a representation with and without access to PI on a relabeled version of ImageNet ( details in §5.2 ) using a ResNet50 ( He et al. , 2016 ) . We then freeze the representation and evaluate using a linear model . Access to PI improves the representations learned .

Summary: The manuscript proposes classification methods utilizing privileged information (PI, which is available only at training time). The main idea is to directly approximate the conditional marginal distribution, i.e., p(y|x) \approx q(y|\pi(x), w), where \pi(x) is a feature extractor and w is the weight parameter. Another main point is to learn the representation (\pi(x)) of the features with access to PI at the training time. To do this, the proposed methods employ knowledge transfer by weight sharing. The proposed methods include several variants by adopting heteroscedastic classification (Het-TRAM) or distillation (Distilled-TRAM).