Year: 2021

Abstract:
Supervised learning datasets often have privileged information, in the form of features which are available at training time but are not available at test time e.g. the ID of the annotator that provided the label. We argue that privileged information is useful for explaining away label noise, thereby reducing the harmful impact of noisy labels. We develop a simple and efficient method for supervised neural networks: it transfers via weight sharing the knowledge learned with privileged information and approximately marginalizes over privileged information at test time. Our method, TRAM (TRansfer and Marginalize), has minimal training time overhead and has the same test time cost as not using privileged information. TRAM performs strongly on CIFAR-10H, ImageNet and Civil Comments benchmarks.

heading: 1 INTRODUCTION
text: Supervised learning problems are typically formalized as learning a conditional distribution p(y|x), y ∈ Y and x ∈ X from (xi, yi), i = 1, ..., N pairs. Yet we often have access to additional features a ∈ A at training time that will not be available at test time. These features are known as privileged information (Vapnik & Vashist, 2009), or PI for short. An example of PI are features describing the human annotator that provided a given label, such as the annotator ID, the length of time to provide the label, the experience of the annotator, etc. Annotators do not always agree on the correct label for a given x, some annotators may be more reliable than others and the reliability of annotators may depend on the location of x in the input domain X (Snow et al., 2008; Sheng et al., 2008). The expanded training dataset consists of (xi,ai, yi) triplets. Given that our test time predictive distribution cannot be conditioned on a, what use is this PI? As a thought experiment, suppose there exists a malicious (or lazy) annotator that provides random labels. It is known that random labels harm the performance of supervised learning models (Frénay & Verleysen, 2013; Song et al., 2020; Cordeiro & Carneiro, 2020). If these random labels can be explained away via access to PI, such as the annotator ID, then this harm can be prevented. In particular we can use the PI to explain away noise in the labels which otherwise would be irreducible aleatoric uncertainty.
More formally, suppose A, the PI random variable, is predictive of Y given X , in the sense that the conditional mutual information I(Y ;A|X) is non-zero. Then, the entropy of Y is reduced if we condition on both X and A rather than X alone, as summarised in Lemma 1.1. Lemma 1.1. I(Y ;A|X)> 0⇒H(Y |X,A)<H(Y |X).
In §2.1 we examine this lemma for a particular model, proving that under certain conditions, PI can be leveraged to lower the expected risk for linear regression problems. Additionally, prior work has proven that PI can lead to generalization bounds with better sample complexity (Vapnik & Vashist, 2009; Lambert et al., 2018).
Inspired by prior work and our theoretical analysis of a simple linear model, we focus on exploiting PI in supervised deep neural networks. The production deployment of such models often has tight latency and memory constraints. Hence a number of methods have been developed to utilize PI with the same test time memory and computation cost as networks trained without PI (Yang et al., 2017; Lambert et al., 2018; Lopez-Paz et al., 2015). Yang et al. (2017) uses PI as a form of input-dependent regularizer. Lambert et al. (2018) train with heteroscedastic Gaussian dropout, with the training-time dropout variance a function of the PI. Lopez-Paz et al. (2015) distill a network trained with PI into a network without access to a.
Below we develop a method, TRAM, which transfers knowledge via weight sharing from the part of the network trained using PI to the test time network which does not have access to PI. At test time, TRAM makes a simple, efficient approximation to the integral p(y|x) = ∫ p(y|x,a)p(a|x)da. Making predictions without PI is no more costly than that with a standard network trained without access to PI. Unlike prior work which requires specific techniques such as Gaussian Dropout, we need not constrain the form of the predictors to make the downstream marginalization possible. Implementation and training are simple.
In summary the paper contributions are the following:
• To better illustrate when PI is useful, we show analytically that, under certain conditions, PI reduces the expected risk for linear regression models. • We provide empirical evidence suggesting that the representations learned with access to PI
are more robust against label noise. • We propose a novel efficient method, TRAM, which exploits PI in supervised deep neural
networks and has zero computational overhead at prediction time. • Empirically, we show that our method performs better than a series of baselines on CIFAR-
10H, ImageNet and CivilComments benchmarks.

heading: 2 EXPLOITING PRIVILEGED INFORMATION
text: To build up intuition and to better illustrate situations where PI can be useful, we start with a simple linear model where a formal analysis can be carried out. Next, we look into non-linear models and provide a motivating experiment suggesting that useful PI can be leveraged in deep networks to improve representation learning.

heading: 2.1 WHEN CAN PI BE HELPFUL? AN ANALYSIS IN A SIMPLE LINEAR MODEL
text: We consider the following regression generative model with target y y = x>w? + a>v? + ε
where x ∈ Rd and a ∈ Rm correspond to standard and PI features respectively, while ε ∼ N (0, σ2) stands for some additive noise. The two unknown parameters (w?,v?) establish the relationships between the target y and the features (x,a). To model the fact that the PI features can themselves depend on the x—e.g., raters having diverging assessments on ambiguous input samples—we assume that a ∼ p(a|x) = N (µ(x)|Σ(x)) for some mean and covariance dependent on x. Let us assume we have n observations from this generative model represented by y ∈ Rn,X ∈ Rn×d, A ∈ Rn×m and µ(X) ∈ Rn×m. We are interested in comparing different predictors τ(X) that can predict y only based on X , as required in the case of PI. To compare the predictors, we use the concept of risk Eε∼p(ε),a∼p(a|x)[R(τ(X))], formally defined in Appendix E, to capture the expected error of τ in predicting y; see Section 3.5 in Bach (2021) for more background about risk analysis.
We defer to Appendix E a rigorous exposition of the results and convey instead here some intuitive messages. We first focus on the comparison between
• (NO-PI) the least-square estimate ŵ0 = (X>X)−1X>y that ignores A, and • (PI) the joint least-square estimate [ŵ1; v̂1] = (Q>Q)−1Q>y with Q = [X,A]
in Rn×(d+m). At prediction time, if we had access to (xtest,atest) for (PI), we would predict with ŵ>1 xtest + v̂ > 1 atest. However, since atest is not available in our context, we use instead its (assumed known) mean µ(xtest), i.e., mean imputation (Little & Rubin, 2019).
Denoting by Πx the orthogonal projector associated with X , defined in Appendix E, our analysis shows that as long as
• Variance of PI: The variance {(v?)>Σ(xi)v?}ni=1 due to PI is large enough and/or • Alignment of PI: The PI features A have a significant average component outside of the
subspace spanned by the features X , i.e., the term below is large enough 1
n ‖(I −Πx)µ(X)v?‖2 (1)
then the estimator (PI) has a lower risk compared to (NO-PI). In other words, it is provably better to exploit the privileged information A at training time instead of ignoring it.
Our analysis further covers the case of (marg.NO-PI) where we marginalize ŵ0 with respect to PI and we predict with XEa∼p(a|x)[ŵ0], which we compare with (marg.PI) the marginalized predictions Ea∼p(a|x)[Xŵ1 + Av̂1]. In that case, we can show the same conclusion as previously, with the exception that the variance term {(v?)>Σ(xi)v?}ni=1 does not have influence anymore, only (1) drives the comparison. Indeed, the proof in Appendix E shows that marginalizing removes from the risk expressions the terms related to the variance of PI.

heading: 2.2 PI HELPS TO LEARN BETTER REPRESENTATIONS: A MOTIVATING EXPERIMENT
text: Our analysis of the effect of PI in a linear model has established key insights into the conditions under which PI is provably useful. We now look at a toy non-linear learning experiment using neural networks, which provides empirical evidence that PI can be helpful in deep networks. In particular, we show that representations learned with access to useful PI can explain away label noise and transfer better than representations learned without access to PI. This motivating experiment forms the basis of our TRAM method.
We simulate a noisy annotator by assuming in this toy example that the PI is a binary indicator, a ∼ Bernoulli(0.3), with a = 1 representing the case where the noisy annotator provides a label independent of x: y = (1− a) · sin(2πx) + a · v + , where x ∈ [0, 1], v ∼ U(−1, 1) and ∼ N (0, 0.1). We then fit two networks to N = 2, 500 training examples generated according to this process. The first network does not have access PI and is a two layer MLP, both layers of dimension 64, with tanh hidden activations and linear output activation; see top left of Fig. 1a for an illustration. The second network has access to PI, and is defined as per the two-step TRAM approach. The part of the network which learns the x representation, φ, is defined exactly as the no-PI MLP. q(y|x,a), the output head with access to PI, see Fig. 1a, is a single layer MLP with 64 units, tanh activation and linear output layer, with the concatenation of a and φ(x) as inputs; see bottom left of Fig. 1a. Both networks are fit for 10 epochs by the Adam optimizer (Kingma & Ba, 2014) with mean squared error loss function.
We then extract the non-linear representations of x learned by both networks and fit a linear model on the dataset {φ(xi), yi}, i = 1, . . . , N . The linear model can be solved exactly using the L2 regularized ordinary least squares solution. We plot the results in Fig. 1. We see that the representations learned by the model with access to PI in step #1 enable a near perfect fit to the true expected marginal distribution, E(a,y)∼p(a,y|x)[y] accross X space. However without access to PI the noise term a · v cannot be explained away. As a result, the linear model fit on top of the representations learned without access to PI is substantially worse than the model fit using the two step procedure. We emphasize that both models have exactly the same capacity.
In Appendix F we extend this representation learning procedure to a large-scale image classification case. We learn a representation with and without access to PI on a relabeled version of ImageNet (details in §5.2) using a ResNet50 (He et al., 2016). We then freeze the representation and evaluate using a linear model. Access to PI improves the representations learned.

heading: 3 METHOD: TRAM
text: We consider learning under privileged information (Vapnik & Vashist, 2009), LUPI. Our proposed method, TRAM, consists of a single neural network with two output heads, providing predictions for both p(y|x,a) and p(y|x); see Fig. 2. There are two key ingredients to TRAM; (i) the p(y|x) head is a simple, yet a provably valid, approximation to the marginal ∫ p(y|x,a)p(a|x)da and (ii) a partition of the parameter space such that the neural network weights are shared between the two output heads, and that these shared weights are updated solely based on the lower variance gradients from the p(y|x,a) head which has access to PI. Below we develop TRAM in the classification setting, in §3.4 we extend the method to the regression setting.

heading: 3.1 INGREDIENT #1: MARGINALIZE OVER PI AT TEST TIME
text: A natural probabilistic approach to LUPI is (i) to learn the conditional distribution p(y|x,a) during training and (ii) then, at test time, marginalize over the A domain, computing p(y|x) =∫ p(y|x,a)p(a|x)da (Lambert et al., 2018).
Predicting with the marginal p(y|x) at test time is motivated by the following key observation. Consider the set of distributions Q over the C class labels, Q = {q(y|·)|∀x ∈ X , q(y|x) ∈ ∆C} where ∆C is theC-dimensional simplex. Among all the distributions q ∈ Q, the marginal distribution x 7→ p(y|x) minimizes the following optimization problem:
min q∈Q
E(x,a)∼p(x,a) [DKL(p(y|x,a)‖q(y|x))] . (2)
See proof in Appendix B. In words, p(y|x) is optimal in the sense that it minimizes the expected KL divergence to p(y|x,a). Access to p(y|x) enables minimizing the Bayes risk (Murphy, 2012) at test time. Note further that the mean imputation scheme which provably reduces the expected risk for a linear model, §2.1, corresponds precisely to marginalizing over PI at test time. Directly computing p(y|x) has two problems; (i) it is typically intractable and (ii) p(a|x) is unknown and therefore must be learned, which is a challenging generative modelling problem in itself, especially as a may have mixed type features. A Monte Carlo estimate of the integral using the samples from A in the training set is only feasible with the independence assumption p(a|x) = p(a), so that p(y|x) reduces to ∫ p(y|x,a)p(a)da ≈ 1S ∑S s=1 p(y|x,as) with as ∼ p(a).
Unfortunately this independence assumption is often violated in practice. In addition the memory and computational cost of MC estimation scales linearly in S, the number of MC samples. This O(S) scaling is undesirable for production deployment with strict latency requirements.
Due to the challenge of computing the integral directly, we propose a simple approximation q(y|x;w) to p(y|x). It exploits the property (2) of p(y|x) as the distribution minimizing the expected KL
divergence to its conditional p(y|x,a). We choose q to be cheap to evaluate at test time. For example, for a multi-class vanilla TRAM classifier q(y|x;w) = softmax(Wφ(x)).

heading: 3.2 INGREDIENT #2: TRANSFER VIA WEIGHT SHARING
text: We partition the parameter space into four disjoint subsets;
1. Let φ(x) be a feature extractor for x ∈ X . 2. Similarly, let ψ(φ(x),a) be a feature extractor jointly applied to (φ(x),a) for (x,a) in X ×A.
3. The weights w parameterize the marginal distribution: q(y|x;w) = q(y|φ(x);w). 4. The weights u parameterize the conditional distribution q(y|x,a;u), namely q(y|x,a;u) = q(y|ψ(φ(x),a);u).
Two-step approach. Motivated by Eq. (2), the connection between LUPI and multi-task learning (Jonschkowski et al., 2016) and our toy representation learning experiments, §2.2, we consider the following two-step approach:
min u,φ,ψ
E(x,a,y)∼p(x,a,y) [ L1(y, q(y|x,a)) ] (3)
min w
E(x,a,y)∼p(x,a,y) [ L2(y, q(y|x)) ] with φ = φ? (4)
L∗ are arbitrary loss functions. We assume φ and ψ are parameterized as neural networks, so minφ,ψ refers to optimizing the network weights.
Crucially φ? is the feature extractor learned in (3) with access to PI. This weight sharing enables knowledge transfer to the network trained without PI. Given Eq. (2), we know that Eq. (4) approximates the true marginal distribution p(y|x) (observe that the KL divergence in Eq. (2) reduces to the cross-entropy loss function for L2 when taking the one-hot training labels for p(y|x,a)).
Merging the two steps. To further simplify the above approach, we propose to merge Eq. (3) and Eq. (4) into a single training procedure. To that end, and reusing the terminology commonly used in deep-learning frameworks, let us define
π(y|x;w) = q ( y|stop gradient(φ(x));w ) which coincides with q(y|x) except that its gradient only depends on w. For some β > 0, we then consider:
min u,w,φ,ψ
E(x,a,y)∼p(x,a,y) [ L2(y, π(y|x)) + βL1(y, q(y|x,a)) ] as the joint training objective. In practice, since the parameters of the two losses are partitioned, we can set β = 1 and fold instead the search over β into the search of the learning rate, hence not introducing an extra hyperparameter.

heading: 3.3 TRAM VARIANTS
text: Privileged information may only explain away some of label noise uncertainty. Below we propose two TRAM variants which combine TRAM with existing noisy labels methods.
Het-TRAM. Heteroscedastic classifiers are capable of modeling label noise that is input-dependent and have been successfully applied in this setting (Collier et al., 2021). Further note that even in some cases where the conditional distribution q(y|x,a) is homoscedastic, the marginal q(y|x) is heteroscedastic, see Appendix C for details.
Hence we propose Het-TRAM, a TRAM variant in which q(y|x) is heteroscedastic. This increases the expressiveness of q, improving the approximation in the second step of our optimization procedure, Eq. (4). We implement the method of Collier et al. (2021) to make q(y|x) heteroscedastic.
Distilled-TRAM. Distillation (Hinton et al., 2015) is a technique for transferring knowledge between two neural networks. Distillation has been previously applied to LUPI (Lopez-Paz et al., 2015). In Distilled-TRAM we use the two step TRAM procedure, setting the loss function, L1 in Eq. (3), to the distillation loss. The teacher network is first trained with access to PI and in the second step the distilled-TRAM model is trained.

heading: 3.4 REGRESSION
text: We developed TRAM and Het-TRAM focussing on the classification setting but our approach is trivial to generalize to regression problems. In the regression case, we can choose the predictive distribution to be Gaussian, q(y|x) = N (µ(x), σ2(x)). For vanilla TRAM we can choose σ2(x) = 1, while for Het-TRAM we can choose σ2(x) = softplus(w>σ φ(x)). µ and σ
2 are parameterized as neural networks with our shared feature extractor φ(x), similar to Kendall & Gal (2017). L1 in Eq. (3) and L2 in Eq. (4) are replaced by the Gaussian negative log-likelihood.

heading: 4 RELATED WORK
text: Vapnik & Vashist (2009) develop a theoretic framework for the LUPI paradigm and introduce the SVM+ method for training Support Vector Machines in this regime. The slack variables for the SVM+ constraints are a function of the PI. SVM+ has been extended in the SVM literature (Lapin et al., 2014; Vapnik & Izmailov, 2015). Jonschkowski et al. (2016) provide a unifying framework that connects together multi-task learning, multi-view learning and LUPI.
Yang et al. (2017) extend the SVM+ approach to neural network models with their MIML-FCN+ method. The authors formulate a two-tower network similar to ours, but without weight sharing between the towers. Both towers make independent predictions given x or a as inputs. The tower with access to PI predicts the loss of the other tower and this prediction is regularized to be close to the true loss. In this way the PI tower outputs a neural network analogue to the SVM+ slack variables.
Lambert et al. (2018) utilize PI by making the training-time Gaussian-dropout variance (Kingma et al., 2015) a function of the PI. At test time the PI is approximately marginalized over by removing the dropout. Similarly Hernández-Lobato et al. (2014) allow the additive Gaussian noise component of a heteroscedastic Gaussian Process Classifier (Rasmussen & Williams, 2006) to be a function of the PI. The classifier is homoscedastic at test time.
Lopez-Paz et al. (2015) propose a distillation (Hinton et al., 2015) style approach to learning with PI. The teacher network is trained with access to PI. In the distillation step the student network is given x as input and a convex combination of soft labels from the teacher network and true labels y as targets. Xu et al. (2020) extend and apply this distillation method to a recommender system.
TRAM implements knowledge transfer via weight sharing, performs efficient approximate marginalization at test time and can be applied to many widely used architectures. Lambert et al. (2018) also share weights and approximate the marginal p(y|x) however they require the use of Gaussian dropout, which is not widely used. The distillation and MIML-FCN+ methods do not transfer via weight sharing and do not approximate p(y|x). Distillation also requires a two-step training procedure. See Table 5, in Appendix F for a comparison of the key features of selected LUPI methods.

heading: 5 EXPERIMENTS
text: Our experiments tackle the general LUPI problem under label noise. There are a few large-scale public datasets with PI We thus use both real-world datasets with PI as well as synthesizing PI for a re-labelled version of ImageNet (Deng et al., 2009).
We evaluate a number of baselines in addition to our method.
• The “No PI” baseline is standard neural network training which directly learns p(y|x) and never uses PI. • Zero and mean imputation learn p(y|x,a) at training time and substitute a = 0 and a = 1N ∑ i ai respectively at test time. For mean imputation, averaging takes place after
feature pre-processing, e.g., one-hot encoding of the annotator ID.
Prior work did not evaluate against these simple imputation baselines or full marginalization (LopezPaz et al., 2015; Yang et al., 2017; Lambert et al., 2018).

heading: 5.1 CIFAR-10H
text: One dataset with annotator features is CIFAR-10H (Peterson et al., 2019), which is a re-labelled version of the CIFAR-10 (Krizhevsky & Hinton, 2009) test set. The new labels are provided by crowd-sourced human annotators. We make use of three annotator features; the annotator ID, the reaction time of the annotator to provide the label and how much experience the annotator had with the task, as measured by the number of labels the annotator had previously provided.
As we only have annotator features for the CIFAR-10 test set, we use this as our training set and evaluate on the official training set. As a result we have only 10,000 images for training. To achieve reasonable performance we start from a MobileNet (Howard et al., 2017) pretrained on ImageNet. Images have on average > 50 annotations each. This is unrealistic for typical applications where 1-3 labels per example is more common. Therefore, we subsample 16,400 labels (1.64 labels per example), see Appendix D for details of the subsampling procedure. The subsampled labels agree with the true CIFAR-10 test set labels 79.4% of the time.
In Table 1 we see the results. First, and as expected, using annotator features via TRAM, marginalization or the imputation methods provides a performance improvement over standard neural network training without PI. Second, we see that TRAM performs on par with full marginalization (which uses 16,400 MC samples of a from the training set), despite having constant time compute and memory requirements w.r.t. the number of MC samples for the full marginalization baseline. Mean imputation is a strong baseline on CIFAR-10H. Het-TRAM improves over TRAM demonstrating the efficacy of making q(y|x,a) heteroscedastic. It is noteworthy that distillation using PI, (Lopez-Paz et al., 2015) does not improve over standard distillation without PI. However Distilled-TRAM with makes use of PI for distillation but then performs approximate marginalization and transfer learning via weight sharing improves over the distillation baselines on both accuracy and log-likelihood metrics.

heading: 5.1.1 QUALITATIVE ANALYSIS OF CIFAR-10H RESULTS
text: We qualitatively analyse how PI is helping improve the performance of TRAM on CIFAR-10H. The PI for CIFAR-10H does not contain a feature for annotator reliability. However the PI does contain features such as the annotator ID and the annotator reaction time from which it may be possible to
learn to trust some annotators more than others. TRAM can learn output a less confident distribution for unreliable annotators, thus reducing the harmful impact of incorrect labels.
In Fig. 3 we show an analysis of the confidence of the TRAM and No PI (i.e., standard) models for each annotator in CIFAR-10H. We see that the trend for TRAM is a strong linear relationship between the reliability of an annotator and the confidence of the model, Fig. 3a. The TRAM model is consistently more confident than the No PI model for reliable annotators while the No PI model is overconfident for unreliable annotators, Fig. 3b

heading: 5.2 IMAGENET ILSVRC12
text: In order to create a large-scale dataset with annotator features, we re-label the ImageNet ILSVRC12 training set by the following procedure. We download 16 different models pre-trained on ImageNet, see Appendix D for further details. We also add a 17th malicious annotator model which picks a label uniformly at random from the 1,000 ImageNet ILSVRC12 classes. For each image in the training set we select the malicious annotator with 10% probability and otherwise sample one of the 16 models with equal probability. We then sample a label from the predictive distribution of that model for that image. This is the label used for training. On average the sampled label agrees with the true ImageNet label 68.3% of the time. The annotator features are the model ID (a proxy for a human annotator ID) and the probability of the label assigned by the model (a proxy for the confidence of a human annotator). The ImageNet image is used as the non-privileged information x. φ is randomly initialized ResNet-50 (He et al., 2016).
See Table 2 for the results. The full marginalization baseline uses 1,000 MC samples of a from the training set. The imputation baselines perform worse than not using PI, perhaps due to the imputed values having low density p(a|x). Again TRAM performs on par with full marginalization and HetTRAM has higher accuracy than both. The ImageNet labels are known to exhibit heteroscedasticity
(Collier et al., 2021), therefore we make both q(y|x) and q(y|x,a) heads heteroscedastic for HetTRAM. Distilled-TRAM has significantly better NLL and accuracy than the two distillation baselines.

heading: 5.3 CIVIL COMMENTS
text: We further evaluate our method on a large-scale text classification dataset. Civil Comments1 is a collection of comments from independent news websites annotated with 7 toxicity labels (identity attack, insult, obscene, severe toxicity, sexually explicit, threat, toxicity). The Civil Comments Identities subset of the Civil Comments data contains privileged information in the form of 24 attributes identified in the comment (male, female, christian and so on). The Identities subset consists of 405,130 training examples, 21,293 validation examples and 21,577 test set examples.
The shared network φ is a pre-trained Universal Sentence Encoder (Cer et al., 2018). Table 3 contains the test set results. We report negative log-likelihood and accuracy averaged over the 7 labels. The TRAM, Het-TRAM and imputation methods perform similarly well in terms of average accuracy, outperforming the No PI baseline as well as the Gaussian Dropout and full marginalization methods.
The poor accuracy of the full marginalization method is interesting to note. The PI is directly derived from the non-PI (in the form of 24 identity human labelled attributes for the non-PI). This is clearly a violation of the independence assumption required for a MC estimate of full marginalization to be computable. The dependence of a on x is most clearly identifiable for the Civil Comments Identities dataset; as a result the relative performance of the full marginalization method is poorest on this dataset. Further note that the TRAM and Het-TRAM methods have lower negative log-likelihood than all other baseline methods. Standard distillation with no PI and Lopez-Paz et al. (2015) style distillation where the teacher network is trained with PI does not provide a performance improvement over the no PI baseline. Distilled-TRAM performs on par with vanilla TRAM.

heading: 6 CONCLUSION
text: We introduced TRAM, a new method for LUPI in supervised neural networks. TRAM (i) learns an efficient, simple distribution to approximately marginalize over PI at test time and (ii) partitions the parameter space enabling transfer via weight sharing of the knowledge learned with access to PI. TRAM can be successfully combined with established methods for dealing with noisy labels; distillation (Distilled-TRAM) and heteroscedastic output layers (Het-TRAM). We have analysed a linear model with PI where deriving analytic results are feasible. In this setting we have shown the utility of using PI and ingredient #1 of our TRAM method, the marginalization over PI. Using a toy low-dimensional problem we have further shown the effectiveness of ingredient #2 of our proposed TRAM method, transfer learning via weight sharing of representations learned with access to PI. We then have empirically validated the single-step TRAM procedure on larger-scale datasets in the image and text domain; CIFAR-10H, a noisy version of ImageNet and Civil Comments Identities.
1https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/data

heading: 7 ETHICS AND REPRODUCIBILITY STATEMENTS
text: Ethics. We present a generic algorithm for classification/regression with privileged information. The method is not application specific and has minimal computational overhead. We do not see ethical issues specific to our method beyond any already associated with LUPI methods and/or deep neural network classifiers.
Reproducibility. We provide detailed information on the data generation process for the CIFAR10H and Imagenet datasets in Appendix D.1. Hyperparameters and further details required to reproduce our results are provided in Appendix D.2. We plan to release publicly Colab notebooks to reproduce our toy transfer learning and Civil Comments experiments. We also plan to make public our Imagenet dataset in Tensorflow Dataset format to be used as a large-scale dataset with annotator features by researchers working on the LUPI problem. Detailed derivations with conditions for our theoretical analysis are presented in Appendix E.

heading: A APPENDIX

heading: B PROOF OF EQUATION (2)
text: As a reminder, we consider C class labels and denote by ∆C the C-dimensional simplex. We define the set of distributions Q over the C class labels by
Q = {q(y|·) | ∀x ∈ X , q(y|x) ∈ ∆C}.
Consider the optimization problem
min q∈Q
Ex∼p(x) [DKL(p(y|x)‖q(y|x))] (5)
whose solution is straightforwardly given by the marginal distribution x 7→ q?(y|x) = p(y|x). We recall that the KL DKL(p(y|x)‖q(y|x)) is defined by
DKL(p(y|x)‖q(y|x))= C∑ j=1 pj(y|x) log (pj(y|x) qj(y|x) ) . (6)
For any x and j ≤ C, we can rewrite the terms of the sum pj(y|x) log (pj(y|x) qj(y|x) ) as
Ea|x∼p(a|x) [ pj(y|x,a) log (pj(y|x) qj(y|x) )] where we have used (i) the fact that log(pj(y|x)qj(y|x) ) does not depend on a and (ii) the definition of the marginal distribution
pj(y|x) = ∫ pj(y|x,a)p(a|x)da
= Ea|x∼p(a|x) [pj(y|x,a)] .
Multiplying and dividing in the argument of the log by pj(y|x,a), we obtain Ea|x∼p(a|x) [ pj(y|x,a) log (pj(y|x,a) qj(y|x) pj(y|x) pj(y|x,a) )] .
Summing over j ∈ {1, . . . , C} to reconstruct the KL term (6), this leads to, for any x,
DKL(p(y|x)‖q(y|x)) = Ea|x [DKL(p(y|x,a)‖q(y|x))] −Ea|x [DKL(p(y|x,a)‖p(y|x))] .
Since the second term above does not depend on q, minimizing (5) is equivalent to minimizing
min q∈Q
Ex [ Ea|x [DKL(p(y|x,a)‖q(y|x))] ] = min
q∈Q E(x,a)∼p(x,a) [DKL(p(y|x,a)‖q(y|x))]
which is equal to (2) and which is, analogously to (5), minimized by the marginal distribution x 7→ q?(y|x) = p(y|x).

heading: C HETEROSCEDASTIC MOTIVATION
text: We consider a simplified special case of our framework in which the conditional model p(y|x,a) is homoscedastic but the optimal variational distribution in the sense of Eq. 2 is heteroscedastic. This motivates Het-TRAM, in which the variational approximations q(y|x) and q(y|x,a) are heteroscedastic.
Suppose we have a regression dataset constructed from labels assigned by M annotators. Each annotator has their own homoscedastic Gaussian model p(y|x, a = m) = N (µθm(x), 1). Here the
PI is a single discrete Categorical feature representing the annotator ID which takes one of M values with equal probability, a ∼ Cat( 1M ). The marginal p(y|x) is a Gaussian Mixture Model. We choose our variational family to be the Gaussian distribution, q(y|x) = N (µ(x), σ2(x)). The values of µ and σ2 that minimize Eq. 2 are: µ∗(x) = 1M ∑ m µθm(x) and σ 2 ∗(x) = (M − µ∗(x)) + 1M ∑ m µ 2 θm
(x) (Lakshminarayanan et al., 2017). Crucially note that despite the conditional distribution being homoscedastic, the best variational distribution is heteroscedastic as the variance depends on the location in X space.

heading: D EXPERIMENTAL DETAILS

heading: D.1 DATA GENERATION PROCESS
text: CIFAR-10H. We use the CIFAR-10 image as the non-privileged information x. The annotator ID, the number of prior annotations the annotator has provided and the reaction time in milliseconds of the annotator, are used as privileged information a. For feature pre-processing the annotator ID is one-hot encoded. The number of prior annotations and the reaction time are independently divided into 10 equally sized quantiles and the quantile ID is one-hot encoded. The image is pre-processed according the the standard MobileNet pre-processing (Howard et al., 2017).
As CIFAR-10H has on average more than 50 annotations per image and the labels are not particularly noisy. We subsample the CIFAR-10H labels by the following procedure. We keep all labels by the 41 annotators that agree with the true CIFAR-10 label less than 85% of the time. We then select a further 41 annotators from the remaining annotators. The average agreement of the bad annotators with the CIFAR-10 label is 63.3%, in the full subset of labels: 79.2% and in the full CIFAR-10H dataset: 94.9%. The subsampling procedure leaves 16,400 labels from 82 annotators while the full CIFAR-10H dataset has 514,200 labels from 2,571 annotators.
ImageNet. The annotator features are the model ID used to re-label x, which is one-hot encoded and the probability of that label being sampled. See main paper for details on the sampling procedure and see Table 4 for the list of models used and their accuracy on the ImageNet training set. The pre-trained models are downloaded from tf.keras.applications2.
2https://www.tensorflow.org/api docs/python/tf/keras/applications

heading: D.2 HYPERPARAMETERS
text: CIFAR-10H. For all methods φ(x) (or equivalent) is a MobileNet (Howard et al., 2017) pre-trained on ImageNet ILSVRC12, followed by a global average pooling layer and a Dense + ReLU layer with 64 units. ψ(x,a) is a two-layer MLP with 64 units per layer and ReLU activation. The first layer takes only a as an input, while the second layer takes the output of the first layer concatenated with φ(x) as input.
All models are trained for 20 epochs with the Adam optimizer with base learning rate= 0.001, β1 = 0.9, β2 = 0.999, = 1e − 07. All models are trained with L2 weight regularization with weighting 1e− 3. Heteroscedastic models are trained using the method of Collier et al. (2021) with 4 factors for the low-rank covariance matrix approximation and a softmax temperature parameter of τ = 3.0. Distilled models are also trained with a softmax temperature of τ = 3.0 to smooth the teacher labels and with the distillation hyperparameter λ = 0.5 which weights the losses from the soft teacher labels and the true labels. A grid search over τ ∈ {1.0, 2.0, 3.0, 4.0} and λ ∈ {0.0, 0.25, 0.5, 0.75, 1.0} was conducted.
ImageNet. For all methods φ(x) (or equivalent) is a randomly initialized ResNet-50 (He et al., 2016) with the output layer removed. ψ(x,a) is a two-layer MLP with 128 units per layer and ReLU activation, the output of this MLP is concatenated with φ(x) and then passed to the output layer. The first layer of the ψ(x,a) MLP takes only a as an input, while the second layer takes the output of the first layer concatenated with φ(x) as input.
All but Het-TRAM models are trained for 90 epochs with the SGD optimizer with base learning rate= 0.1, decayed by a factor of 10 after 30, 60 and 80 epochs. Following Collier et al. (2021), Het-TRAM is trained for 270 epochs with the same initial learning rate and learning rate decay at 90, 180 and 240 epochs. All models are trained with L2 weight regularization with weighting 1e− 4. Heteroscedastic models use 15 factors for the low-rank covariance matrix approximation and a softmax temperature parameter of τ = 1.5. Distilled models are trained with a softmax temperature of τ = 3.0 and with the distillation hyperparameter λ = 0.5. A grid search over τ ∈ {1.0, 2.0, 3.0, 4.0} and λ ∈ {0.0, 0.25, 0.5, 0.75, 1.0} was conducted.

heading: E RISK ANALYSIS
text: Generative model and notations. We assume the following
• a ∈ Rm,x ∈ Rd, • a ∼ p(a|x) = N (µ(x)|Σ(x)) for some mean and covariance dependent on x, • y = x>w? + a>v? + ε with ε ∼ N (0, σ2).
When considering n observations from this generative model, we use the matrix representations y ∈ Rn,X ∈ Rn×d, A ∈ Rn×m and ε ∈ Rn. We also write the zero-mean Gaussian vector
z = (A− µ(X))v? + ε ∈ Rn ∼ N (0, σ2I + Λ)
where we have defined the diagonal covariance Λ = Λ(v?,X) = Diag ( {(v?)>Σ(xi)v?}ni=1 ) ∈ Rn×n.
We list below some notation that we will repeatedly use
• The orthogonal projector associated with X:
Πx = X(X >X)−1X> ∈ Rn×n.
• Similarly, the orthogonal projector associated with A:
Πa = A(A >A)−1A> ∈ Rn×n.
• The projections Xa⊥ = (I −Πa)X and Ax⊥ = (I −Πx)A. • The matrices: H = (X>X)−1X> ∈ Rd×n and G = (A>A)−1A> ∈ Rm×n. • The matrices above when restricted to the projections of X and A respectively, that is,
Ha⊥ = (X > a⊥Xa⊥) −1X>a⊥ ∈ Rd×n and Gx⊥ = (A>x⊥Ax⊥)−1A>x⊥ ∈ Rm×n.

heading: E.1 DEFINITION OF THE RISK
text: We will compare different estimators based on their different risks. We focus on the fixed design analysis (Bach, 2021), i.e., we study the errors only due to resampling the noise ε and the feature a.
Given a predictor τ(X) based on the training quantities (X,A, ε), we consider y′ = Xw?+A′v?+ ε′ (where the prime is to stress the difference with the training quantities without prime) and define the risk of τ as
R(τ(X))) = Eε′∼p(ε′),a′∼p(a′|x) { 1
n ‖y′ − τ(X)‖2
} . (7)
Expanding the square with y′ − τ(X) = Xw? − τ(X) + µ(X)v? + z′, we obtain the expression
R(τ(X)) = 1 n ‖Xw? − τ(X) + µ(X)v?‖2 + 1 n tr(σ2I + Λ). (8)
Following common practices (Bach, 2021), to assess the risk, we finally take a second expectation Eε∼p(ε),a∼p(a|x)[R(τ(X))] with respect to the training quantities (A, ε).
Since we will mostly consider differences of risks, we omit the variance term 1n tr(σ 2I + Λ) in the equations below.

heading: E.2 CAPTURING THE BENEFIT OF PI WITHOUT MARGINALIZATION
text: We first describe when, in absence of any marginalization, ordinary least squares ignoring PI is worse than ordinary least squares using PI with mean imputation at prediction time. Proposition E.1. Assume that X>X is invertible. Moreover, assume that A>A and [X,A]>[X,A] are almost surely invertible. We have that
E[R(τNO-PI(X))] > E[R(τPI(X))] if and only if
1 n ‖(I −Πx)µ(X)v?‖2 +
σ2d
n +
1 n tr(ΠxΛ) >
σ2
n E[‖K‖2]
with K = XHa⊥ + µ(X)Gx⊥. When m = 1 (i.e., A is a column vector), a sufficient condition is 1
n ‖(I −Πx)µ(X)v?‖2 +
1 n tr(ΠxΛ) > 2E
[ ‖ΠxA‖2 + ‖µ(X)‖2
‖(I −Πx)A‖2
] + σ2d
n .
We provide the details of the derivation of the risk for τNO-PI and τPI in Section E.2.1 and Section E.2.2 respectively. Moreover, the second part of the proposition follows from an application of Lemma E.5.

heading: E.2.1 ORDINARY LEAST SQUARES (NO MARGINALIZATION)
text: The solution of min w 1 2 ‖y −Xw‖2 is given by ŵ0 = (X>X)−1X>y = Hy. The corresponding predictions are τNO-PI(X) = Xŵ0 = Πxy = Xw ? + Πxµ(X)v ? + Πxz.
Plugging into (8), we obtain
R(τNO-PI(X)) = 1
n ‖(I −Πx)µ(X)v? −Πxz‖2.
Expanding the square and using that tr(Πx) = d, the final risk expression is
E[R(τNO-PI(X))] = 1
n ‖(I −Πx)µ(X)v?‖2 +
1 n E[‖Πxz‖2]
= 1
n ‖(I −Πx)µ(X)v?‖2 +
σ2d
n +
1 n tr(ΠxΛ). (9)

heading: E.2.2 ORDINARY LEAST SQUARES WITH PI AND MEAN IMPUTATION (NO MARGINALIZATION)
text: We focus on the solution of min w,v 1 2 ‖y −Xw −Av‖2
to construct an estimator. Using Lemma E.3, we have
ŵ1 = Ha⊥y and v̂1 = Gx⊥y.
Using Lemma E.4, we can simplify
ŵ1 = Ha⊥y = w ? + 0 + Ha⊥ε
and v̂1 = Gx⊥y = 0 + v ? + Gx⊥ε.
Since A is not available at prediction time, we impute it instead with its mean µ(X), which is assumed to be perfectly known. This leads to
τPI(X) = Xŵ1 + µ(X)v̂1 = Xw ? + µ(X)v? + Kε,
with K = XHa⊥ + µ(X)Gx⊥.
Plugging into (8) and taking the expectation, we obtain
E[R(τPI(X))] = 1 n ‖0‖2 + 1 n E[‖Kε‖2]
= σ2
n E[‖K‖2]. (10)

heading: E.3 CAPTURING THE BENEFIT OF PI WITH MARGINALIZATION
text: We then describe when, with marginalization, ordinary least squares ignoring PI is worse than ordinary least squares using PI. Proposition E.2. Assume that X>X is invertible. Moreover, assume that A>A and [X,A]>[X,A] are almost surely invertible. We have that
E[R(τmarg. NO-PI(X))] > E[R(τmarg. PI(X))] if and only if
1 n ‖(I −Πx)µ(X)v?‖2 +
σ2d n > σ2 n ‖E[L]‖2
with L = XHa⊥ + AGx⊥. When m = 1 (i.e., A is a column vector), a sufficient condition is
1 n ‖(I −Πx)µ(X)v?‖2 > 2E
[ ‖ΠxA‖2 + ‖A‖2
‖(I −Πx)A‖2
] + σ2d
n .
We provide the details of the derivation of the risk for τmarg. NO-PI and τmarg. PI in Section E.3.1 and Section E.3.2 respectively. Moreover, the second part of the proposition follows from an application of Lemma E.5.

heading: E.3.1 ORDINARY LEAST SQUARES (WITH MARGINALIZATION)
text: Restarting from Section E.2.1, we consider the predictions marginalized with respect to A. We have
τmarg. NO-PI(X) = Ea∼p(a|x)[Xŵ0] = Xw? + Πxµ(X)v? + Πxε. Plugging into (8), we obtain
R(τmarg. NO-PI(X)) = 1
n ‖(I −Πx)µ(X)v? −Πxε‖2.
Expanding the square and using that tr(Πx) = d, the final risk expression is
E[R(τmarg. NO-PI(X))] = 1
n ‖(I −Πx)µ(X)v?‖2 +
1 n E[‖Πxε‖2]
= 1
n ‖(I −Πx)µ(X)v?‖2 +
σ2d
n . (11)

heading: E.3.2 ORDINARY LEAST SQUARES WITH PI AND MARGINALIZATION
text: Restarting from Section E.2.2, we consider the predictions marginalized with respect to A. In particular, we do not impute A by its mean but rather directly take the expectation over A. We have
τmarg. PI(X) = Ea∼p(a|x)[Xŵ1 + Av̂1] = Xw? + µ(X)v? + Ea∼p(a|x)[L]ε,
with L = XHa⊥ + AGx⊥.
Plugging into (8) and taking the expectation, we obtain
E[R(τmarg. PI(X))] = 1 n ‖0‖2 + 1 n E[‖Ea∼p(a|x)[L]ε‖2]
= σ2
n ‖Ea∼p(a|x)[L]‖2. (12)

heading: E.4 TECHNICAL LEMMAS
text: Lemma E.3. Assume that both X>X and A>A are invertible. Moreover, assume that both X>a⊥Xa⊥ and A > x⊥Ax⊥ are invertible.
We can write the solution of
min w,v
1 2 ‖y −Xw −Av‖2
as ŵ = Ha⊥y and v̂ = Gx⊥y.
Proof. The proof follows by applying inversion formula for the block matrix
Q = [ X>X X>A A>X A>A ] where X>a⊥Xa⊥ and A > x⊥Ax⊥ are the two Schur complements of X
>X and A>A. Under the assumptions of the lemma, the matrix is Q is invertible.
Lemma E.4. We have the following properties
• Ha⊥X = (X>a⊥Xa⊥)−1X>(I −Πa)X = (X>a⊥Xa⊥)−1(X>a⊥Xa⊥) = I ,
• Ha⊥A = (X>a⊥Xa⊥)−1X>(I −Πa)A = 0.
Conversely, we have
• Gx⊥A = (A>x⊥Ax⊥)−1A>(I −Πx)A = (A>x⊥Ax⊥)−1(A>x⊥Ax⊥) = I ,
• Gx⊥X = (A>x⊥Ax⊥)−1A>(I −Πx)X = 0. Lemma E.5. Assume m = 1, i.e., A is a column vector. We have
E[‖K‖2] ≤ 2d+ 2E [ ‖ΠxA‖2 + ‖µ(X)‖2
‖(I −Πx)A‖2
] .

heading: Similarly, it holds that
text: ‖E[L]‖2 ≤ 2d+ 2E [ ‖ΠxA‖2 + ‖A‖2
‖(I −Πx)A‖2
] .
Proof. We start by splitting the term into
‖K‖2 ≤ 2‖XHa⊥‖2 + 2‖µ(X)Gx⊥‖2. Notice that Ha⊥H>a⊥ = (X > a⊥Xa⊥) −1 and similarly Gx⊥G>x⊥ = (A > x⊥Ax⊥) −1.
Since ‖M‖2 = tr(M>M), we have
‖K‖2 ≤ 2tr((X>X)(X>a⊥Xa⊥)−1) + 2tr(µ(X)>µ(X)(A>x⊥Ax⊥)−1).
By definition of Ax⊥, when m = 1, we have
(A>x⊥Ax⊥) −1 =
1
‖(I −Πx)A‖2 .
For the term (X>a⊥Xa⊥) −1, the Sherman–Morrison formula leads to
(X>a⊥Xa⊥) −1 = (X>X)−1 +
1
1− b>(X>X)−1b (X>X)−1bb>(X>X)−1
with b = 1/‖A‖ ·X>A ∈ Rd. Further simplifying, we obtain
tr((X>X)(X>a⊥Xa⊥) −1) = tr
( I + ΠxAA >Πx ‖A‖2 − ‖ΠxA‖2 ) = d+ ‖ΠxA‖2 ‖(I −Πx)A‖2 .
For the second part of the proof, we start by applying Jensen inequality:
‖E[L]‖2 ≤ E[‖L‖2].
The rest of the proof follows along the same arguments, replacing µ(X) by A.

heading: F RELATED WORK TABLE

heading: G TWO-STEP TRAM: IMAGENET SCALE REPRESENTATION LEARNING EXPERIMENT
text: We conduct experiment to test two things: 1) does the one-step TRAM procedure, introduced in §3.2, which is easier for practitioners to implement, approximate the two-step TRAM procedure well and 2) can the results of the toy represrntation learning experiment, §2.2, be replicated in a larger scale setting.
We train a feature extractor with and without access to PI on ImageNet, following the same procedure, architecture and dataset used in the main paper. We then freeze the feature extractor and train a single dense/linear layer with softmax activation on top of the fixed features. We then evaluate the efficacy of these features trained with and without PI using this “linear probe” evaluation widely used in the representation learning literature (Chen et al., 2020).
The results are presented in Table 6. We see that the simpler single-step TRAM method approximates the more complicated two-step TRAM method very well. In addition we see that the features learned by the network with access to PI which are then frozen and evaluated using a linear probe protocol perform better in terms of accuracy and log-likelihood.
H TOY EXPERIMENT: VARY
We vary the standard deviation of used in our motivating toy experiment, §2.2. The results can be seen graphically in Fig. 4. Fig. 4 also contains the average RMSE to the true marginal across the data points plotted. The graphical and numerical results demonstrate that even for large levels of noise PI aids with representation learning but as expected, as the level of noise grows the advantage of using PI diminishes as it becomes increasingly difficult to distinguish irreducible noise from noise which can be explained away with PI.
I IMAGENET EXPERIMENT PI ABLATION
We run an ablation, removing PI feature: the probability of the label assigned by the model from the PI set. We are thus left with just one PI feature, the one-hot encoded ID of the model that produced the label.
We see the results in Table 7. As expected (and predicted by our theoretical analysis), removing informative PI reduces the effectiveness of TRAM. Nonetheless, TRAM with the reduced PI feature set still outperforms the No PI baseline, with accuracy and log-likelihood lying between the No PI and full PI feature set TRAM methods.

Review summaries:
This paper suggests a notion that using additional information called privilege information (PI) from annotators will help explain away the noise of the label they annotated and propose a way to implement that idea. The authors give the intuition from a simple linear model and a non-linear sine model that PI is useful. Then, an implementation of that idea called TRAM is proposed to make use of the information from PI through a stop gradient operator in training.  Since there are scant large-scale datasets with PI annotation, to evaluate the proposed approach, the authors have to create a synthetic one relabeled from ImageNet. 

The manuscript proposes classification methods utilizing privileged information (PI, which is available only at training time). The main idea is to directly approximate the conditional marginal distribution, i.e., p(y|x) \approx q(y|\pi(x), w), where \pi(x) is a feature extractor and w is the weight parameter. Another main point is to learn the representation (\pi(x)) of the features with access to PI at the training time. To do this, the proposed methods employ knowledge transfer by weight sharing. The proposed methods include several variants by adopting heteroscedastic classification (Het-TRAM) or distillation (Distilled-TRAM).  

This paper develops a simple and efficient method for training with privileged information and testing without privileged information. The training with privileged information focuses on transferring via sharing the weights of the encoder, which is the knowledge learned with privileged information. The testing without privileged information addresses the practical case where test data may not be equipped with useful privileged information. The corresponding solution is to marginalize over privileged information at test time.