Transfer and Marginalize: Explaining Away Label Noise with Privileged Information
Authors: Mark Collier
Rodolphe Jenatton
Effrosyni Kokiopoulou
Jesse Berent
Venue: ICLR 2022 Submitted (2022)

Abstract: Supervised learning datasets often have privileged information, in the form of features which are available at training time but are not available at test time e.g. the ID of the annotator that provided the label. We argue that privileged information is useful for explaining away label noise, thereby reducing the harmful impact of noisy labels. We develop a simple and efficient method for supervised neural networks: it transfers the knowledge learned with privileged information via weight sharing and approximately marginalizes over privileged information at test time. Our method, TRAM (TRansfer and Marginalize), has minimal training time overhead and has the same test time cost as not using privileged information. TRAM performs strongly on CIFAR-10H, ImageNet and Civil Comments benchmarks.

Under review as a conference paper at ICLR 2022
TRANSFER AND MARGINALIZE: EXPLAINING AWAY LABEL NOISE WITH PRIVILEGED INFORMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Supervised learning datasets often have privileged information, in the form of features which are available at training time but are not available at test time e.g. the ID of the annotator that provided the label. We argue that privileged information is useful for explaining away label noise, thereby reducing the harmful impact of noisy labels. We develop simple and efficient method for supervised neural networks: it transfers via weight sharing the knowledge learned with privileged information and approximately marginalizes over privileged information at test time. Our method, TRAM (TRansfer and Marginalize), has minimal training time overhead and has the same test time cost as not using privileged information. TRAM performs strongly on CIFAR-10H, ImageNet and Civil Comments benchmarks.
INTRODUCTION
Supervised learning problems are typically formalized as learning a conditional distribution p(y|=z). y E Y' and E X' from (=), 1,... N pairs. Yet often have access to additional features a E A at training time that will not be available at test time. These features are known as privileged information (Vapnik & Vashist 2009), or PI for short. An example of PI are features describing the human annotator that provided given label, such as the annotator ID, the length of time to provide the label, the experience of the annotator, etc. Annotators do not always agree on the correct label for given æ, some annotators may be more reliable than others and the reliability of annotators may depend on the location of æ in the input domain X' (Snow et al.. 2008 Sheng al., 2008).
The expanded training dataset consists of (x; ai y:) triplets. Given that our test time predictive distribution cannot be conditioned on a, what use this PI? As a thought experiment, suppose there exists malicious (or lazy) annotator that provides random labels. It is known that random labels harm the performance of supervised learning models (Frénay Verleysen 2013 Song et al. 2020 Cordeiro & Carneiro 2020 If these random labels can be explained away via access to PI, such as the annotator ID. then this harm can be prevented. In particular we can use the PI to explain away noise in the labels which otherwise would be irreducible aleatoric uncertainty.
More formally, suppose A, the PI random variable, is predictive of Y given X, in the sense that the conditional mutual information I(Y; A ) is non-zero. Then, the entropy of Y is reduced if we condition on both X and A rather than X alone, as summarised Lemma 1.1
Lemma 1.1. T(Y: x 0= H X A) < HY))).
In 2.11 we examine this lemma for a particular model, proving that under certain conditions, PI can be leveraged to lower the expected risk for linear regression problems. Additionally, prior work has proven that PI can lead to generalization bounds with better sample complexity (Vapnik Vashist
2009 Lambert et al. 2018).
Inspired by prior work and our theoretical analysis of simple linear model, we focus on exploiting PI in supervised deep neural networks. The production deployment of such models often has tight latency and memory constraints. Hence a number of methods have been developed to utilize PI with the same test time memory and computation cost as networks trained without PI (Yang al. 2017 Lambert et al. [0018] Lopez- Paz et al. 2013). ). Yang et al. (017)) uses PI as a form of input-dependent regularizer. Lambert al. (2018) train with heteroscedastic Gaussian dropout, with the training- time dropout variance a function of the PI. Lopez- et al. (2013) distill a network trained with PI into a network without access to a.

---

Under review as a conference paper at ICLR 2022
Below develop method, TRAM, which transfers knowledge via weight sharing from the part of the network trained using PI to the test time network which does not have access to PI. At test time, TRAM makes a simple, efficient approximation to the integral p(y|æ) Making predictions without PI is no more costly than that with a standard network trained without access to PI. Unlike prior work which requires specifie techniques such Gaussian Dropout, need not constrain the form of the predictors to make the downstream marginalization possible. Implementation and training are simple.
p(y) at a d)pp(a) (z) addda.
In summary the paper contributions are the following:
To better illustrate when PI is useful, we show analytically that, under certain conditions, PI reduces the expected risk for linear regression models.
We provide empirical evidence suggesting that the representations learned with access PI are more robust against label noise.
We propose novel efficient method, TRAM, which exploits PI in supervised deep neural networks and has zero computational overhead at prediction time.
Empirically, show that method performs better than a series of baselines CIFAR- 10H, ImageNet and CivilComments benchmarks.
EXPLOITING PRIVILEGED INFORMATION
To build up intuition and to better illustrate situations where PI can be useful, start with simple linear model where a formal analysis can be carried out. Next, we look into non-linear models and provide motivating experiment suggesting that useful PI can be leveraged in deep networks to improve representation learning.
2.1
WHEN CAN PI BE HELPFUL? AN ANALYSIS IN A SIMPLE LINEAR MODEL
We consider the following regression generative model with target
x: w + a T v² + where æ E Rd and E ET correspond to standard and PI features respectively, while NO, o?) stands for some additive noise. The two unknown parameters (w" v²) establish the relationships between the target y and the features (z, a). To model the fact that the PI features themselves depend on the x e.g.g raters having diverging assessments on ambiguous input samples-we assume that a p(a (a) (u(2)) [()) for some mean and covariance dependent on æ.
Let us assume we have observations from this generative model represented by y E RR" X E R A Raxm and xx E R We are interested in comparing different predictors 7(X) that can predict only based on X, as required in the case of PI. To compare the predictors, we use the concept of risk (a=pppt).aap(aaa) | [R( (X) ]. formally defined in Appendix E capture the expected етror of 7 in predicting y; see Section 3.5 in Bach (202T) for more background about risk analysis.
We defer to Appendix E rigorous exposition of the results and convey instead here some intuitive messages. We first focus on the comparison between
(NO-PI) the least-square estimate wh = (xx - that ignores A, and
(PI) the joint least-square estimate [w0]; 01] = (QQQ)-1 Q y with Q = Xx, A in R=x(d+m)) At prediction time, we had access to (Ziest aress) for (PI), we would predict with w Fiest 6. auea However, since aust is not available in our context,
instead its (assumed known) mean p(xtes), i.e., mean imputation (Little & Rubin 2019).
Denoting by II, the orthogonal projector associated with X, defined in Appendix E our analysis shows that as long as
Variance of Pl: The variance {(v+) (Ex)) due to Pl is large enough and/or
Alignment of PI: The PI features A have a significant average component outside of the subspace spanned by the features X, ic.. the term below is large enough
I II. II)) (X)'''1
(1)
7
2

---

Under review as a conference paper at ICLR 2022
Step Representation learning no Pl Step Luast-squares solution
sont kayers
(60g (00) AI) '0o0'
function Expected marginal
420
10
dod 5
Observation
No
93
P
Step Representation learning with Pl Step 2: Least-squares solution
-05
(00)))01) "oots""
- 9000
-10
hand oubout
02
1.0
(a) Learning the representation (xx) without (top) or using PI (bottom).
(b) Comparing representations learned by No vs. using PI in a two-step procedure.
Figure I: Toy representation learning experiment. In Step 2 (common to both methods) we solve least-squares problem assuming ..,,..., UNl E EN and ((x) = (((x))....0xzxx) E IR vxr.
then the estimator (PI) has a lower risk compared to (NO-PI).. In other words, it is provably better to exploit the privileged information at training time instead of ignoring it.
Our analysis further covers the case of (marg. NO-PT) where we marginalize tho with respect to PI and we predict with X (appp(a))) [zo.]. which we compare with (marg. PT) the marginalized predictions Ea-p(a] X " + Aöj]. In that case, we can show the same conclusion as previously, with the exception that the variance term {(v*) E(x,) )or) In [;=] does not have influence anymore, only (1) drives the comparison. Indeed, the proof in Appendix E shows that marginalizing removes
from the risk expressions the terms related to the variance of Pl.
2.2 PI HELPS TO LEARN BETTER REPRESENTATIONS A MOTIVATING EXPERIMENT
Our analysis of the effect PI in a linear model has established key insights into the conditions under which PI is provably useful. We now look at toy non-linear learning experiment using neural networks, which provides empirical evidence that PI can be helpful in deep networks. In particular, we show that representations learned with access to useful PI can explain away label noise and transfer better than representations learned without access to PI. This motivating experiment forms the basis of our TRAM method.
We simulate a noisy annotator by assuming in this toy example that the PI a binary indicator, (ernoulli with 1 representing the case where the noisy annotator provides label independent x::
== (1 a) sin() + aav
where x E [0, ~~ u(-1,1) and e ~ 0..0..).
We then fit two networks N 2, 500 training examples generated according this process. The first network does not have access PI and is a two layer MIP, both layers dimension 64, with tanh hidden activations and linear output activation; see top left of Fig. [a an illustration. The second network has access to PI, and is defined as per the two-step TRAM approach. The part of the network which learns the x representation, é, is defined exactly as the no-PI MLP. (yyae,, a), the output head with access to PI, see Fig. aa is a single layer MLP with 64 units, tanh activation and linear output layer, with the concatenation of a and ((z) as inputs; see bottom left of Fig. Both networks are fit for 10 epochs by the Adam optimizer (Kingma & Ba 2014) with mean squared error loss function.
We then extract the non-linear representations of learned by both networks and fit a linear model on the dataset (Φ(z)), y. i 1, ... N. The linear model can be solved exactly using the I.2 regularized ordinary least squares solution. We plot the results in Fig. | We see that the representations learned by the model with access to PI in step #1 enable near perfect fit to the true expected marginal distribution, E p(aa)))p(a))=)) [y] accross X space. However without access to PI the noise term a v cannot be explained away. As result, the linear model fit top of the representations learned without access to Pl is substantially worse than the model fit using the two step procedure. We emphasize that both models have exactly the same capacity.
3

---

Under review as a conference paper at ICLR 2022
B
øix]
qly xx
Stop Gradient
PrMllged Information
@1p(x))
niy
Test time
B
g
aly x)
Predicted
Label
Figure 2: The TRAM method in diagrammatic form.
In Appendix we extend this representation learning procedure to a large-scale image classification case. We learn representation with and without access to PI on a relabeled version of ImageNet (details in 5.2 using a ResNet50 (He et al. 2016 We then freeze the representation and evaluate using a linear model. Access to PI improves the representations learned.
METHOD: TRAM
We consider learning under privileged information /Vapnik & Vashist 2009), LUPI. Our proposed
method, TRAM, consists of a single neural network with two output heads, providing predictions for both p(y (z. a) and p(y|z); head is a simple, yet a provably valid, approximation to the marginal / p(y) æc a)p(a(zz) ( da and (ii) a partition of the parameter space such that the neural network weights output heads, and that these shared weights are updated solely based on the lower variance gradients from the p(y (z, a) head which has access to Below we develop TRAM in the classification setting, in $3.4 (.4)va we extend the method to the regression setting.
Fig. 2 There are two key ingredients to TRAM; (i) the p(y ax)
shared between the two
3.1
INGREDIENT #1: MARGINALIZE OVER PI AT TEST TIME
natural probabilistic approach to LUPI is (i) to learn the conditional distribution p(y| wx, a) dur- ing training and (ii) then, at test time, marginalize over the A domain, computing p(y(æ) = p(y|2x, a)p(aa))aaaa (z) ddd (Lambert et al. 2018).
Predicting with the marginal p(y/æz) at test time is motivated by the following key observation. Consider the set of distributions Q over the C' class labels. = [v(l)))\V> X. g(yla))))) E Ac} where Ag is the C-dimensional simplex. Among all the distributions
Q, the marginal distribution
+ (z) minimizes the following optimization problem:
min ((æ,a)<p(a.a)
[Dr (KI(ppyle. a) [[(()=)))
qeQ
See proof in Appendix B In words, p(ylx) is optimal in the sense that it minimizes the expected KL divergence to p(yl x., a.. Access p(y) æ)) enables minimizing the Bayes risk (Murphy 2012) at test time. Note further that the mean imputation scheme which provably reduces the expected risk for a linear model, 2 corresponds precisely marginalizing over PI test time.
Directly computing p(y ar) has two problems; (i) it is typically intractable and (ii) pl a az) is unknown and therefore must be learned, which challenging generative modelling problem in itself, espe- cially as a may have mixed type features. A Monte Carlo estimate of the integral using the samples from A in the training set is only feasible with the independence assumption p(a aa))) = p(a), so that p(y[z) reduces to f ppylz x a) a))(()dd xe E P(y x a.) with a, ~ p(a).
Unfortunately this independence assumption often violated in practice. In addition the memory and computational cost of MC estimation scales linearly in S, the number of MC samples. This ( scaling undesirable for production deployment with strict latency requirements.
Due to the challenge of computing the integral directly, we propose a simple approximation g(yl æ;; w) to plyla)). It exploits the property (2) of p(y(æ=) as the distribution minimizing the expected KI.
4

---

Under review as a conference paper at ICLR 2022
divergence to its conditional p(y|2, a). We choose q to be cheap to evaluate at test time. For example, for a multi-class vanilla TRAM classifier q(y/2; w) = softmax W Wo(x)).
3.2 INGREDIENT #2: TRANSFER VIA WEIGHT SHARING
We partition the parameter space into four disjoint subsets;
1. Let (xx) be feature extractor for E X.
2. Similarly, (((()), a) be feature extractor jointly applied to (ø(œ), a) for (æ, a) in
X' x A.
3. The weights w parameterize the marginal distribution: q(y]<; w) ((vlo(x): w))
4. The weights u parameterize the conditional distribution glyl2r, a; u), namely qlyl,, u) = (()))()())..)))))
Two-step approach. Motivated by Eq. (2) the connection between LUPI and multi-task learn- ing (Jonschkowski et al. (2166) and our toy representation learning experiments, 2.2 consider the following two-step approach:
min R ((z.a.)) [Li (V. a(vla, a))] u.d.v
(3)
min (( (2 [((+))) with Φ = o*
(4)
C. are arbitrary loss functions, We assume and vb are parameterized as neural networks, so mins. refers to optimizing the network weights.
Crucially @* is the feature extractor learned in 3 with access knowledge transfer to the network trained without PI. Given Eq. (2) we know that Eq. (4) approxi-
This weight sharing enables
mates the true marginal distribution p(y|æ) (observe that the KL divergence in Eq. 2 reduces to the cross-entropy loss function for L2 when taking the one-hot training labels for p(y x a.).
Merging the steps. To further simplify the above approach, we propose merge Eq. 3 and Eq. (4) into single training procedure. To that end, and reusing the terminology commonly used in deep-learnigg frameworks, us define
x (y æ w) =
which coincides with ( (z) except that its gradient only depends on w. For some 0, we then consider:
min E (la.e.a)-paa... a.)) [C2(u (y)=)) BB (v, (1. (1)
24. M. P
as the joint training objective. In practice, since the parameters of the two losses are partitioned, can set 3 = 1 and fold instead the search over 3 into the search of the learning rate, hence not introducing an extra hyperparameter.
3.3 TRAM VARIANTS
Privileged information may only explain away of label noise uncertainty. Below propose two TRAM variants which combine TRAM with existing noisy labels methods.
Het-TRAM. Heteroscedastic classifiers are capable of modeling label noise that is input-dependent and have been successfully applied in this setting (Collier et al. 2021). Further note that even in some cases where the conditional distribution q(y) a) homoscedastic, the marginal q æ:) is heteroscedastic, see Appendix C for details.
Hence propose Het- Het-TRAM, TRAM variant in which q( (z) is heteroscedastic. This increases the expressiveness q, improving the approximation in the second step of our optimization procedure, Eq. (4). We implement the method of Collier et al. (2021) to make (yy/==) heteroscedastic.
5

---

Under review as a conference paper at ICLR 2022
DistilledTTRM.. Distillation (Hinton al. 2015) is technique for transferring knowledge between two neural networks. Distillation has been previously applied to LUPI (Lopez-Paz et al. 2015). In DistilledTRRMM we use the two step TRAM procedure, setting the loss function, L₁ in Eq. (3. to the distillation loss. The teacher network is first trained with access to PI and in the second step the distilled-TRAN model is trained.
3.4 REGRESSION
We developed TRAM and Het-TRAM focussing on the classification setting but our approach is trivial generalize regression problems. In the regression case,
choose the predictive distribution For vanilla TRAM we can choose 0²(xx) 1, while for
to be Gaussian, ( æ) = N(((()),o Het-TRAM we can choose o² (z) softplus (w, Φ(x) )). μ and o² are parameterized as neural networks with our shared feature extractor ((x), similar to Kendall & Gal (2017). L1 in Eq. (3) and
L2 in Eq. are replaced by the Gaussian negative log- ligliikllihod..
4 RELATED WORK
Vapnik & Vashist 2009) develop theoretic framework for the LUPI paradigm and introduce the
SVM+ method for training Support Vector Machines in this regime. The slack variables for the SVM+ constraints are function of the PI. SVM+ has been extended in the SVM literature (Lapin)
ct al. 2014, Vapnik& & Izmailov 2015). Jonschkowski et al. (2016) provide a unifying framework that connects together multi-task learning, multi-view learning and LUPI.
Yang et al. (2077) extend the SVM+ approach to neural network models with their MIML-FCN+ method. The authors formulate two-tower network similar to ours, but without weight sharing between the towers. Both towers make independent predictions given æ or a as inputs. The tower with access to PI predicts the loss of the other tower and this prediction is regularized be close to the true loss. In this way the PI tower outputs a neural network analogue to the SVM+ slack variables.
Lambert al. (2018) utilize PI by making the training-time Caussian-dropout variance (Kingma et al.] 2015) a function of the PI. At test time the PI is approximately marginalized over by removing the dropout. Similarly (Heráánezz-Lotee
al. (2014) allow the additive Gaussian noise component of heteroscedastic Gaussian Process Classifier (Rasmussen & Williams 2006) to be function of the PI. The classifier is homoscedastic test time.
Lopez-Pazz et al. (2155) propose a distillation (Hinton et al. 2015) style approach to learning with PI. The teacher network is trained with access to PI. In the distillation step the student network is given input and convex combination of soft labels from the teacher network and true labels as targets. Xu et (220)) extend and apply this distillation method to a recommender system.
TRAM implements knowledge transfer via weight sharing, performs efficient approximate marginal- ization at test time and can be applied to many widely used architectures. [Lambert et al. (2018) also share weights and approximate the marginal p(y xx) however they require the use of Gaussian dropout, which not widely used. The distillation and MIML-FCN+ methods do not transfer via weight sharing and do not approximate p(y|æ)) Distillation also requires two-step training procedure. See Table [] in Appendix F for a comparison of the key features of selected LUPI methods.
EXPERIMENTS
Our experiments tackle the general LUPI problem under label noise. There are few large-scale public datasets with PI We thus use both real-world datasets with Pf as well as synthesizing PI for re-labelled version of ImageNet (Deng al. 2009).
We evaluate a number of baselines in addition to our method.
The "No PI" baseline is standard neural network training which directly learns p(ylæ: and
never uses PI.
Zero and mean imputation learn p(y|æ, a) at training time and substitute and a = * Σ; a; respectively at test time. For mean imputation, averaging takes place after feature pre-processing, e.g., one-hot encoding of the annotator IDD.

---

Under review as a conference paper at ICLR 2022
Table CIFAR-I0 negative log-likelihodd & accuracy (trained on CIFAR-IOH), Averaged over 20 training runs + std. dev.
METHOD
JNLL
FACCURACY
No PI
1.058 + 0.050
67.0 = 1.7
ZERO IMPUTATION
1.009 0.032
68.7 = 1,4
MEAN IMPUTATION
0.963 0.058
70.1 1.5
LAMBERT ET AL. (2018)
1.033 + 0.044
67.1 1.3
FULL MARGINALIZATION
1.119 1 0.058
70.3 = 2.5
TRAM
0.980 0.037
70.1 = 1.4
HET-TRAM
0.972 0.038
70.4 = 1.5
DISTILLATION NO PI
1.118 0.037
70.1 = 1.4
LOPEZ-PAZE ET AL, 2015
1.121 0.040
70.2 = 1.4
DISTILLED- IRAM
0.941 0.039
71.8 = 1.4
The "Full marginalizaaion" baseline is expensive MC estimate of p(y|æ)
gold standard (up to in- dependence assumption error), impractical to compute in many applications.
f plylæ, x a (p(aaz)da at test time, see $3.1 for details.
We also compare against distillation based approaches. "Distillation No PI" is an ablation to see the effect of distillation alone, independent of PI, in which network trained withour access to PI is distilled into another network also without access to PI.
Prior work did not evaluate against these simple imputation baselines or full marginalization (Lopez Paz et al. 2015 Yang et al., 2017: Lambert et al. 2018
5.1
CIFAR-10H
One dataset with annotator features CIFAR-IOH PPteerso et al., 2019), which is a re-labelled version of the CIFAR-IO (Krizhevsky & Hinton 2009) test The new labels are provided by crowd-sourced human annotators. We make use of three annotator features; the annotator ID, the reaction time of the annotator provide the label and how much experience the annotator had with the task, as measured by the number of labels the annotator had previously provided.
As we only have annotator features for the CIFAR-10 test set, we use this as our training set and evaluate on the official training set. As result we have only 10,000 images for training. To achieve reasonable performance we start from MobileNet (Howard et al. 2017) pretrained on ImageNet. Images have on average > 50 annotations each. This is unrealistic for typical applications where 1-3 labels per example is more common. Therefore, we subsample 16,400 labels (1.64 labels per example), see Appendix D for details of the subsampling procedure. The subsampled labels agree with the true CIFAR-10 test set labels 79.4% of the time.
In Table we the results. First, and as expected, using annotator features via TRAM, marginaliza- tion or the imputation methods provides a performance improvement over standard neural network training without PI. Second, 16,400 MC samples a from the training set), despite having constant time compute and memory requirements w.r.t. the number of MC samples for the full marginalization baseline. Mean imputation a strong baseline on 1IFAR-1OH. Het-TRAN improves over TRAM demonstrating the efficacy of making q(y) ar, a) heteroscedastic. It noteworthy that distillation using PI, (Lopez-Paz al., 2015) does not improve over standard distillation without PI. However Distilled- -RAM with makes use of PI for distillation but then performs approximate marginalizaation and transfer learning via weight sharing improves over the distillation baselines on both accuracy and lig-likeldhood metrics.
that TRAM performs par with full marginalization (which uses
5.1.1 QUALITATIVE ANALYSIS OF CIFAR- 10H RESULTS
We qualitatively analyse how PI is helping improve the performance of TRAM CIFAR-10H. The PI for CIFAR-1O -OH does not contain a feature for annotator reliability. However the PI does contain features such the annotator ID and the annotator reaction time from which it may be possible to
7

---

Under review as a conference paper at ICLR 2022
0.5
model
No Pt confideneee TRRAM canfidence
0.3
TRAN more confident
TRAN less confident
0.8
0.2
0.0
-0.2
0.2
94
06
08
10
Prob rater greea connect (pater relubbity)
rater gives correct (rater reliabiity)
(a) Average confidence per model.
(b) Delta in average confidence between models.
Figure 3: How model confidence varies with annotator reliability for CIFAR-1OH. Each point represents a single human annotator, The x-value is the probability the annotator's 's label agrees with the true CIFAR-10 label. See individual captions for y-value meaning. Confidence is defined as the maximum probability given by the model across the 10 labels.
Table 2: ImageNet validation neg-log- likelli-- and accuracy. Avg. over 10 seeds + std. dev.
METHOD
JNLL
TACCURACY
No PI
1 1.264 + 0.007
71.7 0.2
ZERO IMPUTATION
1.895 + 0.008
63.5 0.2
MEAN IMPUTATION
1.619 +.0000
65.1=0.
LAMBERT AL. (2018)
1.264 0.006
71.8
FULL MARGINALIZATION
1.217 0.004
72.6=
TRAM
1.225 0.006
72.5
HET-TRAM
1.207 0.008
72.8
DISTILLATION NO PI
1.207 +
72.6 = 0.2
LOPEZ- PAZ AL. 2015
1.216 1 0.003
72.7 = 0.2
DISTILLED- TRAM
1.154 1 0.004
73.8 + 0.2
learn to trust some annotators more than others. TRAM can learn output less confident distribution for unreliable annotators, thus reducing the harmful impact of incorrect labels.
In Fig. 3 we show an analysis of the confidence of the TRAM and No PI (i.e., standard) models for each annotator CIFAR-1OH. We see that the trend for TRAM is a strong linear relationship between the reliability of an annotator and the confidence of the model, Fig. 5a The TRAM model is consistently more confident than the No PI model for reliable annotators while the No PI model is overconfident for unreliable annotators, Fig. 3b
5.2 IMAGENET ILSVRC12
In order to create a large-scale dataset with annotator features, we re-label the ImageNet ILSVRC12 training set by the following procedure. We download 16 different models pre-trained on ImageNet, see Appendix for further details. We also add a 17th malicious annotator model which picks a label uniformly at random from the 1,000 ImageNet ILSVRC12 classes. For each image in the training set we select the malicious annotator with 10% probability and otherwise sample one of the 16 models with equal probability. We then sample label from the predictive distribution of that model for that image. This is the label used for training. On average the sampled label agrees with the true ImageNet label 68.3% of the time. The annotator features are the model IDD (a proxy for a human annotator ID) and the probability of the label assigned by the model (a proxy for the confidence of human annotator). The ImageNet image is used as the non-privileged information æ. is randomly initialized 5e (He al.) 2016).
See Table 2 for the results. The full marginalization baseline uses 1,000 MC samples of a from the training set. The imputation baselines perform worse than not using PL, perhaps due to the imputed values having low density p(a xx). Again TRAM performs on par with full marginalization and Het- TRAM has higher accuracy than both. The ImageNet labels are known to exhibit heteroscedasticity
8

---

Under review as a conference paper at ICLR 2022
Table 3: Civil Comments Identities test set negative log-likelihood and average accuracy over classes. Averaged over 10 training runs + I std. dev.
METHOD
JNLL
t AvG. ACCURACY
No PI
0.085 + 0.011
97.8 0.12
ZERO IMPUTATION
0.073 0.004
98.2 + 0,01
MEAN IMPUTATION
0.069 0.003
98.2 0.02
LAMBERT ET AL. (2018)
0.084 0.012
778
FULL MARGINALIZATION
0.065 1: 0.004
97.8 0.00
TRAM
0.064 0.002
98.2
HET-TRAM
0.062 0.001
98.1 = 0.1
DISTILLATION NO PI
0.094 0.011
97.8 + 0.1
LOPEZ- PAZ ET AL. 2015
0.089 0.000
97.8 = 0.0
DISTILLED IRAM
0.065 + 0.001
98.2 = 0.0
(Collier et al. 2021). therefore we make both g [z) and q x a) heads heteroscedastic for Het- TRAM. Distilled-TRA. has significantly better NLL and accuracy than the two distillation baselines.
5.3 CIVIL COMMENTS
We further evaluate our method on a large-scale text classification dataset. Civil Comments is collection of comments from independent news websites annotated with toxicity labels (identity attack, insult, obscene, severe toxicity, sexually explicit, threat, toxicity). The Civil Comments Identities subset of the Civil Comments data contains privileged information in the form of 24 attributes identified in the comment (male, female, christian and on). The Identities subset consists of 405,133 training examples, 21, 21,293 validation examples and 21,577 test set examples.
The shared network & is pre-trained Universal Sentence Encoder (Cer al. 2018). Table 3 contains the test set results. We report negative log-likelihood and accuracy averaged over the 7 labels. The TRAM, TeAM and imputation methods perform similarly well in terms of average accuracy, outperforming the No PI baseline as well as the Gaussian Dropout and full marginalization methods.
The poor accuracy of the full marginalization method is interesting to note. The PI is directly derived from the non-PI (in the form of 24 identity human labelled attributes for the non-PI). This is clearly violation of the independence assumption required for MC estimate of full marginalization to be computable. The dependence of a on & is most clearly identifiable for the Civil Comments Identities dataset; as a result the relative performance dataset. Further note that the TRAM and Het- TRAM methods have lower negative lideeiodl than all other baseline methods. Standard distillation with no PI and Lopez-Paz et al. (2015) style distillation where the teacher network is trained with PI does not provide performance improvement over the no Pl baseline. Distilled- TRAM performs on par with vanilla TRAM.
the full marginalization method is poorest on this
CONCLUSION
We introduced TRAM, a new method for LUPI supervised neural networks. TRAM (i) learns an efficient, simple distribution to approximately marginalize PI at test time and (ii) partitions the parameter space enabling transfer via weight sharing of the knowledge learned with access to PI. TRAM can be successfully combined with established methods for dealing with noisy labels; distillation (Distilled-TRAM) and heteroscedastic output layers (Het-TRAM). We have analysed a linear model with PI where deriving analytic results feasible. In this setting have shown the utility of using PI and ingredient #1 of our TRAM method, the marginalization over PI. Using a toy low-dimensional problem we have further shown the effectiveness of ingredient #2 of our proposed TRAM method, transfer learning via weight sharing of representations learned with access to PI. We then have empirically validated the angle-step TRAM procedure on parger-ccale datasets in the image and text domain; CIFAR-1OH, noisy version of ImageNet and Civil Comments Identities.

---

Under review as a conference paper at ICLR 2022
7 ETHICS AND REPRODUCIBILITY STATEMENTS
Ethics. We present a generic algorithm for classification/regressin with privileged information.
The method is not application specific and has minimal computational overhead. We do not ethical issues specific to our method beyond any already associated with LUPI methods and/or deep neural network classifiers
Reproducibility. We provide detailed information on the data generation process for the CIFAR- 10H and Imagenet datasets in Appendix D.I
Hyperparameters and further details required to reproduce our results are provided in Appendix D.2 We plan to release publicly Colab notebooks to reproduce our toy transfer learning and Civil Comments experiments. We also plan make public our Imagenet dataset in Tensorflow Dataset format to be used as a large-scale dataset with annotator features by researchers working on the LUPI problem. Detailed derivations with conditions for our theoretical analysis are presented in Appendix
REFERENCES
Francis Bach. Learning Theory from First Principles. (draft), 2021.
Daniel Cer, Yinfei Yang. Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Steve Yuan, Chris Tar, et al. Universal sentence encoder. arXiv preprint 1rXiv:/803.. 115, 2018.
Mario
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Mark Collier, Basil Mustafa, Efi Kokiopoulou, Rodolphe Jenatton, and Jesse Berent. Correlated input- dependent label noise in large-scale image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Filipe R Cordeiro and Gustavo Carneiro. A survey on deep learning with noisy labels: How train your model when you cannot trust on the annotations? In 2020 33rd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pp. 9-16. IEEE, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scall hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: survey. IEEE Iransactions on neural networks and learning systems, 25(5):845-869, 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Daniel Lemmnndez-Lobtoo Viktoriia Sharmanska, Kristian Kersting, Christoph H Lampert, and Novi Quadrianto. Mind the nuisance: Gaussian process classification using privileged noise. In Neural Information Processing Systems, 2014.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.02533 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint / 2017.
Rico Jonschkowski, Sebastian Hofer, and Oliver Brock. Patterns for learning with side information. arXiv preprint arXiv: 1511: 11..06429, 2016.
Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for computer vision? In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. -5500-559,,00 2017.
10

---

Under review as a conference paper at ICLR 2022
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P Kingma, Tim Salimans, and Max Welling, Variational dropout and the local reparameteri- zation trick. arXiv preprint urXiv:/50.002555 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31st International Conference Neural Information Processing Systems, pp. 6405 6416, 2017.
John Lambert, Ozan Sener, and Silvio Savarese. Deep learning under privileged information using heteroscedastic dropout. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8886-8895, 2018.
Maksim Lapin, Matthias Hein, and Bernt Schiele. Learning using privileged information: Svm+ and weighted svm. Neural Networks, 53:95-108, 2014.
Roderick JA Little and Donald B Rubin. Statistical analysis with missing data, volume 793. John Wiley & Sons, 2019.
David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information. arXiv preprint arXiv:1511.0366 2015.
Kevin P Murphy. Machine learning: a probabilistic perspective. MIT press, 2012.
Joshua Peterson, Ruairidh M Battleday, Thomas Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9617-9626, 2019.
Carl Edward Rasmussen and Christopher KI Williams. Gaussian processes for machine learning.
The MIT Press, 2006.
Victor Sheng, Foster Provost, and Panagiotis G Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 614-622, 2008.
Rion Snow, Brendan O' connor, Dan Jurafsky, and Andrew Y Ng. Cheap and fast-but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 conference on empirical methods in natural language processing, pp. 254 263, 2008
Hwanjun Song, Minscok Kim, Dongmin Park, Yooju Shin, and Jac-Gil Lee. Learning from noisy labels with deep neural networks: A survey, arXiv preprint arXiv:2007.08199) 2020.
Vladimir Vapnik and Rauf Izmailov. Learning using privileged information: similarity control and 2049, 2015.
knowledge transfer. J. Mach. Learn. Res.,
Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged informa- tion. Neural networks, (2(55))):44455555 5574444-5555005 2009.
Chen Xu, Quan Li, Junfeng Ge, Jinyang Gao, Xiaoyong Yang, Changhua Pei, Fei Sun, Jian Wu, Hanxiao Sun, and Wenwu Ou. Privileged features distillation at taobao recommendations. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2590-2598. 2020.
Hao Yang, Joey Tianyi Zhou, Jianfei Cai, and Yew Soon Ong. Miml-fen+: Multi-instance multi-label learning via fully convolutional networks with privileged information. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1577-1585, 2017.
11

---

Under review as a conference paper at ICLR 2022
A APPENDIX
B PROOF OF EQUATION (2)
As reminder, we consider C class labels and denote by Ac the C'-dimensional simplex. We define the set of distributions Q over the C' class labels by
( Vo [z)(() Ac}.
Consider the optimization problem
min pp))) ge
[Dx (p(y)a) [((l))))]
(5)
whose solution is straightforwardly given by the marginal distribution æ + (y((x) p(y(z). We recall that the KI. (xz(p(y)2)) (1)()())) is defined by
Po (2(()az))
Dkz(plyl=) 14 alv x)) 1 (psle
(6)
qj (3(y)()))
j=l
For any a and j S C, we can rewrite the terms of the sum
()(())))
py(((() log
43 (1yla)
Pi (1/(z
Ba p(la)))(a
Py (y le, log
(y(y)r))
where we have used (i) the fact that log( 25 ()(()))) marginal distribution
does not depend on a and (ii) the definition of the
()(()=)= = (ula,, a)p(alx)da
Balesp(ala) (ylll,)))) a)]
Multiplying and dividing in the argument of the log by P3 (yla,, a), we obtain
(1)))( a)
Pa (y(xx
E p(pa)())))) (,(l a) log
(,(v/=))
p,(
Summing over j E {1, O} to reconstruct the KL term DxL(p(yl=)]l<<<)>)))) Eaja [Dx a a) [))]
6 this leads to, for any
-Faj (P a:) p(y]=)))
Since the second term above does not depend on q, minimizing is equivalent to minimizing
min
Em [aa (DKr x a) [y [))]]
q
min
((.,a)~p(e.a)
a) a [])]
q
which is equal to (2) and which is, analogously minimized the marginal distribution
x + 9² (yl x) = x)
C HETEROSCEDASTIC MOTIVATION
We consider a simplified special case of our framework in which the conditional model p(y æ, a) homoscedastic but the optimal variational distribution in the sense of Eq. 2 is heteroscedastic. This motivates Het-TRAM, in which the variational approximations q x and q xx, a) are heteroscedastic.
Suppose we have a regression dataset constructed from labels assigned by M annotators. Each annotator has their own homoscedastic Gaussian model p(y p. m) N (m), 1). Here the
12

---

Under review as a conference paper at ICLR 2022
Table 4: Pre-trained models used re-label ImageNet ILSVRCI2 training and their accuracy on that training set.
Model
Training set accuracy
ResNet50V2
0.70086
ResNet101V2
0.72346
ResNet15522
0.72738
DenseNet121
0.74782
DenseNet169
0.761 84
DenseNet201
0.77344
InceptionResNert22
0.8049
Inception V3
0.77994
MobileNet
0.70594
MobileNetV2
0.71458
MobileNetV33Large
0.75622
MobileNetV3Small
0.68158
NASNetMobile
0.74302
VGG16
0.71178
VGG19
0.71156
Xception
0.79076
PI is single discrete Categorical feature representing the annotator ID which takes one of M values with equal probability, a Cat( ).).
The marginal ply x) is a Gaussian Mixture Model. We choose our variational family to be the Gaussian distribution, g(ylar) (x)) ((())) The values of f and a² that minimize Eq. are: (xx = M Σn Me,,. (ar) and (m) (M nt (x)) + M Σ P. (æ) (Lakssmiinarayanan jt al. 2017). Crucially note that despite the conditional distribution being homoscedastie, the best
variational distribution is heteroscedastic as the variance depends on the location in X' space.
D EXPERIMENTAL DETAILS
D.1 DATA GENERATION PROCESS
We use the CIFAR-10 image as the non-privileged information The annotator ID. the number of prior annotations the annotator has provided and the reaction time milliseconds of the annotator, used privileged information a. For feature pre- pre-processing the annotator ID is one-hot encoded, The number of prior annotations and the reaction time are independently divided into 10 equally sized quantiles and the quantile IDD is one-hot encoded. The image pre-processed according the the standard MobileNet pre-processing (Howard et al., (017).
CIFAR-10H.
As CIFAR-1OH has average more than 50 annotations per image and the labels not particularly noisy. We subsample the CIFAR-1OH labels by the following procedure. We keep all labels by the 41 annotators that agree with the CIFAR-10 label less than 85% of the time. We then select further 41 annotators from the remaining annotators. The average agreement of the bad annotators with the CIFAR-10 label is 63.3%, the full subset of labels: 79.2% and in the full CIFAR- 10H dataset: 94.9%. The subsampling procedure leaves 16,400 labels from 82 annotators while the full CIFAR- 10H dataset has 514,200 labels from 2,571 annotators.
ImageNet. The annotator features are the model ID used re-label æ,, which is one-hot encoded and the probability of that label being sampled. See main paper for details on the sampling procedure and see Table for the list of models used and their accuracy on the ImageNet training set. The pre-trained models are downloaded from tf keaaa..apl
13

---

Under review as a conference paper at ICLR 2022
D.2 HYPERPARAMETERS
CIFAR-IOH. For all methods ((x) (or equivalent) is a MobileNet (Howard et al. (017) pre-trained on ImageNet ILSVRCI2, followed by global average pooling layer and Dense ReLU layer with 64 units. (xx a) is two-layer MLP with 64 units per layer and ReLU activation. The first layer takes only an imput, while the second layer takes the output of the first layer concatenated with (xx) as input.
All models are trained for 20 epochs with the Adam optimizer with base learning rate= 0.001, 31 = 0.9, B2 0.999, c = le 07. All models are trained with L2 weight regularization with weighting le 3.
Heteroscedastic models are trained using the method Collier et al. (022 with 4 factors for the low-rank covariance matrix approximation and a softmax temperature parameter of 3.0. Distilled models are also trained with a softmax temperature of r = 3.0 to smooth the teacher labels and with the distillation hyperparameter = 0.5 which weights the losses from the soft teacher labels and the true labels. A grid search over E {1.0, 2.0, 3.0, 4.0) and A {0.0, 0.25, 0.5, 0.75, 1.0} was conducted.
ImageNet. For all methods ((x) (or equivalent) randomly initialized ResNet-50 (He al. 2016) with the output layer removed. s/(x,, a.) is a two-layer MLP with 128 units per layer and ReLU activation, the output of this MLP is concatenated with of ax) and then passed to the output layer. The first layer of the (xx, a.) MLP takes only an input, while the second layer takes the output of the first layer concatenated with (xx) as input.
All but Het-TRAM models are trained for 90 epochs with the SGD optimizer with base learning rate= 0.1, decayed by factor
after 30, and epochs. Following Collier et al.) (202T), Het-TRAM is trained for 270 epochs with the same initial learning rate and learning rate decay at 90, 180 and 240 epochs. All models are trained with L2 weight regularization with weighting 1e
Heteroscedastic models use 15 factors for the low-rank covariance matrix approximation and softmax temperature parameter of r = 1.5. Distilled models are trained with a softmax temperature of 3.0 and with the distillation hyperparameter À 0.5. A grid search over {1.0, 2.0, 3.0, 4.0) and À E {0.0,04 0.25, 0.5, 0.75, 1.0} was conducted.
E RISK ANALYSIS
Generative model and notations.
We assume the following
a E 1 I E IRd,
a p(a a)) ((((()) for some mean and covariance dependent on æ;
y w TTv + with o2,
When considering observations from this generative model, the matrix representations E R" X E Raxd
E Raxm and & E R.. We also write the zero-mean Gaussian vector
z = (A - o()))o* teE Rd NO
where we have defined the diagonal covariance
A A(v X) = Diag (iag(((u*) ( E(x,)= ) E Baxn
We list below some notation that we will repeatedly use
The orthogonal projector associated with X:
II, X x) -1 T E Rann
Similarly, the orthogonal projector associated with A:
I1 A(A A AT E m
14

---

Under review as a conference paper at ICLR 2022
The projections XT1 XI and A.1 = =I II
The matrices: H = -Xx XX"X)- E Rdx= and = (AT A E Rmxx.
The matrices above when restricted to the projections of X and A respectively, that is,
Ha.1 ( R and G.1 (A A - eR
E.1 DEFINITION OF THE RISK
We will compare different estimators based on their different risks. focus on the fixed design analysis (Bach 2021). i.e., study the errors only due to resampling the noise and the feature
Given predictor 7 rXX) based on the training quantities AX E), we consider y Xx" + A'v"" e' (where the prime is to stress the difference with the training quantities without prime) and define the risk of 7 as
((T(X))) = Epg'mp(e)), (opppppea'aaaae a'
11 - T {xx)11
1)xx 1 + z', we obtain the expression Expanding the square with y/ T =(X) = X 2 7 (X) + X
R(T(X)) X v0" (X) μ v
-r(o2 A).
7
Following common practices (Bach 2021), to assess the risk, we finally take a second expectation (==p(() (X)] with respect the training quantities E).
Since we will mostly consider differences of risks, we omit the variance term n + in the equations below.
E.2 CAPTURING THE BENEFIT OF PI WITHOUT MARGINALIZATION
We first describe when, in absence of any marginalization, ordinary least squares ignoring PI is worse than ordinary least squares using PI with mean imputation at prediction time. Proposition E.1. Assume that X X invertible. AX X A] are almost surely invertible. We have that
Moreover, assume that AT A and
EiR(TNO (X)) > B[R(TH((X))
if and only if
odd + IT(IL,
d² K|]
(I II, 00
n
with K = XHH + (X)) When (i.e. A is column vector), sufficient condition is
[[ A + (x(
o²d
I (I
[I 2E
2
We provide the details of the derivation the risk for TNO-Pt and Tpi in Section E.2.I and Section E.2. respectively. Moreover, the second part of the proposition follows from an application of Lemma E.5
E.2.1
ORDINARY LEAST SQUARES (NO MARGINALIZATION)
The solution of
min yy X |
2
is given by wh (X
X
1 - y Hy. The corresponding predictions are
TNo PI X) X HTy Xwe + II,(()o' + II,
Plugging into
we obtain
1
R(TNO-Mi (X)) [[ II, (X)e** 212
Expanding the square and using that tr([I,))
= d, the final risk expression is
E [[[[(No-M(())
[I II,)a((
n
o2d n + A).
[[ II )
(9)
15

---

Under review as a conference paper at ICLR 2022
E.2.2 ORDINARY LEAST SQUARES WITH PI AND MEAN IMPUTATION (NO MARGINALIZATION]
We focus the solution of
min
y
Xw
Aole
2
to construct an estimator.
Using Lemma
E.3 we have
i Ha 1/ and 61 G.
Using Lemma
E.4
we can simplify
t Ha y = w +0 +0+ + Hal8
and
01 - v² + GZIE.
Since A is not available at prediction time, we impute it instead with its mean (xX), which is assumed to be perfectly known. This leads to
(X) Xw + pl (X) Xw + x(X)0= + Ke,
with
K XH TP + m(XGGz1
Plugging into X and taking the expectation, we obtain
1
E[((Tr((X))
0
2
E[| Ke|l
n
K
-
(10)
72
E.3 CAPTURING THE BENEFIT OF PI WITH MARGINALIZATION
We then describe when, with marginalization, ordinary least squares ignoring PI worse than ordinary least squares using PI.
Proposition E.2. Assume that X X is invertible.
Moreover, assume that A A and
[X, [X, A] are almost surely invertible. We have that
EiR(Tmane NO-PI (X))] > [[ p1( (X))
if and only if
(I - (X odd o² |||L]]
n
with L XHa AG.1- When (i.e., A is column vector), sufficient condition is |1 + ||
1 [ II, ( (X)=11
o²d
2E
n
[[
n
We provide the details of the derivation of the risk for Tiarg. NO and Timagg Pl in Section E.3.1 and Section E.3. respectively. Moreover, the second part of the proposition follows from application of Lemma
E.3. ORDINARY LEAST SQUARES (WITH MARGINAIIZATION!
Restarting from Section E.2.1
we consider the predictions marginalized with respect A. We have Ba-p(ale) [X Xxoo Xww + =I,((X)0' II,..
Tiarg, NO.P1 X
Plugging into we obtain
1 ||[I Il)))(X)' ell
R(Tmary Tmarg NO.PI NO))
Expanding the square and using that tr(II,)
the final risk expression is
E [[R(Tan NO.PI (X))]
1
[I 1I, ) v
+
1
E E|
n
1
add
[[ II X
(11)
n
n
16

---

Under review as a conference paper at ICLR 2022
E.3.2 ORDINARY LEAST SQUARES WITH PI AND MARGINALIZATION
consider the predictions marginalized with respect A. In particular, we do not impute A by its mean but rather directly take the expectation over A. We have
Restarting from Section E.2.2
Tmarg, (x) Ba-p(al))) Xwi + Aöi] Xw" + =x(Xxo" Barp(alx) Lb]e,
with
L X AG.1.
Plugging into
8 and taking the expectation, we obtain
(R(Taa (x))
= |01| + Ell [4]=])
0²
E aap(( [b]l?
(12)
E.4 TECHNICAL LEMMAS
Lemma E.3. Assume that both X X and A A are invertible. Moreover, assume that both X Xa. and A Ar are invertible.
We can write the solution of
min 1ly Xw Avl2 00.0
as
H. y and G.
Proof. The proof follows by applying inversion formula for the block matrix
X X
Q
A ATA
where XT
a_ XA1 and A A are the two Schur complements of X X and A A. Under the assumptions of the lemma, the matrix is Q is invertible.
Lemma E.4. We have the following properties
HarX Xx.) X II,
Xo1) - (a1) I.
Conversely, we have
G.IA (A A -I (AT A. " A+1) I.
G (A Aa1) A (I-
Lemma E.5, Assume m
i.e., A is column vector. We have
A 2 + || 111?
E K 2 < 2d 2E
II(I II, A
Similarly, it holds that
|| + A
L 1 < 2d + 2E
|I(I AA
Proof. We start by splitting the term into
| < 2 XH, 2| 2 +2)ln(X)G, 11..
Notice that H. H. = (X a1 -I and similarly G. G = =[AT. -11
Since || 2 tr(M] M), have
K 2tr((X X X + 2tr(a(X) ( (( -
17

---

Under review as a conference paper at ICLR 2022
By definition of Ard, when 1, we have
(A (m1) I[I II, 2
For the term (X x 1 the Sherman-Morriset formula leads to
1
(X Xa1) 1- = (X TX
x) 1- (X 16 X x)= bb T "0'(x" xx-
with b - 1/||A|| X A E Rd. Further simplifying, we obtain
II,A II,
A|
r X) X x-) =
d
2 A|
1 II, A
For the second part of the proof, we start by applying Jensen inequality:
||[L] < F|||
The rest of the proof follows along the same arguments, replacing (XX) by A.
F RELATED WORK TABLE
Table Comparison related work.
METHOD
p(a REQUIRED
TRAINING
WEIGHT SHARING
APPROXIMATE p(ylz)
IMPUTATION
X
MODEL, STEP
=NO PI
V
DISTILLATION Paz ST 2015
MODELS, STEPS
=NO
HET. DROPOUT
MODEL, STEP
=NO PI
V
MIML-FCN+ LANG 2017
MODEL, STEP
= NO
X
FULL MARGINATIZATION
MODEL, STEP
O(S NO PI)
TRAM (OURS)
MODEL, STEP
NO P1
V
HET-TRAM (OURS)
MODEL, STEP
NO
DISTILLED DISTILEE-TRAA TRAM (OURS)
MODELS, 2 STEPS
=N P1
G TWO-STEP TRAM: IMAGENET SCALE REPRESENTATION LEARNING EXPERIMENT
We conduct experiment to test two things: 1) does the one-step TRAM procedure, introduced 3.2 which is easier for practitioners to implement, approximate the two-step TRAM procedure well and 2) can the results of the toy represrntation learning experiment, 2.2 be replicated in a larger scale setting.
We train a feature extractor with and without access to PI on ImageNet, following the same procedure, architecture and dataset used in the main paper. We then freeze the feature extractor and train single dense/linear layer with softmax activation on top of the fixed features. We then evaluate the efficacy of these features trained with and without PI using this "linear probe" evaluation widely used in the representation learning literature CCen et al. 2020
The results are presented in Table 6
We see that the simpler single-step TRAM method approximates the more complicated two-step TRAM method very well. In addition see that the features learned by the network with access to PI which are then frozen and evaluated using a linear probe protocol perform better in terms of accuracy and log-likelihood.
H TOY EXPERIMENT: VARY 6
18

---

Under review as a conference paper at ICLR 2022
Table 6: Two-step TRAM: scaling up our toy representation learning experiment. ImageNet validation set negative log- oegllikeihood and accuracy. Averaged over 10 training runs + std. dev.
METHOD
JNL.
TACCURACY
ONE-STEP No PI
1.264 + 0.007
71.7 + 0.2
TWO-STEP No PI
1.265 0.008
71.7 - 0.3
ONE-STEP TRAM
1.225 + 0.006
72.5
= 0.2
TWO-STEP TRAM
1.226 1 0.002
72.7 1 0.2
10
13
-46
-05
-10
-00
-3
60
15
60
19
(a) 6 (..0..). RMSE No Pl to marginal: 0.0858
(b) é ~ (O0,0.5 0.5).
(c) E (0.00.0)
RMSE No Pf to marginal: 0.0880
RMSE No PI to marginal: 0.0897
RMSE Pl marginal: 0.0008
RMSE Pl to marginal: 0.0007
RMSE PT marginal: 0.0027
031
-05
-05
koollenikn
(d) e (0, 1.5). RMSE No Pl to marginal: 0.0841
( (0, 2.0). RMSE No Pl to marginal: 0.0977
RMSE Pl to marginal: 0,0472
RMSE PI to marginal: 0.0977
Figure Varying the influence of on our motivating toy experiment.
We vary the standard deviation of used in our motivating toy experiment, seen graphically in Fig. Fig. also contains the average RMSE to the true marginal across the data points plotted. The graphical and numerical results demonstrate that even for large levels of noise PT aids with representation learning but as expected, as the level of noise grows the advantage of using PI diminishes it becomes increasingly difficult to distinguish irreducible noise from noise which can be explained away with PI.
The results can be
I IMAGENET EXPERIMENT PI ABLATION
We run an ablation, removing feature: the probability of the label assigned by the model from the Pl set. We are thus left with just one PI feature, the one-hot encoded ID of the model that produced the label.
We see the results in Table As expected (and predicted by our theoretical analysis), removing informative reduces the effectiveness of TRAM. Nonetheless, TRAM with the reduced PI feature set still outperforms the No PI baseline, with accuracy and log-likelihood lying between the No P and full PI feature set TRAM methods
19

---

Under review as a conference paper at ICLR 2022
Table 7: ImageNet ablation with reduced PI feature set. ImageNet validation set negative log- likelihood and accuracy. Averaged over 10 training runs std. dev.
METHOD
JNLL
1ACCURACY
No PI
1.264 +:2 0.007
71.7 1 0.2
TRAM WITH FULL PI SET
1.225 + 0.006
72.5 + 0.2
TRAM WITH REDUCED PI SET
1.246 + 0.004
72.0 0.2
20


Reviews:
[{"review_id": "Uwn8E1BAQb", "reviewer": "Reviewer_KQ8f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Propose the notion of using PI to improve the performance of a network by explaining away the noise from annotators. \n2. Propose a new way to implement that idea using a stop gradient operator\n\nWeaknesses:\n1. Theoretically, the authors do not give a formal proof of why using PI in a practical network like ResNet helps to improve performance. The authors just give the proof for the very simple edge cases like linear and non-linear sine models. How about the classification case that this paper is targeted for? These simple cases are only true when the feature extraction network is kept unchanged during training. \n\n2. Empirically, the experiments are conducted in a limited and counter-intuitive: (a) When testing on CIFAR-10, it is quite unintuitive that the authors train on the test set and evaluate on the training set. To address the lack of training examples since the test set is used for training, they use the pretrained model on ImageNet; (b) When testing on the synthetic dataset relabeled from ImageNet, the information of the probability of the label assigned by a pretrained model is used as a proxy for the confidence of a human annotator. This information tells a lot about the quality of the label, which is unrealistic in real-world scenarios since, for each example, the annotators are required to additionally answer the confidence question resulting in a significant cost and ambiguity. \n\nMy suggestion is that the authors should collect a real-world large-scale dataset such that the PI is insignificantly expensive to obtain along with the main annotations to prove their claim empirically. Otherwise, this paper has a limited impact on practical scenarios.", "summary_of_the_paper": "This paper suggests a notion that using additional information called privilege information (PI) from annotators will help explain away the noise of the label they annotated and propose a way to implement that idea. The authors give the intuition from a simple linear model and a non-linear sine model that PI is useful. Then, an implementation of that idea called TRAM is proposed to make use of the information from PI through a stop gradient operator in training.  Since there are scant large-scale datasets with PI annotation, to evaluate the proposed approach, the authors have to create a synthetic one relabeled from ImageNet. ", "summary_of_the_review": "This paper proposes an intuitive phenomenon that using PI is helpful and improves the performance of the model. However, the authors have not given any proof for the general case (not the edge cases) theoretically or empirically successfully.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F96Cee6vqG", "reviewer": "Reviewer_6G97", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \n1. The proposed methods have simple architectures (not requiring specific modules, e.g., Gaussian dropout [Lambert et al., 2018], for the marginalization). \n2.  Making prediction for a test point (PI is not available) has the same cost with a standard network trained without PI. \n\nWeaknesses\nThe manuscript could be improved in presentation. I am listing some points which are unclear and need to be addressed.\n- What is the formal definition of label noise for classification? It seems that the current version of the manuscript provides an example for label noise for regression (Section 2.2), but there is not given a clear definition or an example for label noise for classification. \n- I think that the transition from the theoretical analysis using the linear regression model in Section 2 to the motivation of approximating the conditional marginal distribution is not that smooth. It seems that the analysis just supports that PI can be useful. In fact, the motivation of approximating the conditional marginal distribution is directly explained by the equation (2) (the idea of the representation learning with PI is explained by the example in 2.2). I think that the section 2.1 could be revised to address how this theoretical analysis can motivate the approximation of the conditional marginal distribution.\n- In the introduction, it is stated “We provide empirical evidence suggesting that the representations learned with access to PI are more robust against label noise”. What does “empirical evidence” refer to? The motivating example in Section 2.2?  If the answer is yes, I am not sure if this toy example can be convincing empirical evidence. I think that this is just a motivating example to explain the intuitive idea of the representational learning with PI. In addition, the example is for regression (the assumptions here would not be realistic settings for classification).   \n", "summary_of_the_paper": "The manuscript proposes classification methods utilizing privileged information (PI, which is available only at training time). The main idea is to directly approximate the conditional marginal distribution, i.e., p(y|x) \\approx q(y|\\pi(x), w), where \\pi(x) is a feature extractor and w is the weight parameter. Another main point is to learn the representation (\\pi(x)) of the features with access to PI at the training time. To do this, the proposed methods employ knowledge transfer by weight sharing. The proposed methods include several variants by adopting heteroscedastic classification (Het-TRAM) or distillation (Distilled-TRAM).  ", "summary_of_the_review": "I think that the main idea, directly approximating the conditional marginal distribution without requiring any sampling methods or additional specific architectures, is interesting (although I think the proposed methods are not super novel). I also like the way the authors build their methods from theoretical analyses and motivating examples. However, I at the same time think that the manuscript could be improved. Please see the detailed comments above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9_w0rkngAjY", "reviewer": "Reviewer_GTi4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The proposed framework uses privileged information in deep neural networks efficiently.  \n\n2. The method is well-demonstrated with both theoretical analyses and numerical experiments. Thus the result is convincing.\n\n3. Using privileged to deal with label noise may help to better characterize the instance-dependent label noise. Thus this work could contribute a new idea to learning with label noise.\n\n\nWeaknesses:\n\n1. The notations could be better defined. For example, in Page 2, in addition to saying \"Denoting by $\\prod_x$ the orthogonal projector associated with $X$\", it would be better to show the equation as well. In Page 3, $u$ in $q(y|x,a;u)$ is not clearly defined.It is also confusing why $a \\cdot u$ is the noise term. Should it be $a \\cdot v$? \n\n2. The privileged information in the toy example may be too strong since $\\epsilon$ is too small compared with $v$ and knowing $a$ means knowing which point is corrupted by label noise. The privileged information is often side information to help better decisions, rather than critical knowledge as shown in this example.\n\n3. The feature extractor trained with privileged information is supposed to be better than the one trained without privileged information. But the current experiment did not explicitly show it. An ablation study that compares the performance of fine-tuning the last layer using clean data with fixed feature extractors (trained with PI or not) would be helpful.", "summary_of_the_paper": "This paper develops a simple and efficient method for training with privileged information and testing without privileged information. The training with privileged information focuses on transferring via sharing the weights of the encoder, which is the knowledge learned with privileged information.\nThe testing without privileged information addresses the practical case where test data may not be equipped with useful privileged information. The corresponding solution is to marginalize over privileged information at test time.", "summary_of_the_review": "The paper proposes a new perspective on the noisy learning literature. The proposed framework is simple but useful.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3RffeSUz4-7", "reviewer": "Reviewer_Nesa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. the authors present a clear explanation of the underlying intuition of using PI and the motivation behind TRAM. \n2. the idea of integrating privileged information through weight sharing is interesting, and the two-step design is novel (as far as I am concerned).\n3. the proposed method can (in principle) apply to any neural network model and has zero overhead at prediction time.\n\nWeakness/Concerns:\n\n1. In the title and abstract, the authors state that privileged information helps explain away label noise. However, the latter investigations only focus on the accuracy improvement without an in-depth discussion about how and how much the label noise can be \"explained away.\" in realistic deep neural network models.\n2. there lacks quantitative (or even qualitative) evidence about how, how much, and what kind of privileged information is transferred through weight sharing in realistic deep neural network models.\n3. In the discussion of Table 2, it is overstated that \"TRAM provides significant performance improvement\" since a ~5% increase in accuracy does not seem significant at all.\n\nContinuing point 2 the above, I have a question: Is it possible to control the amount of privileged information transferred to the shared network block by designing a specific training scheme? Say (just a rough idea), we may gradually increase the weight of the shared network block during backpropagate through the $q(y|x,a)$ branch.", "summary_of_the_paper": "This paper proposes a method, TRAM, that integrates the privileged information into the learned network weight through weight sharing at training time and approximately marginalizes over the privileged information at test time. Experiments on both realistic and synthetic datasets demonstrate the effectiveness of TRAM in improving model performance on noisy data.", "summary_of_the_review": "The paper proposes TRAM, a method of integrating privileged information in the network weights during training. Experiments on both realistic and synthetic datasets show that TRAM can help improve the model accuracy on noisy data. However, the improvement is significant. Further, the lack of detailed investigations about how the privileged information is integrated into the network weights and how the privileged information help explain away the label noise in realistic deep neural network models limits the contribution of the present work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript proposes (TRansfer and Marginalize) TRAM method that integrates the privileged information into the learned network weight through weight sharing at training time and approximately marginalizes over the privileged information at test time. TRAM can also be combined with methods for dealing with noisy labels, distillation (Distilled-TRAM) and heteroscedastic output layers (Het-TRAM). Experiments are performed on both realistic and synthetic datasets including CIFAR-10H, re-labeled ImageNet, and Civil Comments Identities.

Reviewers agreed on several positive aspects of the manuscript, including:
1. The proposed methods have simple architectures (not requiring specific modules, e.g., Gaussian dropout [Lambert et al., 2018], for the marginalization);
2. The proposed method can in principle be applied to any neural network model and has zero overhead at prediction time.

Reviewers also highlighted several major concerns, including:
1. The analysis is performed on edge cases such as linear and non-linear sine models. There is no analysis for the classification case that this manuscript is targeted for. The simple cases are only true when the feature extraction network is kept unchanged during training;
2. Empirically, the experiments are conducted in a limited and counter-intuitive; 
3. Lack empirical evidence suggesting that the representations learned with access to privileged information are more robust against label noise;
4. Lack quantitative (or even qualitative) evidence about how, how much, and what kind of privileged information is transferred through weight sharing in realistic deep neural network models.

Several new experiments have been added to show, among others: representations learned with privileged information outperform representations learned without access to privileged information (using a linear classification model on ImageNet), better quantitatively and qualitatively understanding how and how much privileged information is transferred in realistic deep networks.

Post-rebuttal, reviewers stayed with borderline ratings, and they have suggested further improvements: simulating with more annotators by using different checkpoints and/or different hyperparameters, collecting a real-world large-scale dataset such that the privileged information is insignificantly expensive to obtain along with the main annotations, and disentangling the effect of the pretraining model on the denoising method.
Decision: Reject