Rethinking Knowledge Transfer in Learning Using Privileged Information
Authors: Danil Provodin
Bram van den Akker
Christina Katsimerou
Maurits Clemens Kaptein
Mykola Pechenizkiy
Venue: Accepted by TMLR ()

Abstract: In supervised machine learning, privileged information (PI) is information that is unavailable at inference, but is accessible during training time. Research on learning using privileged information (LUPI) aims to transfer the knowledge captured in PI onto a model that can perform inference without PI. It seems that this extra bit of information ought to make the resulting model better. However, finding conclusive theoretical or empirical evidence that supports the ability to transfer knowledge using PI has been challenging. In this paper, we critically examine the assumptions underlying existing theoretical analyses and argue that there is little theoretical justification for when LUPI should work. We analyze two main LUPI methods - generalized distillation and marginalization with weight sharing - and reveal that apparent improvements in empirical risk may not directly result from PI. Instead, these improvements often stem from dataset anomalies or modifications in model design misguidedly attributed to PI. Our experiments for a wide variety of application domains further demonstrate that state-of-the-art LUPI approaches fail to effectively transfer knowledge from PI. Thus, we advocate for practitioners to exercise caution when working with PI to avoid unintended inductive biases.

Published in Transactions on Machine Learning Research (02/2025)
Rethinking Knowledge Transfer in Learning Using Privileged Information
Danil Provodin Eindhoven University of Technology
d. d.prouodinStue.l
Bram van den Akker Booking.com
Sww..uuanddnakeerrboooknggg
Christina Katsimerou
Booking.com
Maurits Kaptein Eindhoven University of Technology
m.c.kapleinnttue..l ml
Mykola Pechenizkiy Eindhoven University of Technology
m.pochenizkii@@ue..l
Reviewed on OpenReview: https: openreviee. net/ forum? id= dg1g
Abstract
In supervised machine learning, privileged information (PI) is information that is unavailable at inference, but is accessible during training time. Research on learning using privileged information (LUPI) aims to transfer the knowledge captured in Pl onto a model that can perform inference without PI. It seems that this extra bit of information ought to make the resulting model better. However, finding conclusive theoretical or empirical evidence that supports the ability to transfer knowledge using PI has been challenging. In this paper, critically examine the assumptions underlying existing theoretical analyses and arguc that there is little theoretical justification for when LUPI should work. We analyze two main LUPI methods generalized distillation and marginalization with weight sharing and reveal that apparent improvements in empirical risk may not directly result from PI. Instead, these improvements often stem from dataset anomalies or modifications in model design misguidedly attributed to PT. Our experiments for a wide variety of application domains further demonstrate that state-of-the-art LUPI approaches fail to effectively transfer knowledge from PI. Thus, advocate for practitioners exercise caution when working with PI to avoid unintended inductive biases.
1 Introduction
In supervised machine learning (MT),, we aim to learn the fit between some features X and target y ) The information going into may exist features E Z that are only available during training due to engineering complexities or because this information ion only materializes post-inference. These features z can present. themsseles in many forms, including uncompressed features (c.g., images), third-party expert annotations, non-target post-inference signals (e.g., clicks or dwell time), and metadata about the annotator /label provider. Our work is motivated by a common e-commerce application of optimizing north-star metric, such as product conversion. In
only be used if it is accessible at the time of inference. However, there
The term 'north-star metric" refers to single key metric that success, guiding strategy and Decision-making In the context e-commerce, 'product conversion" often serves as such metric. Product, conversion measures the rate which users perform desired action, such as purchasing product, critical
the primary indicator of organization's long-term

---

Published in Transactions on Machine Learning Research (02/2025)
this context, user interactions that occur after prediction, such as clicks, can be strong indicators of the user's intent to purchase, and evaluating the probability of conversion conditioning a click becomes more straightforward. However, clicks exist as features only in the offline data and not during inference.
For this reason, Vapnik & Vashist (2099) introduced the paradigm of learning using privileged informa- tion (LUPPT.. Since its introduction, LUPI has sparked significant interest within the research community
across various domains, including speech recognition (Markov & Matsui 2016), computer vision (Garcia
et al. 2020 Lee et al. 2020), semi-supervised learning (Gong et al. 2018 Yang et al., 2022b), noisy-labels (Collier et al. 2022 Ortiz-Jimenez et al., 2023). and others (Li et al. 2020 Xu et al. 2020). Given this
widespread interest, it is crucial to develop sound methodologies to conclusively ascertain the effectiveness of privileged information (PI). However, existing methods, being generic, are often mistakenly considered universal solutions. This misconception leads to a lack of thorough theoretical and empirical foundations regarding the impact of privileged information.
The key intuition behind LUPI is that privileged information should be addressed via knowledge transfer transferring knowledge from the space of privileged information (PI model) to the space where the decision
rule is constructed (no-PI model) (Vapnik & Izmailov 2015b). State-of-the-art approaches for LUPI are
largely based on two knowledge transfer techniques: knowledge distillation (Lopez-Paz et al., 2016 Markov & Matsui 2016; Lee et al. 2020 Xu et al., 2020 Yang et al. 2022b) and marginalization with weight sharing
Lambert et al. 2018 Collier et al. 2022 Ortiz-Jimenez et al. 2023). In this work, we analyze these two
popular knowledge transfer techniques for LUPI from both theoretical and practical perspectives.
Recent research suggests that incorporating PT is crucial for enhancing sample efficiency and generalization performance (Lambert ct al., 2018 Yang et al., 2022b; Collier et al., 2022). These studics attempt to cxplain under what conditions LUPI is beneficial. However, theoretical analyses often either assume knowledge transfer occurs or demonstrate it takes place for extreme cases under assumptions that are difficult to verify.
Additionally, empirical analyses in existing studics frequently rely on stylized examples (Collier et al. 2022
Ortix-Jimenez et al.) 2023). specific experimental settings (Lopez-Paz et al., 2016 Xu et al.) 2020: Collier
et al.] 2022), or low-data regimes [Vapnik Ixmailov] 2215bb Markov & Matsui 2016 Lopez-Paue Paz et al.
201 6
Lambert et al., 2018). Therefore, conclusively identifying that knowledge transfer happens and is induced by PI is non-trivial, and there remains gap in understanding PI.
In this paper, we investigate whether knowledge transfer truly takes place in knowledge distillation and meryinatization with weight sharing. To that end, we critically review the theory behind knowledge transfer in LUPI and explicitly discuss assumptions imposed by the existing theoretical analyscs. We argue that the imposed assumptions are overly restrictive and discover that discussions on the robustness of the results to violations of these assumptions are frequently omitted. On the empirical side, we conduct an elaborate ablation study and demonstrate the apparent improvements often result from factors unrelated to PI. We reveal that previous studies tend to misinterpret the observed gains in empirical performance and mistakenly attribute them to PI. Interestingly, when focusing on the mechanisms that disclose PI models' better performance, we obscrve that the gap between PI and no-PI models can be bridged by simply training models longer or replacing PI with a constant.
Back to the real world, we validate the existing methods on four real-life datasets from various application domains, including e-commerce, healthcare, and acronautics. Our results demonstrate that the state-of- the-art approaches fail to outperform model that does not use PI, which adds evidence to the limited contributions of LUPI in practical applications. Overall, our study highlights that, in the current state of research, there is no solid empirical or thcoretical evidence that knowledge transfer takes place in the LUPI paradigm.
Our contribution Our key contributions can be summarized as follows:
We critically review the theory behind knowledge transfer in LUPI and argue that current research provides little theoretical justification for when LUPI should work.
performance indicator for understanding and optimixing customer Journeys. This metric is widely utilized in practice Farris et al. 2016;; Purnomo 2023).
2

---

Published in Transactions on Machine Learning Research (02/2025)
We revisit empirical studies that claim performance improvements due to PI and highlight that these improvements can be explained through mechanisms unrelated to PI.
We conduct experiments on four real-world datasets from various application domains and find out that no improvement from PI model is observed, which adds evidence to the limited contribution of LUPI practical applications.
Concerns about LUPI are not unprecedented. Farlier work by Serra- et al. (2014) discusses ex experiments on SVM+, one of the first algorithms developed for LUPI (Vapnik & Vashist, 2009), that yield identical results to the regular Support Vector Machine (SVM) algorithm with randomly generated features as PI. Our analysis extends to newer algorithms that utilize PI, further advancing our understanding of the practical limitations of LUPI algorithms despite recent developments.
Paper outline The rest of the paper is organized follows. Section discusses the knowledge transfer in LUPI and introduces the techniques of knowledge distillation and marginalization with weight sharing. Section
reviews the theory behind knowledge transfer in LUPI. The common misinterpretations are outlined in Section with claborate analyses of knowledge distillation in Section 4.1 and marginalization with weight sharing in Finally, we conclude with Section 6 The source code of the experiments can be found http:://gttauo.co/maaall///////p
Section 4.2 This followed by our real-world experiments in Section 5
2 Knowledge transfer in LUPI
In this section, we present two popular knowledge transfer techniques that are largely used in LUPI.
Let 'D denote a training dataset, D = {(x,, 24, y:) consisting of triples: features x; X, available during both training and inference, privileged information 2; Z accessible only during training, and labels y E e drawn from the unknown distribution p(-12,, 2). We focus c-class classification task (i.e., Y E {I.....c) c)), although the same ideas apply to regression task.
The LUPI problem often described as an interaction between an intelligent teacher, who has access to PI, and a student, who learns from the teacher's 'explanations' (Vapnik & Vashist 2009). Let G. := [glg: X + y a teacher function class and G. = {glg X + y} a student function class. Vapnik & Izmailov 2005bb formulate two conditions that are required to learn effectively using PI:
the empirical error privileged space Z smaller than the empirical error in the feature space X, i.e. the classification rule y = (x, 2) is more accurate than the classification rule y = g, (z), for some I: E G. and the best 9s Go-
2. the knowledge of the rule 9e 2) in space X' 2 can be represented/transtee ttranffered to improve the accuracy of the desired rule = yo (I) in space X.
Assuming that the first condition holds, which is easy to verify empirically on a given dataset, the difficulty is to verify whether and when the knowledge transfer actually happens. To address this challenge, two main knowledge transfer techniques have been proposed the LUPI literature for improving the accuracy rule y = (x)): knowledge distilation and marginalization with weight sharing.
Knowledge distillation Distillation introduced by Hinton et al. (2015) forms the basis for knowledge
distillation methods using PI (Lopcz-Paz Paz et al. 2016 Markov & Matsui] 2016 Garcia et al. 2020 Lee et al. 2020 Xu et. al. 22200] Yang et al. 2022b). Lopez- Paz el. al. (0066) unifies LUPI with distillation for supervised learning and suggests that the representation learned by the PI model can be effectively distilled to a no-PI model. Their method, called Generalized distillation, procceds in two stages. First, train teacher model that takes both x and z as input to predict y. With slight abuse of notation, we assume that y is represented by one-hot encoded vector, Le., y A. where A is a set of c-dimensional probability vectors. The teacher's
3

---

1 l (ys, (g(x;, Published in Transactions on Machine Learning Research (02/2025)
goal is to learn the representation
arg min 9E9, 1=1
(1)
where & A x A R_ is loss function, and R + A is the softmax operation. In the second stage, student model distills the learned representation de into
Ia arg min gEG,
Σ [1- A)e (((()))) +(s. 1=1
(2)
where s, ((,, 2)) E A is soft label with temperature T provided by the teacher model and A E [0, 1] is the imitation parameter, which balances the importance between imitating the soft predictions and predicting the true hard labels y/-
Intuitively, the teacher reveals the label dependencies to the privileged information by softening the class- probability predictions 9,, and the student distills this knowledge by training using the input-output pairs {(1,, ys) 1. [xr, 8t) The soft labels 8; provided by the teacher assumed contain more information than hard labels yi and allow faster learning (Lopez-Paz al., 2016). After distilling the privileged information, can use the student model 9, G2 for prediction at test time.
Generalized distillation underpins numerous PI algorithms MMakovv & Matsui 2016: Garcia et al. 2020 Lee
et al. 2020 Xu et al. 2020; Yang et al. 2022b introduced with problem- specific adjustments periphera to the knowledge distillation component.
Marginalization and weight sharing Another popular approach of incorporating privileged information is based on marginal distribution p(y|=) J ply|æ, 2)p(=/=)dd (Lambert al., 2018: Collier al. 2022 Ortiz-Jimenez et al. 2023). Consider training problem:
9: arg min gEG, n
1
Σ e ((),,)))
(3)
This is equivalent classical supervised learning problem defined over the privileged space X' Z. In order to solve the inference problem, we can consider the following marginal distribution
g. (2) = [z=p(=])) (xc(2x,))) =)]
(4)
The major problem in this formulation is the intractability of computing the expectation in equation g as p(=(z) unknown. As such, Collier et al. (2022) propose a knowledge transfer technique based on weight sharing to approximate equation 4 Their method, called TRAM (transfer and marginalize), is designed to reduce the harmful impact of noisy labels and facilitate learning. The authors motivate their work by the
ability of PI reduce the effect of malicious or lazy annotators on collected labels.
TRAM is based Specifically, they propose neural network architecture which consists of three parts: shared feature extractor Φ(x), No PI head (x'), and PI head ge (x', 2), where Φ : X' + X' learns representation x' of features x for some representation space X". Then, they consider the following two-step approach:
two-headed model in which head has access to PI, and the other does not
0², gt = arg min .
l (((()))))
(5)
n
1
g4 arg min gEG. n
(v (x)))
(6)
Crucially, feature extractor o² learned in equation 5 with access PI. This weight sharing assumed enable knowledge transfer the network trained without PI in equation head is used for prediction.
At test time, only the No PI
4

---

Published in Transactions on Machine Learning Research (02/2025)
3 When is knowledge transfer in LUPI proven theoretically?
In this section, we review the existing theoretical analyses of the LUPT paradigm. Recent work altempts to explain when LUPI is beneficial, but finding conclusive theoretical evidence for knowledge transfer using PI remains challenging. These theoretical analyses often depend on strong assumptions and lack discussion when these are satisfied or violated.
LUPI was introduced as a technique that can leverage PI to distinguish between easy and hard examples, a concept closely tied to SVMs, where the difficulty of an example can be quantified by the slack variable Vapnik Vashist 2009). For the case of SVMs, Vapnik & Izmailov (2015b) show that utilizing slack variables as privileged information can result. in generalization error bound with rate O instead of (2+)). The motivation behind this that SVM classification becomes separable after we correct for the slack values, which mcasure the degrec of misclassification of training data points. Since it is unlikely that the teacher is able to provide true slack variables, the idea of the SVM+ algorithm is to estimate slack variables and represent them by the teacher's decision rule 9:- Technically, the improved convergence rate holds under two conditions: (i) function class G, has a smaller capacity than student's function class G. and (ii) teachers' explanations p(z[æ)) engender a convergence that is faster than O( ((2²), However, the sets of functions satisfying those conditions are confined to Reproducing Kernel Hilbert Space (RKHS) (Vapnik & Izmailov 2015b), and their theoretical justifications does not generalize beyond SVMs with decision rules defined in RKHS.
On the last point, Lopez-Paz et al. (006)) argue that in Generalized distillation, the rate at which the student learns from the teacher's soft labels is faster than O( (())), since soft labels contain more information than hard labels per example, and should allow for faster learning. This requirement on the learning rate is rather strong and hard to satisfy in a general setting.
Generalized dist.illation was also analyzed in the semi-supervised learning setting by Yang al. 2022b The authors consider problem where two datasets are available: Diabel := {xx, zi, 1.) )))== and Dantabel == {(x;, +)) Their distillation algorithm trains the teacher model using labeled dataset Diabel, which provides pseudo-labels for both the labeled and unlabeled datasets, Diabel and D Duntabel, respectively. Then, the student. model is trained on the combined dataset Diabel U Dumaabel using the imputed pseudo-labels as targets. They theoretically demonstrate that their algorithm reduces estimation variance in the case of linear models with independent regular and privileged features and report improved empirical performance. However, the improvement appears to largely come from the semi-supervised aspect rather than PT-induced knowledge transfer. In Appendix A we show that when have no unlabelled data, the estimation variance of distillation actually slightly increases.
For the marginalization approach, Lambert et al. (018)) demonstrate that the convergence rate can be increased to O for convolutional neural networks undor a strict assumption that the variance of the model can be upper-bounder by for an arbitrarily small value of > The authors leave verilying this assumption as an open problem.
Mcanwhile, Collier et al. 2022)) formulate two conditions under which marginalization can achieve lower empirical risk for a linear regression y x w + z T + where p(z]x) (i) the regression coefficients v have a large variance when explained only by the features x and (ii) privileged features z have a significant average component outside of the subspace spanned by the features x. However, their analysis is intractable beyond this simple case, and hence, we cannot quantify such conditions in the general setting.
Overall, the existing theory cither assumes that knowledge transfer occurs or identifies conditions under which it might happen in stylized linear models. However, conclusive theoretical evidence supporting knowledge transfer through PI remains lacking.
2Separable classification corresponds to a situation when there exists a function that separates the training data with errors and admits generalization error bound of the rate O(z). Conversely, in non-separable classification, there is no function that can separate training data wouh errors, and the generalization error bound of the order O 1) Vapnik 1998
5

---

Published in Transactions on Machine Learning Research (02/2025)
Table 1: Expanding the training size of Experiment (Clean Labels) from Lopez- Paeez-Pazee et of Generalized distillation wears off when the training size surpasses 1000 samples.
2016 The effect
Training size
Privileged Generalized distillation
no-PI
200
0.95 20.01
0.95 10.01
0.87 #0.02
500
0.95 10.01
0.95 10.01
0.92 10.01
1000
0.95 10.01
0.95 10.01
0.94 10.01
2000
0.95 #0.01
0.95 #0.01
0.95 #0.01
4 What does existing empirical evidence show?
In this section, we revisit the original experiments conducted with the introduction of Generalised distillation and TRAM. Our goal is to challenge Pl-induced knowledge transfer for these two methods in the original experiments. In Section 4.1 revisit four experiments from Lopez-Paz et all (2016) (one of the experiments is deferred to Appendix B to highlight potential limitations and misinterpretations from the aforementioned
work. In Section 4.2 we revisit the experiments by Collier et al. (2022) to demonstrate that TRAM fails to explain the annotators' noise, and the observed improvements in empirical risk can be explained by the
architecture of TRAM.
4.1 Generalized distillation
Synthetic experiments from Lopez-Par et al. (2016) Lopez-Paz et al. (2016) ran four experiments to demonstrate the ability of Generalized distillation to transfer knowledge. These are simulations of logistic regression models repeated over 100 random partitions. For the two experiments that see positive effects of using Generalised distillation, the triplets (æi, Fss Wi) are sampled from one of two generating processes:
Experiment 1: Clean labels as PI
Experiment 3: Relevant features as PI
24 NOO, Ia)
F3 ~ NOO,
2, - (a,
2, + Dz,J
6 1)
V: < [((
34 [1 2,) 0}
where d is dimensionality of regular features, d = 50, a E Rd is the separating hyperplanc, and set J, J = 3, is a subset of the variable indices [1,...,d} chosen at random but common for all samples. Both Generalized distillation and no-PI models are trained on 200 samples (n = 200), and the authors report substantial improvement in accuracy (88% vs. 95% for Clean labels as PI and 89% vs. 97% for Relevant features as PI) testing models on 10000 test samples.
The results in Table 1 demonstrate that knowledge transfer via. PI enhance performance in low-data valuable proof-of-conceet. However, this effect rapidly diminishes as the sample size
regimes, serving increases relative to the dimensionality of z. For detailed results for Experiment 3, see Table 3
It also important to note that in both experiments, PI contains (almost) perfect information about the distance of each sample to the decision boundary, and from a practical perspective, obtaining such high-quality PI improbable. In Experiment 1, PI encodes the exact distance, while in Experiment 3, PI encodes the relevant features used to calculate the distance (both cases align with the perfect knowledge of the slack variables in Vapnik & Izmailov 2015a
MNIST experiment from Lopez-Paz et al. (2016) The authors further demonstrate Pl-induced knowledge transfer using an experiment with the MNIST dataset. In this experiment, the teacher learns from full 28x28 images while the student learns from downscaled 7x7 images. They conduct two experiments with 300 and 500 training samples, reporting significant improvement in classification accuracy compared to a model without PI. When revisiting these experiments, found that the original experiment limited training

---

Published in Transactions on Machine Learning Research (02/2025)
0.85
0.80
0.85
Accuracy
0.75
Test Accuracy
0.80
0.70
0.65
0.80
teacher student
teacher
0.55
no-pi
Epochs
Epochs
(a) Results on MNIST for 300 samples
(b) Results on MNIST 500 samples
Figure The effect of sufficient training epochs on the MNIST Generalisod distillation experiment.
0.5
0.5
Observation ons
Observations ons
-0.5
Noise free f-n
-0.5
Noise free D.037
NoPl: D.131
-1.0
TRAM: 0.07
-1.0
TRAM: 0.036
TRAM zeros: 0.055
TRAM zeros: 0.036
0.2
0.6
0.9
0.2
(a) Undertrained (10 epochs)
(b) Sufficiently trained (200 epochs)
Figure 2: TRAM zeros, TRAM, and no-PI for insufficient training 2a and sufficient training 2bb. The
numbers in the legend indicate MSE loss with respect to the noise-free function.
epochs to 50. In Figure 1 we show that the reported effects are indeed visible around 50 epochs but quickly disappear when we allow all models to continue training.
Important to note, is that given the teacher-student setup, when the no-PI and student model performances are reported
epochs in Figure [ the student model actually requires a teacher model that had already completed 50 epochs, thus combined requiring 100 training epochs. Taking this into consideration, there is evidence of either improved sample efficiency computational efficiency by using Generalisod distillation in this setting.
Further discussion knowledge distillation using PI While Generalized distillation shows preliminary evidence of knowledge transfer,
that it takes place only for low data regimes and in highly styled examples. To address those gaps, several attempts have been made from the application side (Markov &
Matsui 2016 Garcia et al. 2020 Lee et al. 2020 Xu et al.) 2020), with Xu et al. (2020) applying generalized distillation to recommendation with privileged in inoomaatoon in e-commerce. Admittedly, all of these works
report marginal improvement over the no-PI model.
4.2 Revisiting TRAM
Collier et al. (2022) and Ortiz-Jimenez et al. (2023) argue that PI can be used to "explain away" label noise. To demonstrate TRAM having this capability, Collier et al. (2022) consider the following synthetic experiment: A noisy annotator is simulated by binary indicator Ber(O.3), such that = represents the case where the noisy annotator provides a random label independent of x
y (1 =) sin(( +2.0+6 +

---

Published in Transactions on Machine Learning Research (02/2025)
Convergence rate
Sample efficiency
0.12
0.12
0.10
0.10
MSE loss
0.08
loss
0.08
0.06
0.06
0.04
0.04
0.02
0.02
0.00
0.00
50
200
10a
10
num epochs
num samples (log)
NoPI
TRAM
Uncorrupted
Figure 3: TRAM and no-PI training dynamics for the synthetic experiment from equation 7 (Left) presents training dynamics over 200 epochs. (Right) shows the resulting models' performances across varying sample sizes trained for 200 epochs. "Uncorrupted" corresponds a regular model fitted to uncorrupted data y = sin(2 t.
where I E 10,1),0 u ~ Unif(-1,1), and E~ 0.10..).
The authors train TRAM and no-PI models on n 2500 training samples using a 2-layer fully connected neural network with tanh activation function. They observe results from Figure 2a and state "We see that the representations learned by the model with access to PI in step #1 ennble near perfect fit to the true expected marginal distribution, B(s.))-(++>))(b)) over X. However, without access to PI, the noise term cannot be explained away."
enable
We regard the expression "explaining away the noise term" as cumbersome in this context: as one can see, neither TRAM nor no-PI effectively explains the noise term & v away. The task of explaining noise term would ideally correspond to learning the noise-free function Efyyar, 0] = sin(2xx); however, as depicted in Figure 2b after sufficient training, TRAM and no-PI converge to a biased function. This effect is more clearly visible in Figure 3 where we compare the TRAM performance to an uncorrupled model (a regular model that is fitted to data without the corrupted labels coming from v). Thus, we can conclude that TRAM does not "average out" or "explain away" label noise; rather, similarly to the no-PI model, it completes the average [[y]xr].
Next, we consider the training dynamics of TRAM against no-PI model for the regression task in equation 7 Figure (left) shows the training dynamics over 200 epochs for n 2500. Figure (right) shows the models' performances trained for 200 epochs across varying numbers of samples. The y-axis represents the MSE loss with respect to the noise-free generating function sin (2mxr.
Although, which was already observed in Figure 2b both TRAM and no-PI models eventually converge to similar performance levels, some disparity is observed in their trajectories (refer to Figure (left)), with TRAM achieving optimal performance generally faster (in Appendix [
we extend our analysis to classífication tasks, which are generally more difficult, and the advantage of TRAM is more noticeable there). This suggests that TRAM has a faster convergence rate. However, from Figure 3 (right), we can see that both models enjoy the same performance after sufficient training, which suggests that TRAM is not more sample efficient than no-PI model. Thus, similar to the MNIST experiment, incrcasing the number of epochs for no-PI model achieves identical performance to TRAM, resulting in both models fitting the expected marginal distribution almost perfectly.
Why TRAM does not leverage PI In order to understand by which mechanisms TRAM enables a faster convergence rate, we consider modification of TRAM, where instead of Pl z, we plug in a zero vector (TRAM zeros). Figures 2a and 2b show that the performance of TRAM zeros is identical to the
sstep #1 corresponds learning feature extractor d² in equation

---

Published in Transactions on Machine Learning Research (02/2025)
AA
37
50
Strpps
21
940
1999 900
24
itronger
stroeger
Stronger?
(a) Gen. dist. relative No PT
(b) TRAM relative to No PT
(c) Teacher relative to No PT
Figure 4: Heatmap of accuracy gain (compared to No PT model; in %): warmer colors indicate positive gains, cooler colors indicate losses, grey cells correspond to statistically not significant results at 0.05. x-axis represents PI strength from stronger to weaker, and y-axis represents the number of samples. The results are averaged over 30 runs.
performance of TRAM using Pl. This suggests that the benefit of TRAM stems from architectural changes rather than Pl-induced knowledge transfer.
This mechanism et al. (2022). In this experiment, the authors reduced the capacity of the network by downsizing the number of parameters by 75% while keeping the number of training samples unchanged. Their observation indicated that TRAM performed equivalently to the no-PI model under these conditions. However, with the full-size network, TRAM exhibited a slight improvement over the no-PI model. This observation suggests that the full-size network might have been in an underfitted regime, where TRAM's archilectural adjustments conferred an advantage.
be traced back to the original TRAM experiments, as outlined in Appendix F of Collier
4.3 Generalized distillation and TRAM for varying dataset sizes and Pl strengths
The goal of this experiment is to evaluate the impact of PI on the overall performance across varying PI strengths and dataset sizes.
We generate a synthotic dataset with features normally distributed around the vertices of a d-dimensional hypercube and a binary label. In this experiment, d = 20 with 10 informative features and 10 redundant. features. The regular features x comprise 10 randomly selected features from the total 20. subset of the features is treated as Pl, which is unavailable to the No P1 model. By varying the size of the P1 features subset, we control the relative strength of PI compared to regular features. The weakest PI configuration includes one informative feature, while the strongest. PT configuration includes all 10 informative features and the true label (!). Additionally, we vary the number of samples from 100 to 10,000.
For each combination of sample size and PI strength, we train each of the No PI, Gen. dist., TRAM and teacher models are used 30 times, and the relative performance reported compared to the No PT model. Figure Figure 4a 4b presents heatmaps of the accuracy gain for the Gen. dist. and TRAM compared to the No PI model. We report the accuracy gain of the teacher model compared to the No PI model in Figure 4c sanity check to illustrate the potential of privileged information.
The results in Figure 4 highlight several key trends. For the teacher model (Figure 4ch, weaker PI harms performance, moderate PI strength has negligible impact, and stronger PI leads to statistically significant. improvements. For Gen. dist. and TRAM, the heatmaps reveal distinct patterns despite generally non-positive accuracy gains. Specifically, Gen. dist. (Figure 4a) performs worse scttings with weaker PI, whereas TRAM (Figure 4a) performs worse with larger sample sizes. Notably, neither Gen. dist. nor TRAM improves upon the No PT model, and both methods fail to demonstrate practical utility in this experimental setup.
9

---

Published in Transactions on Machine Learning Research (02/2023)
Repeated Buyers
Heart Discase
NASA NEO
Smoker or Drinker
0.08
0.73
Verfermance metere
0.70
0.00
0.72
0.70
0.02
0.90
0.68
0.62
0.60
0.68
0.60
0.89
0.56
0.88
0.60
0.24
0.68
0.27
0.22
Cross entropy
0.21
0.26
0.21
0.20
0.60
0.20
0.19
0.55
0.24
0.18
0.53
30
30
Training steps
Training stops
Training steps
Training stops
Pl
TRAM
Gen. dist.
Teacher
Figure 5: Training dynamics of No PI, TRAM, Gen. dist., and Teacher for 4 real-world datasets averaged over 10 runs. (Top row) shows the performance metric on the test set (scaled roc auc score for Repeat Buyers and Heart Disease datasets and accuracy for NASA-NEO and Smoker or Drinker datasets). (Bottom row)
shows cross-cntropy loss on the test
5 Real-world applications
To further validate the described methodologies, we conduct experiments on four real-world datasets from a variely of application domains, including e-commerce, healthcare, and aeronautics:
Motivated by our use-case example, we consider the Repeat Buyers dataset, a large-scale public dataset from the LJCAI-15 competition. The data provides users' activity logs of online retail platform, including user-related features, information about items at sale, and implicit multi-bchavioral feedback such click, add to cart, and purchase. We assign uscr-item features to x, intermediate signals click and add lo carl to 2, and purchase to y-
Repeat Buyers Allibbaa 2024)
Heart Disease BRFSS 2024 This dataset is derived from the 2015 Behavioral Risk Factor Surveillance System, and it contains 260k cleaned responses, focusing the binary classification of heart discase. We use social-demographic features (such as age and income) as privileged information and medical data as regular features x.
NASA nearthest earth object dataset compiles the list of NASA-certified astcroids. It contains 90k samples with various properties of astcroids, and the task is to predict if an asteroid is hazardous. For the purpose of our study, we treat a subset of original features as privileged information.
NASA-NED NASA 2024
2024) This dataset was collected from the National Health Insurance Service in Korea. It compiles medical histories of ~ 900k patients, focusing on their smoking and drinking status. For our study, we treat a subset original fcatures as priviloged information.
Smoker or Drinker
So0
The choice of PT in our experiments was motivated by three practical use cases of PI: (1) PI as features that materialize post-inference, (2) PI as features that are expensive to compute during real-time inference, and
All datasets distributed under CC BY-NC license.
10

---

Published in Transactions on Machine Learning Research (02/2025)
Table 2: Comparison of models' performance on test data. Results represent MEAN = STD. DEV. and are averaged over 10 random sceds. We use scaled roc auc score for Repeat Buyers and Heart Disease datasets and accuracy for NASA-NEO and Smoker or Drinker datasets.
Dataset
Method
+ Cross-entropy loss
Metric
no-PI
0.2189 = 0.0015
63.13 1 0.25
TRAM
0.2194 + 0.0013
62.89 1 0.42
Repeat Buyers
Gen. dist.
0.2183 = 0.0017
62.88 + 0.56
SC. TOC anc
Teacher
0.1938 = 0.0019
73.23 1: 0.38
no-PI
0.2557 = 0.0020
61.64 = 0.47
TRAM
0.2555 = 0.0018
61.62 = 0.30
Heart Disease
Gen. dist.
0.2543 = 0.0013
61.11 = 0.42
Teacher
0.2422 = 0.0018
67.38 = 0.31
no-PI
0.1945 = 0.0009
90.28 = 0.07
TRAM
0.1948 + 0.0009
90.26 +: 0.09
NASA-NEO
Gen. dist.
0.1951 = 0.0010
90.29 = 0.09
accurage recuracy
Teacher
0.1818 = 0.0011
91.35 = 0.10
no-PI
0.5823 = 0.0017
69.05 = 0.15
TRAM
0.6125 = 0.0031
66.54 = 0.44
Drinker Smoker
Gen. dist.
0.5820
69.09 +
Teacher
0.5157 0.0011
73.08 + 0.11
(3) PI as features that might be unavailable (e.g,, due to privacy reasons). These use cases are prevalent practice, and our choice PI experiments was specifically aligned with them. For instance, the Repeated Buyers dataset corresponded to the first use case, the Heart Discase, and NASA-NEO datasets the second, and the Smoker or Drinker datasets to the third.
We consider Generalized distillation, TRAM, and no-PI models, which are 2-layer fully-connected neural networks for all datasets. For reference, we report the teacher's performance for all datasets to indicate that PI could be useful in all cases. We perform a simestamp-based train test split and use 70% of data for training each model and 30% of data for reporting performance. The experiments are repeated over 10 random model initialization. The further experimental details are provided in Appendix
Figure shows the training dynamics for TRAM, Generalized distillation, and no-PI models across the four datascts, and Table 2 reports the resulting performance metric. We use scaled roc aud for Repeat Buyers and Heart Disease datasets and accuracy for NASA-NEO and Smoker Drinker datasets. As we can see, there is no benefit. from using TRAM or Generalized distillation over no-PI model for all datasets, with TRAM performing substantially worse in Smoker or Drinker dataset. Therefore, there is no evidence that TRAM and Generalized distillation transfer knowledge from privileged information, and there is no added value real-world setting with moderate to large data sizes and properly tuned and trained models.
6 Conclusion
Misinterpretations empirical results and attributing performance gains to privileged information are so prevalent in recent literature that they create widcly accepted impression of the uncompromising uscfulness of PI. However, this misconception has resulted in lack of thorough theoretical and empirical foundations regarding the true impact of privileged information.
Our theoretical overview of recent developments in LUPI argues that the existing theory does not provide a sufficient basis for claiming that knowledge transfer occurs and highlights the need for more solid theoretical justification. While this observation only applies to the theoretical analyses discussed in our study, we are also unaware of other work compellingly showing when knowledge transfer possible and effective LUPI.
scaled roc roc
11

---

Published in Transactions on Machine Learning Research (02/2025)
Through experiments, we identify common fallacies of misinterpreting gains in empirical performance as knowledge transfer induced by PT. We demonstrate that after adequate training, state-oftthe-at LUPT methods fail to outperform no-PI model. Surprisingly, we observe that low data regimes and undertrained models (low training epoch regimes) often seem to be confused. While PI is beneficial in low data regimes in highly styled examples, it has yet to be verified that this can be extended realistic settings. So far, existing methods benefit. from other factors unrelated to PI.
Similarly, empirical evidence for TRAM suggests lack of support for the notion that PI accounts for noise originating from corrupted labels. We illustrate that the purported improvements in empirical risk achieved through TRAM can be attributed to alterations in model architecture.
Addressing these concerns, our empirical and theoretical analyses provide compelling evidence that existing methods are insufficient in achieving effective learning using PI in practical, realistic scenarios; thereby rethinking knowledge transfer in learning using privileged information.
Nevertheless, our findings do not definitively disprove the possibility of knowledge transfer induced by privileged information. While the prevailing view knowledge transfer in LUPI methods centers on enhancing general model performance through metrics such as accuracy or loss, our findings invite broader perspective. The anticipated performance gains from LUPI methods may not materialize in general settings; however, PT could still play a critical role in more specific domains. For example, by providing additional context, PI might be leveraged to improve fairness in machine learning models helping mitigate biases in decksion-making processes (Chai et al. 2022 Yan al., 2023). This suggests that sole focus on performance- based metrics may overlook other valuable benefits of LUPI, emphasizing the need for research that explores its potential.
Thus, we believe practitioners and researchers should exercise caution when working with PI to avoid potential performance degradation or unintended inductive biases caused by experiment setup or dataset anomalies. Additionally, we urge the research community to devise more sound mothodologics to conclusively ascortain the presence and effectiveness of knowledge transfer induced by PI.
Acknoledgements
Part of this work carried out during DP's internship at Booking.com This project is partially financed by the Dutch Research Council (NWO) and the ICAT initiative in collaboration with KPN. The authors thank Philip Bocken and Andrey Davydov for discussions on carlier drafts of the paper.
References
Alibaba.
Repeat
buyers
prediction competition. 2024. Retrieved August 1.
https://ijcai-15.org
Susan Athey, Raj Chetty, Guido W Imbens, and Hyunseung Kang. The surrogate index:
Combining short-
term proxies estimate long-term treatment effects more rapidly and precisely. Technical report, National Bureau of Economic Research, 2019.
BRESS. Heart disease health indicators dataset, version 4. sitttebouuhhaar--dssesss---sseee
2024. Retricved August 1.
Rich Caruana. Multitask learning. Machine learning, 28:41-75, 1997.
Junyi Chai, Tacuk Jang, and Xiaoqian Wang. Fairness without demographics through knowledge distillation. In Advances in Neural Information Processing Systems, 2022.
Mark Collier, Rodolphe Jenatton, Effrosyni Kokiopoulou, and Jesse Berent. Transfer and marginalizo: Explaining away label noise with privileged information. In Proceedings of the 39th International Conference on Machine Learning, 2022.
Paul W. Farris, Neil Bendle, Phillip Pfeifer, and David Reibstein. Marketing Metrics: The Manager's Guide to Measuring Markeling Performance. Pearson Education, 2016.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Nuno C. Garcia, Pictro Morerio, and Vittorio Murino. Learning with privileged information via adversarial discriminative modality distillation. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2020.
Chen Gong, Xiaojun Chang, Meng Fang, and Jian Yang. Teaching semi-supervised classifier via generalized distillation. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligncce, IJCAI-18. International Joint Conferences on Artificial Intelligence Organization, 2018.
Gooffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
Rico Jonschkowski, Scbastian Höfer, and Oliver Brock. Patterns for learning with side information, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
John Lambert, Ozan Sener, and Silvio Savarese. Deep learning under privileged information using het- eroscedastic dropout, 2018.
Wonkyung Lee, Junghyup Lee, Dohyung Kim, and Bumsub Ham. Learning with privileged information for efficient image super-resolution, 2020.
Xue Li, Bo Du, Chang Xu, Yipeng Zhang, Lefei Zhang, and Dacheng Tao. Robust learning with imperfect privileged information. Artificial Intelligence, 2020.
David Lopez-Pase Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information, 2016.
Timothy Arthur Mann, Sven Gowal, Andras Gyorgy, Huiyi Hu, Ray Jiang, Balaji Takshminarayanan, and Pray Srinivasan. Learning from delayed outcomes via proxies with applications to recommender systems. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4324 4332. PMLR. 09-15 Jun 2019. URL
mlr. wtt.pppp......//ee/... http////mmmm9m.m/m.mmmm
Konstantin Markov and Tomoko Matsui. Robust Speech Recognition Using Generalized Distillation Framework. In Proc. Interspeech 2016, pp. 2364 2368, 2016. doi: 10.21437/ 11t.33//1nttrspecchh0000 2016-852.
Rishabh Mehrotra, Niannan Xue, and Mounia Lalmas. Bandit based optimization of multiple objectives on a music streaming platform. Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 3224-3233, 2020.
NASA. Nasa - nearest earth objects, version 2. https://www.aagg: haagggll..co//dtttsssst/sseeeepe haaa--nerrettt-aartbb-bjeeee 2024. Retrieved August 1.
Guillermo Ortiz-Jimenez, Mark Collier, Anant Nawalgaria, Alexander Nicholas D'Amour, Jesse Berent, Rodolphe Jenatton, and Efi Kokiopoulou. When does privileged information explain away label noise? International Conference on Machine Learning, pp. 26646 26669. PMLR, 2023.
Yudiyanto Joko Purnomo. Digital marketing strategy to increase sales conversion on e-commerce platforms. Journal of Contemporary Administration and Management (ADMAN), 122):44622 62, Aug. 2023.
Hitesh Sagtani, Madan Gopal Jhawar, Rishabh Mehrotra, and Olivier Jeunen. Ad-load balancing via off-policy lcarning in content marketplace. In Proceedings of the 17th ACM International Conference on Web Search and Dala Mining, pp. 586 595, 2024.
Carlos Serra-Toro V. Javier Traver, and Filiberto Pla. Exploring some practical issues of svm++ Is really privileged information that helps? Pattern Recognition Letters, 2014.
Y. Soo. Smoking and drinking dataset with body signal, version 2. http:///www.kglle.cco//eaaeeeee sooyoungher//mmkiing drinking dataaet/atattttaaaaaaaaaaa 2024. Retrieved August 1.
Trevor Standley, Amir R. Zamir, Dawn Chen, Loonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning?, 2020.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Vladimir Vapnik. Statistical Learning Theory. Wiley-Interscieenee, 1998.
Vladimir Vapnik and Rauf Izmailov. Learning using privileged information: Similarity control and knowledge transfer. Journal of Machine Learning Research, 2015a.
Vladimir Vapnik and Rauf Iamailov. Learning using privileged information: Similarity control and knowledge transfer. Journal of Machine Learning Research, 16(61):2024 2:022-2049, 2015b. URL http://jmlr.or org/
papers/v16tvvapnik55 hmml
Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged information. Neural networks, 22(5-6):544 557, 2009.
Sethu Vijayakumar. The sarcos dataset. Available online, 2000. URL: https://gausiaaprccesssso
gpml/data/
Chen Xu, Quan Li, Junfeng Ge, Jinyang Gao, Xiaoyong Yang, Changhua Pei, Fei Sun, Jian Wu, Hanxiao Sun, and Wenwu Ou. Privileged features distillation at laobao recommendations, 2020.
Siwen Yan, Phillip Odom, Rahul Pasunuri, Kristian Kersting, and Sriraam Natarajan. Learning with privileged and sensitive information: a gradient-boosting approach. Fronliers in Artificial Intelligence, 2023.
Jeremy Yang, Dean Eckles, Paramveer Dhillon, and Sinan Aral. Targeting for long-term tergtermmmnn outcomes, 2022a.
Shuo Yang, Sujay Sanghavi, Holakou Rahmanian, Jan Bakus, and Vishwanathan SVN. Toward understanding privileged features distillation learning-to-rank. Advances in Newral Information Processing Systems, 35: 26658-26670, 2022b.
14

---

Published in Transactions on Machine Learning Research (02/2025)
A Independent features
We follow the proof by Yang et al. (022b) for the special case that m = 0 (no unlabelled instances)
Assuming a linear model generating the label as follows:
y ==x + +zvv+ tc, c~\(0,00))
(8)
where w Rd and v E RR the unknown parameters, the regular features (N, la,), the privileged features (0, Ia.). and represents label noise. The solution of the standard lincar regression is
reg X + Zv* N) = + (tvVVV + +N),
(9)
where N E Rax1 the label noise vector. Therefore, we have
Ex| w'1l = Ex| + +N) +N)12
de (o² 1| 11 - dn -
The last equality holds because x = (XTX) -1 follows the inverse-Wishate distribution, whose expectation
is T
For generalised distillation, the teacher E BRdrtd; we have
Ø = X Xw + Zv + +N) [w' T. w TJT + [X2, (Z
where Xz,1 is the pseudo inverse of the projection of X to the column space orthogonal to Z, and Zx is defined similarly. After distillation, we have that
X Z| 0
- + x'zv" + +X N + XZ N.
We note that Z N has variance of order O ( which is non dominating term. For the other two terms we have
Ex, z/| Wri - w0 = Ex Zv" + X 2.1 NI;
d, ||
dg o²
dr 1 + d2 d2 -
2 Ex WWreg
- -w'
B Generalized distillation: SARCOS experiment
SARCOS experiment from Lopez-Paz al. (2016) The last experiment provided by Lopez-Paz et al.
(016)) based the SARCOS dataset. (Vijayakumar 2000 This dataset. characterizes the joint torques
15

---

Published in Transactions on Machine Learning Research (02/2025)
16-
16-
jemperanie terppeatuee
Cumpecinees
carpesiue
100
Figure 6: Reproducing the SARCOS experiment with the teacher replaced with Jr 0.
of robotic arm given 21 real-valued features. Lopez-Paz al. (006 learns teacher on 300 samples to predict each of the 7 torques given the other 6, and then distills this knowledge into a student who uses her regular input space the 21 real-valued features. They report improvement in mean squared error when using Generalized distillation and conclude, "when distilling at the proper temperature, distillation allowed the student to match her teacher performance."
However, there misalignment between the experiment sctup and the conclusion drawn by the authors. It is observed that the teacher labels approach 0, the student's 's performance improves. In fact, in Figure equation [6] demonstrate that, due to the experiment setup, plugging in all zeros student model corresponds to the best student's performance. In their code, instead of applying T as a softmax temperature to the labels, the authors divide the soft label by T. This means that by increasing the temperature 7 and the imitation parameter in the original experiment, the authors force the Leacher labels closer to and report the observed improvement. Given that this is not reported in the paper, we believe this be unintended by the authors. However, this means that the performance improvement can fully be attributed the temperature scaling and not to successful knowledge transfer of PI.
target for the
C Experiment
Table 3: Expanding the training size of Experiment (Relevant features privileged information) from
Lopez-Paz et al. (0166). The effect of Generalized distillation wears off when the training size surpasses 2000 samples.
Training size
Privileged
Generalized distillation
no-PI
200
0.97 40.02
0.96 10.02
0.84 10.03
500
0.97 40.02
0.97 40.01
0.92 40.02
1000
0.98 10.02
0.97 10.01
0.95 10.01
2000
0.98 +0.02
0.97 +0.01
0.96 40.01
5000
0.98 40.02
0.97 40.01
0.97 #0.01
D Extending the TRAM experiment to classification tasks
Synthetic experiments for classification task To further demonstrate that explaining away harmful noise non-trivial, extend the sctting above to a classification task make more suitable for our use-case example. As such, y is a binary label that represents conversion, and is PI, which represents the nature of the click.
16

---

Published in Transactions on Machine Learning Research (02/2025)
66-
Noma Cloa
Set Otse
BarratDbss
Narmal obe-
Nose
don-
Mit 2.35
Nort DC3
R:P1 0.000
TAMM 0.217
TIAE 2004
TRAM 0.E14
(a) Deterministic (U)
(b) Bernoulli (U)
(c) Random (U)
(d) Cosine (U)
www.teettarreccaoooooo 200 spacta wTw A
wopereetttee and opot's IEEI
1.0
NETTE 08e4
Namsiidhss
Obro
Reoe
leae frue
Not. 5855
RoP: 0.22
RuP1-0 812
TRam 0.333
TRAM seree
(e) Deterministic (S)
(f) Bernoulli (S)
g Random (S)
(h) Cosine (S)
Figure/ 7: Example of TRAM and no-PI for 4 classification lasks. The models trained for 50 epochs and 2500 samples in the top row and for 200 epochs and 10000 samples in the bottom row. The numbers in the legend indicate MSE loss with respect to the noise-free function. (U) corresponds to an undertrained regime,
(S) corresponds to a sufficiently trained regime.
Similarly to Collier et al. 2022 2 ~~ (0r(..3), and the data generating process is as follows:
Vscore (1 z). sin(2Tzz) + y ~ Ber(yscore),
(10)
where E [0, 1] and represents the nature of the click. We consider four scenarios of PI impact the label: Deterministic v= 1, Bernoulli v (er(.. Uniform Unif[-1,1], Cosine ccos(2zz)) The examples of these scenarios and trained TRAM and no-PI models are represented in Figure 7 with Figures
7a 7d
representing models trained for 50 epochs with 2500 samples and Figures 7e representing models trained for 200 epochs with 10000 samples.
Intuitively, Uniform resembles the original setup of Collier al. (0022) but for the classification task. In our setting, it can be motivated by a bot or users that just randomly click on banners. Deterministic might correspond to adversary that, for example, always clicks and never makes purchase. Intuitively, explaining the noise for Delerminislic regime should be more difficult than for Uniform regime because there is no randomness. Bernoulli regime is middle point between Uniform and Deterministic regimes there is still corruption but with some randomness. Finally, Cosine corresponds to a scenario when there are two types users with different click behavior (according to sin for part of the population and to cos for the rest of the population).
Taking closer look at Figures 7a 7d
we can see that TRAM enables faster convergence rate. However, from Figures 7e 7h it is apparent that both models No PI and TRAM eventually converged to the same functions, which do not correspond to the noise-free function sin(2..)..
Finally, we empirically analyze the sample efficiency of TRAM compared to the no-PI model. We train TRAM and No PI models for varions values of n, from 100 to 10000. Both models are trained for 200 epochs for each generated datasct. Figure (right) presents MSE loss across different valies of n. We can see that both models converge to roughly the same value of all data regimes and all values of n, which suggests that TRAM doesn't enhance the sample efficiency.
17

---

Published in Transactions on Machine Learning Research (02/2025)
Convergence rate
Sample efficiency
0.09
0.09
0.08
0.08
0.07
0.07
MSE loss
MSE loss
0.06
0.05
0.05
0.04
0.04
0.03
0.03
0.02
0.02
10a
10
num epochs
num samples (log)
deterministic-NoPl
bernoulli-Nor lNPl bernoulli-TRas
random-Nott - random-TRAM
cosin-NoPI
deterministic-TRAM
cosin-TRAM TRAM
Figure TRAM and no-PI training dynamics for 4 data regimes.
E
Experimental details
This section describes experimental details for sections 4.1
4.2 and 5 The source code for all experiments
is atlached in supplementary materials and will be available publicly upon acceptance of the article. We distribute all runs across 6 CPU nodes (Intel(R) CPU 17-10750H) and GPU Nvidia Quadro T1000 per run for experiments.
Generalized distillation experiments We follow the original setup of Lopez-Pav. et al. 2016) For both Experiment and Experiment 3, as a no-PI, student, and teacher models, we use lincar layer of dimension 50, with softmax activation. The networks were trained using an rmsprop optimizer with mean squared error loss function. The temperature and imitation parameters for Generalized distillation were set to 1.
For MNIST and SARCOS experiments, we use two-layer fully connected neural networks of dimension 20, with ReLU hidden activations and softmax output activation for the no-PI, student, and teacher models. The MNIST experiment corresponds to a classification task with ten labels. The networks were trained using an rmsprop optimizer with mean squared error loss function. The temperature and imitation parameters for Generalized distillation in the MNIST experiment were set and 1, respectively, the best parameter set from the original paper Lopez- Paz al. (2016).
TRAM experiments We consider two tasks regression and classification. For both of them, as a no-PI model, we use two-layer fully connected neural networks of dimension 64, with tanh hidden activations and linear output activation for regression and sigmoid for classification. TRAM model has an extra hidden layer of size 64 with tanh activation function in the PI head. Both TRAM and no-PI networks fit using the Adam optimizer Kingma & Ba (017 with mean squared error loss function. The nunbers of epochs are specified in figure captions for each experiment.
Real-world experiments The experiment design is the same for all datasets unless stated otherwise.
For the no Pl model, activation and a residual connection. For the Generalized distillation model, the teacher and student have the same archilecture as the no-PI model, with teacher models' inputs x and being fed independently to the linear layer first and then concatenated. The temperature and imitation parameters for Generalized distillation were set and 1, respectively, as the best parameter set. For TRAM, the feature extractor d(x) also has an architecture of the no-PI model, and similarly the teacher of Generalized distillation, the PI head of TRAM had independent. inputs x and that goes through linear layer first.
two-layer fully connected neural network with the Gaussian error linear unit
18

---

Published in Transactions on Machine Learning Research (02/2025)
All models trained for 50 epochs with cross-entropy loss function and Adam optimizer with base learning rate of 0.001, $1 = 0.9, 31 0.95, = le 07. All models are trained with L2 weight regularization with a decay weight of 0.1.
We train all models 10 times with the random initialization, and for all models, we report the cross- cross-entropy loss value and performance metric on the test data Heart Disease datasets and accuracy for NASA-NEO and Smoker or Drinker datasets and ROC AUC scaled between and 1 (2*ROC AUC 1) for Repeat Buyers (refer to Table 2 Additionally; performance metric the test data in Figure | demonstrate that PI is indeed useful information.
report the training dynamics of the cross-entropy loss value and
The teacher performance is provided for the reference to
F Other related work
Multi-task learning While not strictly focused on the concept of PI, indications of successful knowledge transfer can be found in the field of multi-task (Caruana 1997) and multi-objective learning (Mehrotra et al., 2020; Sagtani et al., 2024h. The primary goal of this type of research to find some joint- or Pareto optimal solution for multiple tasks or objectives simultaneously. These techniques could also be interpreted as case of LUPI by predicting each privileged feature with an additional task. However, while instances successful knowledge transfer have been reported in the literature, the quality of predictions is often observed to suffer with making multiple prodictions due to phenomenon called negative transfer (Standley al., 2020
Different from multi-task learning, LUPI mainly focuses on improving the learning of the larget task rather than ensuring the performance of all the tasks (Jonschkowski et al. 2016). From the practical point of view, when using dozens of privileged features at once or when estimating the privileged features is more complicated than the original problem, it would be a challenge to tune all the tasks (Xu el al. 2020). For this reason, we focus on methods that can generalize to any type of PT and are not exclusive auxiliary tasks.
Surrogate signals In similar spirit. to LUPI, the proxy or surrogate signals literature (Athey al,, 2019: Mann al. 2019 Yang et al.) 2022a) studies how short-term outcomes can be used for estimating the long-term target outcome (e.g,, in cancer studies). In this setting, the materialization of the target outcome is generally delayed to such an extent that it is unfeasible to use for decision-making By using short-term proxy or surrogate, existing work is able to construct a best-effort estimation of the primary signal before it has fully matured. In contrast to the PI setting, the issue of knowledge transfer is not presented. Additionally, we assume that the primary outcome has fully matured, hence the use such proxies is not desirable.
19


Reviews:
[{"review_id": "DokKIxphxG", "reviewer": "Reviewer_6Bsi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I believe that this critique of the LUPI paradigm is beneficial. It will prevent many users from wasting time because they trust in its universal applicability. The conceptual problem of PI is its non-asymptotic nature. As far as the training data set size grows, a machine-learning algorithm becomes able to work on its own, obtaining the same knowledge from non-privileged information.\n\nOn the other hand, for the same reason, I have to say a bit in defence of the LUPI paradigm. The first two rows of Table 1 (small training size) are valuable as a proof-of-concept. This is what transfer of PI can help with, that is, acceleration of learning at its early stages when the training set is small.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper observes and critiques the justification of the LUPI paradigm from several earlier works.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would not insist on essential changes right now. But in my view, the approach to analysing LUPI should be revised, involving the dependence of the contribution from PI on the training set size as an essential dimension.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AS8WBW7E5C", "reviewer": "Reviewer_DMQt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nS1. The paper is mostly well-written and structured. \n\nS2. The paper discusses an important subject, learning with PI,  as it is being used in research frequently to improve generalization. \n\nS3. The paper performs fairly sound evaluations, including experiments on real-world datasets.\n\n**Weaknesses:**\n \nW1. In the introduction section, par1, l 4, the authors say “… this information only materializes post-inference.” Can you elaborate on this and provide an example? \n\nW2. In the same paragraph, “… optimizing a north-star metric such as product conversion”. Can you provide references?\n\nW3. I believe the first contribution regarding the theoretical review of the LUPI methods is overstated. The paper simply re-states the (admitted) assumptions of the examined works and argues that they are strong or difficult to verify. \n\nW4. In Section 4, in addition to the datasets, please provide more details on the underlying models, the task, and the learning algorithms. \n\nW5. In Section 4.1, the presentation of experiments 1 and 2 could be improved. Embed them in a table for example. \n\nW6. While the authors admit this in conclusion, it is worth repeating in Sections 1 and 4  that this paper only examines two LUPI methods in specific settings and refutes some of their claims.  \n\nW7. I wonder if the authors have any insight into the larger scales -- both larger datasets and larger models. Many works in deep learning use knowledge distillation and report improvements. Do you think larger feature space or parameter space would create the potential for knowledge transfer?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary: The paper discusses learning with privileged information (PI) and raises concerns regarding the knowledge transform capabilities of such learning methods. The paper revisits two known methods: generalized distillation, and marginalization with weight sharing. The authors argue that the assumptions for theoretical analysis in these methods are over-restrictive, and empirically question the conclusions such works have made.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "please see the weaknesses.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "spfBSdl77q", "reviewer": "Reviewer_wfft", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Analysis spanning theoretical and empirical perspectives\n2. Experiments across diverse real-world datasets\n\nWeaknesses:\n1. Limitations in Generalizability:\n   - The theoretical analysis might not fully capture the nuanced ways privileged information could be beneficial\n   - The conclusions might be overly broad given the limited set of methods examined\n2. Potential Methodological Constraints and Confirmation Bias:\n   - The experiments used relatively simple neural network architectures\n   - The definition and selection of privileged information might influence results\n   - The authors seem predisposed to proving the limitations of LUPI, which could lead to a selective interpretation of results\n   - The experimental design might inadvertently be structured to minimize the potential benefits of privileged information\n3. Potential Oversimplification\n   - The conclusion that LUPI is ineffective might be premature\n   - The study doesn't fully explore edge cases or specific domains where privileged information might be crucially important", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript examines the concept of Learning Using Privileged Information (LUPI), a machine learning paradigm where additional information is available during training but not during inference. The authors attempt to challenge the effectiveness of existing LUPI research from both theoretical and empirical approaches. Experimental validation across four real-world datasets from e-commerce, healthcare, and aeronautics domains, demonstrating that state-of-the-art LUPI methods fail to consistently outperform models without privileged information.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Expand the experimental design to include a broader range of LUPI techniques and more varied datasets and problem domains\n2. Include more ablation studies to isolate the specific factors limiting LUPI effectiveness\n3. Develop more sophisticated metrics for assessing knowledge transfer beyond traditional performance measures.", "broader_impact_concerns": "There is no concern on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers’ recommendations are a mix of acceptance and rejection.
We agree that the manuscript could be interesting and insightful for researchers working on privileged information; however, it still requires further refinement. The manuscript examines two Learning Using Privileged Information (LUPI) methods—generalized distillation and marginalization with weight sharing—in specific settings and refutes some of their claims. While Sections 1 and 4 have been revised to emphasize this scope based on the reviewers’ suggestions, the abstract still does not reflect this.

The conclusions might be overly broad given the limited set of methods examined. The concluding statement—“[o]ur work rethinks knowledge transfer in learning using PI and highlights that, in the current state of research, there is no solid empirical or theoretical evidence that LUPI works in realistic scenarios”—might be premature, as the study does not fully explore edge cases or specific domains where privileged information may be crucially important.

For example, Xu, Gao, Yu, and Darrell (“End-to-end Learning of Driving Models from Large-scale Video Datasets,” CVPR 2017) demonstrated how to “drive with privileged information.” In the discussion, the authors mentioned that privileged information could be leveraged to improve fairness in machine learning models; however, they did not include this discussion in the revised manuscript, as it does not “[enhance] general model performance relative to specific metrics such as accuracy or loss.” I encourage the authors to discuss this aspect further. For example, Yan, Odom, Pasunuri, Kersting, and Natarajan (“Learning with Privileged and Sensitive Information: A Gradient-Boosting Approach,” Frontiers in Artificial Intelligence, 2023) showed that their approach leverages sensitive information to improve performance while maintaining fairness.
Decision: Accept (poster)