Toward Understanding Privileged Features Distillation in Learning-to-Rank
Authors: Shuo Yang
sujay sanghavi
Holakou Rahmanian
Jan Bakus
Vishwanathan S. V. N.
Venue: NeurIPS 2022 Accept (2022)
Keywords: Privileged Feature
Learning-to-rank
Distillation

Abstract: In learning-to-rank problems, a \textit{privileged feature} is one that is available during model training, but not available at test time. Such features naturally arise in merchandised recommendation systems; for instance, "user clicked this item" as a feature is predictive of "user purchased this item" in the offline data, but is clearly not available during online serving. Another source of privileged features is those that are too expensive to compute online but feasible to be added offline. \textit{Privileged features distillation} (PFD) refers to a natural idea: train a "teacher" model using all features (including privileged ones) and then use it to train a "student" model that does not use the privileged features.
  
In this paper, we first study PFD empirically on three public ranking datasets and an industrial-scale ranking problem derived from Amazon's logs. We show that PFD outperforms several baselines (no-distillation, pretraining-finetuning, self-distillation, and generalized distillation) on all these datasets. Next, we analyze why and when PFD performs well via both empirical ablation studies and theoretical analysis for linear models. Both investigations uncover an interesting non-monotone behavior: as the predictive power of a privileged feature increases, the performance of the resulting student model initially increases but then decreases. We show the reason for the later decreasing performance is that a very predictive privileged teacher produces predictions with high variance, which lead to high variance student estimates and inferior testing performance.

Toward Understanding Privileged Features Distillation in Learning-to-Rank
Shuo Yang" UT Austin yangshuo ut@utexas.en edu
Sujay Sanghavi
Holakou Rahmanian
Amazon sujayrs@amazon..oom com
Amazon
holakou@amazon. com
Jan Bakus Amazon j jbakus@amazon.com com
S.V.N. Vishwanathan
Amazon
vishy@amazon.ccom
Abstract
In cearning-to-rank problems, privileged feature is one that is available during model training, but not available test time. Such features naturally arise merchandised recommendation systems; for instance, "user clicked this item" as a feature is predictive "user purchased this item" in the offline data, but is clearly not available during online serving. Another source of privileged features is those that are too expensive to compute online but feasible to be added offline. Privileged features distillation (PFD) refers to natural idea: train "teacher" model using all features (including privileged ones) and then use it to train a "student" model that does not use the privileged features.
In this paper. we first study PFD empirically on three public ranking datasets and an industrial-scale ranking problem derived from Amazon's 'ss logs. We show that PFD outperforms several baselines (no-distillation, pretraining-finetunine distiiiiillttonn and generalized distillation) on all these datasets. Next, we an- alyze why and when PFD performs well via both empirical ablation studies and theoretical analysis for linear models. Both investigations uncover an interesting non-monotone behavior: as the predictive power of privileged feature increases, the performance of the resulting student model initially increases but then decreases. We show the reason for the later decreasing performance that a very predictive privileged teacher produces predictions with high variance, which lead to high variance student estimates and inferior testing performance.
Introduction
For recommendation systems, the features at test time are typically a subset of features available during training. Those missing features at test time either expensive compute in real-time, or they are post-event features. For instance, for an e-commerce website, "click" a strong feature for predicting "purchase", but "click" exists feature only in the offline training data, but not during online serving (i.e., one cannot observe "click" before recommendations are generated). Those features that exist only during training called privileged features. Those that exist during both training and testing are called regular features [XLG 20).
The naive approach is to ignore the privileged features and train model that only takes regular features. Such methods inevitably miss the information in the privileged features and lead to inferior
This work done while Shuo Yang interning Amazon.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

performance. A natural instinct to resolve this is to (a) use the privileged features (either by themselves [LPSBV16] or in conjunction with regular features [XLG 20]) to train "teacher" model, and then (b) use it to transfer information via distillation" into "student" model that only uses the regular features. The approach of teacher only using privileged features is named generalized distillation (GenD) [LPSBV16] 16), and the approach of a teacher using both privileged and regular features has been referred to as privileged feature distillation (PFD) [XLG 20].
In this paper we provide a detailed investigation - first via empirical ablation studies on moderate- scale public and industrial-scabe proprietary datasets with deep-learing-to-- -t--rank models, and second via rigorous theoretical analysis on simple linear models into why and when privileged feature distillation works and when it does not. While this paper focuses on learning-to- -t-onn,,--- our results apply to regression/cassiffattinn
problems in general. As a summary, our main contributions are:
We evaluate PFD on three moderate scale public ranking datasets: Yahoo, Istella, and MSLR- Web30k, and an industrial-scale proprietary dataset derived from Amazon search logs.
In all evaluated settings, PFD is better than or as good as the baselines: no-distillation, GenD (teacher model only uses privileged features), self-distillation (teacher model only uses regular features), and pretraining on privileged features then finetuning (when applicable) (Table 2).
We conduct comprehensive ablation studies for PFD. We find that
PFD is effective as long as the teacher loss dominates the distillation loss and the performance is not sensitive to a. Specifically, distillation loss is a linear combination of the loss w.r.t. data and the loss w.r.t. teacher predictions and a the mixing ratio (Figure 3).
While it is known that the gains from self-distillation (over no-distillation one-shot training baseline) are larger when the positive labels are sparser, we see that these gains are further amplified by PFD; i.e. the relative gain PFD over self-distillation also increases as the labels become sparser (Figure 4).
Non-monotonicity in the effectiveness of PFD: the predictive power of privileged feature increases, the resulting student performance initially increases but then decreases (Figure 5).
To provide deeper insight into the landscape of privileged features and distillation, we next rigorously analyze it in stylized setting involving linear models. We show that
PFD works because the teacher can explain away the variance arising from the privileged features, thus allowing the student to focus on the part it can predict. (Theorem l).
The reason that GenD inferior PFD (as seen in our empirical evaluation) is because results in weaker teacher, and also because in the case where the privileged and regular features are independent, the teacher predictions appear as pure noise to the student (who cannot learn from them) (Remark 2).
A very predictive privileged feature induces high variance teacher predictions, which lead to inaccurate student estimates and inferior testing performance. This explains the observation that the most predictive privileged features do not give the best performance (i.e., the non- monotonicity) in our empirical ablation studies (Theorem 2).
The rest of the paper is organized follows: Section 2 covers related works. Section 3 introduces the problem setup, the PFD algorithm and other algorithms for comparison. Section 4 presents empirical evaluation and ablation studies of PFD; and Section presents theoretical insights.
2 Related Work
Privileged features widely exist in different machine learning problems, including speech recognization [MM16], medical imaging [GCA+19] 19], image super-resolution [LLKH20], etc [FA12, FTRSI3, FKSHI4, ALL17J. Privileged features are not accessible during testing either because they are too expensive to compute in real time, or because they are post-event features (thus cannot be used as input) [CM18].
Learning with privileged features is pioneered in [VV09], where they propose framework named "learning using privileged information" (LUPI). At the core, LUPI uses privileged information to
'Here, by distillation we mean the standard practice of labeling the training dataset using teacher predictions. and using these supervision targets in the training of the student model.

---

distinguish between easy and hard examples. The methods are thus closely related to SVM, as the hardness of an example can be expressed by the slack variable. For instance. [VV09, PIVVI0] propose the "SVM+" algorithm which generates slack variables from privileged features and learns an SVM based on regular features with those slack variables; [SQL13] proposes a pair-wise SVM algorithm for ranking, which uses privileged features to distinguish easy and hard pairs. [LHS]] [HH presents variation where the privileged features are used to generate importance weighting for different training samples. Empirically, [SER 14] demonstrates that whether LUPI is effective critically depends on experimental settings (e.g., preprocessing, training/validation split, etc). [VI15] considers transferring the kernel function from teacher SVM that only uses privileged features student SVM that only uses regular features; [LDX* 20] extends the SVM+ algorithm to imperfect privileged features.
Model distillation [HVD* + 15] is a common method for knowledge transfer, typically from large model to a smaller one [PPA18, GYMT21]. Recent works have shown great empirical success in ranking problems [TW18, 18, HAS+20, RPM+21] and even the cases where the teacher model and student model have the identical structure [FLT+18, QYT+21].
Using distillation to learn from privileged features are first proposed in [LPSBV16] as "generalized distillation" (GenD). It provides a unified view of LUPI and distillation. GenD, along with the variants [MM16, GMM19, LLKH20J, train a teacher model with only privileged features and then train a student model mimic the teacher's predictions. PFD is recently proposed in [XLG* XXLG+ where the teacher model takes both regular and privileged features as input. PFD and GenD differ from the standard model distillation as they focus on exploiting privileged features but not on reducing the model size. [XI.G 20] empirically demonstrates the superior performance of PFD for recommendation systems on a non-public data set.
Understanding of privileged features distillation is lacking, despite the aforementioned empirical success. Previously, [PVI0] shows that LUPI brings faster convergence under a strong assumption that the best classifier is realizable with only privileged features. [LPSBV DSSV16,, 16] shows that GenD enjoys fast convergence rate. It assumes that the teacher model has much smaller function class complexity than the student model, which does not match with PFD. [GCFY 88] studies GenD under sumissuper learning and shows that the benefits come from student function class complexity reduction. However, it does not quantify such reduction and the theory does not explain what is the benefit of using privileged features. To the best of our knowledge, there is no empirical or theoretical study explaining why PFD is effective.
Other ways of utilizing privileged features are also previously proposed. [CJFY17] uses privileged information to learn a more diverse representation to improve image classification performance. [LLKH20, WZW+21] propose distillation schemes for better feature extraction from regular features. A more recent work [CJKB22] considers training a model with both regular and privileged features to obtain better internal representation of the regular features.
3 Problem Setup and Algorithms
Consider a learning-to-rank problem where each query-document pair has features x E X and z E 2 and a label E 3' (e.g., click or human-annotated relevance) drawn from an unknown distribution D(y|x, z). Suppose x is the regular feature that is available during both training and testing and z is only available during training. Concretely, privileged feature is defined in the literature below:
Definition 1 (Privileged Feature [CJKB22]). For feature z that exists during training but not testing, we say z is a privileged feature if and only if I [[yzz]x] : H(yl)) - H (3) x, z) > O.
Conditional mutual information Tyy z|x) and conditional entropy H----) follow from the standard notation of information theory. According to Definition 1, the privileged feature z provides extra predictive power of y. For the rest of this paper, we focus on the setting that z is a privileged feature.
Remark 1. An implication of Desinition I is that the privileged feature z can be independent of the regular feature x. In such cases, any transformation of z is not learnable from x, and therefore using z as auxiliary learning target does not help. Interestingly, PFD can still improve the student performance, even when z and x are independent (see Section 5).
We consider the following general learning problem: we are given a labeled training set of size n, Stabel == {(xi, Zi, ys) v.)) se(n) and a unlabeled training set of size m, Suntabel {(x,, zi) se[m]] Our
3

---

Privileged Features Distillation (PFD)
Generalized Distillation
Self-distillation
(GenD)
Teacher
Student
Teacher
Student
Teacher
Student
Privileged Regular
Privileged Features
Regular Features
Regular
Figure Illustration of PFD. generalized distillation (GenD) and self-distillation.
goal is generate good ranking based only regular features For clarity of exposition, only consider pointwise scoring functions = J + } which generates a score for each document, and the ranking is induced by sorting the scores. The results in this paper can be easily extended to models beyond pointwise scoring functions (c.g., DASALC [QYZ+
The distinction between labeled and unlabeled datasets is for generality. The unlabeled dataset naturally appears recommendation systems, where the majority search logs do not contain any user interactions. Instead of taking all such logs as negative samples, it is more proper to view them as unlabeled data due to the lack of user engagement. For the logs that contain click, the documents therein with no click can be treated as negative samples.
3.1 Privileged features distillation
PFD first trains teacher model that takes both x and z as input to predict y. i.c., teacher function class is GPED = {g g X' x 2 - y). For simplicity, we consider pointwise loss 1 ) x y r> IR in this section, while the method pairwise loss in Section 4). The privileged features distillation takes the following two steps:
be easily extended other loss functions (see extension to
Step I: Training teacher model yPFd E GPED by minimizing the loss the labeled dataset: (x,)))))) (g(x;, z;), ys. practice, gradient-based optimizer is used for loss minimization.
Step II: Training student model by distillation. The teacher model gEED trained from Step I is used to generate pseudo labels on Stabel and Suntabet- Let Saul denote the union of Stabel and Sanlabei- The student model is trained by minimizing the following distillation loss:
((((x))ui) + (1-a) a)-
Σ (ES,
((((x,, (((x)))orr((x (xi, 2;)),
data
where a E (0, 1) controls the mixing ratio between the data loss and teacher loss. The student model trained by minimizing the distillation loss in Equation (1).
3.2 Other algorithms for comparisons
Here we introduce two other algorithms for comparison. See illustration in Figure 1.
GenD [LPSBV16, is a distillation method where the teacher model takes only privileged features as input, i.e., the teacher function class is GGrnD (g g: 2 + y}. The teacher model gGonn E GGODD obtained by minimizing (r (g(2.), y:). Similar to PFD, the distillation loss is a linear combination of the data loss and teacher loss.
Self-distillation |FLT+ 18, QYT+21] is distillation method where the teacher model has the same structure as the student model. Specifically, the teacher model Iseff-dist E F obtained by minimizing (x..y.) (g(x,), y.). Notice that F is also the student function class. Similar to PFD, the distillation loss linear combination of the data loss and teacher loss. Comparing PFD against self-distillation separates the benefits of adopting privileged features and distillation.
Experiments
4.1 Main results public datasets
We first evaluate the performance PFD on three widely used public ranking datasets. Specifically, use the Set from "Yahoo! Learn to rank challenge" [CC11]; "Istella Learning to Rank" dataset

---

[DLN't 16]; and Microsoft Learning to Rank WSLL--We330k" dataset [QL13]. We refer to them as "Yahoo", "Istella" and "Web30k" throughout this section.
Datasets overview and preprocessing. The training samples in all three datasets can be viewed as query groups, where each query group contains I query and multiple documents be ranked. Each query-document pair is represented as a real-value feature vector (c.g., user dwelling time, tf-idf of document, etc. See [CC11] for detail). Further, each query-document pair has a human-annotated relevance score E 1,, 3, 4}. All datasets are preprocessed by removing query groups that contain no positive relevance score or have less than 10 documents. The features are transformed by the loglp transformation as in IZWBN20, QYZ+
Binary label generation. In practice, binary label (e.g., click) is more commonly seen and easier to obtain than relevance score. For our experiments, we generate a binary label y for each query- document pair based the human-annotated relevance score Specifically:
= [t - r G1 > t Turget + Go),
(2)
where is a temperature parameter and G1 and Go follow the standard Gumbel distribution. It can be shown that y is I with probability at (r Trarget), where af-) is the sigmoid function (see Appendix A.I for proof). For the rest of our experiments, we set = 4 and Target 4.8 unless otherwise mentioned. We refer to the query groups that contain at least one y 1 to be positive query groups, and other query groups are referred to as negative query groups.
Regular and privileged features split. For each of the datasets, sort the features according the magnitude their correlations with the binary label y and use the top 200, 50, and 40 features as privileged features for Yahoo, Istella, and Web30k, respectively. Other features are used as regular features. Please see Table for dataset statistics after preprocessing and binary label generation.
# features
# qqeery groups
# docs per query group
# positive query group
Data set
regular
privileged
training
test
training
test
training
test
Yahoo
500
200
14,477
4,089
30.02
29.93
2.46%
2.08%
Istella
170
50
23,171
9,782
315.90
319.62
8,83%
9.13%
Web30k
96
40
18,151
6,072
124.35
123.34
3.54%
3.49%
Table 1: Dataset statistics after preprocessing and binary label generation (Equation (2)). Web30k statistics are calculated based on the fold-1 of the official 5 fold splits. Recall that "positive query groups" query groups that contain at least document with y 1.
Ranking model and performance metric. The ranking model is 5-layer fully connected neural network, which maps the query-document feature into a real-value score s E [0, 1]. The ranking मे of documents is obtained by sorting the scores decreasingly, where f(i) represents the ranked order of the i-th document. The ranking performance is measured by the NDCG@ metric:
DCG@X(",y)
2
ADCG@I(f..) DCG@
DCG@k(T,y)
Σ *(t)sk
loge(1 + π (1))
where x² is the optimal ranking obtained by sorting Vi-
PFD is effective for all three
0.25
datasets. We evaluate the effi-
Teating NDCG
0.5
0.20
cacy of PFD on all three afore- mentioned datasets, under both pointwise (RankBCE) and pair- wise (RankNet [BSR +55) loss functions (see definitions in Ap- pendix A.2). Please see the eval- uated algorithms and results in Table 2 (complete results with RankNet loss deferred to Ta- ble 4). Figure 2 shows the testing NDCG@s & curve on Yahoo and Web30k with RankBCE loss.
0.4
0.15
0.3
Dataset: Yahoo
0.10
Dataset: Web30k
0.05
20
40
60
8D
100
0
20
40
64
80
100
Train Training epochs
Training epochs
Baseline
PED
GenD
Self-distilltion
Figure 2: Testing NDCG@8 8 curve Yahoo and Web30k with RankBCE loss. The baseline is no-distillation training with only regular features. It overfits in the later epochs. The performance of self-distillation and GenD depends on dataset. PFD has the best performance in both settings.
5

---

Training method
NDCG @8
NDCG @ 16
NDCG @32
Loss function: RankBCE;
Dataset:
Yahoo
No-distillation
0.517 + 0.005 (+0.0%)
0.557 0.005 (+0.0%)
0.582 0.005 (+0.0%)
Self-distillation
0.522 + 0,004 (+1.0%)
0.559 + 0.003 (+0.3%)
0.585 = 0,003 (+0.5%)
GenD
0.557 = 0.005 (+7.7%)
0.583 + 0.005 (+4.7%)
0.607 0.004 (+4.3%)
PFD
0.566 = 0.004 (+9.5% %)
0.592 + 0.005 (+6.2%)
0.614 + 0.003 (+5.4%)
Loss function: RankBCE;
Dataset: Istella
No-distillation
0.402 + 0,001 (+0.0%)
0.446 + 0.001 (+0.0%)
0.472 0.001 (+0.0%)
Self-distillation
0.402 + 0.001 (+0.0%)
0.446 + 0.002 (+0.1%)
0.472 + 0.001 (+0.2%)
GenD
0.380 + 0.001 (-5.4%)
0.426 + 0.001 (-4.4%)
0.455 + 0.001 (-3.6%)
PFD
0.417 + 0.002 (+3.7%)
0.461 + 0.002 (+3.4%
0.486 + 0.002 (43.1
Loss function: RankBCE;
Dataset: Web30k
No-distillation
0.241 + 0.006 (+0.0%)
0.267 + 0.006 (+0.0%)
0.301 + 0.006 (+0.0%
Self-distillation
0.241 + 0.004 (-0.0%)
0.268 + 0.004 (+0.5%)
0.299 + 0.004 (-0.5%)
GenD
0.205 + 0.004 (-15.0%)
0.233 + 0.005 (-12.7%)
0.269 = 0.005 (-10.6%)
PFD
0.252 = 0.004 (+4.5%)
0.281 + 0.006 (+5.5%
0.314 = 0.005 (+4.3%)
Table 2: Evaluation of PFD and other related algorithms on Yahoo, Istella, Web30k with RankBCE loss function. We set a = 0.5 (Equation (1)) for all evaluated settings. The baseline model is trained without privileged features. We also evaluate "seff-distillationn" [FL.+ 18] and "GenD" [LPSBVI6, for comparisons, see detailed model description Section 3.2. Experiments repeated for independent runs, and the offical 5 fold-splits are used for Web30k experiments. The best checkpoint (measured by testing NDCG@ @ 8) of 100 training epochs is used for evaluation, with mean and standard deviation of 5 runs reported. The results show that PFD has the best performance on all evaluated settings. See Table 4 for complete results with pairwise loss (RankNet) and teacher models' performance.
Yahoo and Istella
Table shows that PFD has the best performance on all evaluated settings. We remark that (1) the only difference between PFD and self-distillation is that the teacher in PFD additionally uses privileged features and therefore has better prediction accuracy than the teacher in self-distillation. Comparing PFD with self-distillation reveals the improvement of using "privileged features" for distillation; (2) the performance of GenD is worse than no-distillation on Istella and Web30k. The reason for such inferior performance is that the teacher model in GenD only uses privileged features (and not regular features). For Istella and Web30k, only using privileged features is not sufficient to generate good predictions. The teachers in GenD also worse than no-distillation, see Appendix A.4.
4.2
Ablation study public datasets
PFD is not sensitive to a.
former experiments, we kept the mixing ratio of teacher loss and
Teating NDCG@8
0.56
data loss to be 0.5. Here
0.54
0.54
evaluate the sensitivity of PFD to parameter The experiments here use the Yahoo dataset and RankBCE loss. From the left- hand side of Figure 3, we see that PFD delivers good performance over large range of a.
0.52
0.52
0.6
10
10
10a
10
Weights the data
teacher over data
Baseline
Privileged Features Distillation
Figure 3: Sensitivity of privileged features distillation to
However, it is worth noting that the teacher loss is typically much larger than the data loss (e.g., about 20 times larger in this set of experiments), since the teacher's predictions are much denser learning targets. The right-hand side plot of Figure 3 takes the scale of both losses into consideration. It shows that PFD yields the best performance only when the teacher loss dominates the distillation loss.
PFD brings larger gain when the positive labels are sparse. Recall that we view negative query groups as unlabeled data, Here we evaluate the performance PFD under different numbers of positive labels. Specifically, by reducing Tragget from 4.8 to 0.4, we can increase the percentage of

---

positive query groups (i.e., query groups with at least one y = 1). The relative improvement over baseline is shown in Figure 4. While it known that distillation works better when there are more unlabeled samples, Figure 4 shows that PFD further amplifies such gains: the relative gain of PFD) over self-distillation also increases as the positive labels become sparser. Such benefit especially favorable in recommendation systems, where the positive labels (e.g., click) are naturally very sparse.
Correlation between privileged features and target. It is believed that privileged features that are discrimina- tive (e.g., high correlation with the target) lead to accu- rate teacher predictions, and thus benefit the distillation 20]. However, we show that PFD has poor perfor- mance when the privileged features are too discriminative.
1.100
Baseline PFD Self- distillation
Scaled NDCG@8
1.075
1.050
Specifically, modify the experiment setting such that all the features in the datasets are used as regular features, while the privileged features z are generated according to l(t (( + G1 > E. Tprivileged + Go), where G₁ and Go have the same values in binary label y generation (Equation (2)). By changing "privileged we can obtain privi- leged features with different correlations with the label y. For instance, when Tpriviteged Target, then 2 can perfectly predict y (since 2 = y by definition); and becomes less discriminative when Tprivileged gets smaller. Using = as the privileged feature, we have the PFD results in Figure
1.025
1.000
0.0
0.2 0.4
0.6
0.8 1.0
Percentage of positive query groups
Figure 4: Distillation performance under different percentages of positive query groups. All testing NDCG @8 scores are scaled w.r.t. the baseline performance. PFD yields larger improvement when the labeled samples are scarce.
Notice that the privileged feature with the largest correla- tion with y does not give the best performance. We believe the reason is that as the correlation of = and y increases, the privileged feature becomes so "discriminative" that it
Testing NDCG@8
0.66
can explain almost all the variance in even the noise, As result, teacher predictions have high variance, which leads to high-variance student estimates and inferior test- ing performance. See Section 5.2 for theoretical insights.
0.64
0.62
10
Privileged feature and target correlation
Baseline
PFD
4.3 Evaluation on Amazon's dataset
Figure 5: The performance of PFD using privileged features with different corre- lations with y. The privileged features with the highest correlation with y do not yield the best distillation performance.
Dataset overview and ranking model. The dataset is derived from Amazon's 's logs which contains query and product title text, the position at which the product was shown, and the user's behaviors click, add-to-cart, and purchase. The ranking model is multi-layer transformer that maps query and product title an estimate of the purchase likelihood. The goal is to rank the products that are more likely to be purchased first.
Efficacy of PFD. Here evaluate the performance of PFD. Notice that the position is a privileged feature as it is not available as an input during online serving (it is the output of the ranking model that becomes position). Further, the click and add-to- add-to-cart are naturally privileged features, since one cannot know which of the product will be clicked or added to cart before showing the products.
The baseline no-distillation model only takes query and product title input, while the teacher models in PFD additionally take positions or clicks or add-to-cart as privileged features. As in public datasets, we only use the positive query groups train the teacher model and use all query groups for distillation. We additionally use pretraining then finetuning as another baseline, as predicting "click" or "add-to-cart" can serve as pretraining tasks. The experiment results are shown in Table 3.
Multi-teacher distillation. Inspired by [FSK* 17, ZXHL18), we also evaluate the multi- teacher distillation, where the student learns from more than one teacher. We adopt three privileged teachers which take positions, clicks, and add-to-cart as input, respectively. We calculate the loss w.n.t. each of the teachers' predictions and use the average as "teacher loss" in Equation (1). Intuitively, the student model is trained to learn from an "ensemble" of teacher models. The multi-teacher PFD yields the best performance, an 1 11.% improvement on testing NDCG @ over the baseline model.
Extension:
7

---

Purchase NDCG
Models
@8
@16
@32
Bascline
No-distillation
1.000
1.000
1.000
Pretrain on click
1.043
1.035
1.031
Pretrain click finetune purchase
1.077
1.060
1.054
Pretrain + finetune
Pretrain on add
1.058
1.046
1.042
Pretrain on add + finetune on purchase
1.059
1.045
1.041
Teacher
1.000
1.000
1.000
Distillation
No privileged feature
Student
1.028
1.021
1.019
Teacher Student
1.182
1.135
1.121
position as privileged feature
1.078
1.057
1.051
Privileged feature distillation
Teacher Student
2.121
1.902
1.827
click as privileged feature add privileged feature
1.082
1.064
1.057
Teacher
2.292
2.053
1.972
Student
1.073
1.056
1.051
Multi-teacher distillation
Distill from position + click + add
1.112
1.085
1.077
Table Evaluation of PFD on Amazon's dataset. We use position (i.e., the position which the product was shown in the logged query group), or click, or add-to-cart as privileged features. The baseline model takes the text of "query" and "product title" inputs. All reported results are scaled w.r.t. the performance of baseline. The results demonstrate that (1) PFD is significantly better than baseline and pretraining-then finettiintnnn (2) using add-to-cart as a privileged feature gives the best teacher model, which, however, does not lead to the best distillation result. This observation agrees with the result from Figure where the most discriminative privileged feature does give the best distillation performance; (3) learning from multiple teachers (i.e., "teacher loss" in Equation (1) being the averaged loss w.r.t. each of the teachers' predictions) further improves the PFD performance and leads to the best ranking performance.
Theoretical Insights
In this section, we present theoretical insights on why and when PFD works via analysis on linear models. While our empirical focus is ranking problems, our theoretical insights more general. Consider the following learning problem: the regular feature x E Rd, drawn from a spherical Gaussian distribution (0, Ia,) and un-observable feature E Rdd is drawn from (N, I.,).
With two unknown parameters w" E ds and v" E Rdd the label is generated as following:
+ 6 (~,(000²).
where represents the label noise. During training time, observe the features
privileged features. Suppose that the labeled training set Stabet = { X E Raaxd Z E Ruxd. , E TR* and the
unlabeled set Suntabe! = (x) E Raxda
[(,) E R mx } are generated according to the afore-
mentioned data generation scheme. Let ((a) X xx,) R(x+m)xd, and Zia) [Z: Z(,)) ((++m))xd xdz be the all the inputs from both labeled and unlabeled datasets. The goal is to learn to predict y with only regular feature input.
5.1 PFD works by reducing estimation variance
Let Wag denote the model learned by standard linear regression and Wpri be the model learned by privileged features distillation. For simplicity, we consider the case with = 0, i.e., only learning from the teacher's prediction during distillation. Specifically, the standard linear regression only uses the set Stabel- and W reg is obtained by regressing y on X. PFD, on the other hand, first uses Stabel to regress y on [X; Z). The learned model is then used to generate predictions ý for Stabel Suntabel- Finally, y is regressed on X()), which gives wr.. We have the following result on the merit of PFD:

---

Theorem 1. For standard linear regression, we have that
( de (a²
( [o2++10
++|lv=
Ex.y w' Wee
For privileged features distillation, we have that
W prrill pr ( dx .02 +
d 1Iv n +
+ O n1
Notice that (yr + ||v 11, where o² corresponds to the label noise and || corresponds to the variance that be explained by the privileged features. The result shows that PFD can explain a proper part of the variance in by privileged features 2. By learning from the teacher's predictions, PFD therefore reduce the variance of Wp by exploiting the privileged features and the unlabeled samples. On the other hand, when learning with plain linear regression, the label variance corresponding
is treated as noise, which leads to estimation with higher variance.
Remark 2. Why GenD has worse-than-baselite performance. Notice that the teacher model in GenD uses privileged features only. GenD has inferior performance for two reasons: (1) the privileged features alone are not enough for the teacher model to generate good predictions; and (2) when z independent of x, the predictions from the GenD's 's teacher are not learnable for the student.
5.2 PFD has inferior performance when the privileged features are too discriminative
To understand the performance of PFD under different privileged features, consider the setting where z E dR is the first dg coordinates of u. When d, = du, it recovers the setting in previous subsection. Notice that the larger d2 becomes, the better (x; z) can predict y. While one might expect that d_ = du (i.e., when the privileged features contain the most information about y) leads to the best distillation performance,
next result shows that such belief is not true in general. Let v be the
part of v* that corresponds to z (i.e., the first d, coordinates of v*), ), we have:
Theorem 2. For privileged features distillation, we have that
dx (02 2 v| 1
d₂ ||
W prril12
+
+0
n - dg dg
n + m d₂
12 m
As increase d_ from to du, I also increases. The teacher, therefore, explains more variance in y and con- tributes to a smaller error in the student estimate Wp
Estimation error of Wer
320
300
However, the denominator of the first term decreases as
de increases, which leads to a higher variance (thus less
280
accurate) student parameter estimate w pri- Combining the two effects, the privileged features z that contain the most information about do not yield the best distillation per- formance. This matches the non-monotone observation in Figure 5; and the results in Table 3 where using add-to-cart (i.e., the most informative feature for predicting purchase) does not give the best PFD result.
260
240
Baseline
PFD
220
100
200 300
400
Variance of y that is explained by
Figure 6: PFD with different of priv- ileged features. The most predictive z 200, and draw w from a spherical Gaussian distribution does not give the best PFD performance. NO N..). Further, the standard linear regression and the privileged features distillation with d, from 0 to 10. The results in Figure 6 shows that the most predictive z does not give the best PFD performance.
Example 1. Consider the data generation as shown in Equation (3). We set dy 10, du 10,n 30, m
sel 15, and let y* [10, 9, ... 2, 1]. We evaluate the performance of
Conclusion
In this paper, we take a step toward understanding PFD in learning-to-rank: We first evaluate PFD on three public ranking datasets (Yahoo, Istella, and MSI.R-Web300 and an industrial-scale ranking problem derived from Amazon's search logs. Our evaluation shows that PFD has the best

---

performance in all evaluated settings. We further conduct comprehensive empirical ablation studies, which demonstrates the efficacy and robustness of PFD and uncovers an interesting non non-monotone behavior as the predictive power of privileged features increase, the performance of PFD first increases and then decreases. Finally, we present theoretical insights for PFD via rigorous analysis for linear models. The theoretical results show that (1) PFD is effective by reducing the variance of student estimation; and (2) a too predictive privileged teacher produces high variance predictions, which lead to high variance (less accurate) student estimates and inferior testing performance.
References
[ALL17] Shuang Ao, Xiang Li, and Charles Ling. Fast generalized distillation for ceni-supervised domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[BSR+05] Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pages 89-96, 2005.
[CC11] Olivier Chapelle and Yi Chang. Yahoo! learning to rank challenge overview. In Proceedings of the learning rank challenge, pages 1-24. PMLR, 2011.
[CJFY17) Yunpeng Chen, Xiaojie Jin, Jiashi Feng, and Shuicheng Yan. Training group orthogonal neural networks with privileged information. arXiv preprint arXiv: 1701.06777 2017.
[CJKB22] Mark Collier, Rodolphe Jenatton, Efi Kokiopoulou, and Jesse Berent. Transfer and marginalize: Explaining away label noise with privileged information. arXiv preprint arXiv:2202.09244 2022.
[CM18] Z Berkay Celik and Patrick McDaniel. Extending detection with privileged information via generalized distillation. In 2018 IEEE Security and Privacy Workshops (SPW). pages 83-88. IEEE, 2018
[DLN+1] 16) Domenico Dato, Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. Fast ranking with additive ensembles of oblivious and non-oblivious regression trees. ACM Transactions on Information Systems (TOIS), 35(2):1-31, 2016.
[FA12] Jan Feyereisl and Uwe Aickelin. Privileged information for data clustering. Information Sciences, 194:4 23, 2012.
[FKSHI4] Jan Feyereisl, Suha Kwak, Jeany Son, and Bohyung Han. Object localization based on structural svm using privileged information. Advances in Neural Information Processing Systems, 27, 2014.
18]
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pages 1607-1616. PMLR, 2018.
[FSK+17] 17] Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jia Cui, and Bhuvana Ramabhadran. Effficient knowledge distillation from an ensemble of teachers. In Interspeech, pages 3697-3701, 2017.
[FTRS13] Shereen Fouad, Peter Tino, Somak Raychaudhury, and Petra Schneider. Incorporating privileged information through metric learning. IEEE transactions on neural networks and learning systems, 24(7):1086-1098 2013.
[GCA+19] Zhifan Gao, Jonathan Chung, Mohamed Abdelrazek, Stephanie Leung, William Kongto Hau, Zhanchao Xian, Heye Zhang, and Shuo Li. Privileged modality distillation for vessel border detection in intracoronary imaging. IEEE transactions on medical imaging,
3945)1152-1554, 2019.
[GCFY18] Chen Gong, Xiaojun Chang, Meng Fang. and Jian Yang. Teaching sumi-supervised classifier via generalized distillation. In IJCAI, pages 2156-2162, 2018
[GMM19] Nuno C Garcia, Pietro Morerio, and Vittorio Murino. Learning with privileged informa- tion via adversarial discriminative modality distillation. IEEE transactions on pattern analysis and machine intelligence, 42(10))2581-255 2019.
10

---

[GYMT21] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge
distillation: A survey, International Journal of Computer Vision, 129(6):179-11811 2021.
[HAS*20] 20) Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan
Hanbury. Improving efficient neural ranking models with cross-architecture knowledge distillation. arXiv preprint arXiv:2000..m 0..02666, 2020.
[HVD+15] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv: 503.02531, 2(7), 2015.
[LDX*20] Xue Li, Bo Du, Chang Xu, Yipeng Zhang, Lefei Zhang, and Dacheng Tao. Robust learning with imperfect privileged information. Artificial Intelligence, 282:103246, 2020.
[LHS14] Maksim Lapin, Matthias Hein, and Bernt Schiele. Learning using privileged information: Svm+ and weighted svm. Neural Networks, 53:95-108, 2014.
[LLKH20] Wonkyung Lee, Junghyup Lee, Dohyung Kim, and Bumsub Ham. Learning with privileged information for efficient image super-resolution. In European Conference on Computer Vision, pages 465-482. Springer, 2020.
[LPSBV16] D. Lopez-Paz, Schölkopf, L. Bottou, and V. Vapnik. Unifying distillation and privileged information. In International Conference on Learning Representutions (ICLR), November 2016.
[MM16] Konstantin Markov and Tomoko Matsui. Robust speech recognition using generalized distillation framework. In Interspeech, pages 2364-2368, 2016.
[PIVV10] Dmitry Pechyony, Rauf Izmailov, Akshay Vashist, and Vladimir Vapnik. Smo-style algorithms for learning using privileged information. In Dmin, pages 241. Citeseer, 2010.
[PPA18] Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. arXiv preprint arXiv: 1802,,8800.000 05668, 2018.
[PV10] Dmitry Pechyony and Vladimir Vapnik. On the theory of learnining with privileged information. Advances in neural information processing systems, 23, 2010.
[QL13] Tao Qin and Yin Liu. Introducing letor 4.0 datasets. ArXiv, abs/ abs/1306.2597, 2013.
[QYT+21] Zhen Qin, Le Yan, Yi Tay, Honglei Zhuang, Xuanhui Wang, Michael Bendersky, and Marc Najork. Born again neural rankers. arXiv preprint arXiv: 15285, 2021.
[QYZ*21] Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Mike Bendersky, and Marc Najork. Are neural rankers still outperformed by gradient boosted decision trees? In International Conference on Learning Representations (ICLR), 2021.
[RPM*21] Sashank Reddi, Rama Kumar Pasumarthi, Aditya Menon, Ankit Singh Rawat, Felix Yu, Seungyeon Kim, Andreas Veit, and Sanjiv Kumar. Rankdistil: Knowledge distillation for ranking. In International Conference on Artificial Intelligence and Statistics, pages 2368-2376. PMLR, 2021.
[SER14] Exploring some practical issues of svm+: Is really privileged information that helps? Pattern Recognition Letters, 42:40-46, 46, 2014.
[SQL13] Viktoriia Sharmanska, Novi Quadrianto, and Christoph H Lampert. Learning to rank using privileged information. In Proceedings of the IEEE international conference on computer vision, pages 825-832, 2013.
[TW18] Jiaxi Tang and Ke Wang. Ranking distillation: Learning compact ranking models with high performance for recommender system. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2299-2298. 2018
[VI15] Vladimir Vapnik and Rauf Izmailov. Learning using privileged information: Similarity control and knowledge transfer. 16(1), 2015.
[VV09] Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged information. Neural Networks, 22(5): 557, 2009. Advances in Neural Networks Research: IJCNN2009.

---

[WZW*21] Shuai Wang, Kun Zhang, Le Wu, Haiping Ma, Richang Hong, and Meng Wang. Priv- ileged graph distillation for cold start recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1187-1196, 2021.
20] Chen Xu, Quan Li, Junfeng Ge. Jinyang Gao, Xiaoyong Yang, Changhua Pei. Fei Sun, Jian Wu, Hanxiao Sun, and Wenwu Ou. Privileged features distillation at taobao recommendations. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2590-2598, 2020.
[Yu20] Hai-Tao Yu. Pt-ranking: A benchmarking platform for neural learning-to- tooninggto-rank.. arXiv preprint arXiv:2008. / 13368, 2020.
[ZWBN20] Honglei Zhuang. Xuanhui Wang, Michael Bendersky, and Marc Najork. Feature transformation for neural ranking models. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1649-1652, 1002,165,2200 2020.
[ZXHL18] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Sections 4 and 5.
(b) Did you describe the limitations of your work? [Yes] Sections 4 and 5 present discussion on when PFD does not work.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] The beginning of Section 5 specifies the setting and assumption.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See supplemen- tal material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A.3.
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [Yes] All results on the public datasets include standard deviation.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section and appendix A.3.
(b) Did you mention the license of the assets? [Yes] See supplemental material.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
12

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "rmTOCB4ZhGf", "reviewer": "Reviewer_GHC5", "summary": "In this work, the authors studied about the Privileged Features Distillation (PFD), where there are indicative features available in training but missing in serving, so as to leverage these privileged features, distillation of a teacher model trained with privileged features is deployed. In specific, in the PFD proposed in the work, the teacher model leverages both privileged features and regular features available in serving. The proposed setting is shown to be better than all baselines on the public and industrial datasets. The authors also provided emprical explaination on why and when PFD works by ablation study and theory on linear models. The main contributions mainly include a practically applicable method PFD and the theoretical understanding of the method.", "questions": "One puzzle to me is in the Appendix Table 4, I find some PFD students perform as good as, sometimes even better than their PFD teachers (on Istella and Web30k). How could the students without the most explainable privileged features outperform the teachers with both regular and privileged features? Because the labelled dataset is much smaller? It would be helpful if the authors could explain this.\n\nHow are the correlation of selected privileged features? In the synthetic data, the authors picked the most correlated features from each public dataset as the privileged features. On the other hand, the authors showed that there is a non-monotonic dependence of student model performance and the correlation of the privileged features. It sounds like selecting the most correlated features are not optimal except they can show the correlations of the most correlated features are arround the range of the optimal performance.", "limitations": "N.A.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper is overall well-written, with all main contributions listed and sufficient ablation study.\n2. First work gives reasonable understanding of why using privileged feature distillation works.\n\nWeaknesses\n1. Figure layouts look a bit wierd. Authors may have used some special template. Usually, a figure will take a full column instead of just a floating panel.\n2. Some notation might be better explained in the main text, for example, RankBCE is short for binary cross entropy loss. It's not very interpretable without checking the appendix.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iVbbAfQFKO-", "reviewer": "Reviewer_XfRZ", "summary": "This paper studies the privileged feature distillation (PFD) problem. The paper consists of two parts - empirical evaluation on public datasets and an industry dataset, followed by some theoretical analysis on linear models. The paper focuses on understanding an existing method instead of proposing new methods. The empirical part confirms that PFD is effective on several datasets. Some ablations are provided in terms of label sparsity, etc. On the three public datasets, the setting is controlled - binary labels are generated and privileged features are manually selected. The evaluation on the industry dataset looks more standard. On the theoretical part, the analysis is done on linear models. The insights found include 1) PFD works by reducing estimation variance. 2) Why too discriminative privileged features can hurt. \n\nOverall the reviewer finds this paper well written in general. The reviewer feels the empirical study meets the bar by performing on multiple datasets and comparing with sensible baselines. The theoretical part looks reasonable but not surprising. \n", "questions": "Please see above.\n\nOne comment is that the reviewer does not find anything specific to learning to rank except for the dataset used. The reviewer does not feel it is a big issue but is a bit surprised. The authors may consider clarifying why learning to rank is the topic, as in the title and such.", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\nThe reviewer has personal interest in the topic (though not sure about the interest from a wider group). The paper is generally well written.\nThe reviewer feels the empirical evaluation meets the bar, by performing on multiple datasets and comparing with sensible baselines. Some ablations look interesting.\nThe theoretical part is clear and focus on important aspects. \n\nWeakness\nThe theoretical analysis is not particularly deep. The conclusions are intuitive and nothing is surprising. Focusing on linear models is ok but may not be very impressive.\nThe controlled setting on public datasets seem a bit artificial and may bias towards the concerned methods. For example, the most correlated features are used as privileged features. Considering other options could be more comprehensive.\nThe paper does not show any online experiments, which is the major motivation of PFD. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bub-PBGjEKx", "reviewer": "Reviewer_76AW", "summary": "The paper empirically evaluates distillation approach for privileged features. Teacher is trained with privileged features that are not available during inference, and student then aims to replicate performance of the teacher without these features. Empirical results on public and proprietary datasets show that this approach achieves better performance than baselines. Authors further analyse theoretical properties of this distillation approach in the case of linear models and show that it has desirable properties.", "questions": "-On line 185 why these specific numbers of features were chosen: 200, 50, and 40? What happens when you use more/less features for each dataset? I would think that this choice has significant effect on performance. \n-Why \"Pretrain on click\" and \"Pretrain on add\" performs better than baseline? These models are not finetuned on purchases, does this mean that clicks and adds are better training targets?\n-Have you tried to use position+click+add in one teacher? Seems odd to only use one of them when a much better teacher can probably be obtained with all three.", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nPaper is well written and easy to follow. Empirical evaluation is quite thorough and I particularly enjoyed the section on the Amazon dataset although these results are not reproducible. Theoretical results provide some insight into properties of PFD and could be used as a stepping stone for more analysis.\n\nWeaknesses\nI find that the paper has limited novelty. Section 4 is all about empirical evaluation and while it has useful insights the novelty limited.  Theoretical analysis in Section 5 is probably the most novel part but it only analyses linear models, and is of limited utility for the complex gradient boosting or deep learning models that are typically used for ranking. Moreover, in most cases it should be possible to use privileged features directly as additional targets. This is cheaper than distillation since it doesn't require training teacher models. Pretrain + finetune results in Table 3 perform similarly to distillation with one teacher, and call into question whether distillation is necessary here at all. I suspect that by carefully tuning weights in a multi-target loss it should be possible to recover multi-teacher performance with just one model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "D3mV5ctQTsS", "reviewer": "Reviewer_iukh", "summary": "The authors provide an empirical study of Privileged Features Distillation (PFD) for Learning-to-Rank problems (LTR), applied on 3 public datasets and one private industrial dataset (Amazon search logs). \nThe principle of PFD is based on two models: 1) one, which learns with all the features available (including the privileged ones) and will play the role of a « teacher » to a 2) second model, the « student » model which is trained using only the regular features and into which teacher information is transferred via distillation. \nPFD is compared against 4 other baselines:\n- no distillation (training only on regular features, no teacher, only one model)\n- pre-training on privileged features followed by fine-tuning with only regular features\n- self-distillation (the teacher model is trained only on non-privileged features)\n- and generalized distillation (GenD, the teacher model is trained only on privileged features)\nExperiments show PFD performs better or as good as the baselines.\nAn ablation study and theoretical analysis focused on linear models finally help to understand when and why PFD works. ", "questions": "Q1: Equation 2: \nFirst comment: By reading the sequel we understand why we need the temperature-based schema to transform human-annotated relevance scores into binary relevance (and not a simple threshold rule): the objective is to have a non-trivial artificial relationship between label and privileged features for the purpose of some demonstration later but this could be clarified before. \n\nHow \\tau target is defined? \n\nQ2: Regarding the sensitivity trend to \\alpha, do we observe the same for other datasets?\n\nQ3: Evaluation on Amazon dataset. How is handled the feature of « product title » in the model? What does the « query » feature look like?\n\nQ4: Table 3: multi-teacher distillation requires 3 teacher models, each of them trained on all regular features +  one privileged feature at a time. What about one single teacher (and one teacher loss) with all the privileged features at the same time? What is the corresponding performance?\n", "limitations": "Yes, limitations are enunciated. This is even the purpose of the paper to determine in which cases PFD performs or does not perform. \nPotential negative societal impact of the work is not mentioned as this is more a generic LTR problem. \n\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nS1 (clarity, quality): The paper is well-written and its clarity makes it easy to follow and enjoyable to read.\n\nS2 (significance): This paper is a first attempt to bridge the gap of lack of performance understanding of PFD.\nThe extensive experiments and theoretical analysis conducted in this paper help to better understand PFD and support some intuitions (ex: PFD cannot do miracles if the most discriminative features for the task at hand are privileged - this shows the superiority of PFD over GenD) and less intuitive results (teacher loss should dominate distillation loss, PFD works better with sparser labels, PFD reduces estimation variance).\nSupplemental work on the use of a teacher model with imputed privileged features during inference is also interesting.\n\n\nWeaknesses:\n\nW1 (clarity, minor): This is minor but as the concept of privileged feature is not limited to learning-to-rank problems, the first sentence of the abstract can be misleading. \n\nTypos:\n-row 54: « PDF »\n-row 178: indicator function mentioned for the first time, always helpful for the reader to name it (and define it)! \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept