Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper empirically evaluates distillation approach for privileged features. Teacher is trained with privileged features that are not available during inference, and student then aims to replicate performance of the teacher without these features. Empirical results on public and proprietary datasets show that this approach achieves better performance than baselines. Authors further analyse theoretical properties of this distillation approach in the case of linear models and show that it has desirable properties.
strengths_and_weaknesses: Strengths
Paper is well written and easy to follow. Empirical evaluation is quite thorough and I particularly enjoyed the section on the Amazon dataset although these results are not reproducible. Theoretical results provide some insight into properties of PFD and could be used as a stepping stone for more analysis.

Weaknesses
I find that the paper has limited novelty. Section 4 is all about empirical evaluation and while it has useful insights the novelty limited.  Theoretical analysis in Section 5 is probably the most novel part but it only analyses linear models, and is of limited utility for the complex gradient boosting or deep learning models that are typically used for ranking. Moreover, in most cases it should be possible to use privileged features directly as additional targets. This is cheaper than distillation since it doesn't require training teacher models. Pretrain + finetune results in Table 3 perform similarly to distillation with one teacher, and call into question whether distillation is necessary here at all. I suspect that by carefully tuning weights in a multi-target loss it should be possible to recover multi-teacher performance with just one model.
questions: -On line 185 why these specific numbers of features were chosen: 200, 50, and 40? What happens when you use more/less features for each dataset? I would think that this choice has significant effect on performance. 
-Why "Pretrain on click" and "Pretrain on add" performs better than baseline? These models are not finetuned on purchases, does this mean that clicks and adds are better training targets?
-Have you tried to use position+click+add in one teacher? Seems odd to only use one of them when a much better teacher can probably be obtained with all three.
limitations: NA
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this work, the authors studied about the Privileged Features Distillation (PFD), where there are indicative features available in training but missing in serving, so as to leverage these privileged features, distillation of a teacher model trained with privileged features is deployed. In specific, in the PFD proposed in the work, the teacher model leverages both privileged features and regular features available in serving. The proposed setting is shown to be better than all baselines on the public and industrial datasets. The authors also provided emprical explaination on why and when PFD works by ablation study and theory on linear models. The main contributions mainly include a practically applicable method PFD and the theoretical understanding of the method.
strengths_and_weaknesses: Strengths
1. The paper is overall well-written, with all main contributions listed and sufficient ablation study.
2. First work gives reasonable understanding of why using privileged feature distillation works.

Weaknesses
1. Figure layouts look a bit wierd. Authors may have used some special template. Usually, a figure will take a full column instead of just a floating panel.
2. Some notation might be better explained in the main text, for example, RankBCE is short for binary cross entropy loss. It's not very interpretable without checking the appendix.
questions: One puzzle to me is in the Appendix Table 4, I find some PFD students perform as good as, sometimes even better than their PFD teachers (on Istella and Web30k). How could the students without the most explainable privileged features outperform the teachers with both regular and privileged features? Because the labelled dataset is much smaller? It would be helpful if the authors could explain this.

How are the correlation of selected privileged features? In the synthetic data, the authors picked the most correlated features from each public dataset as the privileged features. On the other hand, the authors showed that there is a non-monotonic dependence of student model performance and the correlation of the privileged features. It sounds like selecting the most correlated features are not optimal except they can show the correlations of the most correlated features are arround the range of the optimal performance.
limitations: N.A.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors provide an empirical study of Privileged Features Distillation (PFD) for Learning-to-Rank problems (LTR), applied on 3 public datasets and one private industrial dataset (Amazon search logs). 
The principle of PFD is based on two models: 1) one, which learns with all the features available (including the privileged ones) and will play the role of a « teacher » to a 2) second model, the « student » model which is trained using only the regular features and into which teacher information is transferred via distillation. 
PFD is compared against 4 other baselines:
- no distillation (training only on regular features, no teacher, only one model)
- pre-training on privileged features followed by fine-tuning with only regular features
- self-distillation (the teacher model is trained only on non-privileged features)
- and generalized distillation (GenD, the teacher model is trained only on privileged features)
Experiments show PFD performs better or as good as the baselines.
An ablation study and theoretical analysis focused on linear models finally help to understand when and why PFD works. 
strengths_and_weaknesses: Strengths:

S1 (clarity, quality): The paper is well-written and its clarity makes it easy to follow and enjoyable to read.

S2 (significance): This paper is a first attempt to bridge the gap of lack of performance understanding of PFD.
The extensive experiments and theoretical analysis conducted in this paper help to better understand PFD and support some intuitions (ex: PFD cannot do miracles if the most discriminative features for the task at hand are privileged - this shows the superiority of PFD over GenD) and less intuitive results (teacher loss should dominate distillation loss, PFD works better with sparser labels, PFD reduces estimation variance).
Supplemental work on the use of a teacher model with imputed privileged features during inference is also interesting.


Weaknesses:

W1 (clarity, minor): This is minor but as the concept of privileged feature is not limited to learning-to-rank problems, the first sentence of the abstract can be misleading. 

Typos:
-row 54: « PDF »
-row 178: indicator function mentioned for the first time, always helpful for the reader to name it (and define it)! 

questions: Q1: Equation 2: 
First comment: By reading the sequel we understand why we need the temperature-based schema to transform human-annotated relevance scores into binary relevance (and not a simple threshold rule): the objective is to have a non-trivial artificial relationship between label and privileged features for the purpose of some demonstration later but this could be clarified before. 

How \tau target is defined? 

Q2: Regarding the sensitivity trend to \alpha, do we observe the same for other datasets?

Q3: Evaluation on Amazon dataset. How is handled the feature of « product title » in the model? What does the « query » feature look like?

Q4: Table 3: multi-teacher distillation requires 3 teacher models, each of them trained on all regular features +  one privileged feature at a time. What about one single teacher (and one teacher loss) with all the privileged features at the same time? What is the corresponding performance?

limitations: Yes, limitations are enunciated. This is even the purpose of the paper to determine in which cases PFD performs or does not perform. 
Potential negative societal impact of the work is not mentioned as this is more a generic LTR problem.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the privileged feature distillation (PFD) problem. The paper consists of two parts - empirical evaluation on public datasets and an industry dataset, followed by some theoretical analysis on linear models. The paper focuses on understanding an existing method instead of proposing new methods. The empirical part confirms that PFD is effective on several datasets. Some ablations are provided in terms of label sparsity, etc. On the three public datasets, the setting is controlled - binary labels are generated and privileged features are manually selected. The evaluation on the industry dataset looks more standard. On the theoretical part, the analysis is done on linear models. The insights found include 1) PFD works by reducing estimation variance. 2) Why too discriminative privileged features can hurt. 

Overall the reviewer finds this paper well written in general. The reviewer feels the empirical study meets the bar by performing on multiple datasets and comparing with sensible baselines. The theoretical part looks reasonable but not surprising. 

strengths_and_weaknesses: Strength
The reviewer has personal interest in the topic (though not sure about the interest from a wider group). The paper is generally well written.
The reviewer feels the empirical evaluation meets the bar, by performing on multiple datasets and comparing with sensible baselines. Some ablations look interesting.
The theoretical part is clear and focus on important aspects. 

Weakness
The theoretical analysis is not particularly deep. The conclusions are intuitive and nothing is surprising. Focusing on linear models is ok but may not be very impressive.
The controlled setting on public datasets seem a bit artificial and may bias towards the concerned methods. For example, the most correlated features are used as privileged features. Considering other options could be more comprehensive.
The paper does not show any online experiments, which is the major motivation of PFD. 


questions: Please see above.

One comment is that the reviewer does not find anything specific to learning to rank except for the dataset used. The reviewer does not feel it is a big issue but is a bit surprised. The authors may consider clarifying why learning to rank is the topic, as in the title and such.
limitations: NA
review_title: null

Meta-review: metareview: There is a consensus that the insights on the distillation of privileged information presented in the paper are interesting (e.g., possibility of distillation even if the privileged information is independent from x, non-monotonicity of the impact of privileged information vs correlation with the target feature), which is why the paper is recommended for acceptance.

Note that even after the rebuttal, several of the main weaknesses remain, 

- it is not clear why the paper focuses "learning to rank" (apart from the original motivation of the authors), since the claims seem to hold as well in classification or regression
- the value of the theoretical analysis is limited, because it seems the authors considered the easiest setup where the phenomena illustrated in the experiment could be proved. In particular, they study linear least-square regression, which doesn't match any of their experiments.
- no novelty in terms of methods

In the end, the paper is borderline on the side of acceptance because the insights are significant enough.
confidence: Less certain
Decision: Accept