Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Knowledge distillation (KD) is a popular technique to trade-off between performance and efficiency. Student models trained with KD usually work better than those trained from the original labels. However, applying KD to ranking problems is not well studied. One earlier work (Tang & Wang, 2018) on ranking distillation proposes pointwise logistic loss for distillation after unsuccessfully attempting pair-wise approaches on the teacher's top K ranking. The authors take on the challenge of dealing with the following characteristics of the ranking models:
* The ranking scores are neither calibrated nor normalized. They do not directly convey information about the underlying distribution.
* They care about the relative order of score and do not make an iid assumption over documents.
* Large teacher models tend to overfit because it is difficult to apply techniques like data augmentation & small datasets.

The authors extend BAN in which student models are configured with the same capacity as teachers and propose BAR. BAR is referred to the specific setting where ranking loss with original labels, listwise distillation on teacher score, and tunable affine teacher score transformation. Unlike typiical KD models, the primary goal of the paper is to improve ranking performance over state-of-the-art teacher rankers.
main_review: **Strengths**

* The paper clearly motivates the need for distillation specific to LTR problems, explained the need for a score transformation layer and recommends one way (BAR) to achieve good transfer learning on 3 public datasets.
* The primary contribution, i.e. transforming teacher scores using a ReLU + Affine layer and using listwise loss for the student model are simple, easy to understand and replicate. The effect of hyperparameters could have been studied better. 

**Weaknesses**

* The paper is very light on experimentation. e.g. It would have been good to see the comparison of listwise vs pointwise teachers on more than one dataset - ideally with 3-5 neural architectures that have been used in the literature. What was the effect of the scaling factor in the Istella dataset?  How sensitive is $\alpha$ across datasets? Does early stopping play a role in the generalization performance? Does $L_2$ affect the regularization ability of BAR?
* Lack of theoretical justification of the BAR choices. The authors claim "theoretical explanations" as a contribution of the paper. Yet, question like why ReLU, why affine transformation, how to fit parameters of the transformation on new datasets are not justified by theory. The section on Understanding BAR merely provides an intuition and one hypothesis explaining the trend.

summary_of_the_review: The authors empirically show improvements on 3 public LTR dataset using the BAR method over a neural baseline. To clearly support their research questions, they should continue experimenting with a few more baseline neural models to understand the effectiveness of BAR. Additionally, the ablations and design choices for transformation & loss functions should be done on more than one dataset to be able to conclusively make the claims made in the paper. The biggest gap is however, the lack of theoretical justification on why the specific choices in BAR are needed. The paper can just have empirical evidence to show one transformation that works well across datasets. However claiming that BAR is a novel listwise distillation framework with theoretical justification is far-fetched.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a distillation approach for the learning to rank setting. Specifically, the "born again neural ranker" approach is investigated where both teacher and student models are identically parameterised neural networks. Authors propose a listwise loss to incorporate scores from the teacher model during student optimization and show that this leads to improvements on three real world datasets.
main_review: Strengths: 
The paper investigates an interesting and under explored area of model distillation and learning to rank, and is well written and easy to follow. Experiments are done on large real world datasets and compare against leading baselines from both gradient boosting and deep learning domains.

Weaknesses:
My major concern is with the novelty. The proposed approach is a relatively straightforward application of distillation with off-the-shelf loss and optimization. I also have a concerns about section 4.3, ReLU is an odd transformation to use in the learning to rank setting as it squashes all negative scores to 0 making that part of the ranking indistinguishable. This can significantly affect teachers accuracy and also introduces many ties into the students loss. The justification in 4.3 does not support such a transformation, and in section 5.1 authors actually use identity for Web30K and Yahoo datasets, and divide scores by 100 for Istella. These choices again seem hand picked and non-principled, and further justification is needed here since authors claim that normalising teachers scores is important for distillation to work. 
summary_of_the_review: The paper explores an important area in learning to rank but the proposed approach lacks novelty and makes multiple ad hoc choices that need further justification.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces Knowledge Distillation (KD) to Learning to rank (LTR), adapting the idea of Born Again Networks (BAN). The propose model, Born Again neural Rankers (BAR), combines an appropriate teacher score transformation and a novel listwise distillation framework. The authors conduct thorough experiment studies to demonstrate the superior performance of the BAR, and also provide explanations in terms of why it works.
main_review: The paper introduces Born Again neural Rankers (BAR) for Learning to rank (LTR). Unlike traditional Knowledge Distillation (KD) which usually seeks a trade-off before performance and efficiency, BAR targets for better performance without increasing the model capacity. 

In particular: 

1. The authors clearly state two important design choices: 1) a listwise Softmax loss function and 2) a teacher score transformation. In additional, they also explore other alternatives and empirically show why the current choice is warranted.

2. Extensive experiment studies are conducted to demonstrate the effectiveness of the proposed method.

3. The authors also propose an explanation to better understand the superior performance of BAR.

There are several places where I am not certain, or maybe can make the paper better:

1. In the experiment there are several ensembles considered, including DASALC-ens and BAR-ens. For a more fair comparison, I wonder whether the authors can also compare with ensembles of \lambdaMART, since it achieves two best performances across 9 settings. 

2. In (4) there is a hyperparameter \alpha, and in (7) there is a and b for the affine transformation. The experiment results shown in Table 1 only uses fixed values for these hyperparameters. It might be worth to see if we tune more on these three hyperparameters, how much more gain the model can achieve. 

3. Is such complexity necessary or is more complexity helpful? I do not expect the authors to address this point in this paper, but it would be helpful to share some thoughts if they have. Currently, we have the student model the same complexity as the teacher. However, is such complexity really necessary? Can we have a less complexity student while also achieve superior performance? From another perspective, if we have a more complexity student, with the help of knowledge distillation, is it able to perform even better? 
summary_of_the_review: I believe this paper is well written with clear statement of their research questions, extensive experiments, and rigorous arguments. I suggest to accept this paper to ICLR.
review_title: null

Meta-review: The authors are strongly encouraged to elaborate further about the novelty of their method, as well as to give detailed (either theoretical or experimental) justifications for the design choices they make within the paper. Finally, the paper could benefit from additional experiments, as outlined in the reviews.
Decision: Reject