Born Again Neural Rankers
Authors: Zhen Qin
Le Yan
Yi Tay
Honglei Zhuang
Xuanhui Wang
Michael Bendersky
Marc Najork
Venue: ICLR 2022 Submitted (2022)
Keywords: learning to rank
knowledge distillation
neural networks

Abstract: We introduce Born Again neural Rankers (BAR) in the Learning to Rank (LTR) setting, where student rankers, trained in the Knowledge Distillation (KD) framework, are parameterized identically to their teachers. Unlike the existing ranking distillation work, which pursues a good trade-off between performance and efficiency, BAR adapts the idea of Born Again Networks (BAN) to ranking problems and significantly improves ranking performance of students over the teacher rankers without increasing model capacity. By examining the key differences between ranking distillation and common distillation for classification problems, we find that the key success factors of BAR lie in (1) an appropriate teacher score transformation function, and (2) a novel listwise distillation framework, both are specifically designed for ranking problems and are rarely studied in the knowledge distillation literature. Using the state-of-the-art neural ranking structures, BAR is able to push the limits of neural rankers above a recent rigorous benchmark study, and significantly outperforms strong gradient boosted decision tree based models on 7 out of 9 key metrics, the first time in the literature. In addition to the strong empirical results, we give theoretical explanations on why listwise distillation is effective for neural rankers. 

Under review as a conference paper at ICLR 2022
BORN AGAIN NEURAL RANKERS
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce Born Again neural Rankers (BAR) in the Learning to Rank (LTR) set- ting, where student rankers, trained in the Knowledge Distillation (KD) framework, are parameterized identically to their teachers. Unlike the existing ranking distil- lation work which pursues good trade-off between performance and efficiency, BAR adapts the idea of Born Again Networks (BAN) to ranking problems and significantly improves ranking performance of students over the teacher rankers without increasing model capacity. The key differences between BAR and com- mon distillation techniques for classification transformation function, and (2) a novel listwise distillation framework. Both tech- niques are specifically designed for ranking problems and are rarely studied in the knowledge distillation literature. Using the state-of of-the-ar art neural ranking structure, BAR is able to push the limits of neural rankers above recent rigorous benchmark study and significantly outperforms traditionally strong gradient boosted decision tree based models on 7 out of 9 key metrics, the first time in the literature. addition to the strong empirical results, we give theoretical explanations on why listwise distillation is effective for neural rankers.
(1) an appropriate teacher score
INTRODUCTION
Learning to rank (LTR) has become an essential component for many real-world applications such as search (Liu 2009). In ranking problems, the focus is on predicting the relative order of list of items given query. It is thus different from classification problems whose goal is to predict the class label of a single item. While many neural machine learning techniques have been proposed recently, they mostly for classification problems. Given the difference between ranking and classification problems, it is interesting to study how neural techniques can be used for ranking problems.
Knowledge distillation (KD) (Hinton et al. 2015 Gou et al. 2020) is one of such recently popular techniques. The initial goal of KD is to pursue a good trade-of off between performance and efficiency. Given high-capacity teacher model with desired high performance,
compact student model is trained using teacher labels (Heo et al. 2019 [Sun et al.. 2019 [aanh et al. 2019). Student models trained with KD usually work better than those trained from original labels without teacher guidance. How to effectively apply KD to ranking problems is not straightforward and has not been well-studied yet, for the following reasons:
First, teacher models in classification typically predict a probability distribution over all classes. Such "dark knowledge" is believed be key reason why KD works (Hinton et al., 2015). However, a teacher model in ranking does not convey such distribution over pre-defined classes, as ranking models only care about the relative orders of items and simply output a single score for each item in a possibly unbounded candidate list. Ranking scores are typically neither calibrated (comparing to the probabilistic interpretation of classification scores (Guo et al. (017)) nor normalized (comparing to summing up to one for classification scores over all classes for most popular losses (Hinton et al.) 2015)). Thus directly using the outputs of a teacher model as labels for ranking tasks with existing KD methods may be less optimal.
Second, listwise information over all input items should be considered to achieve the best ranking performance, since the goal is to infer the relative orders among them. For example, listwise losses have been shown to be more effective than other alternatives for LTR problems (Cao et al., 2007). On the other hand, classification tasks almost universally treat each item independently based on the

---

Under review as a conference paper at ICLR 2022
i..d. assumption. It is interesting to also consider listwise frameworks when studying KD for ranking problems, which has not been explored in the literature, to the best of our knowledge.
Third, though there is consensus in the classification KD literature that given severe over-fitting larger teacher models usually work better (He et al.] 2016), recent works show that for traditional LTR problems, it is hard for larger models to achieve better performance (Bruch et al. 2019 Qin et al. 2021) due to the difficulty of applying standard techniques such as data augmentation (compared to image rotation in computer vision (Xie al.) 2020)) and the lack of very large human-labeled ranking datasets. This makes it harder to use the standard KD setting to improve the performance of ranking models by distilling from a very large teacher model.
Thus, KD techniques need special adjustments for LTR problems. In this paper, inspired by the Born Again Networks (BAN) (Furlanello et al., 2018) in which student models are configured with the same capacity as teachers and are shown to outperform teachers for classification problems, we study the BAN techniques for ranking problems. To this end, we propose Born Again neural Rankers (BAR) that train the student models using listwise distillation and properly transformed teacher scores, which can achieve new state-of-the-an ranking performance for neural rankers. While existing ranking distillation works such as (Tang & Wang 2018 Reddi et al.) 2021) require a more powerful teacher model and focus on cerformance-efficiene trade-ofis, the primary goal of our paper is to improve ranking performance over state-of of-the--ttht art teacher rankers.
In summary, our contributions are as follows:
We propose Born Again neural Rankers (BAR) for learning to rank. This is the first knowledge distillation work that targets for better ranking performance without increasing the model capacity.
We show that the key success factors of BAR are (1) an appropriate teacher score transfor- mation function, and (2) ranking specific listwise distillation loss. Both design choices are tailored for LTR problems and are rarely studied the knowledge distillation literature.
We provide new theoretical explanations on why BAR works better than other alternatives.
This contributes to both the general knowledge distillation and ranking distillation research.
We verify our hypothesis on rigorous public LTR benchmarks and show that BAR is able to significantly improve upon ofate-of-the-arr neural teacher rankers.
2 RELATED WORK
Knowledge distillation has been popular research topic recently in several areas such as image recognition (Hinton et al. [013 Romero et al. 2014 Park et al. 2019), natural language understand- ing (Sanh et al. [019 Jiao al. 2019 Aguilar et al. 2020), and neural machine translation (Kim & Rush] [2066] Chen et al. 201 lan et al. 2019) as a way to generate compact models, achieving good trade-offs. As we mentioned in the Introduction, both the classical setting (e.g., using pointwise losses on i.i.d. data) and theoretical analysis (e.g.. "dark knowledge" among classes) for classification tasks may not be optimal for the ranking setting.
performance-efficiene
The major goal of this work is to push the limits of neural rankers on rigorous benchmarks. Since the introduction of RankNet (Burges et al. (005) over a decade ago. only recently, neural rankers were shown to be competitive with well-tuned Gradient Boosted Decision Trees (GBDT) on traditional LTR datasets (Qin et al., 202T). We build upon (Qin et al. 202T) and show that BAR can further push the state-of-ther ofrt art of neural rankers on rigorous benchmarks. We also provide extra experiments to show listwise distillation helps neural ranking in other settings.
We are motivated by Born Again Networks (BAN) that were introduced by (Furlanello et al. 201 8). BAN is the first work that shows better performance can achieved by parameterizing the student model the same the teacher. However, BAN only focuses on classification and direct application of
BAN does not help LTR problems. Building upon the general "born again" ideas (Zhang et al. 2019)]
Clark al. 2019), contribution in this paper is in developing specific new techniques and theory that make these ideas applicable for the important LTR setting.
Another closely related work is Ranking Distillation (RD) (Tang & Wang 2018), since it studies knowledge distillation for ranking. There are several marked differences between our work and RD. First, RD focuses on performance-efficinnyy trade-offs, where the student usually underperforms the
2

---

Under review as a conference paper at ICLR 2022
teacher in terms of ranking metries (Reddi et al. 2021). while focus on outperforming the teacher. Second, the work only uses pointwise logistic loss for distillation. The authors state that "We also tried use pair-wise distillation loss when learning from teacher's 's top-K ranking. However, the results were disappointing". without going into more detail or explore listwise approaches, which we show are the key success factor. Also, Tang and Wang (Tang & Wang (018) use hyperparameter K: items ranked as top K are labeled as positive and others as negative for distillation. Besides only working for binary datasets, this setting is not very practical since real-world ranking lists may have very different list sizes or number of relevant items. Our method does not require such a parameter. Furthermore, they only evaluated their methods on some recommender system datasets where only user id, item id, and binary labels are available. It is not typical LTR setting, so the effectiveness of RD over state-of-ther of-the-art neural rankers unclear.
BACKGROUND ON LEARNING TO RANK
For LTR problems, the training data can be represented as a set V/ ((x,)) E x" R") where is a list of n items E X and is a list of n relevance labels y, E R for < S n. We use X as the universe of all items. traditional LTR problems, each 2; corresponds to query-item pair and is represented as feature vector in R$ where & is the number of feature dimensions. With slight abuse of notation, we also use T; as the feature vector and say x E R The objective learn function that produces an ordering of items in x so that the utility of the ordered list is maximized, that is, the items are ordered by decreasing relevance.
Most LTR algorithms formulate the problem as learning a ranking function to score and sort the items in a list. As such, the goal of LTR boils down to finding a parameterized ranking function f(; 0) x" + R", where 9 denotes the set of trainable parameters, to minimize the empirical loss:
C(((=))
Σ (x.))
fx:
where ((-) is the loss function on a single list.
There are many existing ranking metrics such as NDCG and MAP used in LTR problems. common property of these metrics is that they are rank-dependent and place more emphasis on the top ranked items. For example, the widely used NDCG metric is defined as
DCG((T,,)) NDCG(T,,)) = DCG((T",))
(2)
where πy is a ranked list induced by the ranking function f(::8) on x² is the ideal list (where is sorted by decreasing y), and DCG is defined as:
2
DCG( Σ
(3)
logz(1 + π (i))
In practice, the truncated version that only considers the top-k ranked items, denoted as NDCG@k, is often used.
4 BORN AGAIN NEURAL RANKERS
4.1 THE GENERAL FORMULATION OF BAR
In addition to the original training data {(x, y) E x" x R")), we assume there is a well-trained teacher model (:; )²). The goal of BAR is to train student model [(:; where 0 is parameterized the same as 8r, with the following loss:
((f(:0))
1
Σ (
(1()) (4)
where a E [0, 1] is a weighting factor between the two losses, Ip is the additional Born Again loss, and g() is transformation function for teacher scores. The practical necessity of this transform
3

---

Under review as a conference paper at ICLR 2022
Train Teacher
Train Student
Ranking
Ranking labels
Teacher scores
Figure 1: An illustration of the BAR framework. Black arrows show the feed-forward paths and red arrows direct the backward gradients. The teacher ranker can be trained with any ranking loss functions. The student model uses a multi-objective architec- ture that adds a listwise Softmax loss over teacher scores, addition to the original ranking loss.
Ranking
Softmax
Ranking Loss
Teacher Ranker
Teacher Ranker
Student Ranker
Input List
Input List
function is further analyzed in Section 4.3 The value of a can be tuned and we found that wide range of a works well in our experiments.
An illustration of the BAR framework is shown in Fig. T The BAR framework uses multi-objective setting for student score fxx ). In addition to the original loss function, BAR adds an additional distillation loss between student and (transformed) teacher scores.
4.2 LOSS FUNCTIONS
Both the original loss function ((:,)) and the born again loss (B(+, can be any ranking loss, ranging from pointwise to pairwise and listwise losses (Liu 2009). Generally, listwise and pairwise losses are more effective than pointwise ones (Cao al. 2007). For example, the teacher models we leverage from (Qin et al. 202T) use the listwise Softmax cross entropy loss. Specifically, given the labels and scores the Softmax cross entropy loss is defined over list of items:
[Softmax (y,s) = Σ yo In exp(8:)
(5)
Σf=1 exp(s;)
where is the number of items in the ranking list.
In experiments we show that the original loss function can be more flexible (e.g., can be a pointwise listwise loss), but the born again loss is only effective when it is the listwise one.
4.3 TEACHER RANKER SCORE TRANSFORMATION
We formalize the necessity of properly transforming the scores from the teacher ranker. First, many popular ranking losses are translation invariant. For example, in the Softmax cross entropy loss, scores appear in the paired form of 8; 8j-
Softmax (y,s) = Σ Mi In exp(si)
n yi
Σ
(6)
Dj=1 exp(s;)
2 exp(s; - si)
is easy to see that the loss stays invariant after adding a constant to scores. This also the case for other popular ranking losses such ApproxNDCG loss (Bruch et al.) 2019) and the pairwise RankNet loss (Burges) 2010).
Second, since ranking scores not normalized (compared to summing up one for classification over all classes), teacher ranker's insensitivity to score scales may lead to numerical issues for distillation. For example, two documents
be ranked correctly with very close scores (e.g., 1.001 and 1.0) where the student may not differentiate them effectively, even we subtract constant from them. Thus, the scale of the teacher scores also need attention.
4

---

Under review as a conference paper at ICLR 2022
Third, unlike classification problems where predictions are non-negative probabilities, teacher ranker scores can be negative, which will make many ranking losses ill- ill-behaved, such as any cross entropy based losses.
Putting these together, we propose a simple teacher score transformation function which parameterizes g(-) affine transformation function of teacher scores:
(( 0')) ReLU(a f(x; 0*) b).
ReLU(()= max(a, 0) is safeguard to make sure the transformed teacher scores are non-negane non-negative. The slope a(a >> 0) and the intercept b of the affine transformation are treated as hyperparameters. Note that we aim for a general formulation highlighting potential caveats of teacher ranking scores due to the difference from classification problem. The actual tuning depends on teacher ranker behavior and may be simple in practice. For example, when a 1 and b = 0, g(-) becomes the standard ReLU function, which turns out to be effective for two out of three datasets used in experiments.
4.4 ALTERNATIVES
We discuss possible alternatives that we will compare with in experiments. These alternatives used to help highlight that BAR is referred to the specific setting where ranking loss with original labels, listwise distillation on teacher score, and tunable affine teacher score transformation are used.
Our comparison is to help better understand what makes BAR effective in practice.
Pointwise distillation. Instead of the listwise loss, we can perform a pointwise loss for the distillation objective 1B( (-, in 4
= Σ ((o(f(a;;0') f(z; 0*))
ln
(8)
Note that pointwise loss is decomposed across items independently, following the i.idd assumption for regression. Throughout the paper, we use the mean squared error loss as the pointwise loss:
Σ (mx (f)) 0x))1
since the original labels are graded with real values, and as we mentioned Section 4.3 teacher scores be unbounded, for binary labeled datasets.
the range of
Teacher score only distillation. Another way to distill teacher score to ignore the original loss function using original labels (a 1 in Equation
L(f(:0"))
Sofnaax
(10)
(x. Y)W
The original BAN paper (Furlanello et al. 2018) shows that this single objective formulation works better than the two objective formulation in certain classification scenarios.
Softmax teacher score transformation. It is tempting apply the Softmax transformation teacher scores each list, since it converts all labels to be non-negative while preserving their relative order. We study Softmax teacher transform with temperatures, Σ ex=_ in Section and empirically show that it under-perfome affine transformations.
exp(f(x,;8'/1))
5
EXPERIMENTS
We conduct experiments on three public LTR datasets to show that BAR achieves state-of-the-an performance for neural rankers. We also perform various ablation studies to better understand BAR.
5.1
EXPERIMENTAL SETUP
We follow the experimental setup (Qin et al. 2021 that provides a rigorous LTR benchmark. Three widely adopted data sets for web search ranking used: Web30K (Qin & Liu 2013), Yahoo
5

---

Under review as a conference paper at ICLR 2022
(Chapelle & Chang 201 and Istella (Dato et al. 2016). The documents for each query labeled with multilevel graded relevance judgments by human raters. We compare with a comprehensive list of benchmark methods.
AMART (Ke et al., 2017) and AMART RankLib are the two GBDT-based implementations for LTR. RM (Joachims 2006) is classic pairwise learning-to-raan model built on SVM. GSF (Ai et al. 2019) is a neural model using groupwise scoring function and fully connected layers. ApproxNDCG
(Bruch et al.) 2019) is a neural model with fully connected layers and a differentiable loss that approximates NDCG (Qin el al. 2010). DLCM (Ai et al., 2018) is an RNN based neural model that uses list context information to rerank a list of documents based on XMART
as the original paper. SetRank (Pang et al. (2200) is neural model using self- self-attention to encode the entire list and perform a joint scoring. SetRank re (Pag et al. 2020) is SetRank plus ordinal embeddings based on the initial document ranking generated by XMART Ramklib as in the original paper. DASALC (Qin et al. 202T) is the ofate-of-the-art neural rankers combining data transformation and augmentation, effective feature crosses, and listwise context. DASALC-ens is ensemble of DASALC models, significantly outperforming the strong AMART baseline on 4 out of 9 major metrics.
For the main results of BAR, we obtain the teacher model scores and configurations of DASALC (not DASALC-ens) for each dataset from the authors. For the Web30K and Yahoo dataset. the teacher scores are used as they are (i.c., As the teacher scores for the Istella dataset are of larger range (mean = 28.4, std = 299..) and are causing some numeric issues, dampen them by using ((f ))) ReLU 1 100 f(x: 0*)). Better results may be achieved by more tuning. The model architecture configurations, such as the number of hidden layers, are unchanged during BAR training. We simply use a 0.5 assign equal weights between the original objective and distillation objective, and show its robustness in Section 5.5 We do perform syyper-parameee searches over dropout rate, learning rate, and data augmentation noise level on validation sets. Note that all BAR models are only born again once (instead of iteratively born again multiple times by treating previous student model as the new teacher), due to computation overhead and the lack of theoretical guarantee in the original BAN work. BAR is a single neural ranker. B-R is an ensemble of h rankers (3 S k S 5) that is tuned on each dataset with the same architecture from different runs. Note that the candidates in the ensemble for each data set are supervised by the same teacher.
the identity function the affine transformation in g(-)).
5.2 RESEARCH QUESTIONS
We want to answer the following research questions by comparing with competitive methods on LTR benchmarks and performing ablation studies:
RQI: Is the BAR framework able to further push the limits of neural rankers over the state-of-the-an models?
RQ2: Is the listwise distillation loss necessary for BAR to be effective for LTR problems, compared to the common pointwise distillation loss that follows the i.i.d. assumption?
RQ3: Is the dual-objective architecture necessary for BAR, considering that (Furlanello
et al. 2018) shows that using only the teacher objective works better in certain scenarios? How robust is the balaneing parameter between the two objectives?
RQ4: Is the affine teacher score transformation more effective than the Softmax transforma- tion?
5.3 MAIN RESULTS (RQ1)
Our main results are shown in Table L From this table, we can see that the models trained under the BAR framework can significantly push the limits over the tfate-of-the-art neural rankers without sacrificing efficiency. A single BAR model performs best among non- GBDT methods 8 out of 9 metrics. BAR-ens universally outperforms DASALC-ens, and can significantly outperform XMART GDM on 7 out of 9 metrics.
5.4 THE NECESSITY OF LISTWISE DISTILLATION (RQ2)
We compare with pointwise distillation in Table 2 on the Web30K dataset. We use the MSE loss and tune in Eq. 4 for the datasets with graded relevance labels. Results on other datasets are consistent.

---

Under review as a conference paper at ICLR 2022
Table 1: Result on the Web30K, Yahoo, and Istella datasets. means significantly better result, performed against AMART GaM the p < 0.05 level using a two-tailed f-test. Bold numbers are the highest in each column, italicized numbers are the highest in each column for single neural rankers.
Web30K NDCG@k
Yahoo NDCG@k
Istella NDCG@K
Models
691
@10
AMARTROAL
45.35
44.59
46.46
68.52
70.27
74.58
65.71
61.18
65,91
AMARTeaw
50.73
49.66
51.48
71.88
74.21
78.02
74,92
71.24
76.07
RankSVM
30.10
33.50
36.50
63.70
67.40
72.60
52,69
50.41
55.29
41.29
43.74
64.29
73.16
62.24
65.08
ApproxNDOG
46.64
45.38
47.31
69.63
72.32
76.77
65.81
62.32
67.09
DLCM
46.30
45.00
46,90
67.70
69.90
74.30
65.58
61.94
66.80
SetRank
42,90
42.20
44,28
67.11
69.60
73.98
67.33
62.78
67.37
SetRank
45.91
45.15
46.96
70,29
74.53
67.60
63.45
68.34
DASALC
50.95
50.921
52.88
70.98
73.76
77.66
72.77
70.06
75.30
BAR
51,71"
51,567
53,571
70.83
73.89
77.85
72.97
70.24
75.47
(á over DASALC)
(+149%)
(+1.26%)
(+1.30%)
(-0.21%%
(+0.18%%)
(+0.24%)
(+0.27%)
(+0,26%)
(+0.23%)
DASALC-ens
51,897
51,727
53.731
71.24
74.07
77.97
74.40
71.32
76.44"
BAR-ens
52.22'
52.121
54,097
71.45
74.61
78.31
74.51
71.511
76.56
( DASALC-ens)
(+0.64%)
(+0.77%)
(+0.67%)
(+0.29%)
(+0.73%)
(+0.44%)
(+0.15%)
(+0.27%)
(+0.16%)
Method
NDCG@k
Teacher
Student
@
@5
@ 10
Table 2: Results on the Web30K dataset comparing listwise vs pointwise teachers and distillation. "f" means directly using the teacher model (DASALC (Qin et al. 2021 without distillation.
50.67
50.70
52.85
Pointwise
Pointwise Listwise
50.33
50.47
52.60
50.88
51.16
53.26
50.95
50.92
52.88
Listwise
Pointwise
49.78
49.89
52.02
Listwise
51.71
51.56
53.57
In this table, listwise teacher without distillation is the DASALC model, listwise teacher plus listwise student the BAR model. We have the following observations:
By comparing pointwise teacher with listwise teacher (DASALC), we confirm that listwise loss is indeed effective, regardless of model distillations.
We can also see that when listwise distillation is used, it always outperforms the correspond- ing teacher model, even if the teacher model is pointwise On the other hand, when pointwise distillation is applied, performance gets worse. This confirms the necessity of the listwise distillation loss.
Pointwise distillation does not help even if the teacher model uses pointwise loss, eliminating the potential concern that it does not work due to inconsistency between the two objectives.
5.5 THE NECESSITY OF TWO OBJECTIVES (RQ3)
We study the balancing factor a in Eq. 4 with the special case (a = 1) that only uses the teacher-only objective Section 4.4 The results are shown in Table and Figure 2 We can see that student models trained with BAR can outperform the teacher model (a = 0) with a wide range of a, as long as both objectives used. The performance decreases significantly the born again ranker is trained with the teacher objective alone (a = 1), which is different from the observations for some classification problems in (Furlanello al. 2018).
5.6 SOFTMAX SCORE TRANSFORMATION (RQ4)
We empirically show that Softmax score transformation under-performs the simple affine transforma- tion Figure 3 (Left) with varying the temperature T. A possible reason is that Softmax scores are normalized within each query and this may limit their power labels for ranking problems. A better understanding why certain teacher score transformations are more effective for ranking distillation promising future research direction.
7

---

Under review as a conference paper at ICLR 2022
51.75
51.50
$1.25
Web30K NDCG@K
NDCG@85 (%6
Method
$1.00
@
@5
@ 10
50.75
Teacher model
50.95
50.92
52.88
BAR
51.71
51.56
53.57
50.50
Teacher-only obj
50.45
50.53
52.54
90.25
Student Teacher
Table 3: Results on the Web30K dataset
comparing teacher model (a = 0). BAR (a = 0.5), and teacher-only objective born again rankers (a- 1).
0.0
0.2
fraction
0.4
0.6
0.8
1.0
Figure 2: NDCG@5 with varying a.
62.50
60.00
57.50
NDCG@5 (96)
$1.00
55,00
NDCG@5
50.75
92.50
50.50
50.00
Teacher Training
50.25
Softmax transform
47.50
Teacher Validation
Affine transform
Student Training
50.00
45.00
Teacher
Student Validation
49.75
42.50
60
100
temperature
Training step (x1000)
Figure 3: (Left) SDCG@5 on Web30K using Softmax transformation with varying temperatures. (Right) An illustration of ranking metrics over training data and validation data for BAR (student) vs DASALC (teacher).
UNDERSTANDING BAR
Existing theories of Born Again Networks knowledge distillation in general include (1) reweighting the data points by confidence (Furlanello et al. 2018). (2) incorporating dark knowledge from negative classes (Hinton al. 2015), and (3) ensembling of "multi-viww" features in student (Allen-Zhu & Li] 2020). However, these theories could not explain everything we observed in experiments for ranking distillation, especially, why the listwise distillation is necessary Section 5.4 Here provide new theory that helps to explain BAR effectiveness.
Theorem 1. There exists way to combine the teacher prediction score and the label to train student model, such that when the teacher and the student have exactly equivalent capability, the student model can outperform or perform as well the teacher model after being trained with the same amount of resource.
It's easy to show the necessity: we can just train the student model with the true label with a in (4. and the same computation resource and it then performs as well as the teacher model. In the Appendix Section A.I we show the sufficiency of the theorem when there are data points that are hard to fit, or that have erroneous labels.
Here we give an illustrative example of Theorem 1. Consider a training set of three data points with a single input feature
-1, 0, 1 and corresponding labels are y 2, 2, shown red points in Fig. 4 and a test set of two data points (0, 0) and (0.5, 1) in cyan. We fit the training points with a nonlinear model: f(z, b) 2/x bl. shown in green dashed line in Fig. 4 To
this model, two data points (-1,2) and (1,2) are easy-to-fit fit and one data point (0, 1) is hard or mistakenly labeled as it is inconsistent with (0, 0) in test set.
8

---

Under review as a conference paper at ICLR 2022
loss
teacher label
label y or less thee
label y loxe studentt
melel prediction
2.5
student lebel set
1.5
0.
0.3
Figure 4: An illustrative example of born again distillation. Given the training data set (z, y) (red squares), loss functions L(b) (blue solid lines) and optimized models f fax) (green dashed lines) are trained with the original label in red for the teacher model on Left, and are trained with the combined label y² = (y + / (=))/ in yellow for the student model on Right. The models are then evaluated with the test data set in eyan dots.
By minimizing a mean squared error loss, L(b) E (y: f(x;, b))² shown as the blue line in Fig. we find two nontrivial solutions at b -1/6 and 1/6. Suppose we obtain a teacher model with b = -1/ 1/66 6 and scores f 5/3, 1/3, 7/3 for the three inputs and then train student model following the BAR method with a 0.5, which is equivalent
having the student labels 11/6, 2/3, 13/6 respectively, shown as the yellow points in Fig. 4 There are again two nontrivial solutions with b - and 1 1/18 for the student. Ignoring the trivial solution at b 1/66, we find the student solution 6 1/18 achieves better performance by having smaller mean square error metric MSE = 1/81 18 on the test points than MSE = 1/9 of the teacher solution at 1/66.
y" = (y + /
Applying Theorem 1 to learning to rank. Some of the labels in the LTR dataset could be error- prone as a result of the pointwise evaluation nature of human rating. In the listwise ranking sense: (1) Some the labels may be error-prone, such that a document labeled with 2 might not be more relevant than document laheled with 1; (2) The graded relevance labels discrete and may not be able to fully capture the relevance in-between like 1.5; (3) A listwise loss only needs the comparison within each individual list and this can alleviate the impact of noisy labels comparing with a pointwise loss. As result, the listwise distillation is able to capture these mistakenly labeled data points and thus performs significantly better than the examples using pointwise distillation. This deduction is also consistent with the findings in Section 5.4 the most effective distillation is on models trained with the listwise Softmax loss.
The regularization view. Even if the human labels are not noisy but just contain some hard examples, Theorem implies that combination like Eq. reduces overfitting on hard examples and plays regularization role. Qin et al. (Qin et al.] 2021 show that major bottleneck of modern neural rankers overfitting, due to the lack of very large-scale ranking datasets and effective regularization techniques. By examining Eq. 4 the second term be treated regularization to the original objective. We empirically show that the BAR framework works as a very effective regularization technique for LTR problems. As seen in Figure 3 (Right), the BAR student gets lower training data ranking metrics and higher validation data ranking metrics during training, the desired behavior of effective regularization.
CONCLUSION
We propose Born Again neural Rankers (BAR) and further push the limits of neural ranker effec- tiveness without sacrificing efficiency. We show that the key success factors of BAR lie proper teacher score transform and listwise distillation approach specifically designed for ranking prob- lems, which do not follow the common assumptions made in most knowledge distillation work. We further conduct theoretical analysis on why BAR works, filling the gap between existing knowledge distillation theories and the learning to rank setting. As promising directions for future work, we consider advanced teacher transformations for BAR, and label noise reduction techniques.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Chenlei Guo. Knowledge distillation from internal representations. AAAI Conference on Artificial Intelligence, pp. 7350-7357, 2020.
Qingyao Ai, Keping Bi, Jiafeng Guo, and W Bruce Croft. Learning deep listwise context model for ranking refinement. In ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 135-144, 2018.
Qingyao Ai, Xuanhui Wang, Sebastian Bruch, Nadav Golbandi, Michael Bendersky, and Mare Najork. Learning groupwise multivariate scoring functions using deep neural networks. In ACM SIGIR International Conference on Theory of Information Retrieval, pp. 85-92, 2019.
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv: 20/2.09816, 2020.
Sebastian Bruch, Masrour Zoghi, Michael Bendersky, and Mare Najork. Revisiting approximate metric optimization in the age of deep neural networks. In ACM SIGIR Conference on Research and Development Information Retrieval, pp. 1241-1244 2019.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd International Conference on Machine Learning, pp. 89-96, 2005.
Christopher J.C. Burges. From RankNet to LambdaRank to LambdaMART: An overview. Technical Report ASR--RR2010-82, Microsoft Research, 2010.
Zhe Cao, Tao Qin, Tie- Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In International Conference on Machine Learning, pp. 129-136, 2007.
Olivier Chapelle and Yi Chang. Yahoo! learning to rank challenge overview. In Proceedings of the Learning to Rank Challenge, pp. 1-24, 2011.
Yun Chen, Yang Liu, Yong Cheng, and Victor OK Li. teacher-student framework for zero-resource neural machine translation. arXiv preprint arXiv:1705.00753, 2017.
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoc V Le. Bam! born-again multi-task networks for natural language understanding. arXiv preprint 1907.04997. 2019.
Domenico Dato, Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. Fast ranking with additive ensembles of oblivious and non-oblivious regression trees. ACM Trans. Inf. Syst., 2016.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pp. 1607-1616, 2018.
Jianping Gou, Baosheng Yu, Stephen John Maybank, and Dacheng Tao. Knowledge distillation: A survey. arXiv preprint arXiv:2006.05555, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference Machine Learning, pp. 1321-1330, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge distillation with adversarial samples supporting decision boundary. In AAAI Conference Artificial Intelligence, pp. 3771-3778, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv=1503.0253. 2015.
10

---

Under review as a conference paper at ICLR 2022
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: 1909. 10351, 2019.
Thorsten Joachims. Training linear svms in linear time. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 217-226, 2006.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: highly efficient gradient boosting decision tree. In International Conference on Neural Information Processing Systems, pp. 3149-3157, 2017.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. arXiv preprint
1rXiv:1606.079477 2016.
Tie- Yan Liu. Learning to rank for information retrieval. Found. Trends Inf. Retr. 2009.
Liang Pang. Jun Xu, Qingyao Ai, Yanyan Lan, Xueqi Cheng, and Jirong Wen. Setrank: Learning cerrmutationinvariiinn ranking model for information retrieval. ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 499-508, 2020.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
Tao Qin and Tie-Yan Yan Liu. Introducing LETOR 4.0 datasets. CoRR, abs/ abs//1000 1301.2097 2013.
Tao Qin, Tin-Yan Liu, and Hang Li. A general approximation framework for direct optimization of information retrieval measures. Information Retrieval, 13(4):375-397, 2010.
Zhen Qin, Le Yan, Honglei Zhuang, Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Michael Bendersky, and Mare Najork. Are neural rankers still outperformed by gradient boosted decision trees? In International Conference on Learning Representations, 2021.
Sashank Reddi, Rama Kumar Pasumarthi, Aditya Menon, Ankit Singh Rawat, Felix Yu, Seungyeon Kim, Andreas Veit, and Sanjiv Kumar. Rankdistil: Knowledge distillation for ranking. In International Conference on Artificial Intelligence and Statistics, pp. 2368-2376, 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint urXiv: 1412,550, 2014.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1:Xiv:1910.0/ 2019.
Siqi Sun, Yu Cheng, 7he Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv: 09355, 2019.
Xu Tan, Yi Ren, Di He, Tao Qin. Zhou Zhao, and Tie- Yan Liu. Multilingual neural machine translation with knowledge distillation. arXiv preprint arXiv: 1902. 10461, 2019.
Jiaxi Tang and Ke Wang. Ranking distillation: Learning compact ranking models with high per- formance for recommender system. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 2289-2289999 2018.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves ImageNet classification. In IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 10687-10698, 2020.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In IEEE/CVF International Conference on Computer Vision, pp. 3713-3722, 2019.
11

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 PROOF OF THEOREM 1
In this section, we show the student model trained with Eq. can outperform the teacher model with the same capacity at optimal a if there exist hard examples or the labels are noisy.
We first formulate the problem and give the mathematical definitions "hard examples" and "noisy labels". When training converges, deep neural network teacher model f(,, 8*) approximately satisfies
ac 00,-
ally, S) 0f 0);
0
(11)
M
08:
00,
(x.Y)EW
for any parameter 0y in the total P = 10] trainable parameters. The first derivative on the right hand side the loss function dependent gradient. For example:
Mean Squared Error loss: 1(y,s)
Ei= (yr -s:)² we have
ol(y,s) - - -s:);
0si
Pointwise logistic loss: (yy,8) - E Y/ Inp: + - yi) In(1 -pt) with pi exp(s,) we have
I+exp(<:)
olly,s)
--(; -P:); Os,
Softmax loss in Eq. we have
Ol 8) 0s;
-(y/ Ps
yk))
If we normalize the labels by having Σg y/ = 1 in Softmax loss and define the model gradients on parameter fy at data point i, Gy.; = (x, /00, as a P n | matrix, we then have ranking model solutions satisfying P "qquilibrium equations,
(y 0
(12)
with p as a vector of model predictions depending on different losses as shown in above examples. When the rank of is n V we only have trivial solutions with p y. However, noting that the matrix G is nonlinear function of 0, in general we have the rank of G is P < n|V] with non-trivial solutions p # y living in the null space G.
To such solution, we can distinct hard and easy problems in the training dataset by quantifying the deviations from the target labels in the trained teacher model ||y: - p:ll: the larger this deviation is, the harder the corresponding data point is. These data points in the training could be hard in two different senses:
Case One Hard examples: they are hard in nature, meaning that the trained model does not capture the most deterministic feature of the data points.
Case Two Noisy labels: they could be mistakenly labeled, meaning that there exists ground-truth model and the ground-truth model predicts results different from the labels for these data points.
When we train student model from scratch with combined teacher predictions and labels following Eq. 4 with the teacher model solution p' as g(f(x, 0')) so that G- Gy (y))') = O, the student solution should then also satisfy Eq. (2) but with new set of labels y = (1 - a)y + ap'. First of all, it's easy to show that the teacher model with pd is a solution. But for a complex nonlinear model trained from scratch, it would be almost impossible end up with the same solution as the teacher in general, especially when the labels are changed, which determine the initial gradients. We thus focus on the general solutions different from p'.
12

---

Under review as a conference paper at ICLR 2022
In Case One, the model output will be insensitive to the change of parameters for the hard points at initial training stage. After easy points are fit well, y - p' == O, the training enters a stage of fitting the hard points with irrelevant features, which directly leads to overfitting. So in practice, we use predictions p' from a teacher regularized with early-stopping as the distillation labels for students. With such teacher, the initial gradients of the student model are still dominated by easy data points R p and in the later stage the contributions of the hard data points y Pi (1-a) (v: p$) + (p: pi) R (1 - () pt) are lowered by fraction 1 a compared to the training of the teacher model. The overfitting effect will thus be weakened for the student model. In this case, the student model can achieve a better performance on the test data than the teacher model
with >> 0.
In Case Two, the final solution of the teacher model will be balanced by contributions of the correct labels and wrong labels: let's say there are out of n training data points mistakenly labeled and most are correct labels m < n: on average, the teacher model prediction will deviate from the label by (n m)ly" p'1 m - =p', where yf are correct labels and y are wrong labels. We thus have the teacher predictions deviating from the correct labels by Ip² - y01 ~ of and from the wrong labels by Ip' yd1 ( ( 1yy, so the student labels are y = a !i
n y for correct labels and y² (1 a)y + m y for wrong labels. After convergence, the student predictions will thus be p² = y When a n+mm" student predictions on examples with correct labels are consistent with labels and examples with wrong labels approximately independent of the label. Assuming that there is no correlation of making the wrong labels in training and test sets, then the student models validated on the test set will result in better metrics than their teachers as they are making predictions closer the correctly labeled data points.
a) m y m2 y for correct labels and p² = (1 a) "
m. n. for wrong labels.
13


Reviews:
[{"review_id": "re5Ho0pSOX2", "reviewer": "Reviewer_7kAC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n* The paper clearly motivates the need for distillation specific to LTR problems, explained the need for a score transformation layer and recommends one way (BAR) to achieve good transfer learning on 3 public datasets.\n* The primary contribution, i.e. transforming teacher scores using a ReLU + Affine layer and using listwise loss for the student model are simple, easy to understand and replicate. The effect of hyperparameters could have been studied better. \n\n**Weaknesses**\n\n* The paper is very light on experimentation. e.g. It would have been good to see the comparison of listwise vs pointwise teachers on more than one dataset - ideally with 3-5 neural architectures that have been used in the literature. What was the effect of the scaling factor in the Istella dataset?  How sensitive is $\\alpha$ across datasets? Does early stopping play a role in the generalization performance? Does $L_2$ affect the regularization ability of BAR?\n* Lack of theoretical justification of the BAR choices. The authors claim \"theoretical explanations\" as a contribution of the paper. Yet, question like why ReLU, why affine transformation, how to fit parameters of the transformation on new datasets are not justified by theory. The section on Understanding BAR merely provides an intuition and one hypothesis explaining the trend.\n", "summary_of_the_paper": "Knowledge distillation (KD) is a popular technique to trade-off between performance and efficiency. Student models trained with KD usually work better than those trained from the original labels. However, applying KD to ranking problems is not well studied. One earlier work (Tang & Wang, 2018) on ranking distillation proposes pointwise logistic loss for distillation after unsuccessfully attempting pair-wise approaches on the teacher's top K ranking. The authors take on the challenge of dealing with the following characteristics of the ranking models:\n* The ranking scores are neither calibrated nor normalized. They do not directly convey information about the underlying distribution.\n* They care about the relative order of score and do not make an iid assumption over documents.\n* Large teacher models tend to overfit because it is difficult to apply techniques like data augmentation & small datasets.\n\nThe authors extend BAN in which student models are configured with the same capacity as teachers and propose BAR. BAR is referred to the specific setting where ranking loss with original labels, listwise distillation on teacher score, and tunable affine teacher score transformation. Unlike typiical KD models, the primary goal of the paper is to improve ranking performance over state-of-the-art teacher rankers.", "summary_of_the_review": "The authors empirically show improvements on 3 public LTR dataset using the BAR method over a neural baseline. To clearly support their research questions, they should continue experimenting with a few more baseline neural models to understand the effectiveness of BAR. Additionally, the ablations and design choices for transformation & loss functions should be done on more than one dataset to be able to conclusively make the claims made in the paper. The biggest gap is however, the lack of theoretical justification on why the specific choices in BAR are needed. The paper can just have empirical evidence to show one transformation that works well across datasets. However claiming that BAR is a novel listwise distillation framework with theoretical justification is far-fetched.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XDHEBGv_GER", "reviewer": "Reviewer_wTyv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduces Born Again neural Rankers (BAR) for Learning to rank (LTR). Unlike traditional Knowledge Distillation (KD) which usually seeks a trade-off before performance and efficiency, BAR targets for better performance without increasing the model capacity. \n\nIn particular: \n\n1. The authors clearly state two important design choices: 1) a listwise Softmax loss function and 2) a teacher score transformation. In additional, they also explore other alternatives and empirically show why the current choice is warranted.\n\n2. Extensive experiment studies are conducted to demonstrate the effectiveness of the proposed method.\n\n3. The authors also propose an explanation to better understand the superior performance of BAR.\n\nThere are several places where I am not certain, or maybe can make the paper better:\n\n1. In the experiment there are several ensembles considered, including DASALC-ens and BAR-ens. For a more fair comparison, I wonder whether the authors can also compare with ensembles of \\lambdaMART, since it achieves two best performances across 9 settings. \n\n2. In (4) there is a hyperparameter \\alpha, and in (7) there is a and b for the affine transformation. The experiment results shown in Table 1 only uses fixed values for these hyperparameters. It might be worth to see if we tune more on these three hyperparameters, how much more gain the model can achieve. \n\n3. Is such complexity necessary or is more complexity helpful? I do not expect the authors to address this point in this paper, but it would be helpful to share some thoughts if they have. Currently, we have the student model the same complexity as the teacher. However, is such complexity really necessary? Can we have a less complexity student while also achieve superior performance? From another perspective, if we have a more complexity student, with the help of knowledge distillation, is it able to perform even better? ", "summary_of_the_paper": "The paper introduces Knowledge Distillation (KD) to Learning to rank (LTR), adapting the idea of Born Again Networks (BAN). The propose model, Born Again neural Rankers (BAR), combines an appropriate teacher score transformation and a novel listwise distillation framework. The authors conduct thorough experiment studies to demonstrate the superior performance of the BAR, and also provide explanations in terms of why it works.", "summary_of_the_review": "I believe this paper is well written with clear statement of their research questions, extensive experiments, and rigorous arguments. I suggest to accept this paper to ICLR. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "A_2pZvJuzsH", "reviewer": "Reviewer_TTzh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \nThe paper investigates an interesting and under explored area of model distillation and learning to rank, and is well written and easy to follow. Experiments are done on large real world datasets and compare against leading baselines from both gradient boosting and deep learning domains.\n\nWeaknesses:\nMy major concern is with the novelty. The proposed approach is a relatively straightforward application of distillation with off-the-shelf loss and optimization. I also have a concerns about section 4.3, ReLU is an odd transformation to use in the learning to rank setting as it squashes all negative scores to 0 making that part of the ranking indistinguishable. This can significantly affect teachers accuracy and also introduces many ties into the students loss. The justification in 4.3 does not support such a transformation, and in section 5.1 authors actually use identity for Web30K and Yahoo datasets, and divide scores by 100 for Istella. These choices again seem hand picked and non-principled, and further justification is needed here since authors claim that normalising teachers scores is important for distillation to work. ", "summary_of_the_paper": "The paper proposes a distillation approach for the learning to rank setting. Specifically, the \"born again neural ranker\" approach is investigated where both teacher and student models are identically parameterised neural networks. Authors propose a listwise loss to incorporate scores from the teacher model during student optimization and show that this leads to improvements on three real world datasets.", "summary_of_the_review": "The paper explores an important area in learning to rank but the proposed approach lacks novelty and makes multiple ad hoc choices that need further justification.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors are strongly encouraged to elaborate further about the novelty of their method, as well as to give detailed (either theoretical or experimental) justifications for the design choices they make within the paper. Finally, the paper could benefit from additional experiments, as outlined in the reviews.
Decision: Reject