Born Again Neural Rankers

We introduce Born Again neural Rankers ( BAR ) in the Learning to Rank ( LTR ) setting , where student rankers , trained in the Knowledge Distillation ( KD ) framework , are parameterized identically to their teachers . Unlike the existing ranking distillation work which pursues a good trade-off between performance and efficiency , BAR adapts the idea of Born Again Networks ( BAN ) to ranking problems and significantly improves ranking performance of students over the teacher rankers without increasing model capacity . The key differences between BAR and common distillation techniques for classification are : ( 1 ) an appropriate teacher score transformation function , and ( 2 ) a novel listwise distillation framework . Both techniques are specifically designed for ranking problems and are rarely studied in the knowledge distillation literature . Using the state-of-the-art neural ranking structure , BAR is able to push the limits of neural rankers above a recent rigorous benchmark study and significantly outperforms traditionally strong gradient boosted decision tree based models on 7 out of 9 key metrics , the first time in the literature . In addition to the strong empirical results , we give theoretical explanations on why listwise distillation is effective for neural rankers . 1 INTRODUCTION . Learning to rank ( LTR ) has become an essential component for many real-world applications such as search ( Liu , 2009 ) . In ranking problems , the focus is on predicting the relative order of a list of items given a query . It is thus different from classification problems whose goal is to predict the class label of a single item . While many neural machine learning techniques have been proposed recently , they are mostly for classification problems . Given the difference between ranking and classification problems , it is interesting to study how neural techniques can be used for ranking problems . Knowledge distillation ( KD ) ( Hinton et al. , 2015 ; Gou et al. , 2020 ) is one of such recently popular techniques . The initial goal of KD is to pursue a good trade-off between performance and efficiency . Given a high-capacity teacher model with desired high performance , a more compact student model is trained using teacher labels ( Heo et al. , 2019 ; Sun et al. , 2019 ; Sanh et al. , 2019 ) . Student models trained with KD usually work better than those trained from original labels without teacher guidance . How to effectively apply KD to ranking problems is not straightforward and has not been well-studied yet , for the following reasons : First , teacher models in classification typically predict a probability distribution over all classes . Such “ dark knowledge ” is believed to be a key reason why KD works ( Hinton et al. , 2015 ) . However , a teacher model in ranking does not convey such distribution over pre-defined classes , as ranking models only care about the relative orders of items and simply output a single score for each item in a possibly unbounded candidate list . Ranking scores are typically neither calibrated ( comparing to the probabilistic interpretation of classification scores ( Guo et al. , 2017 ) ) nor normalized ( comparing to summing up to one for classification scores over all classes for most popular losses ( Hinton et al. , 2015 ) ) . Thus directly using the outputs of a teacher model as labels for ranking tasks with existing KD methods may be less optimal . Second , listwise information over all input items should be considered to achieve the best ranking performance , since the goal is to infer the relative orders among them . For example , listwise losses have been shown to be more effective than other alternatives for LTR problems ( Cao et al. , 2007 ) . On the other hand , classification tasks almost universally treat each item independently based on the i.i.d . assumption . It is interesting to also consider listwise frameworks when studying KD for ranking problems , which has not been explored in the literature , to the best of our knowledge . Third , though there is a consensus in the classification KD literature that given no severe over-fitting , larger teacher models usually work better ( He et al. , 2016 ) , recent works show that for traditional LTR problems , it is hard for larger models to achieve better performance ( Bruch et al. , 2019 ; Qin et al. , 2021 ) due to the difficulty of applying standard techniques such as data augmentation ( compared to image rotation in computer vision ( Xie et al. , 2020 ) ) and the lack of very large human-labeled ranking datasets . This makes it harder to use the standard KD setting to improve the performance of ranking models by distilling from a very large teacher model . Thus , KD techniques need special adjustments for LTR problems . In this paper , inspired by the Born Again Networks ( BAN ) ( Furlanello et al. , 2018 ) in which student models are configured with the same capacity as teachers and are shown to outperform teachers for classification problems , we study the BAN techniques for ranking problems . To this end , we propose Born Again neural Rankers ( BAR ) that train the student models using listwise distillation and properly transformed teacher scores , which can achieve new state-of-the-art ranking performance for neural rankers . While existing ranking distillation works such as ( Tang & Wang , 2018 ; Reddi et al. , 2021 ) require a more powerful teacher model and focus on performance-efficiency trade-offs , the primary goal of our paper is to improve ranking performance over state-of-the-art teacher rankers . In summary , our contributions are as follows : • We propose Born Again neural Rankers ( BAR ) for learning to rank . This is the first knowledge distillation work that targets for better ranking performance without increasing the model capacity . • We show that the key success factors of BAR are ( 1 ) an appropriate teacher score transfor- mation function , and ( 2 ) a ranking specific listwise distillation loss . Both design choices are tailored for LTR problems and are rarely studied in the knowledge distillation literature . • We provide new theoretical explanations on why BAR works better than other alternatives . This contributes to both the general knowledge distillation and ranking distillation research . • We verify our hypothesis on rigorous public LTR benchmarks and show that BAR is able to significantly improve upon state-of-the-art neural teacher rankers . 2 RELATED WORK . Knowledge distillation has been a popular research topic recently in several areas such as image recognition ( Hinton et al. , 2015 ; Romero et al. , 2014 ; Park et al. , 2019 ) , natural language understanding ( Sanh et al. , 2019 ; Jiao et al. , 2019 ; Aguilar et al. , 2020 ) , and neural machine translation ( Kim & Rush , 2016 ; Chen et al. , 2017 ; Tan et al. , 2019 ) as a way to generate compact models , achieving good performance-efficiency trade-offs . As we mentioned in the Introduction , both the classical setting ( e.g. , using pointwise losses on i.i.d . data ) and theoretical analysis ( e.g. , “ dark knowledge ” among classes ) for classification tasks may not be optimal for the ranking setting . The major goal of this work is to push the limits of neural rankers on rigorous benchmarks . Since the introduction of RankNet ( Burges et al. , 2005 ) over a decade ago , only recently , neural rankers were shown to be competitive with well-tuned Gradient Boosted Decision Trees ( GBDT ) on traditional LTR datasets ( Qin et al. , 2021 ) . We build upon ( Qin et al. , 2021 ) and show that BAR can further push the state-of-the-art of neural rankers on rigorous benchmarks . We also provide extra experiments to show listwise distillation helps neural ranking in other settings . We are motivated by Born Again Networks ( BAN ) that were introduced by ( Furlanello et al. , 2018 ) . BAN is the first work that shows better performance can be achieved by parameterizing the student model the same as the teacher . However , BAN only focuses on classification and direct application of BAN does not help LTR problems . Building upon the general “ born again ” ideas ( Zhang et al. , 2019 ; Clark et al. , 2019 ) , our contribution in this paper is in developing specific new techniques and theory that make these ideas applicable for the important LTR setting . Another closely related work is Ranking Distillation ( RD ) ( Tang & Wang , 2018 ) , since it studies knowledge distillation for ranking . There are several marked differences between our work and RD . First , RD focuses on performance-efficiency trade-offs , where the student usually underperforms the teacher in terms of ranking metrics ( Reddi et al. , 2021 ) , while we focus on outperforming the teacher . Second , the work only uses pointwise logistic loss for distillation . The authors state that “ We also tried to use a pair-wise distillation loss when learning from teacher ’ s top-K ranking . However , the results were disappointing ” , without going into more detail or explore listwise approaches , which we show are the key success factor . Also , Tang and Wang ( Tang & Wang , 2018 ) use a hyperparameter K : items ranked as top K are labeled as positive and others as negative for distillation . Besides only working for binary datasets , this setting is not very practical since real-world ranking lists may have very different list sizes or number of relevant items . Our method does not require such a parameter . Furthermore , they only evaluated their methods on some recommender system datasets where only user id , item id , and binary labels are available . It is not a typical LTR setting , so the effectiveness of RD over state-of-the-art neural rankers is unclear . 3 BACKGROUND ON LEARNING TO RANK . For LTR problems , the training data can be represented as a set Ψ = { ( x , y ) ∈ χn ×Rn ) } , where x is a list of n items xi ∈ χ and y is a list of n relevance labels yi ∈ R for 1 ≤ i ≤ n. We use χ as the universe of all items . In traditional LTR problems , each xi corresponds to a query-item pair and is represented as a feature vector in Rk where k is the number of feature dimensions . With slight abuse of notation , we also use xi as the feature vector and say x ∈ Rn×k . The objective is to learn a function that produces an ordering of items in x so that the utility of the ordered list is maximized , that is , the items are ordered by decreasing relevance . Most LTR algorithms formulate the problem as learning a ranking function to score and sort the items in a list . As such , the goal of LTR boils down to finding a parameterized ranking function f ( · ; θ ) : χn → Rn , where θ denotes the set of trainable parameters , to minimize the empirical loss : L ( f ( · ; θ ) ) = 1 |Ψ| ∑ ( x , y ) ∈Ψ l ( y , f ( x ; θ ) ) , ( 1 ) where l ( · ) is the loss function on a single list . There are many existing ranking metrics such as NDCG and MAP used in LTR problems . A common property of these metrics is that they are rank-dependent and place more emphasis on the top ranked items . For example , the widely used NDCG metric is defined as NDCG ( πf , y ) = DCG ( πf , y ) DCG ( π∗ , y ) , ( 2 ) where πf is a ranked list induced by the ranking function f ( · ; θ ) on x , π∗ is the ideal list ( where x is sorted by decreasing y ) , and DCG is defined as : DCG ( π , y ) = n∑ i=1 2yi − 1 log2 ( 1 + π ( i ) ) . ( 3 ) In practice , the truncated version that only considers the top-k ranked items , denoted as NDCG @ k , is often used .

Summary: Knowledge distillation (KD) is a popular technique to trade-off between performance and efficiency. Student models trained with KD usually work better than those trained from the original labels. However, applying KD to ranking problems is not well studied. One earlier work (Tang & Wang, 2018) on ranking distillation proposes pointwise logistic loss for distillation after unsuccessfully attempting pair-wise approaches on the teacher's top K ranking. The authors take on the challenge of dealing with the following characteristics of the ranking models: * The ranking scores are neither calibrated nor normalized. They do not directly convey information about the underlying distribution. * They care about the relative order of score and do not make an iid assumption over documents. * Large teacher models tend to overfit because it is difficult to apply techniques like data augmentation & small datasets.  The authors extend BAN in which student models are configured with the same capacity as teachers and propose BAR. BAR is referred to the specific setting where ranking loss with original labels, listwise distillation on teacher score, and tunable affine teacher score transformation. Unlike typiical KD models, the primary goal of the paper is to improve ranking performance over state-of-the-art teacher rankers.