Born Again Neural Rankers

We introduce Born Again neural Rankers ( BAR ) in the Learning to Rank ( LTR ) setting , where student rankers , trained in the Knowledge Distillation ( KD ) framework , are parameterized identically to their teachers . Unlike the existing ranking distillation work which pursues a good trade-off between performance and efficiency , BAR adapts the idea of Born Again Networks ( BAN ) to ranking problems and significantly improves ranking performance of students over the teacher rankers without increasing model capacity . The key differences between BAR and common distillation techniques for classification are : ( 1 ) an appropriate teacher score transformation function , and ( 2 ) a novel listwise distillation framework . Both techniques are specifically designed for ranking problems and are rarely studied in the knowledge distillation literature . Using the state-of-the-art neural ranking structure , BAR is able to push the limits of neural rankers above a recent rigorous benchmark study and significantly outperforms traditionally strong gradient boosted decision tree based models on 7 out of 9 key metrics , the first time in the literature . In addition to the strong empirical results , we give theoretical explanations on why listwise distillation is effective for neural rankers . 1 INTRODUCTION . Learning to rank ( LTR ) has become an essential component for many real-world applications such as search ( Liu , 2009 ) . In ranking problems , the focus is on predicting the relative order of a list of items given a query . It is thus different from classification problems whose goal is to predict the class label of a single item . While many neural machine learning techniques have been proposed recently , they are mostly for classification problems . Given the difference between ranking and classification problems , it is interesting to study how neural techniques can be used for ranking problems . Knowledge distillation ( KD ) ( Hinton et al. , 2015 ; Gou et al. , 2020 ) is one of such recently popular techniques . The initial goal of KD is to pursue a good trade-off between performance and efficiency . Given a high-capacity teacher model with desired high performance , a more compact student model is trained using teacher labels ( Heo et al. , 2019 ; Sun et al. , 2019 ; Sanh et al. , 2019 ) . Student models trained with KD usually work better than those trained from original labels without teacher guidance . How to effectively apply KD to ranking problems is not straightforward and has not been well-studied yet , for the following reasons : First , teacher models in classification typically predict a probability distribution over all classes . Such “ dark knowledge ” is believed to be a key reason why KD works ( Hinton et al. , 2015 ) . However , a teacher model in ranking does not convey such distribution over pre-defined classes , as ranking models only care about the relative orders of items and simply output a single score for each item in a possibly unbounded candidate list . Ranking scores are typically neither calibrated ( comparing to the probabilistic interpretation of classification scores ( Guo et al. , 2017 ) ) nor normalized ( comparing to summing up to one for classification scores over all classes for most popular losses ( Hinton et al. , 2015 ) ) . Thus directly using the outputs of a teacher model as labels for ranking tasks with existing KD methods may be less optimal . Second , listwise information over all input items should be considered to achieve the best ranking performance , since the goal is to infer the relative orders among them . For example , listwise losses have been shown to be more effective than other alternatives for LTR problems ( Cao et al. , 2007 ) . On the other hand , classification tasks almost universally treat each item independently based on the i.i.d . assumption . It is interesting to also consider listwise frameworks when studying KD for ranking problems , which has not been explored in the literature , to the best of our knowledge . Third , though there is a consensus in the classification KD literature that given no severe over-fitting , larger teacher models usually work better ( He et al. , 2016 ) , recent works show that for traditional LTR problems , it is hard for larger models to achieve better performance ( Bruch et al. , 2019 ; Qin et al. , 2021 ) due to the difficulty of applying standard techniques such as data augmentation ( compared to image rotation in computer vision ( Xie et al. , 2020 ) ) and the lack of very large human-labeled ranking datasets . This makes it harder to use the standard KD setting to improve the performance of ranking models by distilling from a very large teacher model . Thus , KD techniques need special adjustments for LTR problems . In this paper , inspired by the Born Again Networks ( BAN ) ( Furlanello et al. , 2018 ) in which student models are configured with the same capacity as teachers and are shown to outperform teachers for classification problems , we study the BAN techniques for ranking problems . To this end , we propose Born Again neural Rankers ( BAR ) that train the student models using listwise distillation and properly transformed teacher scores , which can achieve new state-of-the-art ranking performance for neural rankers . While existing ranking distillation works such as ( Tang & Wang , 2018 ; Reddi et al. , 2021 ) require a more powerful teacher model and focus on performance-efficiency trade-offs , the primary goal of our paper is to improve ranking performance over state-of-the-art teacher rankers . In summary , our contributions are as follows : • We propose Born Again neural Rankers ( BAR ) for learning to rank . This is the first knowledge distillation work that targets for better ranking performance without increasing the model capacity . • We show that the key success factors of BAR are ( 1 ) an appropriate teacher score transfor- mation function , and ( 2 ) a ranking specific listwise distillation loss . Both design choices are tailored for LTR problems and are rarely studied in the knowledge distillation literature . • We provide new theoretical explanations on why BAR works better than other alternatives . This contributes to both the general knowledge distillation and ranking distillation research . • We verify our hypothesis on rigorous public LTR benchmarks and show that BAR is able to significantly improve upon state-of-the-art neural teacher rankers . 2 RELATED WORK . Knowledge distillation has been a popular research topic recently in several areas such as image recognition ( Hinton et al. , 2015 ; Romero et al. , 2014 ; Park et al. , 2019 ) , natural language understanding ( Sanh et al. , 2019 ; Jiao et al. , 2019 ; Aguilar et al. , 2020 ) , and neural machine translation ( Kim & Rush , 2016 ; Chen et al. , 2017 ; Tan et al. , 2019 ) as a way to generate compact models , achieving good performance-efficiency trade-offs . As we mentioned in the Introduction , both the classical setting ( e.g. , using pointwise losses on i.i.d . data ) and theoretical analysis ( e.g. , “ dark knowledge ” among classes ) for classification tasks may not be optimal for the ranking setting . The major goal of this work is to push the limits of neural rankers on rigorous benchmarks . Since the introduction of RankNet ( Burges et al. , 2005 ) over a decade ago , only recently , neural rankers were shown to be competitive with well-tuned Gradient Boosted Decision Trees ( GBDT ) on traditional LTR datasets ( Qin et al. , 2021 ) . We build upon ( Qin et al. , 2021 ) and show that BAR can further push the state-of-the-art of neural rankers on rigorous benchmarks . We also provide extra experiments to show listwise distillation helps neural ranking in other settings . We are motivated by Born Again Networks ( BAN ) that were introduced by ( Furlanello et al. , 2018 ) . BAN is the first work that shows better performance can be achieved by parameterizing the student model the same as the teacher . However , BAN only focuses on classification and direct application of BAN does not help LTR problems . Building upon the general “ born again ” ideas ( Zhang et al. , 2019 ; Clark et al. , 2019 ) , our contribution in this paper is in developing specific new techniques and theory that make these ideas applicable for the important LTR setting . Another closely related work is Ranking Distillation ( RD ) ( Tang & Wang , 2018 ) , since it studies knowledge distillation for ranking . There are several marked differences between our work and RD . First , RD focuses on performance-efficiency trade-offs , where the student usually underperforms the teacher in terms of ranking metrics ( Reddi et al. , 2021 ) , while we focus on outperforming the teacher . Second , the work only uses pointwise logistic loss for distillation . The authors state that “ We also tried to use a pair-wise distillation loss when learning from teacher ’ s top-K ranking . However , the results were disappointing ” , without going into more detail or explore listwise approaches , which we show are the key success factor . Also , Tang and Wang ( Tang & Wang , 2018 ) use a hyperparameter K : items ranked as top K are labeled as positive and others as negative for distillation . Besides only working for binary datasets , this setting is not very practical since real-world ranking lists may have very different list sizes or number of relevant items . Our method does not require such a parameter . Furthermore , they only evaluated their methods on some recommender system datasets where only user id , item id , and binary labels are available . It is not a typical LTR setting , so the effectiveness of RD over state-of-the-art neural rankers is unclear . 3 BACKGROUND ON LEARNING TO RANK . For LTR problems , the training data can be represented as a set Ψ = { ( x , y ) ∈ χn ×Rn ) } , where x is a list of n items xi ∈ χ and y is a list of n relevance labels yi ∈ R for 1 ≤ i ≤ n. We use χ as the universe of all items . In traditional LTR problems , each xi corresponds to a query-item pair and is represented as a feature vector in Rk where k is the number of feature dimensions . With slight abuse of notation , we also use xi as the feature vector and say x ∈ Rn×k . The objective is to learn a function that produces an ordering of items in x so that the utility of the ordered list is maximized , that is , the items are ordered by decreasing relevance . Most LTR algorithms formulate the problem as learning a ranking function to score and sort the items in a list . As such , the goal of LTR boils down to finding a parameterized ranking function f ( · ; θ ) : χn → Rn , where θ denotes the set of trainable parameters , to minimize the empirical loss : L ( f ( · ; θ ) ) = 1 |Ψ| ∑ ( x , y ) ∈Ψ l ( y , f ( x ; θ ) ) , ( 1 ) where l ( · ) is the loss function on a single list . There are many existing ranking metrics such as NDCG and MAP used in LTR problems . A common property of these metrics is that they are rank-dependent and place more emphasis on the top ranked items . For example , the widely used NDCG metric is defined as NDCG ( πf , y ) = DCG ( πf , y ) DCG ( π∗ , y ) , ( 2 ) where πf is a ranked list induced by the ranking function f ( · ; θ ) on x , π∗ is the ideal list ( where x is sorted by decreasing y ) , and DCG is defined as : DCG ( π , y ) = n∑ i=1 2yi − 1 log2 ( 1 + π ( i ) ) . ( 3 ) In practice , the truncated version that only considers the top-k ranked items , denoted as NDCG @ k , is often used .

Summary: The paper proposes a distillation approach for the learning to rank setting. Specifically, the "born again neural ranker" approach is investigated where both teacher and student models are identically parameterised neural networks. Authors propose a listwise loss to incorporate scores from the teacher model during student optimization and show that this leads to improvements on three real world datasets.