Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees?

Despite the success of neural models on many major machine learning problems , their effectiveness on traditional Learning-to-Rank ( LTR ) problems is still not widely acknowledged . We first validate this concern by showing that most recent neural LTR models are , by a large margin , inferior to the best publicly available Gradient Boosted Decision Trees ( GBDT ) in terms of their reported ranking accuracy on benchmark datasets . This unfortunately was somehow overlooked in recent neural LTR papers . We then investigate why existing neural LTR models under-perform and identify several of their weaknesses . Furthermore , we propose a unified framework comprising of counter strategies to ameliorate the existing weaknesses of neural models . Our models are the first to be able to perform equally well , comparing with the best tree-based baseline , while outperforming recently published neural LTR models by a large margin . Our results can also serve as a benchmark to facilitate future improvement of neural LTR models . 1 INTRODUCTION . Neural approaches have been dominating in many major machine learning domains , such as computer vision ( He et al. , 2015 ) , natural language processing ( Devlin et al. , 2019 ) , and speech recognition ( Hannun et al. , 2014 ) . However , the effectiveness of neural approaches in traditional Learningto-Rank ( LTR ) , the long-established inter-disciplinary research area at the intersection of machine learning and information retrieval ( Liu , 2009 ) , is not widely acknowledged ( Yang et al. , 2019 ) , especially on benchmark datasets that have only numerical features . Historically , a series of LTR models were developed by researchers at Microsoft , starting with RankNet ( Burges et al. , 2005 ) and LambdaRank ( Burges et al. , 2007 ) , both based on neural networks , and culminating in LambdaMART ( Wu et al. , 2010 ) , which is based on Gradient Boosted Decision Trees ( GBDT ) ; Burges ( 2010 ) provides an overview of this evolution . There are two publicly available implementations of LambdaMART : one provided by the RankLib1 library that is part of the Lemur Project ( henceforth referred to as λMARTRankLib ) ; and the LightGBM2 implementation provided by Microsoft ( Ke et al. , 2017 ) ( henceforth referred to as λMARTGBM ) . As we will show in Section 3 , λMARTGBM substantially outperforms λMARTRankLib . There is strong and continuing interest in neural ranking models , with numerous papers published in the last few years alone . Most of these papers treat RankNet and LambdaRank as weak baselines ( Pang et al. , 2020 ; Bruch et al. , 2019b ) and LambdaMART as the “ state-of-the-art ” ( Bruch et al. , 2019b ; Li et al. , 2019 ; Zhu & Klabjan , 2020 ; Hu et al. , 2019 ) . However , when examining these papers , we note that they either acknowledge their under-performance to λMARTGBM or claim state-of-the-art performance by comparing to a weaker λMARTRankLib implementation . The inconsistency of performance evaluation on benchmark datasets in this field has made it difficult to measure progress ( Lipton & Steinhardt , 2018 ) . It therefore remains an open question whether neural LTR models are as effective as they claim to be , and how to improve them if that is not the case . 1https : //sourceforge.net/p/lemur/wiki/RankLib/ 2https : //github.com/microsoft/LightGBM In this paper , we first conduct a benchmark to show that λMARTGBM outperforms recently published neural models , as well as the λMARTRankLib , by a large margin . While the neural paradigm is still appealing in a myriad of ways , such as being composable , flexible , and able to benefit from a plethora of new advances ( Vaswani et al. , 2017 ; Devlin et al. , 2019 ) , the research progress in neural ranking models could be hindered due to their inferior performance to tree models . It thus becomes critical to understand the pitfalls of building neural rankers and boost their performance on benchmark datasets . Specifically , we investigate why neural LTR approaches under-perform on standard LTR datasets and identify three major weaknesses that are typically ignored by recent work . First , neural models are not as adept at performing effective feature transformations and scaling , which is one major benefit of using tree-based methods ( Saberian et al. , 2019 ) . In ranking data which is typically longtailed , this can be a prohibitive property . Second , standard feed-forward networks are ineffective in generating higher-order features as noted by recent papers ( Wang et al. , 2017b ; Beutel et al. , 2018 ) . More effective network architectures for neural LTR models are needed . Third , recent neural LTR work on benchmark datasets does not employ high-capacity networks , a key success factor of many neural models ( Devlin et al. , 2019 ) , possibly due to a small scale of training data that causes overfitting . On the other hand , there are several potential benefits of neural approaches over LambdaMART for LTR , such as their flexibility to model listwise data and the existence of many techniques to mitigate data sparsity . To that end , we propose a new framework that ameliorates the weaknesses of existing neural LTR approaches and improves almost all major network components . In the proposed framework , we make several technical contributions : ( 1 ) We demonstrate empirical evidence that a simple log1p transformation on the input features is very helpful . ( 2 ) We use data augmentation ( DA ) to make the most out of high-capacity neural models , which is surprisingly the first work in the LTR literature to do so . We show that adding a simple Gaussian noise helps , but only when the model capacity is appropriately augmented ( which probably explains why there is no prior work on such a simple idea ) . ( 3 ) We use self-attention ( SA ) to model the listwise ranking data as context , and propose to use latent cross ( LC ) to effectively generate the interaction of each item and its listwise context . We conduct experiments on three widely used public LTR datasets . Our neural models are trained with listwise ranking losses . On all datasets , our framework can outperform recent neural LTR methods by a large margin . When comparing with the strong LambdaMART implementation , λMARTGBM , we are able to achieve equally good results , if not better . Our work can also serve as a benchmark for neural ranking models , which we believe can lay a fertile ground for future neural LTR research , as rigorous benchmarks on datasets such as ImageNet ( Russakovsky et al. , 2015 ) and GLUE ( Wang et al. , 2018a ) do in their respective fields . 2 BACKGROUND . We provide some background on LTR , including its formulation and common metrics . We review LambdaMART and highlight its two popular implementations which are causes of the inconsistency of evaluations in the recent literature . 2.1 LEARNING TO RANK . LTR methods are supervised techniques and the training data can be represented as a set Ψ = { ( x , y ) ∈ χn × Rn ) } , where x is a list of n items xi ∈ χ and y is a list of n relevance labels yi ∈ R for 1 ≤ i ≤ n. We use χ as the universe of all items . In traditional LTR problems , each xi corresponds to a query-item pair and is represented as a feature vector in Rk where k is the number of feature dimensions . With slightly abuse of notation , we also use xi as the feature vector and say x ∈ Rn×k . The objective is to learn a function that produces an ordering of items in x so that the utility of the ordered list is maximized . Most LTR algorithms formulate the problem as learning a ranking function to score and sort the items in a list . As such , the goal of LTR boils down to finding a parameterized ranking function s ( · ; Θ ) : χn → Rn , where Θ denotes the set of parameters , to minimize the empirical loss : L ( s ) = 1 |Ψ| ∑ ( x , y ) ∈Ψ l ( y , s ( x ) ) , ( 1 ) where l ( · ) is the loss function on a single list . LTR algorithms differ primarily in how they parameterize s and how they define l. There are many existing ranking metrics such as NDCG and MAP used in LTR problems . A common property of these metrics is that they are rank-dependent and place more emphasis on the top ranked items . For example , the commonly adopted NDCG metric is defined as NDCG ( πs , y ) = DCG ( πs , y ) DCG ( π∗ , y ) , ( 2 ) where πs is a ranked list induced by the ranking function s on x , π∗ is the ideal list ( where x is sorted by y ) , and DCG is defined as : DCG ( π , y ) = n∑ i=1 2yi − 1 log2 ( 1 + π ( i ) ) = n∑ i=1 Gi Di ( 3 ) In practice , the truncated version that only considers the top-k ranked items , denoted as NDCG @ k , is often used . 2.2 LAMBDAMART . LTR models have evolved from linear models ( Joachims , 2002 ) , to nerual networks ( Burges et al. , 2005 ) , and then to decision trees ( Burges , 2010 ) in the past two decades . LambdaMART , proposed about ten years ago ( Wu et al. , 2010 ; Burges , 2010 ) , is still treated as the “ state-of-the-art ” for LTR problems in recent papers ( Bruch et al. , 2019b ; Zhu & Klabjan , 2020 ) . It is based on Gradient Boosted Decision Trees ( GBDT ) . During each boosting step , the loss is dynamically adjusted based on the ranking metric in consideration . For example , ∆NDCG is defined as the absolute difference between the NDCG values when two documents i and j swap their positions in the ranked list sorted by the obtained ranking functions so far . ∆NDCG ( i , j ) = |Gi −Gj | · ∣∣∣ 1 Di − 1 Dj ∣∣∣ . ( 4 ) Then LambdaMART uses a pairwise logistic loss and adapts the loss by re-weighting each item pair in each iteration , with s ( x ) |i being the score for item i and α being a hyperparameter : l ( y , s ( x ) ) = ∑ yi > yj ∆NDCG ( i , j ) log2 ( 1 + e −α ( s ( x ) |i−s ( x ) |j ) ) ( 5 ) There are two popular public implementations of LambdaMART , namely λMARTGBM and λMARTRankLib . λMARTGBM is more recent than λMARTRankLib and has more advanced features by leveraging novel data sampling and feature bundling techniques ( Ke et al. , 2017 ) . However , recent neural LTR papers either use the weaker implementation of λMARTRankLib ( Pang et al. , 2020 ; Wang et al. , 2017a ; Ai et al. , 2018 ; 2019 ) , or acknowledge the inferior performance of neural models when compared with λMARTGBM ( Bruch et al. , 2019b ) . Such an inconsistency makes it hard to determine whether neural models are indeed more effective than the tree-based models .

Summary: - The paper argues that neural models perform significantly worse than GBDT models on some learning to rank benchmarks. It first conducts a set of experiments to show that GBDT outperforms some neural rankers. Then, it presents a few tweaks related to feature transformation and data augmentation to improve the performance of neural models. The resulting neural models perform on par with the state-of-the-art GBDT models.