Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees?
Authors: Zhen Qin
Le Yan
Honglei Zhuang
Yi Tay
Rama Kumar Pasumarthi
Xuanhui Wang
Michael Bendersky
Marc Najork
Venue: ICLR 2021 Spotlight (2021)
Keywords: Learning to Rank
benchmark
neural network
gradient boosted decision trees

Abstract: Despite the success of neural models on many major machine learning problems, their effectiveness on traditional Learning-to-Rank (LTR) problems is still not widely acknowledged. We first validate this concern by showing that most recent neural LTR models are, by a large margin, inferior to the best publicly available Gradient Boosted Decision Trees (GBDT) in terms of their reported ranking accuracy on benchmark datasets. This unfortunately was somehow overlooked in recent neural LTR papers. We then investigate why existing neural LTR models under-perform and identify several of their weaknesses. Furthermore, we propose a unified framework comprising of counter strategies to ameliorate the existing weaknesses of neural models. Our models are the first to be able to perform equally well, comparing with the best tree-based baseline, while outperforming recently published neural LTR models by a large margin. Our results can also serve as a benchmark to facilitate future improvement of neural LTR models.

Published as a conference paper at ICLR 2021
ARE NEURAL RANKERS STILL OUTPERFORMED BY GRADIENT BOOSTED DECISION TREES?
Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Michael Bendersky, Marc Najork Google Research {zhenqin, lyyanle, hlz, bi,,yitee yitay, ranakumarakk,r xuanhui bemike, jemmkkkooooo com
ABSTRACT
Despite the success of neural models on many major machine learning problems, their effectiveness on traditional Learning-to-Rei to-iing (LTR) problems is still not widely acknowledged. We first validate this concern by showing that most recent neural LTR models are, by large margin, inferior the best publicly available Gradient Boosted Decision Trees (GBDT) in terms of their reported ranking ac- curacy benchmark datasets. This unfortunately was somehow overlooked in recent neural LTR papers. We then investigate why existing neural LTR models under-perform and identify several of their weaknesses. Furthermore, we propose a unified framework comprising of counter strategies to ameliorate the existing weaknesses of neural models. Our models are the first to be able to perform equally well, comparing with the best tree-based baseline, while outperforming recently published neural LTR models by a large margin. Our results can also serve as benchmark to facilitate future improvement of neural LTR models.
INTRODUCTION
Neural approaches have been dominating in many major machine learning domains, such com- puter vision (He et al. 2015), natural language processing (Devlin et al. 2019). and speech recogni- tion (Hannun al.) 2014). However, the effectiveness of neural approaches in traditional Learning- to-Rank (LTR), the long-establishe inter-disciplinary research area at the intersection of machine learning and information retrieval (Liu 2009), is not widely acknowledged (Yang al. 2019). pecially on benchmark datasets that have only numerical features.
Historically, series of LTR models were developed by rescarchers Microsoft, starting with RankNet (Burges et al. 2005) and LambdaRank (Burges et al. 2007). both based on neural net- works, and culminating Decision Trees (GBDT); Burges (2010) provides an overview of this evolution. There are two pub- licly available implementations of LambdaMART: one provided by the RankLit of the Lemur Project (henceforth referred to as AMART ); and the LightGBM iimpleeentaiion provided by Microsoft (Ke et al. 2017) (henceforth referred as AMART GBM). As we will show in Section 3 AMART substantially outperforms AMART Ramiiib-
LambdaMART (Wu al. (0100) which is based on Gradient Boosted
that part
There is strong and continuing interest in neural ranking models, with numerous papers published in the last few years alone. Most of these papers treat RankNet and LambdaRank as weak base- lines (Pang et al.) 2020 BBuch et al.) (019b) and LambdaMART as the "state-of-theeanr" (Bruch) et al. 2019b Li et al., 2019; Zhu & Klabjan 2020 Hu et al. 2019). However, when examin- ing these papers, we note that they either acknowledge their inder-performance to AMARTGBM or claim state-of-the-an performance by comparing to weaker XMART RamiLab implementation. The inconsistency of performance evaluation on benchmark datasets in this field has made it difficult to measure progress (Lipton & Steinhardi] 2018). It therefore remains open question whether neural LTR models are as effective as they claim to be, and how to improve them if that is not the case.

---

Published as a conference paper at ICLR 2021
In this paper, first conduct benchmark to show that GBM outperforms recently published neural models, as well as the AMART RamkLib. by a large margin. While the neural paradigm is still appealing in myriad of ways, such as being composable, flexible, and able to benefit from a plethora of new advances (Vaswani et al. 2017; Devlin et al., 2019), the research progress in neural ranking models could be hindered due to their inferior performance tree models. It thus becomes critical to understand the pitfalls of building neural rankers and boost their performance on benchmark datasets.
Specifically, we investigate why neural LTR approaches under-perform on standard LTR datasets and identify three major weaknesses that are typically ignored by recent work. First, neural models are not as adept at performing effective feature transformations and scaling, which is one major benefit of using tree-based methods (Saherian et al., (019). In ranking data which typically long- tailed, this can be prohibitive property. Second, standard feed-forward networks are ineffective in generating higher-order features as noted by recent papers (Wang et al. 2017b Beutel et al.) 2018). More effective network architectures for neural LTR models are needed. Third, recent neural LTR work on benchmark datasets does not employ high-capacity networks, key success factor of many neural models (Devlin al., 2019), possibly due to small scale of training data that causes overfitting. On the other hand, there are several potential benefits of neural approaches over LambdaMART for LTR, such their flexibility
model listwise data and the existence of many techniques to mitigate data sparsity. To that end, we propose a new framework that ameliorates the weaknesses of existing neural LTR approaches and improves almost all major network components.
In the proposed framework, we make several technical contributions: (1) We demonstrate empirical evidence that simple log Ip transformation on the input features is very helpful. (2) We use data augmentation (DA) to make the most out of high-capacity neural models, which is surprisingly the first work in the LTR literature to do so. We show that adding a simple Gaussian noise helps, but only when the model capacity is appropriately augmented (which probably explains why there is no prior work on such a simple idea). (3) We use self-attention (SA) to model the listwise ranking data as context, and propose to use latent cross (L.C) to effectively generate the interaction of each item and its listwise context.
We conduct experiments on three widely used public LTR datasets. Our neural models are trained with listwise ranking losses. On all datasets, our framework can outperform recent neural LTR methods by large margin. When comparing with the strong LambdaMART implementation, AMARTGaM. we are able to achieve equally good results, if not better. Our work can also serve benchmark for neural ranking models, which believe lay fertile ground for future neu- ral LTR research, as rigorous benchmarks on datasets such as ImageNet (Russakovsky et al.) 2015) and GLUE (Wang et al. Ba do in their respective fields.
2 BACKGROUND
We provide some background on LTR, including its formulation and common metrics. We review LambdaMART and highlight its two popular implementations which are causes of the inconsistency of evaluations in the recent literature.
2.1 LEARNING TO RANK
LTR methods are supervised techniques and the training data can be represented a set Vp ((x,)) E x" x R))). where x is a list of items I; E X and y is a list of relevance labels y E IR for 1 < isis n. We use x the universe of all items. In traditional LTR problems, each a; corresponds to a query-item pair and is represented as a feature vector in & where k is the number of feature dimensions. With slightly abuse of notation, we also use 2; as the feature vector and say x E R The objective is to learn a function that produces an ordering of items in x so that the utility of the ordered list is maximized.
Most LTR algorithms formulate the problem as learning a ranking function to score and sort the items in list. As such, the goal of LTR boils down finding parameterized ranking function
2

---

Published as a conference paper at ICLR 2021
s(; X" - RR", where @ denotes the set of parameters, minimize the empirical loss:
[(s)
1
Σ
(yy,(()),
(1)
(x,))E
where ((-) is the loss function on single list. LTR algorithms differ primarily in how they parame- terize 8 and how they define I.
There many existing ranking metries such as NDCG and MAP used in LTR problems. A com- mon property of these metrics that they are rank unn-deepndeent and place more emphasis on the top ranked items. For example, the commonly adopted NDCG metric is defined
DCGG(T..))
DCG((",y)
where π, is ranked list induced by the ranking function s on x, x* is the ideal list (where x is sorted by y), and DCG is defined as:
2
G
(T = Σ
Σ
(3)
loga (1 Ti))
D;
1=1
In practice, the truncated version that only considers the top-k ranked items, denoted as NDCG@K, often used.
2.2 LAMBDAMART
LTR models have evolved from linear models (Joachims 2002 to nerual networks (Burges et al, 2003), and then to decision trees (Burges) (010) in the past two decades. LambdaMART, proposed about ten years ago (Wu et al. 2010; Burges 2010). is still treated as the "state-of-the-art" for LTR problems in recent papers (Bruch et al.) 2019b; Zhu & Klabjan 2020). It is based Gradient Boosted Decision Trees (GBDT). During each boosting step, the loss is dynamically adjusted based on the ranking metrie in consideration. For example, ÁNDCG is defined as the absolute difference between the NDCG values when two documents and j swap their positions in the ranked list sorted by the obtained ranking functions so far.
1 D3
ANDCG = IG, G,l
(4)
Then LambdaMART uses pairwise logistic loss and adapts the loss by re-weighting each item pair in each iteration, with ( being the score for item and a being a hyperparameter:
i(y, sx)) = Σ ANDCG((,)) logz(1 +e -
(5)
There are two popular public implementations of LambdaMART, namely XMART and AMART RamtLib- AMARTGви is more recent than AMART RamaLib and has more advanced features by leveraging novel data sampling and feature bundling techniques (Ke et al.) 2017). However, re- cent neural LTR papers either use the weaker implementation of AMART (Pang et al. 2020 Wang et al., 2017a Ai et al., 2018: 2019), or acknowledge the inferior performance neural mod- els when compared with XMART GBM (Bruch al. 2019b). Such an inconsistency makes it hard to determine whether neural models are indeed more effective than the tree-based models.
RamaLib
3 BENCHMARKING EXISTING METHODS
To resolve the inconsistency, we perform benchmark on three popular LTR benchmark datasets to show that: 1) there is a large gap between the two implementations tree-based LambdaMART AMART and AMART RankLiN; 2) Recent neural LTR methods generally significantly worse than the stronger implementation. Then we discuss several weaknesses of recent neural LTR ap- proaches, and point promising directions, which lay the foundation of our proposed framework.
3

---

Published as a conference paper at ICLR 2021
Table 1: The statistics of the three largest public benchmark datasets for LTR models.
#features
#queries
#docs
training
validation
test
training
validation
test
Web30K
136
18,919
6,306
6,306
2,270,296
747,218
753,611
Yahoo
700
19,944
2,994
6,983
473,134
71,083
165,660
Istella
220
20,901
2,318
9,799
6,587,822
737,803
3,129,004
Table All numbers are significantly worse than the corresponding number from AMART at the p < 0.05 level using a two-tailed (-test. Best performing numbers are bold.
Web30K NDCG@k
Yahoo NDCG@k
Istella NDCG@k
Models
Rerank
@
@
10
@
@
@ 10
@
@
@ 10
AMART
X
45.35
44.59
46.46
68.52
70.27
74.58
65.71
61.18
65.91
AMARTcRM CoRM
50.73
49.66
51.48
7L.88
74.21
78.02
74.92
71.24
76.07
RankSVM
30.10
33.50
36.50
63.70
67.40
72.60
52.69
50.41
55.29
GSF
41.29
41.51
43.74
64.29
68.38
73.16
62.24
59.68
65.08
ApproxNDCG
X
46.64
45.38
47.31
69.63
72.32
76.77
65.81
62.32
67.09
DI.CM
46.30
45.00
46.90
67.70
69.90
74.30
65.58
61.94
66.80
SetRank
42.90
42.20
44.28
67.11
69.60
73.98
67.33
62.78
67.37
SetRank re
V
45.91
45.15
46.96
68.22
70.29
74.53
67.60
63.45
68.34
3.1 DATASETS
The three data sets we used in our experiments are public benchmark datasets widely adopted by the research community. They are the LETOR dataset from Microsoft QQin Liu 2013). SSell from the YAHOO LTR challenge (Chapelle & Chang 201T), and Istella (Dato et al. (016)). We call them Web30K, Yahoo, and Istella respectively. All them are data sets for web search ranking and the largest data sets publicly available for LTR algorithms. The relevance labels of documents for each query are rated by human in the form of multilevel graded relevance. See Qin & Liu (2013) for an example list of features, such as the number of URL clicks, or the BM25 scores of the different page sections. An overview of these three datasets is shown in Table
3.2 COMPARISON
We compare comprehensive list of methods in Table XMART GBM Ke et al.) 2017) and AMART RamkLib are the two LambdaMART implementations. RankS VM (Jaachim 2006) is a clas- sic pairwise saaning-to-rank model built SVM. GSF (Ai et al.) 2019) neural model using groupwise scoring function and fully connected layers. ApproXNDCG (Bruch et al. 2019b) is neural model with fully connected layers and differeiable loss that approximates NDCG (Qin al.) 2010). DLCM (Ai et al., 2018) is an RNN based neural model that use list context information to rerank list documents based on AMART RankLiB in the original paper. SetRank (Pang et al., 2020) is a neural model using self-attention to encode the entire list and perform a joint scoring. SetRank (Pang et al. 2020) is SetRank plus ordinal embeddings based on the initial document ranking generated by XMART RambLib as in the original paper.
We choose to compare these methods because they are either popular or recent. The neural mod- els are already leveraging advanced neural techniques such as using neural methods to model the entire ranking list, which difficult for tree-based models to achieve. We reproduced results for AMARTROALA RamkLib. XMARTGBM. RankSVM, GSF, and ApproxNDCG with extensive hyperparameter tuning with more details in Appendix A Results for the DLCM and SetRank methods are from their respective papers where the authors did their tuning. Note that the test set is fixed for all datasets, thus the numbers are comparable.
From Table 2]
see the following. 1) AMART is more appropriate -state-of-the -rt LambdaMARY baseline, as it significantly outperforms AMART Rankia- 2) Recent neural LTR meth- ods, though sometimes outperform XMART Ranktib, are inferior to AMART cam by large margin,
4

---

Published as a conference paper at ICLR 2021
sometimes by as much as 15%, comparatively. These results show the inconsistency of existing methods and validate the concerns on the current practice neural LTR model
4 NEURAL LTR MODELS
A natural question is: why do neural models under-perfome on LTR benchmark datasets compared with LambdaMART, despite their success many machine learning research areas? We first iden- tify few weaknesses of the neural LTR models and then propose our methods to address them.
4.1 WEAKNESSES
By reviewing recent papers and the strength of tree-based models, we give the following hypotheses:
Feature transformation. Neural networks are sensitive to input feature scales and transformations (Saberian et al. 2019). LTR datasets consist of features of diverse scales with long-tail distributions, such as number of clicks of an item. Tree-based models are known to partition the feature space effectively, which is beneficial for datasets (such LTR datasets) with only numeric features. Some recent work already shows the benefits of better input feature transformations than Gaussian normalization (Saberian et al.] 2019 Zhuang et al. 2020). Unfortunately, neither the pioncering neural LTR papers (Burges et al. 2005 2007) nor the most recent ones discuss the impact of feature transformation.
Network architecture. Unless the focus is the neural architecture, neural LTR papers typically standard feed-forward network that consists of stack of fully connected layers. However, fully connected layers are known to be ineffective in generating higher-order feature interactions. The problem has been widely studied in areas such as ads prediction (Wang et al.) (017b) and recommender systems (Beutel et al. 2018), but has not received enough attention for LTR.
Data sparsity. Recent neural LTR models are small and do not employ high-capacity networks (Bruch et al. [019b] Pang et al. 2020), possibly due to the overfitting issue. While large datasets
are key factors to many recent successes of neural models in other domains (He et al. 2015 Devlin
et al. 2019), the publicly available LTR datasets are comparatively small. Popular techniques such as data augmentation to mitigate overfitting in high-capacity networks are commonly used other areas (Perez & Wang 2017). But it is less intuitive on how to do data augmentation for L.TR datasets, compared with, c.g., rotating a cat image in computer vision.
4.2 IMPROVEMENT
We introduce our proposed neural LTT framework that tries address the above mentioned con- cerns. Figure [ summarizes our DASALC framework, which stands for Data Augmented Self- Attentive Latent Cross ranking network.
4.2.1 EXPLICIT FEATURE TRANSFORMATION AND DATA AUGMENTATION
Features in LTR datasets are diverse and can be of different scales. Out of the three datasets we consider, only the Yahoo dataset has been normalized (we leave it not-transformed). It is well known that neural networks are sensitive to input data scale, and we apply a simple "log transformation every element of x and empirically find it works well for the Web30K and Istella datasets:
log, (1 + |x) sign(x).
where is the slement-wise multiplication operator.
We use a very simple data augmentation technique on LTR datasets. We add a random Gaussian noise independently every element of input vector x:
x (0, o²I)
We use the Foldl in Web30K to be consistent with the setup of Yahoo and Istella, Some of the reported results on Wch30K were based on 5-fold vras-validation (CV). We verified on AMART Rmotss that the difference between Fold1 and CV is small and does affect our conclusion.
5

---

Published as a conference paper at ICLR 2021
Ranking Head
Softmax Loss
Score s;
FC
ReLU
5
y1 ...
...
yn
Latent Cross
(C)
Self Attention Layers
FC-ReLU-BN
Layer Normalization
MHSA
FC-ReLU-BN
Gaussian Noise
Log Transform
Document List
Figure 1: An illustration of the DASALC. FC is fully connected layer, ReLU is ReLU activation, and BN indicates batch normalization. Loglp Transform is applied when applicable. Softmax loss is short for softmax output with cross-entropy loss.
where is scalar hyperparameter. The random noise is added after the loglp transformation in online fashion during training (i.e. different perturbations will be added to the same data point seen in different batches). A single scalar for every feature is reasonable because the feature distri- butions are normalized by log Ip. Also data augmentation is added after input Batch Normalization (BN) when applicable. Note that the random noise is added independently every element (later) BN will not cancel it away. We find such a simple data augmentation technique works well in our framework, but as shown experiments, only works when the capacity of the network properly augmented as described in the next section.
For notation simplicity, combine the loglp feature transformation and data augmentation into single function f I x +> a
f log, (1+ (x) o sign(x) + +0, o²1)
(8)
4.2.2 LEVERAGING LISTWISE CONTEXT
For LTR problem, the list of documents be leveraged in neural models. This is the key base to enhance the network architecture for LTR. We leverage the multi-heal self-attention (MHSA) mechanism /Vaswani al. 2017) encode ranking list information. More specifically, we generate a contextual embedding a; for each item i, considering the document similarity between document i and every document the list. For the multi-head self-attention mechanism, we have the input f E | and project f into query (in the context of attention mechanism) matrix Q = fWQ fW- and a value matrix V rw V with trainable projection matrices Wº, w and WV E x where is the attention head size. Then self-attention (SA) head computes the weighted sum the transformed values V as,
key matrix
SA (f) (of)m
(9)

---

Published as a conference paper at ICLR 2021
where similarity matrix between Q and is defined as S(() QK" from the H heads are concatenated to form the output of multi-head self-attention by
y== For each layer, the results
(f) concal (S (SA (f) + bout.
(10)
where W out E H and but E R are trainable parameters. We apply 2 layers of multi- head self-attention followed by layer normalization (Ba et al.) 2016) similarly to (Vaswani et al. 2017
By treating a, as the listwise contextual embedding for item i, we further leverage the simple latent cross idea (Beutel et al. 2018) to effectively generate feature interactions:
h crooss (1 a.) hout (z:),
(11)
where @ is the element-wise multiplication operator (a; will go through a linear projection when the dimensions do not match, omitted in the equation), and (xg) is the output of the final hidden layer of regular network.
Learning to rank can be seen learning induce order over set of items. One desirable property for ranking approaches that use listwise context is to be permutation equivariant: applying per- mutation over input items leads to an equivalent permutation over output scores. DASALC satisfies such a permutation equivariance property.
Proposition 1. Let π be a permutation of indices of [1..., and x E R be the input item representation. DASALC is permutation equivariant for scores generated over input items, i.e, SDASALC(((x))= = π (SDAAALc(x)) See proof at Appendix C
4.3 REMARKS
We compared several popular pointwise. pairwise, and listwise ranking losses. We report all results eti
based on the softmax cross entropy loss (()))(x)) = Σ 1 yo loga Σj since it is simple and empirically robust in general, demonstrated in Appendix B.2
We provided a general framework that can enhance neural LTR models in many components. For each component, purposefully use simple well-known techniques for enhancement because the scope of the current research is to identify the possible reasons why neural LTR is under- pedderperforming when compared with the best traditional tree-based methods. Clearly, each component can use more advanced techniques, such as learning a more flexible data transformation (Zhuang et al., 2020) or using data augmentation policy (Cubuk et al. 2019), which leave as future work.
EXPERIMENTS
We conduct experiments the three LTR datasets (introduced in Sec with proposed frame- work and compare with some methods in Sec 3 For all our experiments using neural network approaches, implemented them using the TF-Ranking (Pasumarthi al., 2019) library.
We use two variants of our proposed approaches. DASALC is a model trained in our proposed framework. DASALC-ens is ensemble of DASALC. By realizing LambdaMART an ensemble method based on boosting, we leverage the randomness of neural model training and simply use the average score of 3-5 models (tuned validation set) from different runs as the final score in DASALC-ens.
Main result. The results are summarized in Table We focus on the comparison with AMART GBM and also include SetRank to highlight the difference with recent neural LTR models. Readers can refer to Table 2 for more results. We tune hyperparameters on the validation sets, with more details in Appendix We have the following observations and discussions: (1) DASALC can sometimes achieve comparable or better results than AMART and outperforms recent neural LTR methods by large margin.
(2) DASALC-ens, though simple, can achieve neutral significantly better results than AMART on all datasets and metrics.
(3) The results on Yahoo dataset are weaker than the other two datasets. One thing to note is Yahoo dataset is already normalized upon release. As we note the importance of input feature transformation, the provided normalization may not be ideal for neural models, thus it should be encouraged release LTR datasets with raw feature values.
7

---

Published as a conference paper at ICLR 2021
Table 3: Result on the Web30K, Yahoo, and Istella datasets. means significantly better result, performanced against AMARTGBM the p < 0.05 level using a two-tailed f-test. Last row is
relative difference of DASALC- over AMARTGBM.
Web30K NDCG@K
Yahoo NDCG@K
Istella NDCG@k
Models
69
WS
@10
@
611
60
W5
w1O
AMARI GaM
50.73
49.66
51.48
71.88
74.21
78.02
74.92
71.24
76.07
SetRank"
45.91
45.15
46.96
68.22
70.29
74.53
67.60
63.45
68.34
DASALC
50.95
50.92
52.88
70.98
73.76
77.66
72.77
70.06
75.30
DASALCCens
51.89
51.72
53.73
71.24
74.07
77.97
74.40
71.32
76.44
(Relative diff)
(+2.29%)
(+4.15%)
(+4.37%)
(-0.89%)
(-0.18%)
(-0.06%)
(-0.69%)
(+0.11%)
(+0.49%)
Table 4: NDCG@5 on Istella when different components are added.
Model
DNN
+loglp
+SA
+LC
+DA
+ens
NDCG@5
64.72
67.09
68.32
68.80
70.06
71.32
Ablation study. We provide some ablation study results in Table 4 to highlight the effectiveness of each component in our framework. Each component is added cumulatively from left to right in the table. We can see that each component helps and the best performance achieved when all components are combined. More detailed ablation study is provided in Appendix B Appendix [ gives more results on the effect of the log Ip transformation. Appendix BanddixB.210 compares different loss functions and shows that listwise ranking loss performs better. Appendix B.3 shows the benefit of effective listwise context modeling. model architectures.
B.4 the effect of data augmentation different
RELATED WORK
We focus on traditional LTR problems when there are only numeric features and human ratings available. Some works (Mitra & Craswell 2018 Nogueira et al. 2019 Han et al. 2020) on doc- ument matching and ranking leverage neural components such as word2vec and BERT when raw text is available, where the major benefit comes from semantic modeling of highly sparse input and tree-based methods become less relevant due to its limitation in handling sparse features.
The pioneering neural LTR models are RankNet (Burges et al. 2005) and LambdaRank (Burges) al. 2007). They use feed-forward networks on dense features as their scoring functions and Sourges,2010/
Recent neural LTR models have explored new model architectures (Pang et al. 2020 Qin et al. 2020b), differetiable losses (Bruch jet al. 2019b), and leveraging more auxiliary information (Ai al. 2018). However, there less work that specifically understands and addresses weaknesses for neural LTR, and a benchmark with strong tree-based baseline is missing. In this work, we show that relatively simple components that aim to address weaknesses of neural models can outperform recent methods significantly.
became less favored than tree-based LambdaMART
The idea of generating new data for LTR has been explored in few work recently, but their focus is train more discriminative ranking models, not to mitigate the data sparsity problem for high- cephccap neural models. For example. Yu & Lam 2019) uses a separate Autoencoder model generate data and then feed them into based models. This work can be treated as orthogonal to our data augmentation technique.
Several LTR papers have leveraged neural sequence modeling based on LSTM (Ai et al. 2018) or self-attention (Pang et al.] 20200 Pasumarthi et al. 20220), which is not easy for tree-based approaches to model. We also leverage listwise context via self-attention to show neural I.TR models are easily extendable. The combination of self-attention based listwise context and latent cross in our work to specifically mitigate the ineffectiveness of neural model generate higher-order feature interactions has not been explored in the literature.
Our work is mostly orthogonal to another line of LTR research, namely unbiased learning to rank from implicit feedback data, such as clicks (Joachims et al.) [0177] ( [u et al. 2019 Qin et al. 2020aa Zhuang et al. 2021). There are also papers that try reproduce tree models using neural architec- tures for tabular data (Saberian et al. 2019 Lee & Jaakkola 2020). Our motivation is different in that our goal is to identify and mitigate weaknesses of neural approaches in general.
8

---

Published as a conference paper at ICLR 2021
7 CONCLUSION AND DISCUSSION
In this paper, we first showed the inconsistency of performance comparison between neural rankers and GBDT models, and verified the inferior performance of neural models. We then identified the weaknesses when building neural rankers in multiple components and proposed methods to address them. Our proposed framework performs competitively well with the strong tree-based baselines. We believe our general framework and the rigorous benchmarking provides critical contribution to facilitate future neural LTR research. In particular, neural models are powerful modeling complex relations (e.g, attention mechanism (Vaswani et al. (017) )) and raw text features (c.g., BERT (Devlin ct al. 2019)). Also, the active research on neural networks in other domains continuously advances neural techniques (e.g., optimizers (Kingma & Ba 2014 All these can be studied in the LTR setting and our work pave ways to avoid pitfalls when leveraging these techniques.
REFERENCES
Qingyao Ai, Keping Bi, Jiafeng Guo, and W Bruce Croft. Learning deep listwise context model for ranking refinement, In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 135-144, 2018.
Qingyao Ai, Xuanhui Wang, Sebastian Bruch, Nadav Golbandi, Michael Bendersky, and Mare Na- jork. Learning groupwise multivariate scoring functions using deep neural networks. In Proceed- ings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, pp- 85 85-92, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 2016.
Alex Beutel, Paul Covington, Sagar Jain, Can Xu, Jia Li, Vince Gatto, and Ed H Chi. Latent cross: Making use of context in recurrent recommender systems. In Proceedings of the Eleventh ACM International Conference Web Search and Data Mining, pp. 46-54, 2018
Sebastian Bruch, Xuanhui Wang, Michael Bendersky, and Mare Najork. An analysis of the softmax cross entropy loss for learning-to-rank with binary relevance. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, pp. 75-78, 2019a
Sebastian Bruch, Masrour Zoghi, Michael Bendersky, and Mare Najork. Revisiting approximate metric optimization in the age of deep neural networks. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1241-1244, 2019b.
Sebastian Bruch, Shuguang Han, Michael Bendersky, and Mare Najork. A stochastic treatment of learning to rank scoring functions. In Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 61-69, 2020.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hul- lender. Learning to rank using gradient descent. In Proceedings of the 22nd International Con- ference on Machine Learning, pp. 89-96, 2005.
Christopher J. Burges, Robert Ragno, and Quoe V. Le. Learning to rank with nonsmooth cost functions. B. Schölkopf, J. C. Platt, and Hoffman (eds.), Advances in Neural Information Processing Systems 19, pp. 193-200. 2007.
Christopher JC Burges. From RankNet to LambdaRank to LambdaMART: An overview. Learning, (1(23-581):811 2010.
Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning, pp. 129-136. 2007.
Olivier Chapelle and Yi Chang. Yahoo! learning to rank challenge overview. In Proceedings of the Learning Rank Challenge, pp. 1-24, 2011.

---

Published as a conference paper at ICLR 2021
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V. Le. Autoaugment: Learning augmentation strategies from data. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 113-123, 123, 2019.
Domenico Dato, Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. Fast ranking with additive ensembles of oblivious and non-oblivious regression trees. ACM Trans. Inf. Syst., 2016.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Aditya Grover, Eric Wang, Aaron Zweig, and Stefano Ermon. Stochastic optimization of sorting networks via continuous relaxations. arXiv preprint
08850, 2019.
Shuguang Han, Xuanhui Wang, Mike Bendersky, and Mare Najork. Seaming-to-rant with bert in tf-ranking. arXiv preprint wxXiv:200.088776 2020.
Awni Hanmun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv: 1412.556, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arXiv preprint arXiv: 152.003385, 2015.
Ziniu Hu, Yang Wang, Qu Peng, and Hang Li. Unbiased lambdamart: unbiased pairwise learning-to-ran tork algorithm. In The Web Conference, pp. 2830-2836 2019.
Thorsten Joachims. Optimizing search engines using clickthrough data. In Proceedings the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 133-142, 2002.
the Eighth
Thorsten Joachims. Training linear svms in linear time. In Proceedings of the 12th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 217-226, 2006.
Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. Unbiased learning-to- -oriing-to-rank with bi- ased feedback. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, pp. 781-789, 2017.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie-
Yan Liu. Lightghm: A highly efficient gradient boosting decision tree. In Proceedings of the 1st International Conference on Neural Information Processing Systems, pp. 3149-3157, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Guang-He Lee and Tommi S. Jaakkola. Oblique decision trees from derivatives of relu networks. In 8th International Conference on Learning Representations, 2020.
Pan Li, Zhen Qin, Xuanhui Wang, and Donald Metzler. Combining decision trees and neural net- the works for learning-to-rant personal search. Proceedings of the 25th ACM SIGKDD Interna- lional Conference on Knowledge Discovery & Data Mining, pp. 2032-2040, 2019.
Zachary C Lipton and Jacob Steinhardt. Troubling trends in machine learning scholarship. arXiv
preprint 1807.038 2018.
Tie-Yan Liu. Learning to rank for information retrieval. Found. Trends Inf. Retr. 2009.
Bhaskar Mitra and Nick Craswell. An introduction to neural information retrieval. Foundations and Trends in Information Retrieval, 13(1):1-126, 2018
Rodrigo Nogucira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. Multi-stage document ranking with bert. arXiv preprint arXiv: 19/0.144/0 2019.
10

---

Published as a conference paper at ICLR 2021
Liang Pang, Jun Xu, Qingyao Ai, Yanyan Lan, Xueqi Cheng, and Jirong Wen. Setrank: Learning a cormuutttioninvvarian ranking model for information retrieval. In Proceedings of the 43rd In- ternational ACM SIGIR Conference Research and Development in Information Retrieval, pp- 499-508, 2020.
Rama Kumar Pasumarthi, Sebastian Bruch, Xuanhui Wang, Cheng Li, Michael Bendersky, Mare Najork, Jan Pfeifer, Nadav Golbandi, Rohan Anil, and Stephan Wolf. TF-Ranking: Scalable tensorflow library for learning-to-rank. In Proceedings of the 25th ACM SIGKDD Interna-tional Conference on Knowledge Discovery & Data Mining, pp. 2970-2978, 2019.
Rama Kumar Pasumarthi, Honglei Zhuang, Xuanhui Wang, Michael Bendersky, and Marc Najork. Permutation equivariant document interaction network for neural learning to rank. In Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval, pp- 145-148, 2020.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning, arXiv preprint arXiv:1712.0400 2017.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In Advances in neural information processing systems, pp. 6638-6648, 2018.
Tao Qin and Yan Liu. Introducing LETOR 4.0 datasets. CoRR, abs/ NeW/1336629 2013.
Tao Qin, Tie-Yan Liu, and Hang Li. A general approximation framework for direct optimization of information retrieval measures. Information retrieval, 13(4):375-397, 2010.
Zhen Qin, Suming J. Chen, Donald Metzler, Yongwoo Noh, Jingzheng Qin, and Xuanhui Wang. Attribute-based propensity for unbiased learning in recommender systems: Algorithm and case studies. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Dis- covery & Data Mining, pp. 1359-23677 2020a.
Zhen Qin, Zhongliang Li, Michael Bendersky, and Donald Metzler. Matching cross network for learning to rank in personal search. In The Web Conference, pp. 2835-2841, 2020b.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJV), 11(3)2211-252, 2015.
Mohammad Saberian, Pablo Delgado, and Yves Raimond. Gradient boosted decision tree neural network. arXiv preprint 19Xiv:1910.093400, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv:1706.03:: 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint
2018a.
Jun Wang, Lantao Yu, Weinan Zhang, Yu Gong, Yinghui Xu, Benyou Wang, Peng Zhang, and Dell Zhang. Irgan: A minimax game for unifying generative and discriminative information retrieval models. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 515-524, 2017a
Ruoxi Wang. Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. In Proceedings of the ADKDD' 17, pp. 1-7. 2017b.
Xuanhui Wang, Cheng Li, Nadav Golbandi. Michael Bendersky, and Mare Najork. The lambdaloss framework for ranking metric optimization. In Proceedings of the 27th ACM International Con- ference Information and Knowledge Management, pp. 1313-1322, 2018b.

---

Published as a conference paper at ICLR 2021
Qiang Wu, Christopher J. C. Burges, Krysta M. Svore, and Jianfeng Gao. Adapting boosting for information retrieval measures. Information Retrieval Journal, 13:254-2700 2010.
Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. Critically examining the "neural hype": Weak baselines and the additivity of effectiveness gains from neural ranking models. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1129-1133 2019.
Qian Yu and Wai Lam. Data augmentation based adversarial autoencoder handling imbalance for learning to rank. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 411-418, 2019.
Xiaofeng Zhu and Diego Klabjan. Listwise learning to rank by exploring unique ratings. In Pro- ceedings of the 13th International Conference on Web Search and Data Mining, 2020.
Honglei Zhuang, Xuanhui Wang, Michael Bendersky, and Mare Najork. Feature transformation for neural ranking models. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. (649-1652) 2020.
Honglei Zhuang, Zhen Qin, Xuanhui Wang, Mike Bendersky, Xinyu Qian, Po Hu, and Chary Chen. Cross-positional attention for debiasing clicks. In The Web Conference, 2021.
APPENDIX
A HYPERPARAMETER TUNING
For AMART GBM. we a grid search for number of trees E [300, 500, 1000), number of leaves {2000, 500, 1000), and learning rate E f0.01, 0.05, 0.1, 0.5).
For our neural models the main hyperparameters are hidden layer size (256, 512, 1024, 2048, 3072, 4096) and number of layers E (3.4.5,0 6} for regular DNN, data augmentation noise E [0, 5.0) using binary search with step 0.1, number of attention layers {3, 4, 6}. and number of attention heads E {2, 3, 4, 5J. The same parameter swept is enabled on the baselines we tried when applicable. One noticeable difference between our work and existing work is that tried large hidden layer size up to 4096 and found that large models work better in general when data augmentation is enabled. We are in the process to release the code and trained models in open-sourced software package.
B ABLATION STUDIES AND ANALYSIS
B. EFFECT OF LOGlP INPUT TRANSFORMATION
We first show that the simple loglp transform can improve performance on the Web30K and Istella datasets (Yahoo dataset has already been normalized). Results in Table based regular DNN models using the softmax cross-entropy loss. The trends are similar for other configurations. We also noted the results in general slightly better than Gaussian normalization due the long-tail nature of LTR dataset features, which we omit here.
Web30K NDCG@k
Istella NDCG@k
Method
@ I
@5
10
@
@
@
10
Without log Ip
44.60
44.99
47.22
67.19
64.72
70.12
With loglp
48.30
48.22
50.35
69.78
67.09
72.49
Table 5: Results on Web30K and Istella using loglp input transformation.
We can see that such simple transformation can bring meaningful gains. In all following sections, use log Ip transformation by default.
12

---

Published as a conference paper at ICLR 2021
B.2 RANKING LOSSES
Many recent progresses of neural LTR are on ranking losses, especially listwise ranking losses (Bruch et al. 2019b 2020 Grover al. 2019). For example, it attractive devise differentiable versions of ranking losses for end-to-end learning. Here we do benchmark of different ranking losses on regular DNN models on different datasets show that (1) Listwise ranking losses are superior choices to pointwise or pairwise losses that are normally used for non-neural LTR models; (2) Performances of state-of-the-ar listwise ranking losses are comparable; (3) The softmax cross entropy loss is simple but robust choice.
We consider the following ranking losses:
SigmoidCrossEnnrroyy: widely used pointwise loss: 1(y, sxx)) Ei=1 -3i8t + +loge (1 + e*))
RankNet (Burges et al. 2005): a popular pairwise loss: 1(y, = Enio, log, (1 +
e*s-a:).
LambdaRank (Burges et al.
2007
Wang et al.
2018b): the pairwise loss with ANDCG
weight, which is a direct imp empettttnoo of the LambdaMART loss in Eq. (5)
Softmax (Cao et al. 2007
Bruch al. 2019a): a popular listwise loss: I(y, sx))
Ei= M/ loge e
ApproxNDCG (Qin et al. entiable approximation of NDCG metrie: ((y))(x))
2010 Bruch et al. 2019b): a listwise loss that is a differ-
Dect= Ei= (1ga(1+*..0))
21
where n. (t) = 1/2 + E, sigmoid( $4m*: with T a smooth parameter.
GumbelApproxNDDG (Bruch et al. 2019b 2020): listwise loss with stochastic treat- ment on ApproxNDCG: scores s in the above NDCG loss function will be substituted by + 9s. with a gumbel noise = log, loge U:)) from U, uniformly sampled in [0, 1].
NeuralSorNDCGGGrrvee et al. 2019): a listwise loss that approximates NDCG metric with the NeuralSort trick: (x))
DCG Eir=1 log, (I+r) where Ps. is an approximate permutation matrix, obtained by NeuralSort trick: P = softmax(((r 2:)s, - E, s, s,D//], with T smooth parameter.
1)
GunbbeNeeuallSortDDGG::: a listwise loss with stochastic treatment of NeuraISortNDDG by replacing the score in neural sort permutation matrix by + Di. where 9i is again sampled from the gumbel distribution. This is new in the literature but not the major focus of this work.
Web30K NDCG@k
Istella NDCG@k
Ranking loss
@ 1
@5
@ 10
@
@5
@ 10
SigmoidCrosEEntropy
47.65
46.85
48.47
67.62
64.46
69.51
RankNet
46.05
46.67
48.98
69.09
66.04
71.81
LambdaRank
45.87
46.55
48.85
68.
65.22
70.88
Softmax
48.30
48.22
50.35
69.78
67.09
72.49
ApproxNDCG
49.31
47.87
49.49
70.05
66.08
70.76
49.53
48.07
49.75
71.78
67.33
71.79
NeGGG
48.66
47.19
48.83
68.92
64.27
69.03
GuumbolNcuralSonNDDG
49.74
48.40
50.22
70.96
67.26
71.92
Table Results on the Web30k and Istella datasets with standard feed-forward network architecture.
The results are summarized in Table 6 For different ranking losses, make grid search over different optimizers with different learning rates: for Adam optimizer, we scan learning rates e (10-4, 10-3 10 1002)))--)) 2); ; for Adagrad optimizer, scan learning rates E {0.01, 0.1, 0.5). When the smooth parameter is applicable, we also scan it E {0.1, 10). We report the results based on best NDCG@5 for different losses.
As we have stated above, we find that: (1) The performance of models trained with listwise losses significantly better than the models trained with pointwise pairwise losses. (2) Different
13

---

Published as a conference paper at ICLR 2021
listwise losses generally comparable, and we found that the softmax cross-entropy loss performs coherently well over different models and different datasets. It is thus used in our main results and following sections. (3) LambdaRank does not work well for neural models. On the other hand, previous work (Bruch et al. 2019a) shows that tree-based models with softmax loss are not as good as LambdaMART, demonstrating that tree-based models and neural LTR models have different behavior on different loss functions. This encourages future work to design neural LTR specific ranking losses.
B.3 EFFECT OF LISTWISE CONTEXT
We study the effect of leveraging listwise context with self-attention, with and without latent cross (concatenation between item feature and context feature will be applied) (Pasumarthi et al. 2020) on the Web30K and Istella datasets. Results are shown in Table 7 We can see that using neural approach to model listwise context, which is difficult for tree- bas- models to do, is quite beneficial. Latent cross, though simple, can help leverage listwise context more effectively.
Wch30K NDCG@K
Istella NDCG@k
Method
@1
@
@ 10
@ 1
@5
@ 10
DNN
48.30
48.22
50.35
69.78
67.09
72.49
Sell-attention
49.89
50.13
52.18
71.77
68.32
73.72
Self-attention and T.C
50.19
50.49
52.47
72.19
68.80
74.21
Table 7: Results on the Wcb30K and Istella datasets using self-attention and latent cross.
B.4 EFFECT OF DATA AUGMENTATION
One of the technical findings in this work is that using a simple Gaussian noise as data augmentation can help neural LTR models. Below we add Gaussian noise with different strength (a) to both DNN model and the DASALC framework with results shown Table
8
We can see that the performance
Web30K NDCG@k
Method (σ)
@
@3
@ 10
DNN(0.0)
48.30
48.22
50.35
DNN(O.1)
48.10
48.01
50.14
DNN(L.0)
46.39
46.15
48.18
DASALC(0.0)
50.19
50.49
52.47
DASALC(O.1)
50.38
50.61
52.56
DASALC(L55
50.95
50.92
52.88
DASALC(2.0)
50.65
50.78
52.68
Table 8: Results the Web30K datasets using different architecture and random noise strength.
of DNN starts to drop as soon as we start to add noise. However, for DASALC, data augmentation helps and the performance looks robust using different levels of noise. The performance peeks around a = 1.5. The optimal a needs tuning for different datasets but the general trends are similar for other datasets. We treat the study of the exact mechanism of how data augmentation works in DASALC and the application of more sophisticated data augmentation techniques as future work.
We also try to add noise to AMART GBM dataset is shown in Table 9 we can see that adding noise leads to worse accuracy.
GBM and see similar results as DNN. The results on the YAHOO
Yahoo NDCG@k
Method (a)
@55
@ 10
AMART (0.0
71.88
74.21
78.02
AMART (0.1)
70.10
72.60
77.19
AMARToaM(1O.
64.96
67.28
72.60
AMART (1.5)
64.39
66.84
72.27
Table 9: Results on the Yahoo datasets using different architecture and random noise strength.
14

---

Published as a conference paper at ICLR 2021
B.5 PERFORMANCE ON CATBOOST
We mainly compared with AMART Ranktab and AMARTGRM in the main content since they are the most popular baselines used in recent papers. There are other GBDT implementations that can also be used for the LTR task. Catboost (Prokhorenkova et al. 2018 is a recently popular GBDT implementation for various tasks. We also evaluate its performance on the three LTR datasets. Note that Catboost is not specific to ranking and does not have a standard LambdaMART implementation the best of our knowledge. We try both the QueryRMSE loss and YetiRank loss, which are the best performing losses on most existing Catboost's 's benchmarks. The results are reported in Table I0)
Table 10: Comparison of Catboost with other methods on the Web30K, Yahoo, and Istella datasets.
Web30K NDCG@k
Yahoo NDCG@k
Istella NDCG@k
Models
@
@5
@ 10
@5
@ 10
@
@5
@
AMART RankLib
45.35
44.59
46.46
68.52
70.27
74.58
67.71
61.18
65.91
XMART GBM
50.73
49.66
51.48
71.88
74.21
78.02
74.92
71.24
76.07
Catboost Rurbooos--QeyyyRMEEE
50.07
50.04
51.97
70.50
74.25
78.31
69.91
67.73
72.18
Catboost- YetiRank
48.92
49.10
51.31
69.86
74.00
78.11
72.06
69.97
74.12
DASALC-ens
51.89
51.72
53.73
71.24
74.07
77.97
74.40
71.32
76.44
We can see that Catboost can produce very decent results, clearly outperforming AMART RankLib but its comparison with AMART is mixed. We encourage researchers to also consider different implementations such as Catboost in future LTR work.
B.6 LAMBDAMART ENSEMBLE
We showed that simple ensemble of neural rankers can bring meaningful gains, leveraging the stochastic nature of neural network learning. On the other hand, LambdaMART itself is an en- semble algorithm using boosting, but it is still interesting to see the effect of ensembling multiple LambdaMART models. We conduct additional experiments on this front using AMART and have two major observations: tion generates very similar results, and ensemble in this setting does not help, whereas neural rankers can benefit from such simple setting; 2) In Table IT we show ensembling LambdaMART with dif- ferent configuration (e.g.. different # trees, # leaves and learning rate) the Istella dataset. We ensemble five LambdaMART models chosen on the validation set. The results on other datasets are similar.
Running LambdaMART multiple times with the same configura-
Istella NDCG@k
Method
@
@
@10
caA GaM
74.92
71.24
76.07
AMART Gan-Rsa
75.04
71.40
76.28
Table 11: Results the Istella datasets using LambdaMART ensembles.
We can that the improvement from ensembling LambdaMART is smaller than that in neural rankers (see Table 3). Our hypothesis is that model ensembles tend to be more effective for neu- ral rankers with stronger stochastic nature, and exploring advanced model ensemble methods with neural rankers is an interesting future direction.
C PERMUTATION EQUIVARIANCE ANALYSIS
For any general scoring function s(x): Raxt - R",, and permutation x over indices [1,..., we call be permutation equivariant iff
((m(x) =(s(x))
(12)
The scoring function for proposed approach, DASALC, can be written as a combination of fea- ture transformation and data augmentation function f, output of multi-headed self-attention
15

---

Published as a conference paper at ICLR 2021
(f) and output of final layer of regular network hout (x).
SDASALC(X) (e +x())) & hour
(13)
Note that per-item transformations, which we refer to univariate transformations, are trivially permutation equivariant. Also, composition of two permutation equivariant functions is also per- mutation equivariant, as the permutation operator and the permutation equivariant functions commutative. Hence linear projection, ReLU activation and (as a function of x) are permutation equivariant. Multi-headed self-attention is shown to be permutation equivariant (Pang et al., 2020). Hence, on applying permutation π to the proposed scoring function, we see that it satisfies the per- mutation equivariance property.
(X)
- =WWTo (eLU((1+axx) @ hour (x))
WTo (eLU(T((1+a(x)) @ Hoar (x))
-WFc (e 2 π (houx (x))
(ECCee(((1+aaTT((( hous (x)))
-SDASALC(T(X))
16


Reviews:
[{"review_id": "qvFAiTJJDFS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- The paper argues that neural models perform significantly worse than GBDT models on some learning to rank benchmarks. It first conducts a set of experiments to show that GBDT outperforms some neural rankers. Then, it presents a few tweaks related to feature transformation and data augmentation to improve the performance of neural models. The resulting neural models perform on par with the state-of-the-art GBDT models.\n- I think this paper establishes an unfair comparison between GBDT and neural-based models. As known,  neural models are good at learning great representations from the raw inputs, such as audio, images, and texts, while GBDT models are good at dealing with sparse features. A more fair comparison could be having the neural models to learn feature representations, which will be concatenated with the normalized sparse features.\n- Finally, the technical contribution of the paper is also quite limited.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lANRuesJ3F", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Originality, Significance: This paper establishes reference points for modern LTR research. The fact that RankLib is a very popular but also a weak baseline has been exploited by too many researchers for too long. When I was reviewing other LTR papers, I often had to point out that the proposed method significantly underperforms LightGBM. This fact has been fairly well known, but apparently not widespread enough. Having an ICLR paper published on this issue will help spreading the fact, which is significant on its own.\n\nQuality: Considering the popularity of RankLib, deeper analysis of why LightGBM outperforms RankLib would've been very nice, however. Authors do mention that LightGBM has more features, but it is unclear which exact feature of LightGBM contributed to such a significant difference between the two. Understanding the reason for LightGBM's superiority could potentially help us to develop better LTR models, neural or not neural. Comparison against Catboost https://github.com/catboost/catboost would've also been useful, as it is often claimed to outperform LightGBM.\n\nProposed DASALC framework is quite simple and uses mostly standard techniques, and this is an advantage as a reference point. Still, DASALC significantly outperforms previous neural LTR approaches. Also, although the idea of applying these standard techniques on LTR seems straightforward, but I argue that's only due to the benefit of hindsight; neural LTR has been a fairly active area of research, yet these techniques haven't yet been widely used in LTR literature. It would've been very interesting to see how these techniques improve the performance of previous neural LTR models; log1p transformation, data augmentation, and model ensembling would straightforwardly apply to other neural models as well.\n\nIn summary, I believe this paper will foster more productive research by establishing the strong baseline on both decision-tree based method (although it has been known) and neural method (on which authors make good technical contributions).\n\nClarity: The paper is quite easy to follow.", "extra_scores": {}, "extra_text": {}}, {"review_id": "J90JUPqt5kh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nIn this paper, the authors study the problem of neural LTR models. They discuss why neural LTR models are worse than gradient boosted decision tree-based LTR models, and introduce some directions to improve neural LTR models.\n\nPros:\nThis paper discusses potential reasons why neural LTR models are worse than gradient boosted decision tree-based LTR models, and uses empirical results to show the effectiveness of the proposed solutions.\n\nConcerns:\nRegarding the proposed solutions, the authors use data augmentation to improve neural LTR models. It seems that some feature engineering work can help improve performance. Comparing to traditional gradient boosted tree-based LTR models, is it really worth putting efforts into studying neural LTR models?\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8EXGTag-_u7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thanks to the authors for their hard work and the nice paper. I both enjoyed this paper and think it's a strong contribution to the ranking literature. It was well written, clear, and nicely organized. The appendices are full of useful experiments.\n\nThere is some room for improvement, particularly in the thoroughness of the experiments:\n\n- Eq (6) - It's nice to have a single transform for everything but what if the data is already, say, standard normal? Or categorical? Do any of your benchmark datasets have categorical variables? How were they dealt with? \n- Sec 3.2 - Hyperparameters for neural networks - learning rate and batch size are usually crucial for neural networks, but are not tuned for any of of the baselines (as far as I could tell). You are likely to get higher performance if these are tuned as well - please do so, especially for the baselines that you implemented yourself.\n- It appears from the ablation study that some of the the bigger performance boosts came from the feature transformation and data augmentation. As this was meant to address issues with the benchmark methods, it would be very worthwhile to apply these techniques to those methods, perhaps while increasing the network capacity.\n- Sec 5 Main Result - for fairness, you should compare to an ensemble of lambdaMARTs.\n- What about training time? LightGBM is very fast. How long does it take to train your model? Is the few percent improvement worth it?\n- Does lambdaMART also get better with Gaussian noise data augmentation?\n\nOne final note: I'm not sure the word \"hitherto\" makes sense in the title.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers are positive, while one is negative. The negative reviewer is well-qualified, but the review is not persuasive. Overall, this paper should be published as a wake-up call to the research community. Unfortunately, the lesson of this paper is similar to that of several previous papers, in particular

Armstrong, T. G., Moffat, A., Webber, W., & Zobel, J. (2009, November). Improvements that don't add up: ad-hoc retrieval results since 1998. In Proceedings of the 18th ACM conference on Information and knowledge management (pp. 601-610).

This submission should be a spotlight, to maximize the chance that future researchers learn its lesson.
Decision: Accept (Spotlight)