Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcity
Authors: Charlie Hou
Kiran Koshy Thekumparampil
Michael Shavlovsky
Giulia Fanti
Yesh Dattatreya
sujay sanghavi
Venue: Accepted by TMLR ()

Abstract: On tabular data, a significant body of literature has shown that current deep learning (DL) models perform at best similarly to Gradient Boosted Decision Trees (GBDTs), while significantly underperforming them on outlier data. However, these works often study problem settings which may not fully capture the complexities of real-world scenarios. We identify a natural tabular data setting where DL models can outperform GBDTs: tabular Learning-to-Rank (LTR) under label scarcity. Tabular LTR applications, including search and recommendation, often have an abundance of unlabeled data, and scarce labeled data. We show that DL rankers can utilize unsupervised pretraining to exploit this unlabeled data. In extensive experiments over both public and proprietary datasets, we show that pretrained DL rankers consistently outperform GBDT rankers on ranking metrics, sometimes by as much as 38%, both overall and on outliers.

Published in Transactions on Machine Learning Research (11/2024)
Pretrained deep models outperform GBDTs in Cearning-To-Rane under label scarcity
Charlie Hou" Deportment of Electrical and Computer Engineering Curnegie Mellon Universily
Kiran K. Thekumparampil
0kf com
Amazon, Palo Alio
Michael Shavlovsky
shavlon@emazon.com
Amazon, Palo Allo
Giulia Fanti
gfanti@andreww cma.. edu
Deportment of Electrical and Computer Engineering Cornegie Mellon University
Yesh Dattatreya Amazon, Palo Alto
ydatta wwwason..cacoooo com
Sujay Sanghavi
sanghavi &maii uuteras. eduu
Department of Computer Science University of Texas at Austin
Reviewed on OpenReview: https: openreview. net/ net/forum? forum? id= 09309/2alt
Abstract
On tabular data, a significant body of literature has shown that current deep learning (DL) models perform best similarly Gradient Boosted Decision Trees (GBDTs), while significantly underperforming them outlier data Gorishniy et al. (2021); Rubachev et al. (2022); McElfresh et al. (2023). However, these works often study problem settings which may not fully capture the complexities of real-world scenarios. We identily natural tabular data setting where DL models can outperform GBDTs: tabular Learning-ter to- Sank (LTR) under label scarcity. Tabular LTR applications, including search and recommendation, often have an abundance of unlabeled data, and scurce labeled data. We show that DL rankers can utilize unsupervised pretraining exploit this unlabeled data. In extensive experiments over both public and proprietary datasets, show that pretrained DL rankers consistently outperform GBDT rankers on ranking and on outliers.
by much 38%-both overall
1 Introduction
The learning-to-rank (LTR) problem aims to train model to rank set of items according to their relevance or user preference (Lin 2009). An LTR model is typically trained on a datasct of queries and associated query groups (i.e., a set of potentially relevant documents or items per query), as well as an associated (generally incomplete) ground truth ranking of the items in the query group. The model trained to output the optimal ranking of documents or items a query group, given a query. LTR. core MT. technology many real world applications most notably in search contexts including Bing web search (Qin & Liu 2013), Amazon
Work partially done while Charlie Hou intern Amazon.

---

Published in Transactions on Machine Learning Research (11/2024)
Set1
Istella
GBDT
0.66-
0.68
0.46
Pretrain Pretrained
0.64
0.66
0.44
0.62
NDCG
NDCG
0.42
0.60
0.40
0.60
GBDT
0.58
GBDT
0.38
0.58
No Pretrain DL
0.56-
No Pretrain DL
0.36
0.56
Pretrained DL
0.54
Pretrained DL
16
10-2
10
10-3
10-2
16-1
100
10
10
10
100
Fraction of training labeled
Fraction of training ags labeled
Fraction of training ags labeled
Figure 1: Main result (Simulated crowdsourcing of labels): We compare NDCG (1) as we sweep the percentage of labeled training query groups (QGs); labeled QGs have relevance scores for every element of the QG. For small enough fractions of labeled query groups, pre-trained DT, rankers (sometimes significantly) outperform both GBDIs (representing the best of supervised GBDTs and semi-supervised GBDTs) and
non-pretrained DL rankers (representing the best of tabular ResNet (Gorishniy al. 2021 ), DeepFM (Guo) et al. 2017), and DCNv2 (Wang et al.
2021 To the left of the black dotted line, pretrained rankers perform
the best (statistically significant at p 0.05 level). Points are averages over 3 trials.
2022). and Netflix movie recommendations [Lamkhede & Kofler 2021 More recently, LTR has also found use in Retrieval Augmented Generation (RAG), where a retrieval system is paired with an LLM (large language model) to ground the latter's responses to truth and reduce hallucination
product search (Yang et al.
Lewis et al. 2020 Glass et al. 20022 Pan al.
2022
Traditionally, LTR models took tabular features numerical or categorical features of queries and documents
as imput features [Burges et al.
2005 Cao et al.)
2007
Chapelle & Chang 2011 Qin & Liu
2013 Lucchese
lel al.
2016 As this continues to be the case for many important. applications, labular LTR is a core problem
in machine learning (Bower et al.
2021 Yang et al. 2022
Yu 2020
Kveton et al. 2022 Pan et al. 2022 Xia
ett al. 2008 Chen et al. 2009). Today, deep models are largely outperformed by gradient boosted decision
trees GBDTss (Friedman 2001 in ML problems with tabular features (Jeffares al. 2023 Qin al.
2021
In contrast, deep models are state-of-the-art by significant. margin in domains like text DDellin et al.
2018
and images
(He et al. 2016).
Recent breakthroughs in modeling non-tabular data like text and images have been driven by first training a deep neural network to learn from unlabeled data (unsupervised pretraining, or pretraining) (Devlin et al., 2018 Chen et al.) 2020), followed by supervised training (finctuning). Models that are pretrained in this way can perform significantly better than models that were only trained on existing labeled data. This remarkable success stems from three factors: (I) limited access to labeled data, (2) large, available sources of unlabeled text and image data, and (3) pretraining methods that can exploit unlabeled data.
A natural question is whether deep models can similarly use unsupervised pretraining to outperform GBDTs on the tabular LTR problem. To this end, many papers have studied unsupervised pretraining techniques both
using deep learning and GBDTs for tabular data Yoon et al. (2020]: Ucar et al. (2021) 05 Bahri et al. 2021
Verma et al. (021)) : Majmundar et al. [202)]
|: Duh & Kirchhoff (2008): ; Pseudo-Label (001)). However, none
of these mothods have shown convincingly that deep learning (with or without unsupervised pretraining)
consistently outperforms GBDTs (semi-supervised or supervised); at best, deep methods appear achieve
roughly the same performance GBDTs (Jeffares et al. 2023 Rubachev et al. 2022; Gorishniy et al. 2021
Shwartz-Ziv & Armon 2022 Grinsztajn et al. 2022). Furthermore, recent work showed that GBDIE Is are significantly more robust against outliers in the datasct than deep learning models McElffesh et al. 2023 We note that most of these works study idealized problem settings which may fail to capture complexities of real-world scenarios such as label scarcity and noisy data. Prior work in other domains show that pretrained deep models can outperform purely supervised models Chen al.) (2020) and former models are more robust to noise Hendrycks al. (2019). Therefore, we hypothesize that pretrained deep models can outperform GBDTs in tabular settings that more accurately reflect real-world challenges.
2

---

Published in Transactions on Machine Learning Research (11/2024)
Through extensive empirical evaluation, we confirm this hypothesis and show that deep models which use unsupervised pretraining consistently outperform GBDTs sometimes by a large margin- when labels are scarce, By scarce, we mean that only a small number of query groups contain labels. This setting is more standard in most real-world deployments of LTR, rather than the idcalized settings studied in prior work. For instance, scarcity can arise when organizations: (a) crowdsource labels for a limited number of query groups due to cost constraints, and/or (b) use rare user feedback (e.g,, clicks, purchasos) as implicit labels. Our experiments consider both sources of scarcity. They are run on the three standard public datasets in the ranking literature: MSLRWEB30K (Qin & Liu 2013), Yahoo (Chapelle & Chang 2011). Istella S (Luechese) et al. 2016), as well as an industry-scale proprictary ranking dataset.
Contributions: (1) We demonstrate for the first time that unsupervised pretraining can produce deep models that outperform GBDTs in ranking. Specifically, pretrained deep rankers can achieve up to 38% higher NDCG (normalized discounted cumulative gain) score (Burges 2010) than GBDTs when labels are scarce (Figure D For instance, Figure | shows on three datasets that as the labels got scarcer, GBDTs start to substantially underperform pretrained deep rankers (experimental setup in Section 4.1.2
(2) We preseribe empiricalll-jusstfeedd LTR-specific pretraining strutegies, including a new ranking-specii pretraining loss, SimCLR-Rank, which is a modification of the widely used SimCLR (Chen et al. 2020 loss. In experiments, SimCLRRRaak achieves similar or better performance than most existing pretraining methods at an order-of magunitude speedup. This allows MauCLR-Rande to scale to large our proprietary industry-scale ranking dataset.
datasets such as
(3) We demonstrate that when lobels are scarce, pretrained deep rankers can perform significantlyy better than GBDTs on outlier query groups, boosting NDCG by much 80% (Figure 8 Appendix [A.3.4p. This is perhaps surprising, given prior results in the label-rich regime showing that GBDTs are significantly more robust to data with irregularities than deep rankers (McElfresh et al. 2023).
The main novelty of this work lics in the identification of label-scarce LTR as a practical real-world tabular learning problem and the results which challenge the prior understanding of the optimal modeling choices for tabular learning. We demonstrate that pretraining strongly help deep models to consistently outperform GBDTs in this label-scarce problem. Of note, these improvements over GBDTs happen even when the labeled data is plentiful in absolute numbers; up to 25% labeled query groups in public datasets and 3 million+ labeled query groups in the large industry-scale ranking dataset. Prior to our work, the MT. and ranking literature had not yet identified a widely applicable setting where (pretrained) deep models could consistently outperform GBDTs (Qin et al.] 2021: McElfresh et al.) 2023). Due to the prevalence of label scarcity in practical settings, our results suggest. & positive change in how practitioners may approach real-world tabular learning problems, especially in scarch and recommendation systems.
Related work: We focus the traditional LTR setting where the features are all mumeric (tabular data). In this setting, gradient boosted decision trees (GBDTs) (Friedman 2001b have historically been the de-facto models, though there is currently great interest in applying deep neural networks to tabular data. Borisov
et al.
(2022) categorize existing techniques for using deep neural networks over tabular data into four lypes:
(T) Encoding-based methods such as VIME Yoon et al.] 2020). SCARF (Bahri et al. 2021), IGTD (Zhul
ct al.
2021), , and SuperTML. (Sun et al.
2019):
; (2) Novel hybrid architectures such as DocpFM (Guo
ee al.
2017,, , xDeepPM (Lian et al. 2018
and many others Cheng et al. (2016); Frosst & Hinton| (2017);
Ke et al. (2018) 2019); Popov et al. (2019) Luo et al. (2020); Liu et al. (2020); Tvanov & Prokhorenkova
(2021);
Luo et al. 20021 (3) Transformer-Dee based architectures including SAINT (Somepalli et al. 2021
TabNet (Arik& & Pfister 2021), Tab Transforrrr (Huang et al. (020),, and ARM-Net Cai et al. [2021 0); (4)
Regularized DNNs
(Shavitt & Segal 2018 Kadra. el al. 2021
Within these categories, proposed architectures include factorization machines (Rendle 2010), wide & deep
architectures (Cheng et al. 2016), and similar architectures designed for recommendation systems (Guo et al. 2017 Wang et al. 2021 Lian et al. 2018 Naumox al. 2019: Qu et al. 2016). While our paper is architecture agnostic, we perform comparisons of pretrained models against non-pretrained DeepFM Guo et al. (017 and DCNv2 Wang et al. (022]) models to show that pretraining is needed to putperform GBDTs
consistently. Self-supervised learning (SSL) or unsupervised pretraining has improved overall performance
3

---

Published in Transactions on Machine Learning Research (11/2024)
and robustness to noise (Hendrycks et al. 2019) in settings where there is significant source of unlabeled
data like text (Devlin et 2018: Howard & Ruder 2018 Kitaev et al. 2018 Liu et al. 2017 Song & Roth
2015) and images (Chen et al. 2020
Chen & He 2021 Grill et al. 2020 Ge et al. 2021 Xie et al. 2020
Inspired by the success of SSL in images and text, many unsupervised pretraining tasks have been proposed for tabular data. We describe some of the most well-known methods. SubTab (Ucar et al. 2021 proposes to train an autoencoder to reconstruct a table using a subset of the columns. VIME-self ( (Yoon et al. 2020) is a
similar method. SCARF and DACL+ (Bahri et al. 2021 Verma et al., 2021] propose augamentations for the
SimCLR pretraining task (Chen et al. 2020 that work for the tabular setting. RegCLR (Wang et al. 2022a
is a method for extracting tables from images, which is not directly applicable to our setting.
A smaller class of approaches has studied regularized DNNs, linding that they require limited changes to perform well on tabular data Shavitt & Segal (2018); Kadra et al. (2021). However, the most competitive of these methods require extensive hyperparameter optimization over a 14-dimensional grid Kadra et al. (2021)
While the body of work studying neural network architectures for tabular data is exlensive, deep models have yet to outperform GBDIs convincingly (Qin et al.) 2021 Joachims 2006 Ai et al. 2019 Bruch et al.
2019 Ai ct al. 2018 Pang et al. 2020 McElfresh et al. 2023 In this work, we consider encoding-based
methods, which alter the representations over which a deep model can learn to rank; we do not focus on introducing new algorithms in this work, but rather evaluate mostly existing methods applied to tabular data. We comprehensively evaluate seven representative DL methods: SubTab, SCARF, DACL+, VeMEEself, MOCCL--Ranke SAINT, SimSiam (Chen & He 2021), and self-supervised approaches for GBDTs in Section 3.1 Our results show that multiple pretrained deep rankers can consistently outperform GBDTs in the label-scarce tabular LTR setting, with SimCLR-Rank and SimSiam (Chen & Hc. 2021) generally performing the best. We provide detailed supplemental discussion of related works in the Appendix A.1
2 Learning- Searning-To-Rans problem and its metrics
In the LTR problem, samples are query groups (QGs) consisting of a query (e.g. shopping query) and L potentially relevant items (c.g. products) for this query. Although an LTR datasct contains multiple QGs, for the sake of simplicity, the notation in this section only handles a single QG. The i-th relevant item in this QG is represented by a feature vector at; Rd. which captures information about this query-item pair, subsuming any query related features. If and only if the QG is labeled, such as the ones used in supervised training or testing, we also have an associated scalar relevance label ys, which could be a binary, ordinal, or real-valued measurement of the relevance of this item for this query (Qin et al., 2021). The objective is to learn a function that ranks these L items such that the items with highest true relevance are ranked at the top.
Most LTR algorithms formulate the problem learning a scoring function fo Rd R that maps the feature vector associated with each item to score, and then ranking the items by sorting the scores in descending order. To measure the quality of a ranking induced by our scoring function So on a labeled test QG, a commonly-used (per-QG) metric is NDCG (normalized cumulative discounted gain) (f):
NDCG(T),. DCG(T for (u:) D Hi)
[0. 1].
(1)
where T/o : [L] - (1.),) is a ranking of the L elements induced by the scoring function fo on (ar)/e (i.e. (i) is the rank of the i-th item), while n² is the ideal ranking induced by the true relevance labels (2 and discounted cunulative gain (DCG) is defined as DCG(TT, (w)) = Σt [ogg[1+ Here, (t) indicates that larger metric values are preferred. Note that for summarizing this metric over the full
2
test set, we simply take the average across all the QGs in the set. Typically, a truncated version of NDCG is
used that only considers the top-k ranked items, denoted as NDCG@k. In the rest of our paper, we will refer to NDCG@5 as NDCG; this will be our evaluation metric.
We consider a natural label-scarce LTR setting with large amounts of unlabeled data and a limited supply of labels for subset of this data. The labels that do exist can arise from multiple sources, such as crowdsourced
4

---

Published in Transactions on Machine Learning Research (11/2024)
workers paid to label QGs, or through implicit user feedback (e.g., clicks purchases); we model both in our evaluation. Unless specified otherwise, we use all the query groups for unsupervised pretraining, but we use only the labeled query groups for supervised training.
2.1 Outlier-NDCG for outlier performance evaluation
Motivated by a recent observation that GBDTs outperform DL models in datasets with irregularitios McElfrosh et al.] (2023), we also study their performance on outliers. In interactive ML systems like search, performing well on outlier queries is particularly valuable as it empowers users to search for more outlier queries, which in turn allows the modeler to collect more data and improve the model. To this end, we evaluate the outlier performance by computing the average NDCG outlier queries groups and refer to it as Outlier-NDCG for brevity. In practice, some outlier queries may already be known, and the modeler can define the outlier datasets accordingly, like in our proprietary dataset. For example, since industry data pipelines often have missing data/featune /features one could identify samples with missing features as outliers. When outliers are unknown, detecting them (especially higher-dimensional data) is challenging. Hence, we create a heuristic algorithm by assuming outlicrs are rare values that are separated from the bulk of the data. Using it we systematically selected outlier query groups for the public datasets that we use. More details about the outliers are provided in Appendix [A222]] Separately measuring the performance the outliers is justified by our later observation that performance on the full dataset and the outliers can significantly differ (Tables and
3 Design: Unsupervised pretraining for LTR
Qualitatively, the ranking relevance of tabular records to query is typically determined by a combination of fields. For example, if person searches for product (say pair of sandals), we can think of the fields in the LTR problem as representing properties of the results, such as material, color, and price. Qualitatively, we expect that if a pair of records are the same except for small differences between fields (e.g., missing a feld), then the two products should generally have similar rankings in the LTR problem. In this section, we discuss how existing methods fail to capture this qualitative intuition.
Semi-supervised learning for GBDTs:
While to the best of our knowledge there are no methods
to perform unsupervised pretraining in GBDTs, one can use semi-supervised methods like consistency regularization (Jeong & Shin 2020: Sajjadi et al., 2016 Miyato et al.] 2018 Oliver et al., 2018 Berthelot
et al. 2019). pseudo-labeling beabbliing/eef---iee /sss/sfffftrree training (Rizve et al. 2021 Pseudo-Label 2013) Shi et al. 2018 Yarowsky
McClosky et al.) 2006), and PCA Duhh & Kirchhofi 2008) to address limited size of labeled data. In consistency regularization, one increases the size of the training set by using data augmentations. In geeuud-labllingg/el---aaainnnnni
1995
a model trained on the available labeled data used to label unlabeled data. Then, the final model is trained on the resulting "labeled" dataset. In PCA, one projects the features along directions of maximum variability in the union of labeled and unlabeled datasets. Since prior work has found that consistency regularization decreases the performance of GBDTs (Rubachev et al. 2022), evaluate only pseudo-labeling and PCA. None of these methods explicitly ensures that semantically similar records have similar representations in the learned models.
To address this gap, there have been many unsupervised pretraining methods proposed for tabular data, which can be applied to the tabular L.R setting. In this paper, we will evaluate (1) SCARF (Bahri et al., (021)) and (2) DACL+ (Verma et al.] 2021 which are contrastive learning methods (Chen et al., 2020 (3) VIME-self (Yoon et al.) 2020) and (4) SubTab (Ucar)
Pretraining methods for tabular data:
et al. )021) which are autoencoder based methods. The four most commonly-evalue
baselines in the pretraining literature for tabular data are [Car et al. (2021); Hajiramezanali et al. (2022); Levin et al. (2022); Majmundar el al. (2022). Our results suggest that many of these methods outperform GBDTs in the label-scarce regime (Section 3.1,, , in accordance with the main theme of this paper. However, the consistently best-performing methods our experiments were more basic, donain-agnostic pretraining methods, which can be viewed as more primitive building blocks of some of the above methods. Surprisingly, they use less domain knowledge of tabular data than these more sophisticated methods.
5

---

Published in Transactions on Machine Learning Research (11/2024)
Specifically, in the image setting, SimCLR (Chen et al. 2020 and SimSiam (Chen & He 2021 are two widely-adopted unsupervised preiraining methods. They use the idea that two variations of the same data point should have similar representations. This idea generalizes across domains, including tabular LTR.
SimCLR [Chen et al. 20200: : In a batch of B query-groups, the feature vector Eg of q-th query group and its i-th item is stochastically augmented twice to obtain 2 and x: = which are called positive pair. In this paper, use combinations of the following two augmentation strategies for all domain-agnostic pretraining methods: (a) Zeroing: Zero out features selected randomly and i.i.d. with fixed probability. (b) Gaussion noise: Add Gaussian noise of a fixed scale to all features. Further details are given in Appendix A.2.3
Next, base encoder h(-) and projection head g(-) map (a) to = g(h(zz)))) for 0, 1. Then by minimizing the InfoNCE loss (Oord et al. 2018) in this projected representation space, we push the positive pair, (z 0 =(1),, closer while pushing them farther from all other augmented data points in the batch, which are called their negatives. Formally, for Fgii we minimize e + qai where:
e(l)
B L
(() 4,4 = -cos(2 (a) 200)// 1/T + log ΣΣΣ d===0==l'=0
2. 19,1129, 2g
( + (a,i,a)} exp(cos(2 (a) +/)/+ +
(2)
1 a, is the temperature parameter, Ln is the number of items in q'-th query group, and cos(2, 2) (2.21//|==|][]] denotes the cosine similarity. After pretraining, the encoder h is used in down- stream applications. SimCLR achieves superior performance in many domains (Chen & Hee 2021 Wang et al. 2022b a). It is known that contrasting a pair of data samples which are hard to distinguish from each other, called hand negatives can help an encoder learn good representations (Robinson et al.) 2020 Oh Song et al.)
2016 Schroll et al., 2015 Harwood et al. 2017 Wu el al., 2017 Ge 2018 [Suh el al.] 2019). SimCLR simply contrasts all pairs of data samples in the batch (including from other query groups) with the assumption that
large enough batch likely contain a hard negative.
SimSiam Chen & He, 2021;: Similar to SimCLR, for each data point zq,i, this method produces stochastically- augmented positive pairs and their projected representations. However, we pass these representations further through predictor pred(-); to get p pred(2 for a 0, Finally, we maximize the similarity between the projected and predicted representations: (os(,,))))(((,)))) + cos(p (gga)sg(2])
(1) =y(=;)))) where sg is a
ttop-gradient. Unlike SimCLR, there are no negatives, i.e., the loss function does not push the representation of an augmented sample away from that of other samples' augmentations. The asymmetry in the loss due to the stop gradient and the predictor prevents representation collapse to make negatives unnecessary Tian et al. ); Zhang et al. (2021; ; Wang et al. 2222bb.. Tf the number of items in each query group is L., then the time/space complexity for SimSiam is only O(BL) per batch, whereas is O(B² for SimCLR. Therefore, SimSiam is more efficient and it can scale to larger batchsizes and data.
Pretraining method for LTR: Motivated by (a) SimCLR's high complexity and (b) the efficacy of contrasting with hard negatives, we additionally propose SimCLR-Rand an LTR-specific alternative. Recall that the SimCLR loss for an item in equation 2 uses all the other items in the batch as negatives. Good pretraining strategies usually exploit the structure of the data, e.g. text sequences motivates masked token prediction Devlin et al. (2018). Studying the query group structure of LTR data, we notice that high-quality hard negatives are freely available as the other ilems from the same query group. These items are retrieved and deemed potentially similarly relevant for this query by an upstream retrieval model, making these items harder negatives than other items in the batch. So, we propose SimCLR- Rank, which modifies SimCLR to contrast only with the other items in the same query group, as illustrated in Figure 2 Formally, modify the per-augmentation loss pla) q. as:
Lg 1
pla) 9. = cos((2 (a) qui 2
//T + log Σ
(o) Σ [1((()a)) + (i,a)) exp(cos(2 ( )/1]
(3)
#=10'=0
Empirically, SimCLR-Rand better separates embeddings within query groups relative to sampling negatives uniformly from each batch (Appendix A.5 The SimCI.R-Rank loss has time/space complexity of O(B.2),
6

---

Published in Transactions on Machine Learning Research (11/2024)
Query Groups (QG) Batch
Augmented Views
Contrastive Learning
View
View
Positive
pair
Pasitive-Positen Embeddings made similar
SimC.R Negatives
QG1: "Cats"
SimCLR-Rank Negatives (-]
Random
Perturbations
Positive-Negative Embeddings made dissimilar
QG2: "Dogs"
In SimCLR and its variants, positive pairs, or augmented versions of the data sample, are trained to have similar embeddings. Positive-negative pairs, augmented samples originating from two different data points or classes, are trained to have distant embeddings. In vanilla SimCLR, each positive pair is contrasted with all other ilems in the batch, denoted by the data points contained in the brown dashed line above. In SimCLR-Ramd (QG), denoted by the data points inside the red solid line above. Table Pretraining methods:
Figure 2:
each positive pair is contrasted with only the items in the some Query Group
comparison across unsupervised pretraining methods on NDCG, averaged over 3 trials. Here only 0.1% of the query groups have labels for finetuning. Bold numbers denote the best in column, and underlined numbers are within the margin of error of the best. Given that SCARF/DACL++SinCCII
are too slow to scale to large LTR datasets (for example the industrial-scale proprietary online shopping dataset), we find that MMCRLRRaad and SimSiam are the best pretraining methods for tabular LTR.
Method
MSLR (t)
Yahoo Sct1 (t)
Istella (t)
Supervised GBDT
0.2801 + 0.0002
0.5083 0.0197
0.5450 0.0000
Sumi-supe GBDT
0.2839 + 0.0004
0.5061 0.0267
0.4656 0.0310
CLn- GBDT
0.3165 + 0.0050
0.5504 + 0.0135
0.5397 + 0.0098
SimSiam + GBDT
0.3158 + 0.0030
0.5620 + 0.0167
0.5297 + 0.0051
SCARF
Bahri et al.
2021
0.3807 + 0.0016
0.5884 0.0129
0.5542 0.0024
DACL+ Verma
2021
0.3833 +'2 0.0037
0.5887 +-2 0.0012
0.5626 + 0.0024
VIME-self
Yoon
2020
0.3834 + 0.0011
0.5839 +\2 0.0058
0.5514 + 0.0025
SubTabb
Ucar al.
2021
0.3748 + 0.0025
0.5814 + 0.0062
0.5082 + 0.0050
SAINT
Somepalli li et al.
2021
0.3355 + 0.0043
0.5890 + 0.0075
0.5560 + 0.0066
Sim
Chen et al.
2020
0.3827 + 0.0027
0.5837 + 0.0093
0.5602 + 0.0055
SimCLR-SS CLR-Sampl
0.3498 + 0.0541
0.5803 0.0070
0.5479 = 0.0016
SimCLR-Rane
0.3868 + 0.0026
0.5843 0.0062
0.5609 + 0.0040
SimSiam
(Chen & Hc
2021
0.3790 + 0.0028
0.6100 0.0072
0.5189 0.0096
making it faster than SimCLR's O() complexity, and enjoying SimSiam's linear complexity in B. Next, we empirically study these methods.
3.1 Evaluation of unsupervised pretraining techniques
In this subsection, compare the discussed pretraining strategies in the following tabular LTR setting.
7

---

Published in Transactions on Machine Learning Research (11/2024)
Dataset. We use MSLRWEB30K (or MSLR. for brevity) QQn & Liu 2013)
Yahoo Set1 (or Set1 for brevity)
[Chapelle& & Chang 2011), and Istella S (Lucchese et al. 2016) (a smaller version of Istella (Dato et al.
2016)). In the tabular LTR literature, these three datasets are generally considered the standard; most recent
papers evaluate (only) on these datasets (Qin et al. 2021 Ai ct al. 2019 Pang et al. 2020 Yang et al. 2022
Yu 2020). To simulate scarcity of labels, we let. 0.1 % of the training query groups each dataset keep their labels, while the rest of the query groups have no labels. Dataset statistics are provided in Appendix A.2.1
Methodology. For the neural pretraining approaches (which include SimCLR-Rande SimSiam, SimCLR CLR, SubTab, VeMEEself, DACL+, SAINT, and SCARF), the base of the model is the tabular ResNet (check the footnote for our explanation on the architecture choice). We also evaluate "SimCLR-SSmple"" pretraining baseline we create to demonstrate the value of negatives from the same query group. SimCLR- but at the beginning of each epoch the items are first randomly permuted
neural
Sample is like and grouped into fake query groups. Then we apply SimCLR- Rank.. Thus SimCLR-Sample has the same time/space complexity as SimCLR-Ranin We pretrain on all the available query groups and finctune/ /train on only the labeled query groups.
For the GBDDT-based approaches, we use the implementation in lightgbm Ke et al. (017) and adopt a tuning strategy similar to Qin et al. (2021). We report "Supervised GBDT" as GBDT utilizing only the labeled data, and in "Semi-superviser" GBDT" we report the best among pseudo-labeled (Pseudo-Labell 2013) and PCA-enriched (Duh & Kirchhofi 2008) GBDTss as well as the combination of the two techniques. We also evaluate GBDTs enriched with SimCLR-Rank and SimSiam embeddings as additional features, which we call "SimCLR- Rank + GBDT" and "SimSiam + GBDT" We describe our experimental setup in detail in Appendix A.2.3 This includes hyperparameters, procedures for data augmentation, GBDT pseudo-labeling, Melliig/PAAAA// /PCA details, model training, and finetuning, For comparison to prior baselines on tabular data, our experimental setup is chosen to mimic the original setup in them.
Results. Table D charts the test NDCG for each of the pretraining methods we study. We highlight a few observations: (1) The cettraininn-based DL methods almost all outperform GBDTs (both supervised and semi-supervised) in this label-scarce setting. Surprisingly, the semi-supervised GBDT baselines tend to achieve lower NDCG than the supervised GBDT baseline without semi-supervised learning. (2) When GBDTs are allowed to use representations Iearned from our pretraining methods, their performance consistently improves, though they do not outperform pretraining with DL. (3) SimCLR-Rank and SimSiam perform the best or the second best among these methods for most of the datasets. In particular, SimCLR- outperforms SimCLR-Sample, demonstrating the value and efficiency of negatives from the same query group. As expected, SimCLR-Rand is also 7-14x faster than SimCLR SimCLR/SCARF/ SOR//SAAF//DD /DACL+ (Table 6 in Appendix AA.31] 1. Given these advantages terms of test NDCG and speed, we select SimCIRR-Ran. and SimSiam as the best, overall pretraining methods for tabular LTR, and use them for our continued evaluation.
4 Empirical evaluation
Next. we compare the best DL pretraining methods for labular LTR (SimCLR-Rank and SimSiam) against GBDT and non-pretrained DL models on (1) the public datasets under different label scarcity patterns to simulate real-world scenario and (2) a private largo-scale ranking datasct which naturally label-scarce.
4.1 Public datasets results
For the public datasets (MSLRWEB30K, Yahoo Set1, Istella SS details given in Section 3.1 we simulate label scarcity occurring in two scenarios: (1) crowdsourcing of labels for a limited number of query groups, and (2) using scarce implicit binary user feedback (c.g. clicks) as labels.
We emphasize that we do not prescribe any specific deep learning architecture choice; the choice for the best DL model may depend on the specific LTR task. Our proposed workflow of pretraining deep models in LTRR is agnostic to the specific choice of DL architecture. Prior work has propused DL architectures designed for click-through rate (CTR)/LTR based factorization machines [Itenlle] 2010), such DeepI'M and DCNv2 (Guo et al. 2017 Wang et al. 2222]) which compare to in Scction 4.1.2ann and Section 4.1.2 against our pretrained models. Noto that it is unclear currently how do unsupervised pretraining using these CTR/LTR specific architectures at the moment. 2Code provided at http://tww..ooo/oocccaatttttrree
8

---

Published in Transactions on Machine Learning Research (11/2024)
4.1.1 Simulated crowdsourcing of labels
We simulate a setting where labels are crowdsourced for only some query groups due to limited resources.
Datasct. For cach of the three public datascts, we vary the fraction of labeled query groups in the training set in {0.001, 0.002, 0.005, 0.1, 0.5, 1.0). Note that within each labeled query group all items are labeled.
Methodology. We repeat the methodology from Section 3.1 for the GBDT and the SimCLR- R-R-Rakk and SimSiam pretraining methods. For each dataset and labeling fraction, we report the test metric of the best GBDT ranker among semi-supervised GBDTs and supervised GBDTss under "GBDT". Similarly, under "no pretrain DL" we report the best supervised DL model amongst tabular ResNet (Gorishniy et al.] 2021), p. DeepFM (Guo et al., 2017), and DCNv2 (Wang et al., 2021) under the name "no pretrain DL". Finally, we report the best pretrained ranker amongst SimCLR- Rank and SimSiam under the name "pretrained DL Pretrained DL rankers use tabular ResNet (see Appendix A.2.3 for more details). During hyperparameter tuning, we tune for NDCG and report the resulting test NDCG and test Outlier-NDGG.
Results. We compare pretrained rankers, non-pretrained DL, rankers, and GBDTs on public datasets in Figure ! The results for Outlier-NDCG are provided in Figure S in Appendix A.3.4 We find that pretrained rankers outperform non-pretrained methods (including GBDTs) on NDCG and OutGier-NDGG across all public datasets, up to a clllasetddpeendett fraction of QGs labeled, as shown in Figure (for NDCG) and Figure g (for Outlier-NDCG). We also note that GBDTs have better NDCG than non-pretrained DL models in most of these regimes. This shows that there exist scenarios with limited supply of labeled query groups, where self-supervised pretraining is the factor that allows deep learning models to outperform GBDTss We provide a detailed comparison between semi-supervised and supervised GBDT models in Appendix A.4
4.1.2 Simulated implicit binary user feedback
Here we simulate scenario when users of an LTR system provide binary implicit feedback for the items, like the clicking of an advertisement on a webpage. This kind of feedback is usually infrequent in an LTR system, as it is the result of a user choosing to spend more time or resources on an item in a presented list. Therefore, most query groups will not have any labels, and even labeled query groups will only have a few positively labeled items. On the other hand, such labels are often cheaper to collect than crowdsourced labels, so they are used in many industry use cases.
Datasct. To simulate this scenario, we follow the methodology from Yang et al. (2022) to generate independent stochastic binary labels for each ilem from its true relevance label for training and validation sets of each of the public datasets. Note that we still use the true (latent) relevance labels in the test set for evaluation. This models a scenario where we observe binary labels that are noisy obsorvations of a true latent label, but the task is to use these noisy labels to learn to rank according to the true labels. The details of this binarizing transformation are provided in Appendix A.2.4 Tn this transformation, a parameter, Tharget: implicitly controls how sparse the binary labels are (larger is more sparse). We select Tearget = 4.5, to obtain 8.9%, 5.3%, and 25.2% labeled query groups which contain at least one positive label in MSLR, Yahoo Set1, Istella S datasets, respectively. We present detailed data statistics in Table 11 of Appendix.
Methodology. We reuse the methodology of Section 4.1.1 except we use linear scoring head for pretrained models to improve stability and performance (see Appendix A.2.3). Similar to Section 4.1.1 we report the best of multiple models for each method and dataset (see: Section 4.1.1 for details).
Results. In Table 2 we present the test NDCG and Outlier-DDGG metrics of the models under this setting. We see that pretrained rankers significantly outperform GBDTs and non-pretrained DL rankers in this simulated binary user feedback setting with up to, even with 25.2% labeled query groups (Istella S). For MSLRWEB30K and Yahoo Sct1 datascts, the second best model is the GBDT, and the non-pretrained DL ranker has a much worse OuDGier-NDCG. This aligns with recent observations that GBDTs perform better than DL models in supervised training with datasets containing irregularities McEllfesh et al. (2023). However, pretraining helps DL rankers close this gap, and at times beat GBDTs in outlier performance. We also present results for Trargtt E (4.25, 5.13 in Appendix A.3.5 There, we find that making the labels sparser (Ttarget = 5.1) increases the relative improvement of pretrained DL rankers, while in a less sparse
9

---

Published in Transactions on Machine Learning Research (11/2024)
Table 2: Main result (Simulated implicit binary user feedback): When X% (MSLBWEB30K-53%% Yahoo Sct1=5.3% and Istella S=25.2% of query groups are assigned few binary labels, pretrained DL method outperform both GBDT and non-pan non-pretrained DL models. We generate labels with Ttarget 4.5 (Section 4. means that pretrained models are significantly better in NDCG than non-pretrained models, measured by t-test with siguificance p 0.05.
Method
MSLRWEB30K (t)
Yahoo Set1 (1)
Istella (t)
NDCG (t)
GBDT No Pretrain DL
0.346 + 0.001
0.617 + 0.000
0.602 + 0.000
0.333 + 0.001
0.621 = 0.000
0.608 + 0.002
Pretrained DL
0.356 + 0.004
0.622 + 0.001
0.613 + 0.002
Outlier-NDCG (t)
GBDT No Pretrain DL
0.288 + 0.000
0.540 = 0.000
0.678 + 0.000
0.238 + 0.026
0.536 + 0.003
0.687 + 0.015
Pretrained DL.
0.289 0.012
0.543 + 0.002
0.699 + 0.003
Table 3: Main result (Real-world dataset): % improvement, of rankers pretrained by SimSiam and SimCLR- over GBDTs on large-scale proprietary online shopping dataset. Unsupervised pretraining improves performance (1) overall (full datasct) (internally, 2% is big improvement), and (2) on outliers.
Method
NDCG (1)
Outlier-NDCG (t)
% change w.r.t. GBDT
GBDT No Pretrain DL
+0.00%
+0.00%
+3.82% + 0.14%
-6.97% + 0.43%
SimCLR-Rank
+4.00% + 0.14%
-4.85% + 0.44%
SimSiam
5.64% + 0.14%
+2.75% + 0.43%
setting (Ttarget = 4.25) GBDTs dominate. This suggests that pretrained DL models out putperform GBDTs when learning from rare implicit binary user feedback, as is common in search and recommendation systems.
4.2 Large-scale real-world dataset results
In this section we test whether the results from the simulated label scarce settings above, especially Section 4.1.2 translate to a large-scale real-world LTR problem.
Dataset. Our proprietary dataset is derived from the online shopping logs of large online retailer, numbering in the tens of millions of query groups. In this dataset, query groups consist of items presented to shopper if they enter search qucry. We assign each item a purchase label: label is 1 if the shopper purchased the item, 0 if the shopper did not purchase the ilem. Hence, this is real-world instance of the implicit binary user feedback setting simulated in Section 4.1.2 Further, only about 10% of the query groups have items with non-zero labels. See Appendix 1 for more dataset details.
Methodology. We compare the performance pretrained DL. models SimCTRR-Rak and SimSiam with GBDT and non-pretrained DI models. SimSiam and SimCLR-Ram are first pretrained on all the query groups before being finetuned on the labeled query groups with some non-zero purchase labels. The GBDT and the non-pretinn poetrrntr models are only trained on the labeled data. More details about these models and training strategies is given in Appendix A.2.5
Results. Our results are summarized in Table and all numbers are given as relative percentage improvements (A%) over the GBDT ranker. First, note that all the DL models outperform GBDT models terms of NDCG. Amongst them, the pretrained model using SimSiam performs the best with a gain of over 5.5%. This is substantial, given that in large-scale industry datasets, even a 2% improvement in NDCG is considered significant. In terms of Outlier-NDCG, the story is very different. Here, the non-pretrained DL. model and
10

---

Published in Transactions on Machine Learning Research (11/2024)
SimCLR- performs up -7% worse than the GBDT. However, SimSiam pretraining allows the DL ranker to overcome this shortcoming and beat the outlier performance of the CBDT. This verifies our hypot from Section 4.1.2 that pretraining can help DL models outperform GBDTs on large-scale real-world LTR problems, especially with implicit user feedback. Lastly, we see a significant performance difference between SimSiar and SimCI.R-Rank. We investigate this an ablation with public datasets in Section 4.3
4.3 Ablations: Choices for pretrained models in LTR
Our experiments suggest that the best pretraining and finetuning strategies for the LTR problem are different from the known best practices for image or text tasks. We make a few observations. (1) SimSiam and SimCLR-Rank perform significantly differently depending on the dataset. (2) Combining models pretrained by SimCLR-Rant and SimSiam can produce a model that is competitive against SimCLR-Rand and SimSiam across all datasets, producing unified approach that may be performant across many settings. (3) Full finetuning of the model performs much better than linear probing, which only finetunes linear layer over frozen pretrained embeddings. In the remainder of this section, elaborate these findings in more detail.
(1) SimCLR- RaRk and SimSiam perform substantially differently depending on the dataset. SimSiam and SimCLR-Rank perform significantly differently depending on the dataset, which is different. from the observations Chen & He (2021), who proposed SimSiam and found that performs similarly to SimCLR derivatives. We next compare SimCLR-Ran with SimSiam at different levels of label scarcity.
MSLRWEB3OK
Yahoo Set1
Istella S
0.66
0.46
0.64-
0.44
0.64
0.62
NDCG
0.42
0.62
NDCG
0.60
0.40
0.60
0.58
SimCLR-Rande SimSiam
SimCLR-Rank + SimSiam
SimCLR-Rank SimSiam SimCLR-Rank
0.38
SimCLR-Rank SimSiam
0.58
SimCLR-Rank
0.56
0.36
SimSiam
0.54
SimSiam
10-3
10-2
10-1
10a
10
10-2
10-1
10a
10
10-2
10-1
10"
Fraction of training qgs labeled
Fraction of training qgs labeled
Fraction training qgs labeled
Figure Comparison of SimCLR-Rank and SimSiam pretraining strategies on three public datasets with varying fraction of labeled query groups. Neither method consistently dominate the other in all the datasets. However, by combining SimCLR-Ran and SimSiam encoders, can produce models that are competitive across all datasets. Data points are averages over 3 trials.
Methodology. The dataset and methodology follows that of Section 1.1.1 except we let (1) SimCLR- use the Gaussian augmentation with scale 1, and (2) SimSiam use the zeroing augmentation with probability 0.1. These augmentations generally perform well for their respective methods. Experiments illustrating the effect of each augmentation method are included in Appendix A.3.6
Results. In Figure 3 SimCLR-Rank performs better on MSLRWEB30K and Istella S, while SimSiam performs better on Yahoo Set1 when labels are scarce. Figure g shows the t-SNE projections of sub-sampled item embeddings learned by SimCLR--Rank SimSiam, and pre-final layer of fully supervised model for Yahoo Set.l. SimSiam exhibit qualitatively more clustered embeddings than SimCLR-Rank, particularly for high relevance labels. Similar t-SNE plots for other datasets are in Appendix A.3.3
(2) Combining models pretrained by SimCLR-Rank and SimSiam produces model that has the strengths of both. To make steps towards a single recommendation for unsupervised pretraining in LTR, we unily SimCLRR-Rak and SimSiam inlo a single method "SimCLRR-Rank + SimSiam".
Methodology. To produce a CamcCLRRaank + SimSiam" model, we finetune a lincar layer over the embeddings from a finetuned SimCLR- concatenated with the embeddings from finetuned SimCLR R-Rank model. More details about how combine the models is given in Appendix A.3.7
11

---

Published in Transactions on Machine Learning Research (11/2024)
SimCLR Rank Rankk
SimSiam L-SNE Setl
pretrained
40
60
40
20
4.0
20
x2
20
0-
20
20
40
-40
-60
-40
-20
20
40
60
-40
-20
0
20
40
60
-60
-40
-20
20
40
60
Figure 4: t-SNE plots of SimCLR-Rank, SimSiam, and a non-pretrained model trained on 0.1% of the data for the Yahoo Setl dataset. Both SimCLR-Rank and SimSiam use the zeros augmentation (p = 0.1) to remove augmentation choice as a confounding variable. Relevance label is indicated by marker size/color. We see that SimCLR-Rank and SimSiam embeddings are qualitatively very different. Qualitatively, SimCLR-Rank and SimSiam embeddings separate different relevance labels better than the non-pretrained model. For this setting, SimSiam has the best downstream performance, and also gives the most clustered embeddings.
Resulls. In Figure 3 we find that "SimCLR-Rank + SimSiam" performs equal to or better than SimCLR-Raad and SimSiam across many different percentages of labeled query groups. While the combined model does not uniformly dominate across all labeled query group percentages, is a promising first step towards a single unified method for unsupervised pretraining in ranking.
(3) Full finctuning outperforms lincar probing. Linear probing is a popular finctuning strategy in text
and image domains, where it produces good results (Chen & He 2021 Chen et al. 2020 Peters et al. 2019 Kumar et al. 2022). We next compare it with the full finetuning strategy in tabular LTR.
Methodology. We use the methodology from Section 3.1 to compare 3 finctuning strategics: lincar probing (LP), full-fnetuning (FF), and multilayer probing (MP) for SimSiam and SinCLR-Rank, MP tunes a 3-layer ML.P head on top of the frozen pretrained embeddings. Experimental details are in Appendix A.2.3.
Results. Table 4 provides results MSLRWEB30K dataset. We find that in terms of NDCG of both SimCLR-Rand and SimSiam, LP is the worst and FF is the best. Interestingly, MP comes second in Sim CL.RRank, but it performs poorly on SimSiam. Our results on other datasets are similar (Appendix A.3.2). We thus recommend full fine-tuning as a stable and performant strategy. To explain the above observations, we we can look again at the t-SNE projections of the embeddings generated by the SimSiam and SimCT.R-Rank encoders (pretrained using the zeros augmentation with = 0.1 to remove augmentation choice as confounding factor, finctuned on 0.1% of the data) and the pre-final layer of a model trained only on 0.1% of the dala on a randomly sampled 1000 samples from the full training set (Figure 4
close
These plots offer the following explanations for the two phenomena. (1) The embeddings produced by the SimSiam "/SSmLLLRRaaan are not sorted by relevance score in the projection space, unlike the fully supervisod encoder. This suggests that a linear ranker cannot use the pretrained embeddings to predict the true relevance labels. (2) MP works better for SimCLR-Rand -aRkRakk than SimSiam because SimCLR-Rank embeddings are more evenly spread out (less collapsed) than those of SimSiam. This allows an MLP trained on the SimCLR-Rank embeddings to distinguish the items with different relevance labels.
5 Conclusion
We study the learning-to-rank (LTR) problem with tabular data under a scarcity of labeled data common scenario in real-world practical LTR systems. Prior works on supervised learning with tabular data have shown that GBDTs outperform deep learning (DL) models, especially on datascts with outliers. In this paper, we find that the label-scarce setting, DL pretraining methods can exploit available unlabeled data to obtain state-of-the art performance. Through experiments with public LTR datasets and wollworld large-scale
12

---

Published in Transactions on Machine Learning Research (11/2024)
Table Finetuning: Comparison of fine tuning strategies on MSLRWEB30K dataset (LP = Linear Probing, MP Multilayer Probing, FF Full Finetuning). Full finctuning consistently performs best or bestb while linear probing and multilayer probing perform poorly.
Method
SimSiam
SimCLR-Rank
NDCG (t)
LP
0.2679 + 0.0007
0.3219 0.0224
MP
0.2764 + 0.0001
0.3890 + 0.0011
FF
0.3935 + 0.0034
0.3959 + 0.0022
OullierNDDG (1)
LP 0.1803 + 0.0033
0.2304 + 0.0332
MP
0.1749 + 0.0023
0.2969 + 0.0009
FF
0.3149 + 0.0119
0.2892 + 0.0025
online shopping datasct, we show that pretraining allows DL rankers to outperform GBDT rankers, especially on outlier data. Finally, we provide guidelines for pretraining on LTR datasets. There remain several open questions around pretraining for tabular and LTR data. Tt. is still unknown whether we can achieve knowledge transfer across different tabular datasets, as is commonly done in image and text domains. It will also be useful to investigate LTR-specific pretraining methods which can further improve the ranking performance.
Acknowledgments
GF and CH gratefully acknowledge the support of Google, Intel, the Sloan Foundation, and Bosch.
References
Qingyao Ai, Keping Bi, Jiafeng Guo, and W Bruce Croft. Learning a deep listwise context model for ranking refinement. In The 1sst international ACM SIGIR conference on research & development in information retrieval, pp. 135 144, 2018.
Qingyao Ai, Xuanhui Wang, Sebastian Bruch, Nadav Golbandi, Michael Bendersky, and Mare Najork. Learning groupwisc multivariate scoring functions using deep neural networks. In Proceedings of the 2019 ACM SIGIR international conference on theory of information retrieval, pp. 85 92, 2019.
Sercan Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. 2021.
Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. arXiv preprint wxXx::216.155447 2021.
David Berthelot, Nicholas Carlini, Tan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel.
Mixmatch: A holistic approach semi-supervised learning. Advances in neural information processing systems, 32, 2019.
Vadim Borisov, Tobias Loemann, Kathrin Scßler, Johannes Hang, Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE transactions on neural nelworks and learning systems, 2022.
Amanda. Bower, Hamid Eftekhari, Mikhail Yurochkin, and Yuekai Sun. Individually fair ranking. arXiv preprint ar Xiv:21108. 11028, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 1901, 2020.
Scbastian Bruch, Masrour Zoghi, Michael Bendersky, and Marc Najork. Revisiting approximate metric optimization the age of deep neural networks. In Proceedings of the 42nd international ACM SIGIR conference research and development. in information retrieval, pp. 1241-1244 2019.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pp. 89 96, 2005.
Christopher JC Burges. From ranknet to lambdarank to lambdamart: An overview. Learning, 11(23-581):81, 2010.
Shaofeng Cai, Kaiping Zheng, Gang Chen, HV Jagadish, Beng Chin Ooi, and Meihui Zhang. Arm-net: Adaptive relation modeling network for structured data. In Proceedings of the 2021 International Conference Management of Data, pp. 207 220, 2021.
Zhe Cao, Tao Qin, Tic-Yan Liu, Ming- Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning, pp- 129-136, 2007.
Olivier Chapelle and Yi Chang. Yahoo! learning to rank challenge overview. In Proceedings of the learning to rank challenge, pp. 1-24. PMLR, 2011.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Wei Chen, Tie- Yan Liu, Yanyan Lan, Zhi-Ming Ma, and Hang Li. Ranking measures and loss functions in learning to rank. Advances in Neural Information Processing Systems, 22, 2009.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750 15758, 2021.
Heng- Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. In Proceedings of the 1sl workshop deep learning for recommender systems, pp. 7-10, 2016.
Sajad Darabi, Shayan Fazeli, Ali Pazoki, Sriram Sankararaman, and Majid Sarrafzadeh. Contrastive mixup: Self-and semi-supervised learning for tabular domain. arXiv preprint arXiv:2108. 10008.1220 12296, 2021.
Domenico Dato, Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. Fast ranking with additive ensembles of oblivious and non-oblivious regression trees. ACM Transactions Information Systems (TOIS), 35(2):1-33) 33, 2016.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint srXie:1810.04805 2018.
Kevin Duh and Katrin Kirchhoff. Learning to rank with partially-labeled data. In Proceedings of the 31st anmual international ACM SIGIR conference on Reseorch and development in information retrieval, pp. 251-22 251-258, 2008
Jerome Н Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pp- 1189-1232, 2001.
Nicholas Frosst and Geoffrey Hinton. Distilling neural network into soft decision tree. arXiv preprint arXin:1711.09788, 2017.
Songwei Ge, Shlok Mishra, Chun-Liang Li, Haohan Wang, and David Jacobs. Robust contrastive learning using negative samples with diminished semantics. Advances in Neural Information Processing Systems, 34:27356 27368, 2021.
Weifeng Ge. Deep metric learning with hierarchical triplet loss. In Proceedings of the European conference on computer vision (ECCV), pp. 269 269-285, 2018.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Michael Glass, Gactano Rossicllo, Md Faisal Mahbub Chowdhury, Ankita Rajaram Naik, Pengshan Cai, and Alfio Gliozzo. Re2g: Retrieve, rerank, generate. arXiv preprinl arXiv:2207.066 06300, 2022.
Gregory Goren, Oren Kurland, Moshe Tennenholtz, and Fiana Raiber. Ranking robustness under adversarial document manipulations. Tn The 4 41st International ACM SIGIR Conference on Research & Development in Information Retricval, pp. 395-404 2018.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Adsances in Neural Information Processing Systems, 34:18932-18943, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to sulf-supe learning. Advances in neural information processing systems, 33:21271-21288 2020.
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances in Newrol Information Processing Systems, 35:507 520, 2022.
Huifeng Guo, Ruiming Tang, Yumming Ye, Zhenguo Li, and Xiuqiang Hc. Deepfm: factorization-machine based neural network for ctr prediction. arXiv preprint 8rXiv:1703.000 042404277 2017.
Paul Hager, Martin J. Menten, and Daniel Rueckert. Best both worlds: Multimodal contrastive learning with tabular and imaging data. In Proceedings of the IEEE/CCS /CFF Conference on Computer Vision and Pattern Recognation (CVPR), pp. 23924-23935, 23935, June 2023.
Ehsan Hajiramezanali, Nathaniel Lee Diamant, Gabriele Scalia, and Max W Shen. Stab: Self-supervised learning for tabular data. In NearIPS 2022 First Toble Representation Workshop, 2022.
Shuguang Han, Xuanhui Wang, Mike Bendersky, and Marc Najork. Learning-to-rant with bert in tf-ranking. arXiv preprint Xrx::20004.0 08476, 76, 2020.
Ben Harwood, Vijay Kumar BG, Gustavo Carneiro, lan Reid, and Tom Drummond. Smart mining for deep metric learning. In Proceedings of the IEEE international conference computer vision, pp. 2821-2829, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large langnage models. In International Conference on Artificial Intelligence and Slatistics, pp. 5549 5581. PMLR, 2023.
Dan Hendrycks, Mantas Mazcika, Saurav Kadavath, and Dawn Song. Using self-supervieed lcarning can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019.
David Holmmüleer, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned mips and boosted trees tabular data. arXiv preprint 00kk::2407.04491,, 2024.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint ar arxi:18811.001 2018.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.066 066788 2020.
Sergei Ivanov and Liudmila Prokhorenkova. Boost then convolve: Gradient boosting meets graph neural networks. arxiv preprint arXic:2101.085. 2021.
Alan Jeffares, Tennison Liu, Jonathan Crabbé, Fergus Imric, and Mihacla van der Schaar. Tangos: Regu- larizing tabular neural networks through gradient orthogonalization and specialization. arXiv preprint "xXi:2230300500 2023.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Jongheon Jeong and Jinwoo Shin. Consistency regularization for certified robustness of smoothed classifiers. Aduances in Newral Information Processing Systems, 33:10558-1000 10570, 2020.
Thorsten Joachims, Training linear svms in linear time. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 217-226, 2006.
Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. Aduances in neural information processing systems, 11123928-23944 2021.
Guolin Ke, Qi Meng, Thomas Finley, Taileng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in newral information processing systems, 30, 2017.
Guolin Ke, Jia Zhang, Zhenhui Xu, Jiang Bian, and Tie- Yan Liu. Tabnn: A universal neural network solution for tabular data. 2018
Guolin Ke, Zhenhui Xu, Jia Zhang, Jiang Bian, and Tie- Yan Liu. Deepgbm: A deep learning framework distilled by gbdt for online prediction tasks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Dala Mining, pp. 384 394, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXio:1412.6900 2014.
Nikita Kitaev, Steven Cao, and Dan Klein. Multilingual constituency parsing with self-attention and pro-training. arXiv preprint Xiv: 11760, 2018.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Aduances in neural information processing systems, 35:22199 222133, 2022.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution In International Conference Learning Representations, 2022.
Branislav Kveton, Ofer Meshi, Masrour Zoghi, and Zhen Qin. On the value of prior in online learning to rank. In International Conference on Artificial Intelligence and Statistics, pp. 6880 6892. PMLR, 2022.
Sudarshan Dayaneshwar Lamkhede and Christoph Kofler. Recommendations and results organization netflix search. In Proceedings of the 15th ACM Conference on Recommender Systems, pp. 577 579, 2021.
Kyungmin Lee and Jinwoo Shin. Rényiel: Contrastive reprosentation learning with skew rényi divergence. Advances in Newral Information Processing Systems, 35:6463 6477, 2022.
Roman Levin, Valeriaa Cherepanova, Avi Schwarzschild, Arpit Bansal, C Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, and Micah Goldblum. Transfer learning with deep tabular models. arXiv preprint arXiv:2206.1559 2022.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. angreevel- generation for knowledge- intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-944 9474,, 2020.
Haitao Li, Jia Chen, Weihang Su, Qingyao Ai, and Yiqun Liu. Towards better web search performance: pre-training, fine-tuning and learning to rank. arXiv preprint arXiv:2303.0/ 04 2023a.
Yuchen Li, Haoyi Xiong, Linghe Kong, Zeyi Sun, Hongyang Chen, Shuaiqiang Wang, and Dawei Yin. MPGraf: a Modular and Pre-trained Graphformer for Learning to Rank at Web-scale In 2023 IEEE International Conference on Data Mining (ICDM), pp. 530-348, Los Alamitos, CA, USA, December 2023b. IEEE Computer Society, doi: 10.1109 10.1109/ICDM5852)) (CDMM58522.0033.. 00043. URL HRLL/ttps:///oot ieeecomptterssciieyy..oggg 10. Dt.119//CCM58522..O 2023. 00043
16

---

Published in Transactions on Machine Learning Research (11/2024)
Yuchen Li, Haoyi Xiong, Linghe Kong, Qingzhong Wang, Shuaiqiang Wang, Guihai Chen, and Dawei Yin. S2phere: Semi-supervised pre-training for web search over heterogeneous learning to rank data. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, pp. 4437-4448, New York, NY, USA, 2023c. Association for Computing Machinery. ISEN 9798400701030.
doi: 10. 05555/55003533509999999
URL httpss://doi org/ 10. 1we.11114/338800 3599933
Yuchen Li, Haoyi Xiong, Qingzhong Wang, Linghe Kong, Hao Liu, Haifang Li, Jiang Bian, Shuaiqiang Wang, Guihai Chen, Dejing Dou, and Dawei Yin. Coltr: Semi-supervised learning to rank with co-training and pweretrrretttiiiiitio for web search. IEEE Transactions on Knowledge and Data Engineering, 35(12): 12542-12555, 2023d. doi: 10.1109/TKDDI TKDE.2003.3220000000
Yuchen Li, Haoyi Xiong, Linghe Kong, Jiang Bian, Shuaiqiang Wang, Guihai Chen, and Dawci Yin. Gs2p: 8 generative pre-trained learning to rank model with over-paraneetrriaaion for web-scale search. Mach. Learn., 113(8):5331 5349, January 2024. ISSN 0885-6125. doi: 10.1007/s10994-000 023-009999-------66-----9---9-99 URL
https://ao1.org100..00 1tt///0000000//110990000333
Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. xdeepfm:: Combining explicit and implicit fcature intcractions for recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1754 1763, 2018.
Jianghao Lin, Yanru Qu, Wei Guo, Xinyi Dai, Ruiming Tang, Yong Yu, and Weinan Zhang. Map: A model-agnostic pretraining framework for click through rate prediction. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1384-1395, 2023.
Guang Liu, Jie Yang, and Ledell Wu. Plab: Using the pre-trained language model for modeling tabular data. arXiv preprint Xrxiv:2209. 08060, 2022.
Hao Liu, Yixin Chen, Bradley Fritz, and Christopher King. Tabcontrast: A local-global level method for tabular contrastive learning. In NewrlPS 2023 Second Table Representation Learning Workshop, 2023.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Adversarial multi-task learning for text classification. arXiv preprint 1xXix:1704.057 2017.
Tie- Yan Liu. Learning to rank for information retrieval. Foundations and Trends3 in Information Retrieval, 3(3):225 331, 2009.
Zhaocheng Liu, Qiang Liu, Haoli Zhang, and Yuntian Chen. Dnn2lr: Interpretation-tnspire feature crossing for real-world tabular data, arXiv preprint arXiv:2008.0977 2020.
Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Fabrizio Silvestri, and Salvatore Trani. Post-learning optimization of tree ensembles for efficient ranking. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pp. 959-952, 2016.
Haoran Luo, Fan Cheng, Heng Yu, and Yuqi Yi. Sdtr: Soft decision tree regressor for tabular data, IEEE Access, 9:55999-5601 56011, 2021.
Yuanfei Luo, Hao Zhou, Wei-Wei Tu, Yuqiang Chen, Wenyuan Dai, and Qiang Yang. Network on network for tabular data classification in real-world applications. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2317-2326, 2020.
Kushal Majmundar, Sachin Goyal, Pranecth Netrapalli, and Prateek Jain. Met: Masked encoding for tabular data. Xiv preprint arXiv:2206.0885 2022.
David McClosky, Eugene Charniak, and Mark Johnson. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pp. 152 159, 2006.
Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Ganesh Ramakrishnan, Micah Goldblum, Colin White, al. When do neural nets outperform boosted trees on tabular data? arXiv preprint 0rX2v22305.02997, 2023.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Bhaskar Mitra, Nick Craswell, et al. An introduction to neural information retrieval. Foundations and Trendæ0 in Information Retrieual, 13(1):1-126, 2018.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii Virtual adversarial training: regular- ization method for supervised and semi-supervised learning. TEEE transactions on pattern analysis and machine intelligence, 1998919979 2018.
Jaehyun Nam, Woomin Song, Seong Hyeon Park, Jihoon Tack, Sukmin Yun, Jaehyung Kim, and Jinwoo Shin. Semi-supervised tabular classification via in-context learning of large language models. In Workshop on Efficient Systems for Foundalion Models@ ICML2023, 2023a.
Jaehyun Nam, Jihoon Tack, Kyungmin Lee, Hankook Lee, and Jinwoo Shin. Stunt: Few-shot tabular learning with self-generated tasks from unlabeled tables. arXiv preprint arXiv:2303.0090 00918, 2023b.
Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G Azzolini, et al. Deep learning recom- mendation model for personalization and recommendation systems. arxio preprint 2019.
Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. Multi-stage document ranking with bert. arXiv preprint arXi:: 1ix 1910.11044444 2019.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEFE conference computer vision and pattern recognition, pp- 4004-4012, 2016.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and lan Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. Advances in neural information processing systems, 31, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation lcarning with contrastive predictive coding. Xixi preprinl aiv: vrxXi:1807.000 307.03748, 2018.
Feifci Pan, Mustafa Canim, Michael Glass, Alfio Gliozzo, and James Hendler. End-to-end table question answering via retrieval-uugmented generation. ar preprint ar Xri: 16714, 2022.
Liang Pang, Jun Xu, Qingyao Ai, Yanyan Lan, Xueqi Cheng, and Jirong Wen. Setrank: Learning a permutation-invarien ranking model for information retrieval. In Proceedings of the 43rd international ACM SIGIR conference research and development in information retrieval, pp. 499-508, 2020.
Gustavo Penha, Arthur Câmara, and Claudia Hauff. Evaluating the robustncss of retricval pipelines with query variation generators. In European conference on information retrieval, pp. 397 412. Springer, 2022.
Matthew E Peters, Sebastian Ruder, and Noah A Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. arXiv preprint arXiv:1903.0500 2019.
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arXie preprint arXiv:1909, 06312, 2019.
Dong-Hye Hyun Lee Pseudo-Label The simple and efficient supirvuper learning method for deep neural networks. In ICML 2013 Workshop: Challenges in Representation Learning, pp. 1 1-6, 2013.
Tao Qin and Tie Yan Liu. Introducing LETOR 4.0 datasets. CoRR, 2597, 2013. URL http: //trxiv.org/aas///// 1306/ 2597
Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Mike Bendersky, and Mare Najork. Are neural rankers still outperformed by gradient boosted decision trees? 2021.
Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. Product-basok neural networks for user response prediction. In 2016 IEEE 16th international conference on data mining (ICDM), pp. 1149 1154. TEFE, 2016.
18

---

Published in Transactions on Machine Learning Research (11/2024)
Steffen Rendle. Factorization machines. In 2010 IEEE International conference on data mining, pp. 1000. TEEE, 2010.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh Rawat, and Mubarak Shah. In defense of pseudo-labeling An uncertainty-aware pseudo-label selection framework for semi-supervised learning. arXiv preprint arXiv:2101.06329 2021.
Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanic Jegelka. Contrastive learning with hard negative samples. orXiv preprint orXiv:201000499,, 2020.
Ivan Rubachev, Artem Alckberov, Yury Gorishniy, and Artem Babenko. Revisiting pretraining objectives for tabular deep learning. Xixi preprint urXiv:2207.03200 2022.
Mchdi Sajjadi, Mchran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. Advances in neural information processing systems, 29, 2016.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815-823, 2015.
Maximilian G Schuh, Davide Boldini, and Stephan A Sieber. Twinbooster: Synergising large language models with barlow twins and gradient boosting for enhanced molecular property prediction. ar preprint arXiv:2401.044 704448. 2024.
Ira Shavitt and Eran Segal. Regularization learning networks: deep learning for tabular datasets. Advances in Newral Information Processing Systems, 31, 2018.
Weiwei Shi, Yihong Gong, Chris Ding, Zhiheng MaXiaoyu Tao, and Nanning Zheng, Transductive semi- supervised deep learning using min-max features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 299 315, 2018.
Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep lcarning is not all you need. Information Fusion, 81:84 90, 2022.
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprinl arXiv:2106.01344 2021.
Yangqiu Song and Dan Roth. Unsupervised sparse vector densification for short text similarity. In Proceedings of the 2015 conference of the North American chapter of the association for computational linguistics: human language technologies, pp. 1275 1280, 2015.
Yumin Suh, Bohyung Han, Wonsik Kim, and Kyoung Mu Lee. Stochastic class-based hard example mining for deep metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Poltern Recognition, pp. 7251-7259, 2019.
Yi Sui, Tongzi Wu, Jesse Cresswell, Ga Wu, George Stein, Xiaoshi Huang, Xiaochen Zhang, Maksims Volkovs, et al. Self-supervised representation learning from random data projectors. arXiv preprint arXiv:2310.07755, 2023.
Baohua Sun, Lin Yang, Wenhan Zhang, Michael Lin, Patrick Dong, Charles Young, and Jason Dong. Supertml: Two-dimensional word embedding for the precognition on structured tabular data. In Proceedings of the IEEE/CVF conference on compuler vision and paltern recognilion workshops, pp. 0 0, 2019.
Robin Swezey, Aditya Grover, Bruno Charron, and Stefano Ermon. Pirank: Scalable learning to rank via differentiable sorting. Advances in Newral Informalion. Processing Systems, 0921644-2165. 2021.
Tahir Syed and Behroz Mirza. Self-supervision for tabular data by learning to predict additive homoskedastic gaussian noise pretext. ACM Transactions on Knowledge Discovery from Data, 2023.
19

---

Published in Transactions on Machine Learning Research (11/2024)
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pp. 10268-1022 10278. PMLR, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint or Xxx::307.009288 2023.
Talip Ucar, Fhsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of tabular data for self- supervised representation learning, Advances in Neural Information Processing Systems, 34:18853 18865, 2021.
Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoc Lc. Towards domain-agnostic contrastive learning. In International Conference on Machine Learning, pp. 10530-1054h PMLR, 2021.
Ellen M Voorhoes. The tree robust retrieval track. In ACM SIGIR Forum, volume 39, pp. 11-20. ACM New York, NY, USA, 2005.
Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. Den v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In Proceedings of the web conference 2021, pp. 1785-1797, 2021.
Weiyao Wang, Byung-Hay Hak Kim, and Varun Ganapathi. Regelr: A self-supervised framework for tabular representation learning in the wild. arXiv preprint 1r1.:2211..0115,, 2022a.
Xiao Wang, Haoqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen. On the importance of asymmetry for siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16579, 2022b.
Zifeng Wang and Jimeng Sun. Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 35:2902 2915, 2022.
Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. In Proceedings of the IEEE international conference on computer vision, pp. 2840-2848, 2017.
Chen Wu, Ruqing Zhang, Jialeng Guo, Wei Chen, Yixing Fan, Maarten de Rijke, and Xueqi Cheng, Certified robustness to word substitution ranking attack for neural ranking models. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 2128 2137, 2022a.
Chen Wu, Ruqing Zhang, Jiafeng Guo, Yixing Fan, and Xueqi Cheng, Are neural ranking models robust? ACM Transactions on Information Systems, (122)11-36, 2022b.
Chen Wu, Ruqing Zhang, Jiafeng Guo, Maarten De Rijke, Yixing Fan, and Xueqi Cheng. Prada: practical black-box adversarial attacks against neural ranking models. ACM Transactions on Information Systems, 41(4):1-27, 2023.
Fen Xia, Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwisc approach to learning to rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning, pp. 1192-1199, 2008.
Qizhe Xie, Minh- Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10687-10698, 2020.
Enqiang Xu, Yiming Qiu, Junyang Bai, Ping Zhang. Dadong Miao, Songlin Wang, Guoyu Tang, Lin Liu, and MingMing Li. Optimizing e-commerce search: Toward a generalizable and rank-consistent pre-ranking model. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '24, pp. 2875-2879, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704414.. doi: 10. 10.115//3626772.361133 0826772.366344
URL http::///ooooo/1..11111
20

---

Published in Transactions on Machine Learning Research (11/2024)
Jiahuan Yan, Jintai Chen, Qianxing Wang, Danny Z Chen, and Jian Wu. Team up gbdts and dnns: Advancing efficient and effective tabular prediction with tree-hybrid mlps. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3679 3689, 2024a.
Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny 7. Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great on tabular prediction. arXiv preprint Xxi:22003.01841, 2024b.
Shuo Yang, Sujay Sanghavi, Holakou Rahmanian, Jan Bakus, and Vishwanathan SVN. Toward understanding privileged features distillation learning-to-rank, Advances in Neural Information Processing Systems, 35: 26658 26677, 2022.
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pp. 189-196, 1995.
Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. Pretrained transformers for text ranking: Bert and beyond. In Proceedings of the 14th ACM International Conference on web search and data mining, pp. 1154 1156, 2021.
Chao Ye, Guoshan Lu, Haobo Wang, Liyao Li, Sai Wu, Gang Chen, and Junbo Zhao. Ct-bert: learning better tabular representations through cross-table pre-training. arXiv preprint arXiv:2307.00 04377,44400, 2023.
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33:11033-11000 2020.
Hai-Tao Yu. Pt-ranking: A benchmarking platform for neural learning-to-rank, 2020.
Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X Pham, Chang D Yoo, and In So Kweon. How does simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning. In International Conference on Learning Representations, 2021.
Peng Zhang, Dawei Song, Jun Wang, and Yuexian Hou. Bias-variance decomposition of ir evaluation. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information relrieval, pp. 1021-1024, 2013.
Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. arXiv preprint or arXim:2305.06090 2023.
Yitan Zhu, Thomas Brettin, Fangfang Xia, Alexander Partin, Maulik Shukla, Hyunseung Yoo, Yvonne A Evrard, James H Doroshow, and Rick L Stevens. Converting tabular data into images for deep learning with convolutional neural networks. Scientific reports, 11(1):11325, 2021.
Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. Rankt5: Fino-tuning t5 for text ranking with ranking losses. In Proceedings of the 46th Internotional ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2023.
A Appendix
A.1 Detailed discussion of related work
Maaring-To-Rann In our paper, we focus on the traditional LTR setting where the features are all numeric (tabular data). However, there is a line of work in LTR where raw text is also an input. In this case, one can
leverage large language models in the ranking setting (Zhuang et al. 2023 Han et al. 2020 Yates et al.
2021 Nogueira et al. 2019 Mitra et al. 2018
In tabular LTR problems, the dominant models currently used are gradient boosted decision trees (GBDTs) Friedman 2001 which are not deep learning models. GBDT models, which perform well on tabular data,
21

---

Published in Transactions on Machine Learning Research (11/2024)
are adapted to the LTR setting via losses that are surrogates for ranking metrics like NDCG. Surrogate losses (including Raaallaaann//RnnkkNaaa (Burges 2010) and PiRank (Swezey et al. 2021 are needed because many important ranking metrics (like NDCG) are non-differentiable. The combination of tree-based learners and ranking losses has become the de-facto standard in ranking problems, and deep models have yet to
outperform them convincingly (Qin et al., 2021 Joachims 2006
Ai et al. 2019
Bruch el al. 2019 Ai el al.
2018
Pang et al. 2020).
Deep tabular models. Given the success of neural methods in many other domains, there have been many attempts to adapt deep models to the tabular domain.
Borisov et al. (2022) categorize existing techniques for using deep neural networks over tabular data into four
types: (1) Encoding-base
methods such as VIME (Yoon el al.)
2020), SCARF [Bahri et. al. 2021), IGTD
Zhu et al.
2021), and SuperTTML (Sun et al.
2019);
(2) Novel hybrid architectures such as DeepFM
Guo ct 2017], xDccpFM (Lian et al. 2018). ). and many others Cheng et al. (2016); Frosst & Hinton [2017):
Ke et al. (2018) 2019); Popov et al. (2019) Luo et al. (2020); Liu el al. (2020); Ivanov & Prokhoren kova
(2021); Luo et al. [2021] (3) Transformer- beseemeer---dddddd architectures including SAINT (Somepall et al.
2021
TabNet (Arik & Pfister 2021). TTT-Traasformrr Gorishniy et al. (2021), TabTTansformer (Huang et al.
2020
and ARM-Net Cai et al. (2021p); (4) Regularized DNNs (Shavitt & Segal, 2018 Kadra et al.
2021
For instance, TANGOS introduced special tabular-specific regularization to try to improve deep models"
performance (Jeffares et al. 2023). We also note that a more recent work, Tireehhybit simple MLP
YYan
et al. 2024 proposes to combine MLPs and GBDTs together for the tabular prediction problem.
Self-supervised learning. Self-suervised learning (SST.) or unsupervised pretraining has improved performance in scttings where there is significant source of unlabeled data like text (Devlin et al., 2018) and images (Chen et al.) 2020). In SSL, deep models are first pretrained on perturbed unlabeled data using self-supervised tasks to learn useful representations for the data. Then these models are finetuned for a downstream task with labeled data. Finctuning often takes one of two forms: (1) linear probing (a popular finetuning strategy in text and images (Chen & He 2021 Chen et al., 2020 Peters et al. 2019) ). wheeee we freeze the pretrained model and only update the linear head during supervised finetuning, and (2) full finetuning, where we update the whole model during supervised finetuning (Devlin et al. 2018). Sometimes a mix of the two is used (Kumar et al.) 2022). The core idea. behind prominent SSL approaches like SimSiam and SimCLR is to carelully perturb input training samples, and train representation that is consistent for pertur bations of the same sample. This provides robustness to natural perturbations and noise in data Hendrycks et al. 2019).
Inspired by the success of pretraining and self-supervised learning in images and text, several works show how to apply SSL to unlabeled tabular data. One strategy is to corrupt tabular data and train deep model to reconstruct it (Yoon al. 2020: Majmundar al. 2022: Ucar al. 2021 Nam et al. 2023b Lin et al.) 2023; Syed & Mirza 2023 Hajiramezanali et al.) 2022 Another approach is to use contrastive losses, which
have been highly successful in the image domain CChn et al. 2020 These methods are also applicable to
our case bocause tabular data, like image data, is often composed of fixed-dimensional real vectors (Verma
let al. 2021 Bahri et al. 2021 Lee & Shin 2022: Hager el al.) 2023 Liu el al., 2023 Darabi et al. 2021
Concurrent work has also investigated training tabular-specific LLMs for tabular tasks (Yan et al. 2024b
Schuh ct al. 2024). Concurrent to us, Holzmüller et al. (2024) also proposcs protraining (with a modified
architecture) to improve labular deep learning performance. Rubachev al. (2022) evaluate variety of
different pretraining methods for tabular learning across many different datasets, finding that there is not
clear state of the art.
Robustness in LTR. There has been prior work on studying worst-case behavior (robustness) of rankers
(Voorhees 2005 Zhang et al. 2013 Goren et al., 2018 Wu et al. 2022b a Penha et al.
2022). Some previous
metrics measure a model's robustness against adversarial attack (Goren et al. 2018
Wu et al. 2022a
2023
Others measure the model's per-query performance variance on a dataset (Voorhees) 2005 Zhang et 2013
Wu et al. 2022b). Our outlier metric, OuDG is departure from previous work because it is not. directly measure of robustness and it is possible for model to perform better on outlier data.
Transfer learning. We summarize some related work on transfer learning in tabular domain. One direction
revolves around pretraining models on common columns across many datasets (Zhu et al., 2023 Wang &
22

---

Published in Transactions on Machine Learning Research (11/2024)
Sui et al. 2023: Ye et al. 2023 Another direction leverages LLMs (large language models) to do lew-shot tabular learning (Hegselman et al. 2023 Liu et al. 2022 Nam al. 2023a Pretrained model features are often useful across datasets. For example, text features are transferable between text datasets Devlin et al. 2018 which has contributed to the recent success of foundation models (Devlin et al. 2018 still unclear how to achieve transfcrability tabular LTR, where previous work has leveraged common columns (or at least column documentation) to achieve transfer learning (Levin et al. 2022 Hegselmann al.) 2023). However most
Sun 2022
Touvron et al. 2023 Kojima el al. 2022 Brown et al., 2020). Unfortunately,
LTR datasets do not have common columns or publicly-documentee column meanings (Qin & Liu 2013 Chapelle & Chang 2011 Lucchese et al. 2016). Developing method to transfer features in tabular L.TR is
8n important open problem.
Pretrained models in learning to rank. Concurrent to the submission/review of this paper, recent works have also validated the benefits pretraining in LTR. S2phere (Li et al. 2023ch proposes an elaborate recipe combining pscudo-labeling, contrastive learning, and transformer- based masked autocncoding to pretrain LTR model, and shows that it performs well in offline and online metrics. MPGral (Li et al., 2023b)
proposes pretraining method for LTR that combines both the query-document regression formulation of LTR and the graph-based formulation of LTR into a single model. GS2P (Li ct al. 2024 proposes to models to generate pseudo-labels for the final model to learn. Like us, GRACE (Xu et al.) 2024) uses contrastive learning for pre-ranking (retrieval); unlike our method, they use this process to align item embeddings with pretrained embeddings during supervised training phase. COLTR (Li et al. 2023d) proposes increase the dimensionality of the features with Fourier features and alternative pseudo-labeling between listwise and pointwise ranker to produce better ranking results.
use an ensemble of differently-traneen
Li
ct al. 2023a finetune pretrained language models for the document ranking task.
A.2
Experimental Details
A.2.1
Dataset statistics
In table Table provide the details of our three public datasets and the proprietary real-world dataset. Note that for the public dataset experiments we sub-sample the pre-training dataset in differrnt ways (as described in Sections 3 4.1.1 and 4.1.2) to create the training dataset and simulate scarcity of labeled samples. However, we use the same validation dataset without and with the labels for the self-supervised pretraining and supervised learning stages, respectively. Note that in all these datasets, the higher the label value the higher the relevance of the item.
Table 5: Datasct statistics on the MSLRWEB30K, Yahoo Set1, Istella S, and proprictary datascts.
Number of query groups Supervised learning
Number of features
List of labels
Pretraining
Dataset
Pretrain
SSL Val
Train
Val
Test
MSLRWEBB3KK
18151
6072
Subset Pretrain
SSL Val
6072
136
(0,1,2,3,4)
Yahoo Set1
14477
2147
Subset Pretrain
SSL Val
5089
700
{0,1,2,3,4)
Istella__
19200
7202
Subset Pretrain
SSL Val
6523
220
(0,1,2,3.4]
Proprictary dataset
35.6M
1.62M
3.68M
171k
1.08M
63
(0.1)
A.2.2 Outliers query group details
Here detail how select the outlier query groups for the Outlier-NDCG metrics (Section 2.1 For our proprictary online shopping datasct, there are known outlier features that cause low quality predictions. These outliers arise from noise introduced in various slages in the data pipeline. We have mitigation strategies in place against these feature outliers, but they are often imperfect. Because we already know some of the outliers features, we deem any query group with items with outlier features as an outlier. These outliers comprise roughly 10% of the test set, which is still a large number of query groups (much larger than the number of query groups in the outlier datasets of the public datasets).
23

---

Published in Transactions on Machine Learning Research (11/2024)
For our public datasets, we systematically select outliers follows: generate histogram with 100 bins for each feature across the validation dalasel. For example, Istella (Tamkhede & Kofler 2021) has 220 features, so we have 220 different histograms. For each histogram, we scan from left to right on the bins until we have encountered at least G empty bins in a row, and if there is less than 1% of the validation set above this bin, then all the leature values above this bin are considered outliers. We also repeat this process right to left. Any test query group containing items with outlier feature values is labeled an outlier query group, and placed in the test outlier dataset. In Figure we present an example histogram of feature in Istella S dataset.
Histogram for Istella-validation dim=174
10"
102
10
Frequency
10a
10
Outliers
10
Feature value
Figure 5: An example of outlier detection in Istella Lucchose al. 2016 for our Outlier-NDCG motric.
Because different datasets have differently-sizeed typical gaps, we tune for each dataset (MSLR, Yahoo, Istella) such that the resulting percentage of outlier queries is as close to 1% of the test set as we can get. MSLRWEB30K has G = 5, with 0.65% (40/6072) outlier queries, Yahoo has G 20, with 1.4% (30/2147) outlier queries, and Istella has G = 32, with 0.46% (34/7202) outlier queries. 1% is hyperparameter that can be tuned according to the user's 's goals.
A.2.3 Model Settings and Training Details
Pretrained DL model.
Pretraining: (1) the encoder used for pretraining the tabular RosNet (Gorishniy et al. 2021 with three ResNet blocks, with the final linear layer removed,
(2) pretraining is done on the entire datasel with learning rate 0.0005 using Adam (Kingma Ba 2014 (3) with SimSiam/Sinnt RaLLRRRRnn/SiCLL Sample, we tuned among four different augmentations for each pretraining method: randomly zcroing out features ("Zcroing") with probabilities 0.1 0.7, and Gaussian noise with scale 1.0 or 2.0, (4) for SCARF we tune the corruption probability in (0.3, 0.6, 0.9}, (5) for VIMF-self we tune the corruption probability in {0.3, 0.5, 0.7). (6) for DACL+ we tune the mixup amount in {0.3, 0.6, 0.9}, (7) for SubTab we divide input features into 4 subsets with 75% overlap, and tune the masking probability in f0.3, 0.6, (8) we pretrain for 300 epochs for all methods, and (9) we use a batch size of roughly 200000 items for SimCLRRRRan, SimSiam, VIMF-self, and SubTab as they are less GPU memory intensive while using batch size of roughly 2000 items for MAACLR//DA++/SSE
which are more GPU memory intensive.
We also evaluate SAINT (Somepalli et al. 2021 architecture, we follow (Somepalli et al.) 2021) using L = 4 layers, dropout of 0.1, attention heads h = 4, with self-attention muery/key/val value dimension of 16 and a intersample attention query/key /value dimension of 64. For augmentation, we follow (Somepalli et al.) 2021 and use a CutMix mask parameter of 0.3 and MixUp parameter of 0.2. Again following (Somepalli et al.] 2021) we weight the denoising loss 10 times more than the contrastive loss. For all other minor architectural details, we follow the code of (Somepalli al. 2021 We pretrain for 5 epochs with batch size of roughly 100 items due to GPU memory contraints and
transformer-based pretrainining method. For the
24

---

Published in Transactions on Machine Learning Research (11/2024)
also the time cost (one epoch of saint took 300x longer than epoch of SimCLR- Rank in MSLRWEB30K, 6000x longer in Yahoo 1 and 1000x longer in Istella). We use learning rate of 5e-5.
Pinetuning: (1) finetuning is done on the labeled train set by adding three-layer MLP to the top of the pretrained model and training only this head for 100 epochs and then fully finetuning for 100 epochs using Adam with a learning rate 5e-5, (2) we use an average batch size of roughly 1000 items (may vary based on query group size), (3) we use the LambdaRank loss (Burges 2010), (4) we use the validation set perform early stopping (i.e. using the checkpoint that performed best the validation set to evaluate the test set).
For SAINT we follow the finetuning process of (Somepalli et al. 2021 in using the output of the CLS embedding as the representation for an item, and tune the learning rate in (5e-5, le-5, 5e-63 as we found the performance to be sensitive to this setting.
Implicit binary user feedback setting. We reuse the methodology of Section 4.1.1 except we use a scoring head of one linear layer (as opposed a three layer MLP) for pretrained models. We found that this improved stability and performance in the binary label setting.
SimCLR-Rank
SimSiam tSSE MSLRWEB30K
Supervised t-SNE MSL RWEB3OK
60
0.0
0.0
1.0
1.0
20
20
2.0
2.0
3.0
3.0
4.0
4.0
-20
20
20
3.0
-40
-60
4.0
-60
-60
-60
-40
40
60 -60
-40
-20
0
40
60
-60
-40
-20
40
x1
x1
x1
Figure 6: We plot the t-SNE plots of embeddings produced by three different encoders for the MSLRWEB30K
dataset: (1) pretrained by SimCLR-Rank, (2) pretrained by SimSiam, (3) trained via supervised training on the entire training set on roughly 1000 samples from MSLR. Marker size and color indicates relevance, We find (1) MOnCT.R-Ran//Siiii supervised encoder. (2) SimCLR-Rank produces more spread-out embeddings with less defined clusters than SimSiam.
cluster different relevances effectively but do not order them well as the
SimCLR-Rank t:SNE Istella
SimSlam t:sNE Istella
Supervised t-SNE Istella
60
60
60
40
1.0
40
2.0
20
20
3.0
0-
4.0
20
2.0
-20
-40
3.0
-40
-40
3.0
4.0
-60
4.0
16.
-40
-20
20
60
-40
-20
20
40
60
-60
-40
-20
x1
x1
x1
Figure 7: We plot the t-SNE plots of embeddings produced by three encoders: (1) pretrained by SimCLR-Rank, (2) pretrained by SimSiam, (3) trained via supervised training on the entire training set on roughly 1000 samples from Istella Marker size and color indicates relevance. We find (1) SimCLR- Rank/SimSian cluster different relevances effectively but do not order them well as the supervised encoder. (2) SimCLR-Rank produces more spread-out embeddings with less defined clusters than SimSiam.
No Pretrained DL model. There are a few rankers we evaluated for the no- no-pretrain DL bascline: (1) a 3-layer tabular ResNet. from (Gorishniy al. 2021 with three-layer ML.P top of it, and was trained
25

---

Published in Transactions on Machine Learning Research (11/2024)
for 300 epochs on the labeled training set with learning rate tuned in {0.01, 0.001, 0.0001) using Adam, with early stopping using the validation set, (2) DeepFM (Guo et al. 2017 model following the model architecture in the paper with learning rate tuned in {0.05, 0.005, 0.0005), and (3) a DCNv2 (Wang et al, 2021 model with learning rate tuned in {0.05, 0.005, 0.00053 and cross layers tuned in (2,3,4). DcepFM and DCNv2 were designed to use calegorical features, and in MSL.WEEB30K/ /Yahoo / S do not document their categorical features (if any). We performed an analysis of the datasots and found 5, 48, 6 categorical features in ISLRWWB3OK/ / Yahoo Set1 /Istellllla S respectfully each with cardinality at most 3. We embedded these categorical features into embeddings of dimension 8, following the of-tumbb in (Wang
et al. 2021)
GBDT model. The GBDT ranker is the one from lightgbm (Ke et al. 2017 and we grid search the number of leaves in {31, 96, 200} minimum data in leaf in {20, 60, 200}, and column sample in {0.5, 0.9, 1.0] individually for each data point (for total of difference choices), while letting the rest of the parameters be the default in lightgbm (our tuning strategy similar to Qin et al. (2021)). We use the LambdaRank loss to train the models.
GBDT + pseudolabeling. We add pseudo-labeled GBDTss as semi-supervised GBDT baseline. For GBDTs with pseudo-labeling (GBDT-pseudo) we propose the following algorithm as there are no references to our knowledge. First, we train the GBDT on the labeled train set, and then use this GBDT to pseudo-labels for the unlabeled train set. A GBDT ranker outputs real-valued scores. To convert real-valued scores into relevance scores, we take the resulting scores and evenly bin them into 31 buckets. The scores converted to O are the lowest bucket, the scores converted to 1 are the second lowest bucket, and so on. We choose 31 because that is the largest number of labels allowed for GBDT ranking in lightghm (the reason is because the LambdaRank loss requires the calculation of 2label which can get very large).
GBDT + PCA. For GBDT+PCA, we take the entire training set's features and learn a PCA on it with {5, 10, 15) components, similar to (Duh & Kirchhoff, 2008), to produce (5, 10, 15) new features (i.c. we tune among 5, 10, 15). We then train on the labeled train set, with both the original features and the new PCA features logether.
GBDT + PCA+ + pscudolabeling. We also introduce a baseline where we perform PCA on the full training dataset, and then perform pseudolabeling as above. We tune the PCA similarly to the GBDT + PCA baseline.
SimCLR-Rank + GBDT//imSian + GBDT. We also explore using pretrained neural embeddings as features in the GBDT. To do this, we used the pretrained SimCLR-Rand (or SimSiam) model to produce embeddings over the labeled data, concatenated the original features with the embeddings, and then trained a GBDT on this new dataset. We use the same tuning grid as for the original GBDT model.
Comparing finetuning strategies. When linear probing (LP), we freeze the pretrained model and update only a linear head on top of it for 200 epochs. In multilayer probing (MP), we freeze the pretrained model and update a 3-layer MLP head on top of it for 200 epochs. In full finetuning (FF), we use the finetuning strategy from Section 4.1.1 We use the validation set for early stopping.
All reported values for Outlier-NDGG are those achieved by the corresponding ranker with best overall NDCG. We report all results as averages over trials, and use single V100 GPUs from shared cluster to run the experiments.
A.2.4 Details of Binary Label generation used in Section 4.1.2
Following the methodology from Yang et al. 2022 we generate stochastic binary labels y from the relevance labels r as y 1(t r + G1 > t. Tharget + Gob where t is a temperature parameter, G1, Go are standard Gumbels, and Tearget is a parameter controlling how sparse the binary labels are. Higher Trarget leads to higher label sparsity. Yang et al. (2022) show that y is with probability a(t (r - Furgeet) where o is the sigmoid function. We set datasct for each of MSLRWEB30K, Yahoo Set1, and Istella S where we convert the relevance labels in the moining/valldaatee sets to binary labels and keep the true relevance labels in the test set.
as in Yang et al. (2022). For a given Traggt: we produce produce a new
26

---

Published in Transactions on Machine Learning Research (11/2024)
A.2.5 Large industry-scale dataset experimental details
For our DL ranker, first we pass the features (numerical and categorical) through some featurizers. Then we use a simple 4 hidden layer MLP, with and ReLU activation, to predict a single scalar score for cach (query, item) pair. We also use dropout and batch normalization in input and hidden layers.
For the SimCLR pretraining stage, we remove the last linear layer of the above ranker to create the main encoder to produce the pretrained embeddings. Projector encoder here is 2 hidden layer MLP, with ReLU activation and no bias, which out puts an input dimensional embeddingg We also do not use any normalization or dropout in this projector.
For SimSiam we use the same embedder and a similar projector. The only difference in the projector being batch normalization in the hidden layers and batch normalization without affine learnable parameters in the output layer. Additionaly, SimSiam also employs predictor MLP, with a single hidden layer and ReLU activation, and with the same input and output sizes. It uses batch normalization and no bias in the hidden layer and input and hidden dropouts.
For pretraining, we used an augmentation which randomly zeroed out some features. We also tried randomly swapping some features between items in the same batch or query group, but they didn't perform as well as zeros augmentation. During the fine-tuming stage we discard the projectors and predictor and add back the linear layer complete the non-pretrained ranker structure. DI. rankers are trained trained/fne-tuned ffine-tuned using PiRank loss Swezey et al. (2021), batchsize of 1000, AdamW optimizer with learning rale of 0.00001, weight decay of 0.01 and cosine scheduling with warmup. All training stages and experiments were conducted machines with 4 GPUs, where ran 100k iterations with ddp and picked the best checkpoint based on validation metric.
GBDT rankers were trained with LambdaRank using the lightgbm package. They were tuned with Bayesian hyperparameter optimization strategy by tuning number of leaves, learning rate, and minimum data in leaf in reasonably large range with maximum of 150 jobs. We used default values of the library for the other hyperparameters.
A.3
Additional results
A.3.1
Runtime comparisons of pre-training strategies
In Table G provide the runtime comparisons between SimSiam, SimCLR, and SimCLR-Rank (new method).
Table 6: Scconds per epoch comparison between pretraining methods. Average over 3 trials. The encoder
use for pretraining is the tabular ResNet. (Gorishniy el al. 2021) with the final linear layer taken out.
Seconds per epoch
Method
Complexity
MSLRWEB30K
Yahoo Set1
Istella
SimCLR
O(B²
102.81 + 1.14
14.45 + 0.04
69.26 + 0.16
Sim MemCCRRRami (new)
O(BL²)
9.4700 - 0.02
1.970 + 0.01
5.950 + 0.04
SimSiam
O(BL)
0.8700 + 0.01
0.130 + 0.00
0.730 + 0.00
A.3.2 Additional Results on linear probing vs finetuning
Here we provide more results on the question of linear probing vs full finetuning, as discussed in Section 4.3 In Tables and we compare different probing strategies for all datasets when using SimSiam and Rank, respectively, the pretraining method.
27

---

Published in Transactions on Machine Learning Research (11/2024)
Table 7: SimSiam under different finetuning strategies on NDCG/Outlier-DDCG, averaged over trials (LP Linear Probing, MP Multilayer Probing, FF Full Finctuning). We find that linear probing and MLP probing perform extremely poorly (except Yahoo Setl, where MLP probing performs well).
Method
||
MSLRWEB33K (t)
Yahoo Set1 (t)
Istella (t)
NDCG (f)
LP
0.2679 + 0.0007
0.6089 0.0032
0.3805 + 0.0042
MP
0.2764 + 0.0001
0.6137 0.0022
0.4484 + 0.0020
FF
0.3935 + 0.0034
0.6107 0.0035
0.5618 + 0.0049
Outlior-DDCG (t)
LP
0.1803 + 0.0033
0.5088 + 0.002
0.4407 + 0.0388
MP
0.1749 + 0.0023
0.5157 + 0.0080
0.5324 + 0.0002
FF
0.3149 + 0.0119
0.52 + 0.0133
0.6348 + 0.0164
Table SimCLR-Rait under different finctuning strategics on NDCG/Outlie-NDDN averaged over 3 trials (LP Linear Probing, MP Multilayer Probing, FF Full Finetuning). Multilayer probing performs moderately worse than full finetuning, and linear probing performs much worse than all other strategies.
Method
MSL.RWEB30K (t)
Yahoo Set (t)
Tstella (t)
NDCG (t)
LP
0.3219 + 0.0224
0.5202 0.0093
0.4029 + 0.0073
MP
0.3890 + 0.0011
0.5942 0.0016
0.5813 + 0.0006
FF
0.3959 + 0.0022
0.6022 0.0013
0.5839 + 0.0013
Outlier-NDGG (t)
LP
0.2304 + 0.0332
0.3811 + 0.0048
0.4717 + 0.0166
MP
0.2969 + 0.0009
0.4888 + 0.0046
0.6369 + 0.0045
FF
0.2892 + 0.0025
0.5143 = 0.0055
0.6352 + 0.0140
A.3.3 Additional Results on SimCLR-Rank vs SimSiam
In this section we give additional t-SNE plots of the embeddings learned by SimCLR-Rank and SimSiam encoders, and the pre-final layer of the fully supervised ranker (trained with all labels) on MSLRWEBB3K dataset (Figure 6), and Istella__ dataset (Figure 7 For the Yahoo Set1 dataset, SimSiam performs the best, but for the above two datasets SimCLR-Raad is the best.
A.3.4 Additional Results: Simulated crowdsourced label setting
Here we provide the results label scarcity for Oulier-NDCG in Figure Result interpretation is given in Section 4.1.1
A.3.5 Additional Results for Simulated Implicit Binary user feedback setting
In this part of the appendix we provide additional results on binary label generation as detailed in Section 4.1.2 In Tables and 10 we give the dataset statistics when using Ttarget 4.25 and Trarget 5.1, respectively. For these two settings, the GBDT baseline does not contain semi-supervised runs we found that semi-supervision tends to generally degrade GBDT performance; see Appendix A.4 The "No Pretrain" baseline uses the tabular ResNet architecture.
In Tables 12 and 13 we give the comparison between pretrained and non-pretrained rankers following the methodology in Section 4.1.2 and find that GBDTs perform the best when binary labels are less sparse (Ttarget 4.25) and pretrained DL. rankers perform the best when binary labels are sparser (Ftarget 5.1).
28

---

Published in Transactions on Machine Learning Research (11/2024)
0.42
GBDT
0.74
Outler-NDCG
0.36
0.38
Pretrain Pretrained
Outller-NDCG
0.58
Gutlier-NDCG
0.70
0.72
0.34
0.54
0.68
0.32
0.52
0.66
0.30
0.50
GBDT
GBDT Pretrain DL
0.28
0.48
No Pretrain DL
Pretrained DL
0.62
Pretrained DL
0.26
16
10
10-5
10-2
10-1
109
0.60
10
100
Fraction of training labeled
Fraction of training age labeled
Fraction of training ags labeled
Figure 8: Simulated crowdsourcing of labels: We compare Outlier-NDCG (f) pretrained rankers, non-pretrained DL rankers, and GBDT rankers as we change the percentage of training query groups that are labeled. To the left of the black dotted line, pretrained rankers perform the best. Points are averages over three trials. To the left of the black dotted vertical line, pretrained rankers are (1) significantly better on outliers than GBDTs at the p = 0.05 level using a one-sided I-test, and (2) on average belter on outliers than all other non non- pretrained methods.
Table 9: Train and validation dataset statistics on MSLRWEB30K, Yahoo Set and Istella after binary label generation with Ttarget 4.25. The test set left untouched. Labeled QGs are those query groups that have least one item with y =
Dataset
# query groups train
# labeled QGs
# itemss per QG train
# positives per QG train
val
train
val
val
val
MSLRWEB30K
18151
6072
15.9%
16.2%
124.35
122.38
1.4%
1.4%
Yahoo Set1
14477
2147
11.1%
10.8%
30.02
30.32
6.2%
6.0%
Istella
19200
7202
49.0%
49.5%
106.36
94.98
1.4%
1.6%
This strongly suggests that pretrained DL models outperform GBDIs when the labels arise from binary user feedback that is moderately uncommon or rare (as in online shopping, recommendations, or search).
A.3.6 Ablation: Data Augmentation Techniques
we show the performance of each augmentation choice on SimCLR-Rank and SimSiam, respectively. We use the methodology described in Section 4.1.1 and have 0.2% of training QGs labeled.
In Tables 14 and 15
A.3.7 Combining SimCLR-RR Rank and SimSiam
Here we describe how we constructively combined the SimCLR-Rank and SimSiam to get the competitive results in Section 4.3 Following the setting of Appendix A.2.3 we train (1) a SimCLR-Ran model with gaussian augmentation and noise of scale 1.0, and (2) a SimSiam model with zeroing augmentation and corruption probability 0.1.
After pretraining the SimCLR-Rank and SimSiam models, we finetune each of them the labeled data and keep the checkpoints with the best validation NDCG. Then combine these two models by concatenating the final embedding layers togother-if cach model's penultimate embedding dimension is h (h 136 for us), then the concatenated embedding is of dimension 2 x h. Then we pass this concalenated model through a batchnorm (to normalize the scales of each embedding to prevent scale issues), pass it through dropout layer of probability 0.7 (if the datasct MSLRWEB30K or Istella_ S) or 0.0 (if the dataset is Yahoo Sct1), and map this vector (of dimension 2 h) to a score of dimension 1 using a linear layer. We finetune only the final linear layer.
29

---

Published in Transactions on Machine Learning Research (11/2024)
Table 10: Train and validation dataset statistics on MSLRWEB30K, Yahoo Setl, and Istella after binary label generation with Ttarget = 5.1. The test set left untouched. Labeled QGs are those query groups that have least one item with y
Dataset
# query groups train
# labeled QGs
# items per QG train
# positives per QG train
val
train
val
val
MSLRWEB30K
18151
6072
1.1%
1.3%
124.35
122.38
0.8%
0.7%
Yahoo Set1
14477
2147
0.6%
0.6%
30.02
30.32
5.4%
5.1%
Istella__
19200
7202
3.0%
2.7%
106.36
94.98
1.1%
1.3%
Train and validation dataset statistics MSLRWEB30K, Yahoo Set1, and Istolla SS after binary label generation with Ttarget = 4.5. The test set is left untouched. Labeled QGs are those query groups that have at least one item with y = 1.
Table 11:
Dataset
# query groups val
# labeled QGs
# items per QG train
# positives per QG train
train
train
val
val
val
MSLRWEB30K
18151
6072
8.9%
8.9%
124.35
122.38
1.1%
1.1%
Yahoo Setl
14477
2147
5.3%
5.4%
30.02
30.32
5.9%
5.4%
Istella
19200
7202
25.2%
25.2%
106.36
94.98
1.3%
1.3%
A.4 Semi-supervised GBDT
Here we provide a more detailed comparison between Supervised GBDTs, GBDT + pseudolabeling, and GBDT + PCA + pseudolabeling
Figure | for the simulated crowdsourcing of labels setting and Table for the simulated implicit binary user feodback setting. We follow the experimental methodology in Appendix A.2.3 We find that for the most part, semi-supervised learning does not benefit. GBDT rankers.
A.5 Quantitative measures of embedding quality
We provide a quantitative measure of embedding quality this section. We attempt to verify that the focus on in-query group negatives helps improve embedding quality. To do this, we compare the embeddings produced by SimCLR-Ran Rank model and SimCLR-Sample model (SamCL--Sample is the same SimCLR-Ran except the negatives randomly taken from the batch rather than being in-query group), both under zeroes augmentation with 0.1, and measure how spread out the embeddings are within a query group for cach method. In Table 16 we find that SimCLR-Rank embeddings are consistently more spread out within a query group than SimCLR SimCLR-Sample, showing that in-query group focus leads to a measurable impact on the distinguish-abile of embeddings within query group. This may help explain why SimCLR-Ran performs better than Sim CLR-SSammle.
30

---

Published in Transactions on Machine Learning Research (11/2024)
Table 12: We compare pretrained models to non-pretrained models in the binary label setting with Tranget 4.25 (Section 4.1.2) on NDCG averaged over three trials. We follow the methodology in Section 4.1.2
Method
MSLRWEB30K (t)
Yahoo Set1 (1)
Istella (t)
NDCG (t)
GBDT
0.3616 + 0.0000
0.6233 + 0.0000
0.6251 + 0.0000
No Pretrain DL
0.3552 + 0.0015
0.6297 + 0.0005
0.6147 + 0.0006
Pretrained DL
0.3602 + 0.0007
0.6272 = 0.0024
0.6243 + 0.0007
OutGier-NDGG (f)
GBDT No Pretrain DL
0.2792 + 0.0000
0.5352 = 0.0000
0.7393 +'2 0.0000
0.3142 + 0.0063
0.5389 = 0.0036
0.6631 + 0.0102
Pretrained DL
0.2923 + 0.0079
0.5471 = 0.0050
0.6945 + 0.0041
Table 13: We compare pretrained models to non- non-pretrained models in the binary label setting with Trarget 5.1
(Section 4.1.2) on NDCG averaged over three trials. We follow the methodology in Section 4.1.2 indicates metrics on which pretrained rankers outperform non con-pretrained rankers significantly via p 0.05 t-test.
Method
MSLRWEB30K
Yahoo Set1
Istella
NDCG (t)
GBDT
0.2844 + 0.0000
0.5782 + 0.0000
0.5638 +'t 0.0000
No Pretrain DL
0.3432 + 0.0065
0.5703 = 0.0246
0.5722 + 0.0080
Pretrained DL.
0.3564 + 0.0054
0.6031 + 0.0064
0.599 + 0.0014
NDCiier-NDGG (t)
GBDT
0.2416 + 0.0000
0.5011 + 0.0000
0.6640 + 0.0000
No Pretrain DL
0.2441 + 0.0089
0.4593 = 0.0317
0.6334 + 0.0054
Pretrained DL
0.2553 + 0.0161
0.54 + 0.0055
0.6730 + 0.0090
Table 14: We compare different augmentation strategies for SinCLR-Rank. The methodology used is the one in Section 4.1.1 with 0.2% of training QGs labeled.
Angmentation
MSLRWEB30K
Yahoo Set1
Istella
NDCG (t)
Zeroing (p=0.1)
0.3830 + 0.0007
0.6022 + 0.0013
0.5820 + 0.0024
Zeroing (p=0.7)
0.3737 + 0.0025
0.5998 + 0.0053
0.5646 + 0.0048
Gaussian noise (scale=1.0)
0.3959 = 0.0022
0.5998 = 0.0026
0.5839 + 0.0013
Gaussian noise (scale-2.0)
0.3907 = 0.0025
0.5953 + 0.0043
0.5809 + 0.0010
Outlier-NDCG (t)
Zeroing (p=0.1)
0.2782 - 0.0036
0.5143 + 0.0055
0.6408 + 0.0063
Zeroing (p=0.7)
0.2730 0.0054
0.5062 + 0.0070
0.6141 0.0064
Gaussian noise (scale=1.00 1.0)
0.2892 0.0025
0.4963 - 0.0024
0.6352 0.0140
Gaussian noise (scale=2.0)
0.2886 0.0096
0.4875 + 0.0054
0.6327 + 0.0191
31

---

Published in Transactions on Machine Learning Research (11/2024)
Table 15: We compare different augmentation strategies for SimSiam. Scction 4.1.1 with 0.2% of training QGs labeled.
The methodology used is the one in
Augmentation
MSLKWEB3OK
Yahoo Set1
Istella
NDCG (t)
Zeroing (p=0.1)
0.3935 + 0.0034
0.6107 = 0.0035
0.5618 + 0.0049
Zeroing (p=0.7)
0.3911 + 0.0003
0.6076 + 0.0072
0.5660
0.0047
Gaussian noise (scale=1.0)
0.3782 0.0093
0.6010 + 0.0026
0.5612
0.0060
Gaussian noise (scale=2.0)
0.3860
0.0011
0.6100 + 0.0089
0.5587
0.0047
Outlier-NDCG (t)
Zeroing (p=0.1)
0.3149 0.0119
0.5200 + 0.0133
0.6348
0.0164
Zeroing (p=0.7)
0.3002 0.0060
0.5081 + 0.0097
0.6201 + 0.0104
Gaussian noise (scale=1.0) 1.0)
0.2585 = 0.0371
0.4893 = 0.0084
0.6145 + 0.0160
Gaussian noise (scale=2.0)
0.2929 + 0.0021
0.5163 + 0.0101
0.5905 + 0.0146
Table 16: The ratio of the in-query group mean absolute deviation over the overall mean absolute deviation of embeddings from SimCLR-Rank and SimCLR-Sample These embeddings were calculated over the validation set of each dataset. We find that SimCLR-Rand R-Rank has a higher ratio, meaning the embeddings are more spread out within query group.
Method
MSTRWEBB3K
Yahoo Set1
Istella
0.9953
0.9904
0.9915
Sim CLR-Sapeee
0.9945
0.9881
0.9797
3SL
Yahoo Set1
0.475
0.675
0.65
0.650
0.425
0.625
0.60
NDCG
0.600
NDCG
0.375
0.575
0.55
0.350
GBOT
GB DBDT
0.325
GBDT PCA
MBDT+pssedde
0.550
0.525
GpDudo
GBDT+PCA
0.50-
GBD
+
0.300
GBDT+PCA+pseudo
0.500
G
PCDT+eAA
10-7 0.275
0.45
10
10-1
10-2
10
10
10-2
100
Fraction of training ags labeled
Fraction of training labeled
Fraction of training labeled
Figure 9: A comparison of Supervised GBDT, GBDT + PCA, GBDT + pseudolabeling, and GBDT + PCA + pseudolabeling, We find that semi-supervised learning may sometimes benefit GBDT rankers, though is not very consistently beneficial.
Table 17: We compare GBDT models in the binary label sctting with Ttrrget 4.5 (Section 4.1.2 on NDCG averaged over three trials. We follow the methodology in Section 4.1.2
Method
MBLKWEB30K
Yahoo Set.
Istella
NDCG (t)
Supervised GBDT
0.3335 + 0.0000
0.6168
0.0000
0.6024
0.0000
GBDT PCA
0.3105 + 0.0236
0.6141 0.0022
0.5508
0.0410
GBDT pseudolabeling
0.3467 = 0.0011
0.5868 + 0.0012
0.5178 = 0.0068
GBDT PCA + pseudolabeling
0.3241 + 0.0270
0.5944 + 0.0012
0.5315 + 0.0120
32


Reviews:
[{"review_id": "vDuDVCUVY4", "reviewer": "Reviewer_fFBs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. **Comprehensive empirical evaluation:** good selection of baselines (including both GBDTs with a variety of data augmentation approaches as well as supervised DL models), with reasonable exploration of hyperparameters and loss objectives.\n2. **Demonstrates potential of pretraining for tabular LTR in settings with sparse labels:** the empirical results clearly support the main claim, that pretraining DL models can be leveraged for better tabular LTR models in settings with sparse labels.\n3. **Observation of more efficient contrastive learning objective (SimCLR-Rank):** while not extremely methodologically novel, the authors do note that in the LTR setting the available targets to rank for a given query (what the authors call a \"query group\") provide a natural source of hard negatives, and as such the contrastive learning objective in SimCLR can be made more efficient by summing over only those available targets.\n4. **Clear writing:** The authors present the motivation for their approach clearly, and the technical details are also presented well.\n\n### Weaknesses\n1. **Introduction and focus on SimCLR-Rank as a novel contribution slightly weakens main storyline:** As described in my summary, my interpretation of the main contribution of this paper is the observation that pretraining for tabular LTR is beneficial in a practical label-scarce setting. On the other hand, my interpretation of SimCLR-Rank is that it is a fairly straightforward adaptation of the existing SimCLR loss which primarily makes it more efficient. This is supported by the fact that SimCLR obtains similar results in Table 1. The observation that ranking tasks generally provide a natural source of \"hard negatives\" is worth mentioning, but beyond this it seems more like an implementation detail which makes it possible to run on larger datasets then I think the authors could de-emphasize it as a novel contribution.\n2. **Discussion of binary user feedback in main paper is too brief:**  While the details were available in the appendix, the writing in the paper itself (Section 4.1.2) could be expanded slightly to give the reader a better understanding of how the binary labels were generated. In particular, it might be useful to simply mention that individual elements from the query groups are sampled according to some stochastic process, and the \"percentage of labeled query groups\" means \"percentage of query groups with at least one label\".\n3. **Ablation is somewhat limited:** While I found most of the paper very well written, section 4.3 was a bit weaker, both in coverage and depth of the experiments and the extent to which the experiments supported the claims therein. (More comments on this in the \"Requested Changes\" section.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors present empirical results which suggest that deep learning (DL) models can outperform gradient-boosted decision trees (GBDTs) on the task of learning-to-rank (LTR) on tabular data when the labels are sparse. This contrasts with the standard view in the field, which holds that GBDTs are generally SOTA on tabular data, as opposed to domains such as images and text where DL is the standard. The primary explanation for DL's success in domains such as images and text is that, for these modalities, we have huge amounts of unsupervised data, and have developed architectures and self-supervised pretraining objectives demonstrably capable of learning good internal representations of the data which can be transferred to other tasks with a relatively small amount of supervised fine-tuning data. Despite repeated attempts to replicate this finding in the tabular domain, GBDTs remain state-of-the-art.\n\nThe authors of this paper present empirical results suggesting that the primary reason for this perceived limitation is that the relative sparsity of labeled data in the standard academic benchmarks is not representative of realistic industry datasets. As such, they explore the performance of pretrained DL models which are then fine-tuned on downsampled labeled data. The authors compare this with both GBDTs and non-pretrained DL models, and find that, as in other modalities, DL models can exploit pretraining and even surpass GBDTs when the sparsity is low, as is often the case for industry datasets in practice.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Since you use different forms of augmentation for SimCLR-Rank and SimSiam, it's unclear whether the differences observed are due to the loss or the augmentation strategy. I realize these augmentation strategies were the ones which were chosen because they performed best for each loss, respectively, however it seems to me that it would be better to use the same augmentation strategy when comparing embeddings qualitatively using tSNE, for example, to rule out the possibility that what you are observing is due to the augmentation strategy as opposed to the loss. This is particularly the case when, based on the results in Table 14 and 15, the alternative augmentation was still quite reasonable.\n2. When combining models you finetuned a linear layer over concatenated embeddings from each model. Is there any reason not to simply train this whole combined model end-to-end using the supervised data? (I realize each model was already separately fine-tuned, but perhaps this could even be avoided, and replaced by a single round of fine-tuning the concatenated model directly.)\n3. Is there a reason the GBDT and non-pretrained models were not trained using data augmentation for the large-scale real-world dataset?\n4. Is there any reason not to try GBDT on the learned embeddings from either the SimCLR-Rank, SimSiam, or concatenated version?", "broader_impact_concerns": "This work is largely empirical in nature, demonstrating efficacy of deep learning techniques for tabular learning-to-rank tasks. I find no broader impact concerns for this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tGCsp6E8Gc", "reviewer": "Reviewer_DqZd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: LTR with limited labels is an [unfortunately] common situation in real-world applications. This work bridges a gap in the literature by demonstrating a setting where deep learning models seem to consistently outperform GBDTs on tabular data. Challenges the status quo are extremely valuable, and I commend the authors for having a go at this one in particular.\n\nS2: SimCLR-Rank has a clever architecture--specifically the grouping of items within queries is an interesting ways of improving constrastive learning. That seems to me a very effective way of improving on the representation learning problem in these settings, and it's the kind of method that relies on the implicit structure of the data (and thus might generalize effectively in real settings).\n\nS3: The authors conduct extensive experiments on multiple public datasets (`MSLRWEB30K`, `Yahoo Set1`, `Istella_S`) as well as a large-scale (though unfortunately proprietary) dataset, providing a comprehensive evaluation of their approach. The experiments cover various scenarios (simulated crowdsourcing, implicit binary user feedback, etc.), and that gives a lot of strength to the narrative of the manuscript. Furthermore the thorough analysis of the proposed methods with plenty of ablation studies over different pretraining / finetuning approaches is very good to see.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the use of unsupervised pretraining for deep learning models in tabular learning-to-rank (LTR) tasks, particularly in settings with scarce labeled data.\n\nThe authors demonstrate that these models can outperform gradient boosted decision trees (GBDTs), currently regularly considered state-of-the-art for tabular data, in LTR tasks when labels are relatively scarce. They introduce a new LTR-specific pretraining method called SimCLR-Rank and evaluate various pretraining strategies on both public LTR datasets and a large-scale proprietary dataset. Overall, the paper argues that pretrained deep models can achieve better performance (especially against outliers), and challenges the current understanding of optimal modeling choices for tabular learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "W1: Some experimental choices and analyses don't seem to be well-justified. For example, the rationale behind choosing specific architectures for pretrained and non-pretrained models is unclear to me. It is not evident why DeepFM and DCNv2 were not evaluated with pretraining, and if that'd make a significant difference.\n\nW2: It is mechanistically difficult to reationalize how SimCLR-Rank, SimSiam, and the combined model work. This is especially striking due to the authors' observation that these models seem to have a high variance in performance depending on which task they get trained on.\n\nW3: There's generally a lack of analysis over the inductive biases of these methods and how they align with the characteristics of tabular LTR data. The methods are often simply introduced and motivated due to their success in other domains, but it's not clear to me what led to them specifically being applied.\n\nW4: it's not totally clear from the manuscript how SimCLR-Rank compare against other somewhat recent neural tabular learning methods like SAINT, TabNet, etc. -- The manuscript seems to be trying to walk the line between making a general statement about DL methods in the setting, and raising the profile of the proposed method; but the former requires a more thorough analysis of existing methods for the statement to generalize.", "broader_impact_concerns": "C1: It would be good to see a deeper analysis of why the proposed DL methods work well in this context.\n\nFor instance:\n\n1. Does the contrastive learning objective related well to learning ranking functions? Are the learned representations capturing important notions in these domains? (e.g. relevance, preference)\n2. In what way does the data augmentation + contrastive learning help with the limited data? How does the model actually change under the constraint and additional data?\n3. Does the focus on whitin-query comparisons lead to more informative representation for ranking? If so, is there a qualitative / quantitative difference that can be observed by probing the model?\n\nC2: section 3 does perhaps too terse of a job at introducing the methods the general ML reader in the context of tabular LTR.\n\nFor instance, semi-supervised learning for GBDTs and pretraining methods for tabular data are introduced without a smooth transition or clear connection between these two concepts.\n\nA straightforward way to address this would be to add a discussion on the motivation for pretraining methods for deep models in LTR, after discussing the limitations of semi-supervised approaches for GBDTs, but the section could also be overhauled to provide a top to bottom story that connects the methods.\n\nSimiarly, SimCLR and SimSiam are also introduced without particularly delving into why exploring domain-agnostic methods is important in this setting. i.e. somewhere in this sesction or beforehand it would be valuable to start convincing the reader that the core idea of learning similar representation for augmented data may generalize across domains. It would also be great to see a paragraph or two on what makes the standard vanilla SimCLR loss limited for the new setting.\n\nThese are just a few examples. Overall section 3 could use a rewrite so that the context of the model design choices are well communicated wrt. the problem and the relevant limitation of the baselines.\n\nC3: there's a fair amount of work in deep learning for tabular data (and even just transformer-based approaches) -- see https://arxiv.org/abs/2110.01889. It would be good to see a discussion of these, and perhaps an attempt at evaluating some of the more popular methods in the literature in the setting.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Vfr700o8Me", "reviewer": "Reviewer_7Esm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths:__\n\nThe goals of this research work are clearly described in the context of the existing literature, and the main results are presented with clear evidence. Whether the empirical results are surprising or not is not something I can comment on with confidence, but there appears to be a clear take-away in terms of an effective neural network-based strategy for learning to rank under scarce labels.\n\n\n__Weaknesses:__\n\nOverall, I do not have any major complaints. For better or for worse, this is applied machine learning research, running numerical tests to study the validity of a particular hypothesis. I personally did not find the results particularly surprising, due to the plethora of techniques for effectively pre-training neural networks (in contrast with GBDTs), but that does not invalidate the authors' results, which are likely to be of value for practitioners interested in the ranking problem using tabular data under scarce labels.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is at its core a straightforward empirical study comparing two families of methods in the setting of a particular learning task, namely the task of ranking, where the quality of ranking is evaluated using \"normalized discounted cumulative gain\" (NDCG), which assumes relevance \"labels\" associated with each feature vector. More precisely, the authors are interested in learning scenarios in which labels (relevance scores upon which the \"ideal\" ordering is assumed to be determined) are *scarce*, i.e., a large fraction of the feature vectors observed are not paired with labels. Furthermore, the data is assumed to be \"tabular data,\" i.e., the feature vectors are not assumed to have any special structure as is found in for example image or text data.\n\nThe two families of methods being compared are, roughly speaking, gradient-boosted decision trees (GBDT) and deep neural network models. Since most of the data is assumed to be unlabeled, each \"method\" here is really a pair, namely a model and a pre-training strategy. The context for this paper is that for tabular data, the general consensus based on the existing literature is that GBDTs outperform deep neural networks. This work positions itself as having identified a natural and important learning scenario in which neural networks tend to be superior. In particular, they show that on certain idiosyncratic \"outlier\" data points, pre-training NNs is a particularly effective strategy.\n\nTheir main results are essentially empirical analysis which elucidates effective strategies for using NNs to beat GBDTs in the scenario of interest, though the paper includes a ranking-specific pre-training technique, which adapts the core idea of the well-known SimCLR to the tabular data setting here.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I don't have any critical questions, just a handful of points I tripped up on while reading.\n\n- NDCG is given as an acronym everywhere in the paper except in the expositions of \"contributions\" early in the paper. Please define it again near the formal definition in equation (1).\n- The authors use the symbol \"($\\\\uparrow$)\" throughout the paper; does this mean \"larger is better\"? This notation is not something I am familiar with; perhaps it is common in applied research papers, but I think a proper definition should be given at the time of the first occurrence in the paper.\n- After equation (1), it says *\"...is a ranking of the $L$ elements induced by the scoring function $f\\_{\\\\theta}$ on $\\\\{\\\\boldsymbol{x}\\_{i}\\\\}\\_{i=1}^{L\\_{k}}$...\"*, but this $L\\_{k}$ should be just $L$, right?\n- I find the formulation of the data in the ranking problem a bit sub-standard. There are several points which I think need revision:\n  - The authors talk about \"query groups,\" but the key formulation in section 2 doesn't make this concrete. There is only one query group present in section 2, correct? Is the notion of a \"query\" even relevant here? I get the intention of course, and I assume this is standard jargon in this domain, but in the present paper, it is just undefined jargon.\n  - When we have multiple query groups, how is evaluation done? Are NDCG values just averaged over query groups? I think this should be formulated in section 2, since later on pre-training techniques with a \"batch\" of $B$ query groups appear.\n  - The \"labels are scarce\" bit is a key characteristic of the problem being studied here, and yet it is not formalized in section 2. We have $L$ labels and $L$ feature vectors; formalizing the unlabeled data points in section 2 seems far more natural to me.\n- Regarding the basic \"hypothesis\" underlying this work, the authors say *\"We hypothesize that pretrained deep models can beat GBDTs in settings that more accurately reflect real-world challenges.\"* Upon what grounds do the authors make this hypothesis? Previous work is said to study \"idealized problem settings,\" but it is not clear to the reader why we would expect DNNs to be any better than GBDTs in such a setting, given the strong empirical track record of GBDTs on tabular data.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I congratulate the reviews and the authors for the discussion. First, it centred on identifying the contribution. The discussion added some empirical details, increasing the paper's scholarship. The changes in the manuscript made the paper more clear.
Decision: Accept (poster)