TabR: Tabular Deep Learning Meets Nearest Neighbors
Authors: Yury Gorishniy
Ivan Rubachev
Nikolay Kartashev
Daniil Shlenskii
Akim Kotelnikov
Artem Babenko
Venue: ICLR 2024 poster (2024)
Keywords: tabular
tabular data
architecture
deep learning
neural networks
Primary area: general machine learning (i.e., none of the above)
TL;DR: TabR is a new tabular DL model with a k-nearest-neighbors-like component and strong results on public benchmarks.

Abstract: Deep learning (DL) models for tabular data problems (e.g. classification, regression) are currently receiving increasingly more attention from researchers.
However, despite the recent efforts, the non-DL algorithms based on gradient-boosted decision trees (GBDT) remain a strong go-to solution for these problems.
One of the research directions aimed at improving the position of tabular DL involves designing so-called retrieval-augmented models.
For a target object, such models retrieve other objects (e.g. the nearest neighbors) from the available training data and use their features and labels to make a better prediction.

In this work, we present TabR -- essentially, a feed-forward network with a custom k-Nearest-Neighbors-like component in the middle.
On a set of public benchmarks with datasets up to several million objects, TabR marks a big step forward for tabular DL: it demonstrates the best average performance among tabular DL models, becomes the new state-of-the-art on several datasets, and even outperforms GBDT models on the recently proposed "GBDT-friendly" benchmark (see Figure 1).
Among the important findings and technical details powering TabR, the main ones lie in the attention-like mechanism that is responsible for retrieving the nearest neighbors and extracting valuable signal from them.
In addition to the higher performance, TabR is simple and significantly more efficient compared to prior retrieval-based tabular DL models.

Published as a conference paper at ICLR 2024
TABR: TABULAR DEEP LEARNING MEETS NEAREST NEIGHBORS
Yury Gorishniy
Ivan Rubachey
Nikolay Kartashev
Daniil Shlenskii!
Akim Kotelnikoy
Artem Babenko
ABSTRACT
Deep learning (DL) models for tabular data problems (e.g. classification, regres- sion) are currently receiving increasingly more attention from researchers. However, despite the recent efforts, the non-DL algorithms based gradient-boosted boosted de- cision trees (GBDT) remain a strong go-to solution for these problems. One the research directions aimed at improving the position of tabular DL involves designing so-called etrieval-augment retrieve other objects (e.g. the nearest neighbors) from the available training data and use their features and labels to make better prediction.
models. For target object, such models
In this work, present TabR essentially, ferd-forward network with custom Beirest-Neighbors--ie component in the middle. On set of public benchmarks with datasets up to several million objects, TabR marks big step forward for tabular DL: it demonstrates the best average performance among tabular DL models, becomes the new state-of-the-art on several datasets, and even outperforms GBDT models on the recently proposed "GBDT-friendly" benchmark (see Figure 1). Among the important findings and technical details powering TabR, the main ones lie in the attention-li like mechanism that is responsible for retrieving the nearest neighbors and extracting valuable signal from them. In addition to the higher performance, TabR is simple and significantly more efficient compared to prior retrieval-basee tabular DL models. The source code published: link.
DL
L'Ties
UXGoood wins
MLP (< 2021)
6
28
FT-Trranformer (Gorishniy et al., 2021) MI.P-PIR (Ciorishniy et al., 2022)
17
TabR (Ours, 2023)
23
Figure 1: Comparing DI. models with XGBoost (Chen and Guestrin, 2016) on 43 regression and classification tasks of middle scale (S 50K objects) from "Why do tree-based models still outperform deep learning typical tabular data?" by Grinsztajn al. (2022). TabR marks a significant step forward compared to prior tabular DL models and continues the positive trend for the field.
INTRODUCTION
Machine learning (ML) problems tabular data, where objects described by set of heteroge- ncous features, are ubiquitous in industrial applications medicine, finance, manufacturing, and other fields. Historically, for these tasks, the models based on gradient- boosted decision trees (GBDTT) have been a go-to solution for a long time. However, lately, tabular deep learning (DL) models have been receiving increasingly more attention, and they are becoming more competitive (Klambauer et al., 2017; Popov et al., 2020; Wang et al., 2020; Hazimeh et al., 2020; Huang et al., 2020; Gorishniy al., 2021; Somepalli al., 2021; Kossen al., 2021; Gorishniy al., 2022).
In particular, several attempts to design retrieval-augeee auggeentdd tabular DL model have been recently made (Somepalli al., 2021: Qin al., 2021: Kossen et al., 2021). For target object, retrieval- augmented model retrieves additional objects from the training set (e.g. the target object's 's nearest
"Corresponding author: fi risnameel @tt.aaammaasttte@@@@@@@ee astname@gmai com
Yandex
1HSE

---

Published as a conference paper at ICLR 2024
neighbors, even the whole training set) and uses them to improve the prediction for the target object. In fact, the retrieval technique is widely popular in other domains, including natural language processing (Das et al., 2021; Wang et al., 2022; Izacard et al., 2022), computer vision (Jia et al., 2021; Iscen et al., 2022; Long et al., 2022), CTR prediction (Qin et al., 2020; 2021; Du et al., 2022), and others. Compared to purely parametric (i.e. retrieval-free) models, the retrieval-based ones can achieve higher performance and also exhibit several practically important properties, such as the ability for incremental learning and better robustness (Das al., 2021; Jia et al., 2021).
models for tabular data problems exist, in our experiments, we show that they provide if only minor benefits over the properly tuned multilayer perceptron (MLP; the simplest parametric model), while being significantly more complex and costly. Nevertheless, in this work, we show that, with certain previously overlooked design aspects in mind, it possible to obtain a retrieval-based tabular architecture that is powerful, simple and substantially more efficient than prior retrieval- models. We summarize our main contributions as follows:
While multiple
1. We design TabR simple aurrival-auummeted tabular DL model which, on set of public benchmarks, demonstrates the best average performance among DL models, achieves the new state-of-the-art on several datasets and is significantly more efficient than prior deep retrieval- based tabular models.
2. In particular, TabR achieves notable milestone for tabular DL by outperforming GBDT on the recently proposed benchmark with middle-scale tasks (Grinsztajn et al., 2022), which was teaoon-ttre-bbseee models over DL models. Tree-based models, in turn, remain a more efficient solution.
originally used to illustrate the superiority of
3. We highlight the important degrees of freedom of the attention mechanism (the often used module in retrieval-based models) that allow designing better retrieval-based tabular models.
RELATED WORK
Gradient boosted decision trees (GBDT). GBDT-based ML models are non-DL solutions for supervised problems on tabular data that are popular within the community due to their strong performance and high efficiency. By employing the modern DL building blocks and, in particular, the retrieval technique, our new model successfully competes with GBDT and, in particular, demonstrates that DL models can be superior on non-big data by outperforming GBDT on the recently proposed benchmark with small-to-middle scale tasks (Grinsztajn et al., 2022).
Tabular deep learning. Tabular DL is a rapidly developing field with the recent advances covering parametric architectures (Klambauer et al., 2017; Wang et al., 2020: Gorishniy et al., 2021; 2022) (and many others), regularizations (Jeffares et al., 2023), pretraining (Bahri et al., 2021) and other methods (Hollmann et al., 2023). In this work, we focus specifically on architectures. In particular, the recent studies reveal that MLP-like backbones are still competitive (Kadra et al., 2021: Gorishniy al., 2021; 2022), and that embeddings for continuous features (Gorishniy et al., 2022) significantly reduce the gap between tabular DL and GBDT. In this work, we show that a properly designed retrieval component can boost the performance of tabular DL even further.
Retrieval-augmmened models in general. Usually, the retrieval-based models are designed as follows. For an input object, first, they retrieve relevant samples from available (training) data. Then, they process the input object together with the retrieved instances to produce the final prediction for the input object. One of the common motivations for designing retrieval-based schemes the local learning paradigm (Bottou and Vapnik, 1992), and the simplest possible example of such model the k-nearest neighbors (kNN) algorithm (James al., 2013). The promise of retrieval-based approaches was demonstrated across various domains, such as natural language processing (Lewis et al., 2020; Guu et al., 2020; Khandelwal et al., 2020; Izacard et al., 2022; Borgeaud et al., 2022), computer vision (Iscen et al.. 2022; Long et al., 2022), CTR prediction (Qin et al., 2020; 2021; Du et al., 2022), and others. Additionally, eetrieval-augeene as better interpretability (Wang and Sabuneu, 2023), robustness (Zhao and Cho, 2018) and others.
models often have useful properties such
Retrieval-augmmened models for tabular data problems. The classic example of non-deep retrieval- based tabular models are the neighbor- based and kernel methods (James et al., 2013; Nader et al., 2022). There are also deep sarieval-basee models applicable to (or directly designed for) tabular data problems (Wilson et al., 2016; Kim et al., 2019; Ramsauer et al., 2021; Kossen et al., 2021;
2

---

Published as a conference paper at ICLR 2024
Somepalli al., 2021). Notably, some of them omit the retrieval step and use all training data points as the "retrieved" instances (Somepalli et al., 2021; Kossen et al., 2021; Schäfl et al., 2022). However, show that the existing retrieval-ber -ladd tabular DL models are only marginally better than simple parametric DI. models, and that often comes with a cost of using heavy Transformer-like architectures. Compared prior work, where several layers with multiple multi-head vanilla attention modules are often used (Ramsauer et al., 2021; Kossen et al., 2021; Somepalli et al., 2021), our model TabR implements its retrieval component with just one single-head attention-like module, customized in way that makes it better suited for tabular data problems. As result, TabR substantially outperforms the existing retrieval- -asee DL models while being significantly more efficient.
TABR
In this section, we design eetrieval-augmentee deep learning model for tabular data problems.
3.1 PRELIMINARIES
Notation. For a given supervised learning problem on tabular data, we denote the dataset as {I ((c,,y)) y=) where T; E X represents the i-th object's features and yi Y represents the i-th object's label. Depending on the context, the i index can be omitted. We consider three types of tasks: binary classification Y {0, 1}. multiclass classification Y fl.... C} and regression simplicity, in most places, we will assume that contains only numerical (continuous) features, and we will give additional comments on binary and categorical features when necessary. The dataset split into three disjoint parts: 1, Train U Teat U Test,, where the "train" part is used for training, the "validation" part is used for early stopping and hyperparameter tuning, and the "test" part is used for the final evaluation. An input object for which given model makes prediction is referred to as "input object" or "target object".
R. For
When the retrieval technique is used for given target object, the retrieval is performed within the set of "context candidates" or simply "candidates": Ieand S Itrasn. The retrieved objects. in turn, are called "context objects" or simply "context". Optionally, the target object can be included in its context. In this work, unless otherwise noted, we use the same set of candidates for all input objects and set Teand Itain (which means retrieving from all training objects).
Experiment setup. We extensively describe our tuning and evaluation protocols in subsection D.6. The most important points are that, for any given algorithm, each dataset, following Gorishniy et al. (2022), (1) we perform hyperparameter tuning and early stopping using the validation set; (2) for the best hyperparameters,
the main text, we report the metric on the test set averaged over 15 random seeds, and provide standard deviations in Appendix E; (3) when comparing any two algorithms, we take the standard deviations into account as described in subsection D.6; (4) to obtain ensembles of models of the same type, we split the 15 random seeds into three disjoint groups (i.e., into three ensembles) each consisting of five models, average predictions within each group, and report the average performance of the obtained three ensembles.
In this work, we mostly use the datasets from prior literature and provide their summary in Ta- ble (sometimes, we refer to this set of datasets "the default benchmark"). Additionally, in subsection 4.2, we use the recently introduced benchmark with middle-scale tasks (S 50K objects) (Grinsztajn et al., 2022) where GBDT reported be superior to DL solutions.
Table 1: Dataset properties. "RMSE" denotes root-mean squure error, "Acc." denotes accuracy.
CH
CA
HO
AD
DI
OT
HI
BL
WE
CO
MI
#objects
10000
20640
22784
48842
53940
61878
98049
166821
397099
581012
1200192
#num.fcature
7
8
16
6
6
93
28
4
118
10
131
#bin.features
0
1
1
fcal.faalures
0
0
7
3
0
0
0
0
metric
Acc.
RMSE RMSE
Acc.
RMSE
Acc.
Acc.
RMSE
RMSE
Acc.
RMSE
#classes
2
majority class
79%
76%
26%
52%
48%
3

---

Published as a conference paper at ICLR 2024
3.2 ARCHITECTURE
To build a retrieval-bann tabular DL model, we choose an incremental approach, where we start from simple retrieval-free architecture, and, step by step, add and improve retrieval component.
Let's consider a generic feed-forward retrieval-free network ((x = P(E(z)) informally partitioned into two parts: encoder E X + R and predictor P: R + Y. To incrementally make it retrieval- based, we add retrieval module R in a residual branch after E as illustrated Figure 2, where T E R4 is the intermediate representation of the target object, (E R are the intermediate representations of the candidates and {yke Heloand
C Y are the labels of candidates.
Input object
E
P
Predictor
Encoder (shared)
candidate labels
{y:}
R
Retrieval module
Information flow
candidates {x.)
F
i E I Land S Itrain
Figure 2: The generic eetrieval-based architecture introduced in subsection 3.2 and used build TabR. First, a target object and its candidates for retrieval are encoded with the same encoder E. Then, the retrieval module I enriches the target object's representation by retrieving and processing relevant objects from the candidates. Finally, predictor ! makes prediction. The bold path highlights the structure of the feed-forward retrieval-free model before the addition of the retrieval module R.
Encoder and predictor. The encoder E and predictor P modules (Figure 2) are not the focus of this work, so we keep them simple illustrated Figure 3.
E
Input
Module
Linear
Block
xNE g
P
Block
LayerNorm
ReIU
Linear
XNp
Block
LayerNorm
Linear
ReLU
Dropout
Linear
Figure 3: Encoder E and predictor P introduced in Figure 2. Ng and Np denote the number of Block modules in E and P. respectively. The Input Module encapsulates the input processing routines (feature normalization, one-hot encoding, etc.) and assembles vector input for the subse- quent linear layer. In particular, Input Module e can contain embeddings for continuous features (Gorishniy al., 2022). LayerNorm is omitted in the first Block of E.)
Retrieval module. We define the retrieval module R in the spirit of k-nearest neighbors as illustrated Figure 4. the figure, the following formal details are omitted for clarity:
1. If the encoder E contains at least one B1 ock (i.e. NØ > 0), then, before being passed to R, a and all 74 are normalized with shared layer normalization (Ba al., 2016).
2. Optionally, the target object itself can be unconditionally (i.e. ignoring the top-m operation) added as the (m + 1)-th object to its set context objects with the similarity score S(ž, T).
3. Dropout is applied to the weights produced by the softmax function.
4. We use m = 96 (ablated subsection A.6) and, unless otherwise noted, Icand = Israin-
Now, iterate
possible designs of the similarity module S and the value module V (introduced in Figure 4). During this process, we do not use embeddings for numerical features (Gorishniy et al., 2022) in the Input Modull le of the encoder E and set Ng Np (see Figure 3).
4

---

Published as a conference paper at ICLR 2024
similarities
Information Nlw
weights
valuess
0
X
0
top-m softmax
> 0
X
Σ
R
0
X
0
X
{S( (ā, 3:)) HEE
{V 3i, y1 3E)
Figure 4: Simplified illustration of the retrieval module R introduced in Figure 2 (the omitted details are provided in the main text). For the target object's s representation म, the module takes the m nearest neighbors among the candidates { according to the similarity module S (RR ,R)) + R and aggregates their values produced by the value module V (RR, Rd, Rd.
baseline. The self attf-attention operation (Vaswani et al., 2017) was often used prior work to model the interaction between a target object and candidate/context objects (Somepalli et al., 2021; Kossen et al., 2021; Schäfl et al., 2022). Then, instantiating retrieval module as the vanilla self-attention (modulo the top-m operation) is a reasonable baseline:
Step-0. The
S (ā, T.) (a(")"Wkil
d/
Vū, wb Wv(
(1)
where Wo. WK. and Wv are linear layers, and the target object is added as the (m + 1)-th object to its own context (i.e., ignoring the top-m operation). As reported Table the Step-0 configuration performs similarly to MLP, which means that using the vanilla self-attention is a suboptimal strategy.
Step-1. Adding context labels. A natural attempt to improve the Step-0 configuration is to utilize labels of the context objects, for example, by incorporating them into the value module as follows:
S(i, i.) = ( ( d/
V R., ys) = Wy ( (ā:))
(2)
where the difference with Equation is the underlined addition of Wy Y + Rd, which is an embedding table for classification tasks and a linear layer for regression tasks. Table shows no improvements from using labels, which is counter-intuitive. Perhaps, the similarity module S taken from the vanilla attention does not allow benefiting from such valuable signal as labels.
Step-2. Improving the similarity module S. Empirically, we observed that removing the notion of queries (i.e. removing Wo) and using the La distance instead of the dot product significantly improves performance several datasets in Table 2 (subsection A.1 provides more discussion):
S(ā, 2.) = W x (i) - Wx ( d
V ã4, y:) = Wy (3:) + Wv
(3)
where the difference with Equation is underlined. This change a turning point in our story, which was overlooked in prior work. Crucially, in subsection A.3, we show that removing any of the three ingredients (context labels, key-only representation, L2 distance) results a performance drop back to the level of MLP. While the L2 distance is unlikely be the universally best choice (even within the tabular domain), it seems to be a reasonable default choice for tabular data problems.
Step-3. Improving the value module V. After improving S on Step-2, turn to the value module V. Inspired by DNNR (Nader et al., 2022) the recently proposed generalization of the kNN algorithm, make V more expressive by taking the target object's 's representation ã into account:
S(ē, R4) = | ( Wx (a:)) d V(a. 7., v4) = Wy (vi) + T(W (2) Wx (x))
T(-) = Li LinearrWi thout httpttttttttttppppppp.oeaueteeaa
where the difference with Equation is underlined. Table shows that the new value module further improves the performance on several datasets. Intuitively, the term Wy(v:) (the embedding of the context object's label) can be seen as the "raw" contribution of the i-th context object. The term T(Wx (.) Wx (ā:)) can be seen the "correction" term, where the module T translates the differences in the key space into the differences in the label embedding space. We provide further analysis on the new value module in subsection A.2.
5

---

Published as a conference paper at ICLR 2024
Table 2: The performance of the implementations of the retrieval module R, described in subsec- tion 3.2. If number is underlined, then it is better than the corresponding number from the previous step least by the standard deviation. Noticeable improvements over MLP start at Step-2. Notation: I corresponds to RMSE, 1 corresponds to accuracy.
CH f CA 4 HO 1. AD DI OT HIt BL 4 WE 1 CO 1
ML.P
0.854 0.499 3. 0.853 0.140 0.816 0.719 0.697 1.905 0.963
(Step-0) The vanilla attention baseline
0.855 0.484 3.234 0.837 0.142 0.814 0.719 0.699 1.903 0.957
(Step-1) Context labels
0.855 0.489 3.205 0.857 0.142 0.814 0.719 0.698
1.906 0.960
(Step-2) New similarity modulc
0.860 0.418 3.153 0.858 0.140 0.813 0.720 0.692
1.804 0.972
(Step-3) 3)+ New value module V
0.859 0.408 3. 158 0.863 0.135 0.810 0.722 0.692
1.814 0.975
(Step-4) Technical tweaks TabR 0.860 0.403 3.067 0.865 0.133 0.818 0.722 0.690
1.747 0.973
Step-4. TabR. Finally, empirically, we observed that omitting the scaling term d' in the similarity module and not including the target object to its own context leads to better results on average as reported in Table 2. Both aspects can be considered hyperparameterss, and the above notes can be as our default recommendations. We call the obtained model "TabR" (Tab ~ tabular, R ~ retrieval). The formal complete description of how TabR implements the retrieval module is as follows:
k == =W (ā), K2 (x, S(ž,
= - || la V(ā,ū,,))) = (y T kj) (5)
where Wx is linear layer, Wy is an embedding table for classification tasks and a linear layer for re- gression tasks, (by default) target object not included in its own context, (by default) the similarity scores are not scaled, and (-)) = L LinearWit houthhtiiiaaa (DrooDot (BeLU(Linear()))
Limitations. TabR has standard limitations of retrieval-augmenntr models, which we describe in Appendix B. We encourage practitioners to review the limitations before using TabR in practice.
4 EXPERIMENTS ON PUBLIC BENCHMARKS
In this section, we compare TabR (introduced in section 3) with existing strrevallbasse solutions and
parametric models. In addition to the fully-fledger configuration of TabR (with all degrees of freedom available for E and P as described in Figure 3), we also use TabR-S ("S" stands for "simple")- a simple configuration, which does not use feature embeddings (Gorishniy et al., 2022), has a linear encoder (Ng = 0) and a one-block predictor (Np = 1). We specify when TabR-S is used only in tables, figures, and captions but not in the text. For other details TabR, including hyperparameter tuning, see subsection D.8.
4.1 EVALUATING RETRIEVAL-AUGMENTE DEEP LEARNING MODELS FOR TABULAR DATA
In this section, we compare TabR (section 3) and the existing retrieval- augmmeted solutions with fully parametric DL models (see Appendix D for implementation details for all algorithms). Table indicates that TabR is the only satrieval-based model that provides a significant performance boost over MLP on many datasets. In particular, the full variation of TabR outperforms MLP-PLR (the modern parametric DL model with the highest average rank from Gorishniy al. (2022)) on several datasets (CA, OT, BL, WE, CO), and performs on par with it on the rest except for the MI dataset. Regarding the prior aetrieval-bsee solutions, we faced various technical limitations, such as incompatibility with classification problems and scaling issues (e.g., as we show in subsubsection 1.4.1, it takes dramatically less time to train TabR than NPT (Kossen et al., 2021) the closest retrieval-based competitor in Table 3). Notably, the retrieval component is not universally beneficial for all datasets.
The obtained results highlight the retrieval technique and embeddings for numerical features (Gorish- niy al., 2022) (used in MLP-PLR and TabR) as two powerful architectural elements that improve the optimization properties tabular DL models. Interestingly, the two techniques are not fully orthogonal, but none of them can recover the full power of the other, and it depends given dataset whether one should prefer the retrieval, the embeddings, or a combination of both.
The main takeaway. TabR becomes and demonstrates a good potential of the retrieval- based approach. TabR demonstrates strong average performance and achieves the new state-of-the-art on several datasets.
strong deep learning solution for tabular data problems

---

Published as a conference paper at ICLR 2024
Table 3: Comparing TabR with existing retrieval-augmenned tabular models and parametric DL models. The notation follows Table 2. The bold entries are the best-performing algorithms, which are defined with standard deviations taken into account as described in subsection D.6.
CH1
CA L
HO
AD
DI
OT 1
HI 1
BL
WE J
C0 1 MI M
Avg. Rank
KNN
0.537
0.588
3.744 0.831
0.256
0.774
0.665
0.712
2.296
0.927
0.761
6.0 = 1.7
DNNR (Nader ul., 2022)
0,430
3.210
0.145
0.704
1.913
0.765
4.8 + 1.9
DKL, (Wilson et al., 2016)
0.521
3.423
0.147
0.699
6.2 +: 015
ANP (Kim et al., 2019)
0.472
3.162
0.140
0.705
1.902
4.6
= 2.5
SAINT (Somepalli al., 2021)
0.860
0.168
3.212
0.860
0.137
0.812
0.724
0.693
1.933
0.964
0.763
3.8 + 1.5
NPT (Kossen et al., 2021)
0.858
0.474
3.175
0.853
0.135
0.815
0.721
0,692
1.917
0.966
0.753
3.6
1.0
MLP
0.854
0.199
3.112
0.853
0.140
0.816
0.719
0.897
1.905
0
0.748
3.7 1.3
MI.--P.R
0.860
0.476
3.056
0.184
0.819
0.729
0.687
1.860
0.970
0.744
2.0 1.0
TahR-S
0.403
3.067
0.865
0.133
0.818
0.722
0.690
1.747
0.973
0.750
1.9 4: 0.7
TabR
0.862 0.400
3.105
0.870
0.133
0.825
0.729
0.676
1.690
0.976
0.750
1.3 + 0.6
4.2 COMPARING TABR WITH GRADIENT-BOOSTED DECISION TREES
In this section, we compare TabR with models based on gradient-boosted decision trees (GBDT): XGBoost (Chen and Guestrin, 2016), LightGBM (Ke et al., 2017) and CatBoost (Prokhorenkova et al., 2018). 8). Specifically, we compare ensembles (e.g. an ensemble of TabRs vs. an ensemble of XGBoosts) for a fair comparison since gradient boosting is already an ensembling technique.
The default benchmark. Table 4 shows that, on the default benchmark, the tuned TabR provides noticeable improvements over tuned GBDT on several datasets (CH, CA, HO, HI, WE, CO), while being competitive on the rest, except for the MI dataset. The table also demonstrates that TabR has competitive default configuration (defined subsection D.8).
The benchmark from Grinsztajn et al. (2022). Now, we go further and use the recently proposed benchmark with small-to-middle--cale tasks Grinsztajn et al. (2022). Importantly, this benchmark was originally used to illustrate the superiority of GBDT over parametric DL models datasets with < 50K objects, which makes an interesting challenge for TabR. We adjust the benchmark to our tuning and evaluation protocols (see subsection C.2 for details) and report the results in Table 5. While MI.--LR (one of the best parametric DL models) indeed is slightly inferior to GBDT on this set of tasks, TabR makes significant step forward and outperforms GBDT average.
Additional analysis: subsection A.5, we try augmenting XGBoost with a retrieval component; in subsection A.4, compare training times and batch inference efficiency of TabR and GBDT models.
The main takeaway. After the comparison with GBDT, TabR confirms its status of new strong solution for tabular data problems: it provides strong average performance and can provide noticeable improvement over GBDT on some datasets.
Table 4: Comparing ensembles of TabR with ensembles of GBDT models. See subsection D.8 to leamn how the "default" T was obtained. The notation follows Table 3.
CHt t CA
HO AD 1 DI OT 1
HI BLJ
WE 1
CO
MI L Avg. Rank
Tuned hyperpurameters
XGBoost 0.861 0.432 3.164 0.872 0.136 0.832 0.726 0.680 1.769 0.971
0.741
2.5
= 0.9
CatBoost
0.859
0.426
3.106 0.872 0.133 0.827 0.727 0.681
1.773
0.969
0.741
2.5
1.1
LightGBM
0.860
0.434
3.167
0.872 0.136 0.832 0.726 0.679
1.761
0.971
0.741
2.4 = 0.9
TabR
0.865
0.391
3.025
0.872
0.131 0.831 0.733
0.674
1.661
0.977
0.748
1.3 1 0.9
Default hyperparumeters
XGBoost 0.856
0.471
3.308
0.871
0.143 0.817 0.716 0.683
1.920
0.966
0.750
3.4 0.9
CatBoost
0.861
0.432
3.108
0.874
0.132 0.822 0.726 0.684
1.886
0.924
0.744
2.1 0.8
LightGBM
0.856
0.449
3.222
0.869
0.137
0.826 0.720 0.681
1.817
0.899
0.744
2,5 0.9
TabR-S
0.864
0.398
2.971
0.859
0.131
0.824
0.724
0.688
1.721
0.974
0.752
2.0
2.0113 1.3
7

---

Published as a conference paper at ICLR 2024
Table 5: Comparing ensembles of DL models with ensembles of GBDT models on the benchmark from Grinsztajn et al. (2022) (e.g., an ensemble of MLPs vs ensemble of XGBoosts; note that in Figure we compare single models, hence the different numbers). See subsection D.8 for the details on the "default" TabR-S. The default configuration of TabR-S is compared against the default configurations of GBDT models. The comparison is performed in a pairwise manner with standard deviations taken into account as described in subsection D.6.
vs. XGBoost
vs. CatBoost
vs. LightGBM
Win / Tie Loss
Win / Tie Loss
Win / Tie / Loss
Tuned hyperparameters
MLP
6
11
26
6
8
29
5
11
27
MLP-PLR
12
17
14
10
11
22
14
15
14
S
21
13
9
17
11
15
21
15
7
TabR
26
14
3
23
13
7
26
14
3
Default hyperparameters
T
28
10
5
17
16
10
9
9
5 ANALYSIS
5.1 FREEZING CONTEXTS FOR FASTER TRAINING OF TABR
In the vanilla formulation of TabR (section 3), for each training batch, the most up-to-date contexts mined by encoding all the candidates and computing similarities with all of them, which can be prohibitively slow on large datasets. For example, it takes more than 18 hours to train a single TabR on the full "Weather prediction" dataset (Malinin et al., 2021) (3M+ objects; with the default hyperparameters from Table 4). However, as we show in Figure 5, for an average training object, its context (i.e. the top-m candidates and the distribution over them according the similarity module S) gradually "stabilizes" during the course of training, which gives an opportunity for simple optimization. Namely, after a fixed number of epochs, we can perform "context freeze": i.e., compute the up-to- date contexts for all training (but not validation and test) objects for the one last time and then reuse these contexts for the rest of the training. Table 6 indicates that, on some datasets, this simple technique allows accelerating training of TabR without much loss in metrics, with more noticeable speedups on larger datasets. In particular, on the full "Weather prediction" dataset, we achieve nearly sevenfold speedup (from 18h9min to 3h15min) while maintaining competitive RMSE. See subsection D.2 for implementation details.
Table 6: The performance of TabR-S with the "context freeze" as described in subsection 5.1. TabR-S (CF-N) denotes Ta with the context freeze applied after N epochs. In parentheses, we provide the fraction of time spent on training compared to the training without freezing (the last row).
CA 4
DI
HI
BL L
WE
CO WE (full)
TabR- S (CF-1) 0.414 (0.72) 0.137 (0.47) 0.718 (0.80) 0.692 (0.61)
1.770 (0.57) 0.973 (0.49) 1.325 (0.13)
TabR S (CF-4)
0.409 (0.71)
0.136 (0.51) 0.717 (0.73) 0.691 (0.62)
1.763 (0.56)
0.973 (0.59)
TabR-S 0.406 (1.00) 0.133 (1.00) 0.719 (1.00) 0.691 (1.00) 1.755 (1.00) 0.973 (1.00)
1.315 (1.00)
5.2 UPDATING TABR WITH NEW TRAINING DATA WITHOUT RETRAINING
Getting access to new unseen training data after training a machine learning model (e.g., after collecting yet another portion of daily logs of an application) is a common practical scenario. Technically, TabR allows utilizing the new data without retraining by adding the new data to the set of candidates for retrieval. We test this approach on the full "Weather prediction" dataset (Malinin et al., 2021) (3M+ objects). Figure 6 indicates that such "online updates" may be viable solution for incorporating new data into an already trained TabR. Additionally, this approach can be used to scale TabR to large datasets by training the model Overall, we consider the conducted experiment as a preliminary exploration and leave a systematic study of continual updates for future work. See subsection D.3 for implementation details.
subset of data and retrieving from the full data.
8

---

Published as a conference paper at ICLR 2024
2.1
0.4
1.9
Acomeet
Dunsset
RMSE
1.7
0.2
BL
Trained of Iminnin dats
2
0.1
1.5
0
100
1.3
40
Epoch
98 of truining data used candidutes on inference
Figure A-context (explained below) aver- aged over training objects until the early stopping while training Ta On a given epoch, for a given object, A-context shows the portion of its context (the top-m candidates and their weights) changed compared to the previous epoch (i.c., the lower the value, the smaller the change; see subsection D.2 for formal details). The plot shows that context updates become less intensive during the course of training, which motivates the optimization described in subsection 5.1.
Figure Training TabR-S S on various portions of the training data of the full "Weather predic- tion" dataset and gradually adding the remaining unseen training data to the set of candidates with- out retraining as described in subsection 5.2. For each curve, the leftmost point corresponds to not adding any new data to the set of candidates after the training, and the rightmost point corresponds to adding all unseen training data to the set of candidates.
5.3 FURTHER ANALYSIS
In the appendix, we provide a more insightful analysis. non-exhaustive list of examples:
in subsection 1, we analyze the key-only L2-basel similarity module S introduced on Step-2 of subsection 3.2, which was turning point in our story. We provide intuition behind this specific implementation of S and perform an in-depth comparison with the similarity module of the vanilla attention (the dot product between queries and keys).
in subsection A.2, we analyze the value module V introduced on Step-3 of subsection 3.2. On regression problems, we confirm the correction semantics of the module T from Equation 4. in subsubsection 1, baselines. We show that compared to prior retrieval retrieval-based tabular models, TabR makes a big step forward in terms of efficiency. While TabR is relatively slower than simple retrieval-free models, within the considered scope of dataset sizes, the absolute training times of TabR are affordable for most practical scenarios.
compare training times of TabR with training times of all the
in subsection A.8, we highlight additional technical properties of TabR.
6 CONCLUSION & FUTURE WORK
In this work, we have demonstrated that retrieval-based deep learning models have great potential in supervised machine learning problems on tabular data. Namely, we have designed TabR retrieval-augmented tabular DL architecture that provides strong average performance and achieves the new state-of-the-art
several datasets. Importantly, we have highlighted similarity and value modules as the important details of the attention mechanism which have a significant impact on the performance of attention- based retrieval components.
An important direction for future work is improving the efficiency of to make them faster in general and in particular applicable to tens and hundreds of millions of data points. Also, in this paper, we focused more on the aspect of task performance, so some other properties of TabR remain underexplored. For example, the retrieval nature of TabR provides new opportunities for interpreting the model's predictions through the influence of context objects. Also, TabR may enable better support for continual learning (we scratched the surface of this direction in subsection 5.2). Regarding architecture details, possible directions improving similarity and value modules, as well as performing multiple rounds of retrieval and interactions with the retrieved instances.
models

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
To make the results and models reproducible and verifiable, we provide our full codebase, all the results, and step-by-step usage instructions: link. In particular, (1) the results and hyperparameters reported the paper is just summary of the results available at the provided URL (with minor exceptions); (2) implementations of TabR and all the baselines (except for NPT) are available; (3) the hyperparameter tuning, training and evaluation pipelines are available; (4) the hyperparameters are available; (5) the used datasets and splits are available; (6) hyperparameter tuning and training times available; (7) the used hardware is available; (8) within fixed environment (i.e. fixed hardware and software versions), most of the results are bitwise reproducible.
REFERENCES
T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, Optuna: A next-generation hyperparameter optimization framework. In KDD, 2019. 23, 24, 27, 28, 29, 30
J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv, 1607.06450v1, 2016.
D. Bahri, H. Jiang. Y. Tay, and D. Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. In ICLR, 2021. 2
P. Baldi, P. Sadowski, and D. Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature Communications, 5, 2014. 20
J. A. Blackard and D. J. Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and Electronics in Agriculture, 131111111111,,, 2000. 20
S. Borgeaud, A. Mensch, Hoffmann, T. Cai, E. Rutherford, K. Millican, G. van den Driessche, J. Lespiau, B. Damoc, A. Clark, D. de Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, Cassirer,
Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan, J. W. Rae, E. Elsen, and L. Sifre. Improving language models by retrieving from trillions of tokens. In JCML, 2022.
S. Huang, Maggiore, C. Jones,
L. Bottou and V. Vapnik. Local learning algorithms. Neural Computation, 4, 1992. 2
Chen and C. Guestrin. Xgboost: scalable tree boosting system. SIGKDD, 2016. 7,
R. Das, M. Zaheer, D. Thai, A. Godbole, E. Perez, J. Y. Lee, L. Tan, L. Polymenakos, and A. McCal- lum. Case-hased reasoning for natural language queries over knowledge bases. In EMNLP, 2021.
K. Du. W. Zhang, R. Zhou, Y. Wang, X. Zhao. J. Jin, Q Gan, Z. Zhang, and D. P. Wipf. Learning enhanced representation for tabular data via neighborhood propagation. In NeurlPS, 2022. 2
J. R. Gardner, G. Pleiss, D. Bindel, K. Q. Weinberger, and A. G. Wilson. Gpytorch: Blackbox matrix- matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems, 2018. 26
Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko. Revisiting deep learning models for tabular data. In NeurIPS, 2021. 1, 2, 20, 28, 29
Y. Gorishniy, I. Rubachev, and Babenko. On embeddings for numerical features in tabular deep learning. In NeurlPS, 2022. 1, 2, 4. 6, 14, 20, 22, 24, 28, 29
L. Grinsztajn, E. Oyallon, and G. Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In NeurlPS, the "Datasets and Benchmarks" track, 2022. 1, 2, 3, 8, 20, 23, 24, 30, 33
K. Guu, K. Lee, Z. Tung, Pasupat, and M. Chang. Retrieval augmented language model pre-training. In ICMI, 2020. 2
10

---

Published as a conference paper at ICLR 2024
H. Hazimeh, N. Ponomareva, P. Mol, Z. Tan, and R. Mazumder. The tree ensemble layer: Differen- tiability meets conditional computation. JCML, 2020.
N. Hollmann, S. Müller, K. Eggensperger, and F. Hutter. Tabpfi: A transformer that solves small tabular classification problems in second. ICLR, 2023.
X. Huang, A. Khetan, M. Cvitkovic, and Z. Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv, 2012.06678v 2020.
A. Iscen, Bird, M. Caron, A. Fathi, and C. Schmid. memory transformer network for incremental learning. arXiv, abs/2210.04485v.. 2022.
G. Izacard, P. S. H. Lewis, M. Lomeli, I.. Hosseini, F. Petroni, T. Schick, J. Dwivedi- Yu, A. Joulin, S. Riedel, and E. Grave. Few-shot learning with retricval augmented language models. arXiv, Abs/2208.03299933 2022. 2
G. James, D. Witten, T. Hastie, and R. Tibshirani. An Introduction Statistical Learning. Springer, 2013. htwps::tp..... wwww www.stttttttnnngeeeeeee com/
A. Jeffares, T. Liu, J. Crabbé, F. Imrie, and M. van der Schaar. Tangos: Regularizing tabular neural networks through gradient orthogonalization and specialization. ICLR, 2023.
M. Jia, B.-C. Chen, Z. Wu, C. Cardie, S. Belongie, and S.-N. Lim. Rethinking nearest neighbors for visual classification. arXiv preprint arXiv:2112.0844 2021. 2
A. Kadra, Lindauer, Hutter, and Grabocka. Well-tuned simple nets excel on tabular datasets.
In NeurIPS, 2021. 2
G. Ke, Q- Meng. T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu. Lightghm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30: 3146-3154, 2017. 7
R. Kelley Pace and R. Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3):
291-297, 1997. 19
U. Khandelwal, O. Levy, D. Jurafsky, I.. Zettlemoyer, and M. Lewis. Generalization through memorization: Nearest neighbor language models. In ICLR, 2020. 2
H. Kim, A. Mnih, J. Schwarz, M. Garnelo, S. M. A. Eslami, D. Rosenbaum, O. Vinyals, and Y. W. Teh. Attentive neural processes. ICLR, 2019. 7,
G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter. Self-normalizing neural networks. In NIPS, 2017. 1,2
R. Kohavi. Scaling up the accuracy of naive- naive-bayes classifiers: a decision-tree hybrid. In KDD, 1996. 20
J. Kossen, N. Band, C. Lyle, A. Gomez, T. Rainforth, and Y. Gal. Self-attention between datapoints: Going beyond individual input-output pairs deep learning. In NeurlPS, 2021. 1, 2, 3, 5, 6, 7, 16, 27
H. Lewis, E. Perez, Piktus, Petroni, V. Karpukhin, Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurlPS, 2020.
A. Long, W. Yin, Ajanthan, V. Nguyen, Purkait, R. Garg, A. Blair, C. Shen, and A. van den Hengel. Retrieval augmented classification for long-tail visual recognition. In CVPR, 2022. 2
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019. 22
A. Malinin, N. Band, G. Chesnokov, Y. Gal, M. F. Gales, A. Noskov, A. Ploskonosov,
L. Prokhorenkova. I. Provilkov, V. Raina, V. Raina, M. Shmatova, P. Tigas, and B. Yangel. Shifts: A dataset of real distributional shift across multiple large-scale tasks. ArXiv, abs/2107.00455v33,
2021. 8, 20
11

---

Published as a conference paper at ICLR 2024
Y. Nader, L. Sixt, and T. Landgraf. Dnnr: Differential nearest neighbors regression. In JCML, 2, 5, 7, 14
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research,
12:2825-2888
2011. 22
S. Popov, S. Morozov, and Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In ICLR, 2020. 1
L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin, Catboost: unbiased boosting with categorical features. NeurlPS, 2018.
J. Qin. W. Zhang, X. Wu, J. Jin, Y. Fang, and Y. Yu. User behavior retrieval for dick-throughh rate prediction. In SIGIR, 2020. 2
Qin, W. Zhang, R. Su, Z. Liu, W. Liu, R. Tang, X. He, and Y. Yu. Retrieval & interaction machine for tabular data prediction. In KDD, 2021. 1,2
T. Qin and T. Liu. Introducing LETOR 4.0 datasets. arXiv, 1306.2597v1, 2013. 20
H. Ramsauer, B. Schäff, J. Lehner, P. Seidl, M. Widrich, L. Gruber, M. Holzleimerr, T. Adler, D. P. Kreil, M. K. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter. Hopfield networks is all you need. In ICLR, 2021, 2,
B. Schäfl, L.. Gruber, Bitto-Nemling, and S. Hochreiter. Hopular: Modern hopfield networks for tabular data, arXiv, abs/2206.00664, 2022. 5, 27
G. Somepalli, M. Goldblum, A. Schwarzschild, C. B. Bruss, and T. Goldstein. SAINT: improved neural networks for tabular data via row attention and contrastive pre-training. arXiv, 1,44v0,34 2021. 1, 3, 7.
Vanschoren, N. Rijn, B. Bischl, and L. Torgo. Openml: networked science in machine learning. arXiv, 1407.7722v1, 2014. 20
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. NIPS, 2017.
A. Q. Wang and M. R. Sabuncu. A flexible nadaraya-watson head can offer explainable and calibrated classification. In TMLR, 2023. 2
R. Wang, R. Shivanna, D. Z.. Cheng, S. Jain, D. Lin, I. Hong, and E. H. Chi. Den v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems, arXiv, 2020. 1, 2
Wang, Y. Xu, Y. Fang, Y. Liu, Sun, Xu, C. Zhu, and M. Zeng. Training data more valuable than you think: A simple and effective method by retrieving from training data. arXiv preprint arXiv:2203.0877. 2022. 2
A. G. Wilson, Z. Hu, R. Salakhutdinov, and P. Xing. Deep kernel learning. AISTATS, 2016. 2, 7
J. Zhao and K. Cho. Retrieval-augmmntted convolutional neural networks for improved robustness 2018. 2
against adversarial examples. arXiv preprint
12

---

Published as a conference paper at ICLR 2024
SUPPLEMENTARY MATERIAL
A ADDITIONAL ANALYSIS
A.1 SIMILARITY MODULE OF TABR
A.1.I MOTIVATION
Recall that in Step-2 of subsection 3.2, the change from the similarity module of the vanilla attention to the new key-only Lg-driven similarity module was a turning point in our story, where a retrieval- based model started showing noticeable improvements over MLP on several datasets. In fact, in addition to the empirical results (in Table 2 and subsection A.3), this specific similarity module has reasonable intuitive motivation, which we now provide.
First, aligning two (query and key) representations of target and candidate objects is an additional challenge for the optimization process, and there is no clear motivation for introducing this challenge in our case. And, as demonstrated in subsection A.3, avoiding this challenge is not just beneficial, rather necessary.
Second, during the design process in subsection 3.2, the similarity module S operates over linear transformations of the input (because, at that point, encoder E just a linear layer, since fixed Ng 0). Then, reasonable similarity measure in the original feature space may remain reasonable in the transformed feature space. And, for tabular data, L2 is usually a better similarity measure than the dot product in the original feature space. Note that the case of shallow/lnnear encoder specific, but very important case: since E is applied to many candidates each training step, E. is better to be lightweight to maintain adequate efficiency.
Combined, the above two points motivate removing query representations and switching to the L2 distance, which leads to the similarity module introduced in Step-2 of subsection 3.2.
A.1.2 ANALYZING ATTENTION PATTERNS OVER CANDIDATES
In this section, analyze the similarity module S introduced in Step-2 of subsection 3.2, which greatly improved the performance on several datasets in Table 2.
Formally, for given input object, the similarity module defines distribution candidates ("weights"" in Figure 4) with exactly m + 1 non-zero entries (m the context size; +1 comes from adding the target object to its context in Step-2). Intuitively, the less diverse such distributions are on average, the more frequently different input objects are augmented with similar contexts. In Table demonstrate that such distributions are more diverse average with the new similarity module compared to the one from the vanilla attention. The implementation details are provided in subsection D.4.
Table 7: Entropy of the average distribution over candidates (the averaging is performed over individual distributions for test objects). The distributions are produced by the similarity module as explained in subsubsection 1.2. The trained Step-1 and Step-2 models are taken directly from Table 2. The similarity module introduced at Step-2 of subsection 3.2 produces more diverse contexts.
CH
CA
HO
AD
DI
OT
HI
WE
Step-1
6.6 6.1
7.0
7.1
5.8
5.3
8.5
8.9
Step-2
8.4 9.0
9.3
9.7
10.3
10.1
10.5
9.5
Uniform 8.8 9.5
9.6
10.2
10.4
10.6
11.0
12.6
A.1.3 CASE STUDIES
In this section, we consider three datasets where the transition to the key-only L2 similarity module from the vanilla dot mance. Formally, this is the transition from "Step-1" 1" to "Step-2" in Table 2. For each of the three datasets, first, we notice that, for a given input object, there is a domain -peciifc notion of "good neighbors", i.e., such neighbors that, from human perspective, are very relevant the input object
demonstrated the most impressive perfor-
13

---

Published as a conference paper at ICLR 2024
and provide strong hints for making better prediction for the input object. Then, we show that the new similarity module allows finding and exploiting those natural hints.
California housing (CA). On this dataset, the transition from the "vanilla" pot-proddut-betweeet queries-and aanries--ndkkyys similarity module to the key-only L2 similarity module resulted in a substantial performance boost, as indicated by the difference between "Step-1" and "Step-2" in Table 2. On this dataset, the task is to estimate the prices of houses in California. Intuitively, for a given house from the test set, the prices of the training houses in the geographical neighborhood should be strong hint for solving the task. Moreover, there are coordinates (longitude and latitude) among the features, which should simplify finding good neighbors. And the "Step-2" model successfully does that, which is not true for the "Stepp1" model. Specifically, for an average test object, the "Step- 2" model concentrates approximately 7% of the attention mass on the object itself (recall that "Step-2" includes the target object in the context objects) and approximately 77% on the context objects within the 10km radius. The corresponding numbers of the "Step-1" model are 0.07% and 1%.
Weather prediction (WE). Here, the story is seemingly similar to the one with the CA dataset analyzed in the previous paragraph, but in fact has a major difference, Again, here, for a given test data point, the dataset contains natural hints in the form of geographical neighbors from the training set which allow making a better weather forecast for a test query; and the "Step-2" model (Table 2) successfully exploits that, while the "Step-1" 1 model cannot pay any meaningful attention to those hints. Specifically, for an average object, the "Step-2" model concentrates approximately 29% of the attention mass the object itself (recall that 'Step-2" includes the target object in the context objects) and approximately 25% on the context objects within the 200km radius. The corresponding numbers of the "Step-1" model are 0.25% and 0.5%. However, there is crucial distinction from the CA case: in the version of the dataset WE that we used, the features did not contain the coordinates. In other words, to perform the analysis, after the training, we restored the original coordinates for each row from the original dataset and observed that the model learned the "correct" notion of "good neighbors" from other features.
Facebook comments volume (FB). In this paper, this is the first time when we mention this dataset, which was used in prior work (Gorishniy et al., 2022) and which we also used for some time in this project. Notably, on this dataset, TabR was demonstrating unthinkable improvements over competitors (including GBDT and the best-in-class parametric DL models). Then we noticed a strange pattern: often, for a given input, TabR concentrated abnormally high percentage of its attention mass on just one context object (a different one for each input object). This is how we discovered that the dataset split that we inherited from Gorishniy et al. (2022) contained a "leak": roughly speaking, for many objects, it was possible to find their almost exact copies in the training set, and the task was dramatically simpler with this kind of hint. In practice, it was dramatically simpler for the TabR, but not for other models. Specifically, for an average object, the "Step-2" model concentrates approximately 20% of the attention mass on the object itself (recall that "Step-2" includes the target object in the context objects) and approximately 35% on its leaked almost-copies The corresponding numbers of the "Step- model are 0.5% and 0.09%.
A.2 ANALYZING THE VALUE MODULE OF TABR
In this section, we analyze the value module V of TabR (see Equation 5).
A.2.1 MOTIVATION
Formally, we note that the output of the value module V of the vanilla attention (as defined in Equation 3, that is, before the Step-3 modification) does not depend on the target object representation E. This gives opportunity for making V more expressive by taking ă into account. While there are numerous technical ways to use this opportunity, we decide to take inspiration from DNNR (Nader et al., 2022) the recently proposed generalization of the kNN algorithm for regression problems.
Conceptually, while kNN captures only local label distributions, DNNR also captures local trends (formally, derivatives). Technically, contrary KNN, in DNNR, neighbor contributes to the prediction not only its label, but also an additional correction term that depends on the difference between the target object and the neighbor in the original feature space.
14

---

Published as a conference paper at ICLR 2024
This is how arrive at Equation 4 - the value module V. Similarly to DNNR, this module builds its output out of two terms:
1. The embedding of the context object's 's label.
2. The "correction" term, where the module T translates the differences in the key space into the differences in the label embedding space.
A.2.2 QUANTIFYING THE "CORRECTION" SEMANTICS
Recall the formal definition of the value module V of TabR (see Equation 5):
(6) Intuitively, for given context object, its label Y/ can be an important part of its contribution the prediction. Let's consider regression problems, where, in Equation 6, E R is embedded by Wy to W C IR.. Since Wy is a linear layer, Y is just a line, and each point on this line can be mapped back to the corresponding label from R. Then, the projection of the correction term T ((Ak;) on Y can be translated to the correction of the context label y::
Vāï, F., v4) (v. + T(k ks) (v:)) + T(Ak))
(āz, ā;. M6) = (y:) + proje ((kk)) + projy_ T(ak;) Wy + Ay: + proje_ p (7)
To check whether the underlined correction term projy. (or Ay;) important, we take trained TabR, and reevaluate it without retraining while ignoring this projection (which is equivalent to setting Αyy = 0). As a baseline, we also try ignoring the projection of T(Ak;) on random onc-dimensional subspace instead of Y. Table indicates that the correction along plays a vital role for the model. The implementation details provided in subsection D)5.
Table 8: Evaluating RMSE of trained TabR-S while ignoring projections of T(Ak;) on different one-dimensional subspaces as described in subsection A.2. The first column shows the projection on which one-dimensional subspace is removed from T(A:;) The first row corresponds to not removing any projections (i.e., the unmodified TabR-S). Ignoring the projection on Y (the label embedding space) breaks the model while ignoring a random projection does not have much effect.
CA HO DI
BL J WE
0.403 3.067 0.133 0.690 1.747
random
0.403 3.071 0.133 0.690 1.754
Y
0.465 3.649 0.364 0.695 2.003
For classification problems, we tested similar hypotheses but did not obtain any interesting results. Perhaps, the value module and specifically the T module should be designed differently to better model the nature of classification problems.
A.3 ABLATION STUDY
Recall that on Step-2 of subsection 3.2, we mentioned that it was crucial that all changes from Step-2 compared to Step-O (using labels not using queries using the Lę distance instead of the dot product) are important to provide noticeable improvements over MLP on several datasets. Note that using queries is equivalent to sharing weights of Wo and Wx: We Wx. Table contains the results of the corresponding experiment and indeed demonstrates that the Step-2 configuration cannot be trivially simplified without loss metrics (see the CH. CA. BL. WE datasets).
Overall, we hypothesize that both things are important: how valuable the additional signal is (Step-1) and how well we measure the distance from the target object the source of that valuable signal (Step-2).
A.4 EFFICIENCY
A.4.1 COMPARING TRAINING TIMES
While TabR demonstrates strong performance, these benefits do not come for free, since, as with all retrieval-augmenntd models, the retrieval component of TabR brings additional overhead. In this
15

---

Published as a conference paper at ICLR 2024
Table The ablation study as described in subsection A.3. Wº WK means using only keys and not using queries. Step-2 is the only variation providing noticeable improvements over MLP on the CH. CA, BL, WE datasets.
T2 Wq= WK, Wy CH
CA
HO
AD
DI
OT
HI
BL
WF
Avg. Rank
MLP
0.499 3.112 0.853 0.140 0.816 0.719 0.697 1.905 [0.854
2.4 = 1.4
Step-0
X
0.855
0.484 3.234 0.857 0.142 0.814 0.719 0.699
1.903
2.4 0.9
Step-1
X
X
0.855 0.489 3.205 0.857 0.142 0.814 0.719 0.698
1.906
2.4 1.2
X
J
X
0.853 0.495 3.178 0.857 0.143 0.808 0.719 0.698
1.903
2.9 0.8
X
J
J
0.857 0.495 3.217 0.857 0.141 0.808 0.717 0.698 1.881
2.7 0.7
0.855 0.488 3.170 0.857 0.143 0.813 0.719 0.698
1.901
2.3 1.0
0,856 0.498 3.206 0.858 0.142 0.812 0.721 0.699
1.900
2.4 1.1
J
0,856 0.442 3.154 0.856
0.141 0.811 0.722 0.698
1.896
2.0 0.7
Step-2 V
V
J 0.860 0.418 3.153 0.858
0.140
0.813
0.720
0.692
1.804
1.2 0.4
section, we aim to quantify this overhead by comparing the training times of TabR with those of all the baselines. Table 10 shows two important things:
first, TabR significantly more efficient (i.e. provides a significantly better trade-off between the downstream performance and training times) than prior retrieval-augmenned tabular models. In particular, TabR is significantly (and, sometimes, dramatically) more efficient than NPT (Kossen et al., 2021)- the closest retrieval-based competitor according to Table 3. second, within the considered scope of dataset sizes, the absolute training times of TabR will be affordable in practice. Moreover, the reported execution times achieved with naive implementation which lacks even some of the basic optimizations.
To sum up. compared to prior work on retrieval-brit -assd tabular DL, TabR makes big step forward in terms of efficiency. TabR is relatively slower than simple models (GBDT, parametric DI. models), and improving its efficiency is important research direction. However, given the room for technical optimizations and techniques similar to context freeze (subsection 5.1), the future of retrieval-based tabular DL looks positive.
Table 10: Training times of the tuned models (from Table 3, Table 4 and Table 6) averaged over the random seeds. The format is hh :mm: ss. TabR-S (CF-4) is TabR-S with the context freeze (subsection 5. applied after four epochs. Colors describe the following informal tiers:
<5 minutes
<30 minutes
<2 hours
<10 hours
>10 hours
CH
CA
HO
AD
DI
OT
HI
BL
WE
CO
MI
XGBoost
0:00:01
0:00:20
0:00:05
0:00:05
0:00:02
0:00:35
0:00:15
0:00:08
0:02:02
0:01:55
0:03:43
LighiGBM
0:00:00
0:00:04
0:00:01
0:00:01
0:00:03
0:00:34
0:00:10
0:00:07
0:06:40
0:06:22
0:06:45
MLP
0:00:02
0:00:18
0:00:09
0:00:17
0:00:15
0:00:31
0:00:24
0:01:38
0:00:29
0:04:01
0:02:09
MLP-PLR
0:00:03
0:00:43
0:00:14
0:00:24
0:00:25
0:02:09
0:00:17
0:00:52
0:20:01
0:03:32
0:30:30
Netrieval-augmenne models
TabR-S (CF-4)
0:00:08
0:00:25
0:00:30
0:00:34
0:00:43
0:00:57
0:01:02
0:03:08
0:09:08
0:23:13
TabR-S
0:00:20
0:01:20
0:01:23
0:03:04
0:01:44
0:01:17
0:02:09
0:11-22
0:12:11
0:49:59
0:53:04
TabR
0:00:16
0:00:40
0:00:55
0:01:30
0:01:24
0:01:47
0:06:22
0:04:14
1:03:18
0:37:03
1:46:07
DKI.
0:06:15
0:03:55
0:21:59
1:04:10
ANP
0:37:40
0:42:16
2:14:38
1:32:27
6:00:11
SAINT
0:00:23
0:06:04
0:01:44
0:00:58
0:01:55
0:05:37
0:03:47
0:06:22
2:55:51
6:17:20
5:39:37
NPT
0:08:44
0:06:58
0:12:21
0:11:22
0:54:55
10:45:2
3:26:47
0:55:04
5:28:56
12:05:28
8:07:36
A.4.2 COMPARING INFERENCE EFFICIENCY: TABR VS. XGBooST
In this section, we compare the inference efficiency of TabR and XGBoost. Importantly, our current implementation of TabR is naive and lacks even basic optimizations.
Table 11 indicates that the inference speeds are mostly comparable. More nuanced observations are as follows:
16

---

Published as a conference paper at ICLR 2024
On "non-simple" tasks (CA, OT, WE, CO), TabR is faster (informally, ''nn-simple" means that XGBoost needs many trees AND/OR XGBoost needs high depth AND/OR dataset has more features).
On "simple" tasks, XGBoost is faster (informally, "simple" means that XGBoost is shallow AND/OR dataset has few features).
With the growth of training size (e.g. see the MI dataset Table 11), TabR may become slower because of the retrieval, however, there significant room for optimizations:
Caching candidate key representations instead of recomputing them on each forward pass.
Performing the search in f instead of float32.
Using approximate search techniques instead of the current brute force.
Using only subset of the training data candidates.
etc.
Table 11: Inference throughput (batch size 4096) of tuned TabR-S and XGBoost from Table 4 on NVIDIA 2080 Ti. The last row reports the ratio between XGBoost's throughput and TabR-S's throughput.
CHI
CA
I10
AD
OT
BL
WE
CO
MI
#ohjects
6400
13209
14581
I 26048
34521
39601
62751
106764
296554
371847
7234
#fcatures
11
8
16
14
93
28
9
119
54
136
XGBoost #trees
121
3997
1328
988
802
524
1040
1751
3999
1258
3814
XGBoost maximum depth
10
13
XGBoost throughput /sec./sec.)
2197k
33k
131k
417k
19k
84k
15k
10k
TabR-S throughput (sbj./sec.)
35k
35k
55k
33k
43k
40k
37k
27k
34k
23k
11k
Overhead
62.3
0.9
3.3
3.9
9.6
0.5
1.9
3.1
0.5
0.4
1.2
A.5 AUGMENTING XGBooST WITH RETRIEVAL COMPONENT
After the successful results of TabR reported in subsection 4.2, we tried augmenting XGBoost with simple retrieval component to ensure that we do miss this opportunity to improve the baselines. Namely, for a given input object, we find m = 96 (equal to the context size of TabR) nearest training objects the original feature space, average their features and labels (the label as-is for regression problems, the one-hot encoding representations for classification problems), concatenate the target object's s features with the "average neighbor's" features and label, and the obtained vector is used the input for XGBoost. The results in Table 12 indicate that this strategy does not lead any noticeable profit for XGBoost. We tried to vary the number of neighbors but did not achieve any significant improvements.
Table 12: Results for ensembles of tuned models. "XGBoost + retrieval" stands for XGBoost augmented with the "average neighbor's" features and label as described in subsection A.5.
CH CA L HOJ J AD DI
OT f HIT BL
WE
CO
MI
Avg. Rank
XGBoost
0.861
0.432
3.164 0.872 0.136 0.832 0.726 0.680
1.709
0.971
0.741
1.9 0.7
XGBoost retrieval 0.855
0.436
3.134
0.871
0.133 0.815 0.724 0.687
1.788
0.962
0.743
2.5 + 0.5
TabR
0.865 0.391
3.025
0.872
0.131 0.831 0.733 0.674
1.661
0.977
0.748
1.2 122400 0.6
A.6 HOW THE PERFORMANCE OF TABR DEPENDS ON THE CONTEXT SIZE m.?
Recall that, throughout the paper, we used the fixed m 96 as the context size (the number of neighbors) for TabR. We evaluate other values of m in Table 13, Crucially, the choice of m must be made based on the performance on validation sets (not on the test tests). The results indicate that m 96 is reasonable default value.
A.7 ADDITIONAL RESULTS FOR THE "CONTEXT FREEZE" TECHNIQUE
We report the extended results for subsection 5,1 in Figure 7, Table 14 and Table 15. For the formal definition of the A A-context metric, subsection D.2.
17

---

Published as a conference paper at ICLR 2024
Table 13: The average ranks over datasets from Table of default T with different values of
m=1 m=2
m=4
m=8
m=16 m=32
m=64
m=96
m= m=256
Validation Set
Avg. Rank
4.5
4
3.25
2.75
2.12
2.12
1.88
1.88
1.88
Rank Std.
2.45
1.87
1.73
1.71
1.56
1,45
1.45
0.93
1.05
1.36
Test Set
Avg. Rank
4.12
3.62
3.5
3.12
2.38
2.12
2
1.62
1.75
1.75
Rank Std.
2.2
1.8
1.5
1.05
1.41
1.27
1.41
0.99
0.97
1.09
0.4
Dataset CH
A-contett
HD
BL
WE
0.1
C0
10
20
40
50
Epoch
Figure 7: The extended version of Figure with more datasets.
Table 14: The extended version of Table 6. Freezing after 0 epochs means freezing with a randomly initialized model. The speedups are provided Table 15
CH 1
CA
HO
AD 1
DI
OT
HI 1 BL 1
WB L.
CO 1 WE (full) ||g. Rank
MLP
0.854
0.499
3.112
0.853
0.140 0.816 0.719 0.697
1.905
0.963
2.9 1: 1.5
TabR-S (CF-0) 0.857 0.424
3.075
0.857 0.137 0.816 0.718 0.700
1.787
0.969
1.387
2.3
+ 1.4
TabR-S (CF-1) 0.856 01.14
3.065
0.856
0.137
0.816 0.718 0.692
1.770
0.973
1.325
1.8 + 1.0
TabR-S (CF-2) 0.856 0.411
3.074
0.856
0.137
0.816 0.718 0.691
1.767
0.973
1.7 1 0.8
TabR-S (CF-4) 0.858 0.409
3.087
0.857
0.136
0.816
0.717 0.691
1.763
0.973
1.3 + 0.5
TahR-S (CF-8) 0.858
0.857
0.135
0.817
1.761
0.973
0.5
TabR-S
0.859 0.406
3.093
0.858
0.133
0.816
0.719
0.691
1.755 0.973
1.315
1.0 0.0
Table 15:
Fraction of time spent on training in Table 14, relative to the training time without the context freeze (the last row; the format is hourssminuttsssccomss))
CI1
CA
OT
111
CO
(full)
TabR-S (CF-0)
0.96
0.78
0.79
0.83
0.75
0.87
1.03
0.64
0.53
0.52
0.13
TabR-S (CF-1)
0.72
0.47
0.80
0.80
0.49
TabR-S (CF-2)
0.91
0.65
0.78
0.86
0.82
0.57
0.60
TabR (CF-4)
1.01
0.71
0.73
0.73
0.51
0.97
0.73
0.62
0.56
0.59
T (CF-8)
1.03
0.76
0.71
0.82
0.61
0.90
0.78
0.67
0.59
0.59
T
0:00:08
0:00:36
0:00:42
0:00:46
0:01:25
0:00:58
0:01:24
0:05:03
0:16:19
0:39:13
18:08:39
18

---

Published as a conference paper at ICLR 2024
A.8 ADDITIONAL TECHNICAL NOTES ON TABR
We highlight the following technical aspects of TabR:
1. Because of the changes introduced in the Step-3 in subsection 3.2, the value representations Vāž, 24 91) of the candidates cannot be precomputed for a trained model, since they depend on the target object. This implies roughly twice less memory usage when deploying the model to production (since only the key representations and labels have to be deployed for training objects), but V(ž, Iz. y/i has to be computed in runtime.
2. Despite the attention-like nature of the retrieval module R. contrary to prior work, TabR does not suffer from the quadratic complexity w.r.t. the number of candidates, because it computes attention only for the target object. but not for the context objects.
3. In Equation 4, T uses LinearWis whoutiitttttt Bias in its definition. Strictly speaking, from the perspective of expressiveness, adding bias (i.e. using L,i Line)r) would be redundant in the presence of Wy in Equation 4. And, just in case, we avoid this redundancy (we did not test using the simple Linear instead of LinearkithoutBiss))
B LIMITATIONS & PRACTICAL CONSIDERATION
The following limitations and practical considerations are applicable to retrieval-augmented models in general. TabR itself does not add anything new this list.
First, for given application, one should carefully evaluate from various perspectives (business logic, legal considerations, ethical aspects, etc.) whether using real training objects for making predictions is reasonable.
Second, depending on an application, for given target object, one may want to retrieve only from subset of the available data, where the subset is dynamically formed for the target object based on application-speee specific filters. In terms subsection 3. means Loand Leand C Itrain-
Third, ideally, retrieval during training should simulate retrieval during deployment, otherwise, retrieval- model can lead to (highly) suboptimal performance. Examples:
For time series, during training. TabR must be allowed to retrieve only from the past. Moreover, perhaps, this "past" should also be limited to prevent the retrieval from too old data and too recent data. The decision should be made based on the domain expertise and business logic. Let's consider a task where, among all training objects, there are some "related objects". For example, when solving ranking problem be obtained as query-document pairs corresponding to the same query, but different documents. In some cases, during training, for given target object, retrieving from "related objects" can be unfair, because the same will not be possible in production for new objects that do not have "related objects" in the available data. Again, this design decision should be made based on the domain expertise and business logic.
point-wise regression, such "related objects" can
Lastly, while TabR is significantly retrieval module R still causes overhead compared to purely parametric models, so TabR may not scale truly large datasets as-is. We showcase a simple trick scale TabR larger datasets in subsection 5. We discuss the efficiency aspect in more detail in subsection A.4.
efficient than prior retrieval-basee tabular DL. models, the
C BENCHMARKS
C. THE DEFAULT BENCHMARK
In Table 16, provide more information the datasets from Table 1. The datasets include:
Churn Modeling
Califoria Housing (real estate data, (Kelley Pace and Barry, 1997))
House 26H
19

---

Published as a conference paper at ICLR 2024
Table 16: Details on datasets from the main benchmark. "# Num", "# Bin", and "# Cat" denote the number of numerical, binary, and categorical features, respectively. The "Batch size" is the default batch size used to train DL-based models.
Abbr
Name
Train
# Validation
# Test
# Num
Bin # Cat Task type
Batch size
CH
Churn Modelling
6400
1600
2000
10
3
Binclass
128
CA
California Housing
13209
3303
4128
0
Regression
256
HО
House 16H
14581
3646
4557
16
0
Regression
256
AD
Adult
26048
6513
16281
1
Binclass
256
Diamond
34521
8631
10788
0
Regression
512
OT
Otto Group Products
39601
9901
12376
93
0
Multiclass
512
HI
Higgs Small
62751
15688
19610
28
0
Binclass
512
BL
Black Friday
106764
26692
33365
1
Regression
512
WE
Shifts Weather (subsct)
296554
47373
53172
118
Regression
1024
CO
Covertype
371847
92962
116203
44
Multiclass
1024
WE (full)
Shifts Weather (full)
2965542
47373
531720
118
1
Regression
1024
Adult (income estimation, (Kohavi, 1996))
Diamond
Otto Group Product Classification
Higgs (simulated physical particles, (Baldi et al., 2014); we use the version with 98K samples available in the OpenMI.. repository (Vanschoren et al., 2014))
Black Friday²
Weather (temperature, (Malinin et al., 2021)). We take 10% of the dataset for our experiments
due to its large size.
Weather (full) (temperature, (Malinin et al., 2021)). Original splits from the paper.
Covertype (forest characteristics, (Blackard and Dean., 2000))
Microsoft (search queries, (Qin and Liu, 2013)). We follow the pointwise approach to learning to rank and treat this ranking problem regression problem.
C.2 THE BENCHMARK FROM GRINSZTAJN ET AL. (2022)
In this section, we describe how exactly used the benchmark proposed in Grinsztajn et al. (2022).
We use the same train-val-ted val-test splits.
When there are several splits for one dataset (i.e., when the --toliltroos-validaatiinn was performed in Grinsztajn et al. (2022)), we first treat each of them as separate datasets while tuning and evaluating algorithms
described in Appendix D, but then, we average the metrics over the splits to obtain the final numbers for the dataset. For example. if there are five splits for given dataset, then we tune and evaluate a given algorithm five times, each of the five tuned configurations is evaluated under 15 random seeds on the corresponding splits, and the reported metric value the average over 15 75 runs.
When there multiple versions of one dataset (e.g., the original regression task and the same dataset but converted to the binary classification task or the same dataset, but with the categorical features removed, etc.), keep only one original dataset.
We removed the "Eye movements" " dataset because there is a leak in that dataset. We use the tuning and evaluation protocols prior works on tabular DL (Gorishniy et al., 2021; 2022). Crucially, we tune hyperparameters of the GBDT models more extensively than most (if not all) prior work in terms of both budget (20 warmup iterations of random sampling followed by 180 iterations of the tree-structured Parzen estimator algorithm) and hyperparameter spaces (see the corresponding sections Appendix D).
described in Appendix D, which was also used in
petps:/wwww.oppmmlooood/22222222 https:/wwww.openmlloor
20

---

Published as a conference paper at ICLR 2024
D IMPLEMENTATION DETAILS
D.I HARDWARE
We report the used hardware in the results published along with the source code. In nutshell, the vast majority of experiments on GPU were performed on one NVidia A100 GPU, the remaining small part of GPU experiments was performed portion of rums performed on CPU (e.g. all the experiments on LightGBM).
Nvidia 2080 Ti GPU, and there also small
D.2 IMPLEMENTATION DETAILS OF SUBSECTION 5.1
In subsection 5.1, used TabR-S with the default hyperparameters (see subsection 1.8). To compute A A-context, we collect context distributions for training objects between training epochs. That is, after the i-th training epoch, we pause the training, collect the context distributions for all training objects, and then start the next (i + 1) training epoch.
A-context. Intuitively, this heuristic metric describes in single number how much, for given input object, the context attention mass was updated compared to the previous epoch. Namely, it is sum of two terms:
1. the novel attention mass, i.c. the attention mass coming from the context objects presented on the current epoch, but not presented on the previous epoch
2. the increased attention mass, i.e. we take the intersection of the current and the previous context objects and compute the increase of their total attention mass. We set it to 0.0 if actually decreased.
Now, we formally define this metric. For a given input object, a E RRlaaal and E R Rtrall denote the two distributions over the candidates from the previous and the current epochs, respectively. Let denote the sets of non-zero entries as A {i a, > 0} and = {i a, > 0}. Note that |A| = B m 96. In other words, A and B are the contexts from the epochs. Then:
A-context = novel + increased
(8)
novel Σ b; iEBYA
(9)
increased = max
Σ b
Σ
ag 0,,0.0 0.0
(10)
VEBnA
iEBnA
D.3 IMPLEMENTATION DETAILS OF SUBSECTION 5.2
In subsection 5.2, we used T with the default hyperparameters (see subsection D.8).
D.4 IMPLEMENTATION DETAILS OF SUBSURSECTION A.1.2
In subsubsection A.1.2, we performed the analysis over exactly the same model checkpoints that we used to assemble the rows "Step-1" and "Step-2" in Table 2.
To reiterate, this is how the entropy in Table computed:
First, we obtain individual distributions over candidates for all test objects. One such distribution contains exactly (m + non-zero entries.
2. Then, we average all individual distributions and obtain the average distribution.
3. Table reports the entropy of the average distribution.
Note that, when obtaining the distribution over candidates, the top-m operation is taken into account. Without that, if the distribution is always uniform regardless of the input object, then the average distribution will also be uniform and with the highest possible entropy, which would be misleading in the context of the story in subsubsection A.1.2.
Lastly, recall that the Step-1 and Step-2 models, an input object is added to its own context. Then, the edge when all input objects pay 100% attention only themselves would lead the highest
21

---

Published as a conference paper at ICLR 2024
possible entropy, which would be misleading for the story in subsubsection A.1.2. In other words, for the story in subsubsection A.1.2, we should treat the "paying attention to self" behavior similarly for all objects. To achieve that, the first step of the above recipe, reassign the attention mass from "self" to new virtual context object, which is the same for all input objects.
D.5 IMPLEMENTATION DETAILS OF SUBSECTION A.2
To build Table 8, we used TabR-S with the default hyperparameters (see subsection D.8).
D.6 EXPERIMENT SETUP
For the most part, we simply follow Gorishniy et al. (2022), but we provide all the details for completeness. Note that some of the prior work may differ from the common protocol that describe below, but we provide the algorithm-speciff implementation details further in this section.
Data preprocessing. For each dataset, for DL-based solutions, the same preprocessing was used for fair comparison. For numerical features, by default, we used the quantile normalization from the Scikit-learn package (Pedregosa et al., 2011), with rare exceptions when turned out to be detrimental (for such datasets, we used the standard normalization categorical features, we used one-hot encoding. Binary features (i.e. the ones that take only two distinct values) are mapped to {0, }} without any further preprocessing.
normalization). For
DL-based algorithms, we minimize cross-entropy for classification problems and mean squared error for regression problems. We use the AdamW optimizer (Loshchilov and Hutter, 2019). We do not apply learning rate schedules. We do not use data augmentations. used predefined dataset-specific batch size. We continue training until
Training neural networks.
For each dataset, there are pat ence + 1 consecutive epochs without improvements on the validation set; we set patience = 16 for the DL models.
How we compare algorithms. For a given dataset, first, we define the "preliminary best" algorithm the algorithm with the best mean score. Then, we define set of the best algorithms (i.e. their results are in bold in tables) as follows: a given algorithm is included in the best algorithms if its mean score differs from the mean score of the preliminary best algorithm by no more than the standard deviation of the preliminary best algorithm.
D.7 EMBEDDINGS FOR NUMERICAL FEATURES
fi
27
22
272
fz
Figure 9: (Copied from Gorishniy et al. (2022)) The same MLP as Figure 8, but now with embeddings for numerical features.
Figure 8: (Copied from Gorishniy et al. (2022)) The vanilla MLP. The model takes two numerical features as input.
In this work, we actively used embeddings for numerical features from (Gorishniy et al., 2022) (see Figure 8 and Figure 9), the technique which was reported to universally improve DI models. In nutshell, for given scalar numerical feature, embedding module is trainable module that maps this scalar feature to a vector. Then, the embeddings of all numerical features are concatenated into one flat vector which is passed further layers. Following the original paper, when embeddings for numerical features, the same embedding architecture is used for all numerical features.
In this work, we used the LR (the combination of a linear layer and ReLU) and PLR (the combination of periodic embeddings, a linear layer, and ReI.U) embeddings from the original paper. Also,
22

---

Published as a conference paper at ICLR 2024
introduce the PLR (lite) embedding, simplified version of the PLR embedding where the linear layer is shared across all features. We observed it to be significantly more lightweight without critical performance loss.
Hyperparameters tuning. For the LR embeddings, we tune the embedding dimension in Uniform 96]. For the PLR and PLR (lite) embeddings, tune the number of frequen- cies in Uniform [16, 96] (in Uniform(8. (8. 96] for TabR on the datasets from Grinsztajn et al. (2022)), the frequency initialization scale in LogUniform (0.01, 100.0] and the embedding dimension in Uniform [6, 64] (in Uniform{4 4, 64] for TabR on the datasets from Grinsztajn et al. (2022)).
D.8 TABR
The implementation, tuning hyperparameters, evaluation hyperparameters, metrics, execution times, hardware and other details are available in the source code. Here, we summarize some of the details for convenience.
Embeddings for numerical features. (see subsection D.7) For the non-simple configurations of TabR, datasets CH, CA, HO, AD, DI, OT, HI, BL, and on all the datasets from Grinsztajn et al. (2022), we used the PLR (lite) embeddings as defined subsection D.7. For other datasets, we used the LR embeddings.
Other details. We observed that initializing the Wy module properly may be important for good performance. Please, the source code.
Default TabR-S. The default hyperparameters for TabR-S were obtained at some point in the project multiple datasets. The specific set of datasets
by literally averaging the tuned hyperparameters for averaging included all datasets from Table 16 plus two datasets that used to be a part of the default benchmark, but were excluded later. So, in total, 13 datasets contributed to the default hyperparameters.
Formally, this is not 100% fair evaluate the obtained default TabR-S the datasets which contributed to this default hyperparameters as in Table 4. However, we tested the fair leave-one-out approach well (i.e. for given dataset, averaging tuned hyperparameters over all datasets except for this one dataset) and did not observe any meaningful changes, so we decided to keep things simple and to have one common set of default hyperparameters for all datasets. Plus, the obtained default TabR-S demonstrates decent performance in Table 5 as well, which illustrates that the obtained default configuration not strongly "overfitted" to the datasets from Table 16. The specific default hyperparameter values of S are follows:
265
Attention dropout rate 0.3892207715445444335
Dropout rate in FFN = 0.3885527974779168777
Learning rate 0.00031212733641115166
Weight decay = 0.00001226052000444411
Hyperparameters. The output size of the first linear layer of FFN and of T is 2d. We performed tuning using the tree-structured Parzen Estimator algorithm from the Akiba et al. (2019) library. The same protocol and hyperparameter spaces were used when tuning models in Table 2 and Table 9.
23

---

Published as a conference paper at ICLR 2024
Table 17: The hyperparameter tuning space for TabR. Here (A) (CH, CA, HO, AD, DI, OT, HI. BL). (B)= (WE, CO, MI). For the datasets from Grinsztajn et al. (2022), the tuning space is identical to (A) with the only difference that d is tuned in UniformInt[16, 384].
Parameter
(Datasets) Distribution
Comment
Width d
(A,B) UniformInt [96, 384]
Attention dropout rate
(A,B) Uniform[0.A) (0.0, 0.6]
Dropout rate in FFN
(A,B) Uniform (ifformm0.00000 0.6]
Learning rate
(A,B)
1e-3]
Weight decay
(A.B) {0, Log 1e-3]}
NE
(A,B) Uniformlnt(0,11
Const(0] for TabR-S
Np
(A,B) (1,
Const[1] for TabR-S
Tuning iterations
(A) 100 (B) 50
D.9 MLP
The implementation, tuning hyperparameters, evaluation hyperparameters, metrics, execution times, hardware and other details are available in the source code. Here, we summarize some of the details for convenience.
used the implementation from Gorishniy et al. (2022).
Hyperparameters. We use the same hidden dimension throughout the whole network. We performed tuning using the tree-structured Parzen Estimator algorithm from the Akiba et al. (2019) library.
Table 18: The hyperparameter tuning space for MLP
Parameter
Distribution
layers
UniformInt[11 6]
Width (hidden size)
UniformInt (64, 1024]
Dropout rate
{0.0, Uniform 0.5]
Learning rate
LogUniform[3e-53 1e-3]
Weight decay
{0, LogUniform (1666, le-3)
Tuning iterations
100
D.10 FT-TRANSFORMER
The implementation, tuning hyperparameters, evaluation hyperparameters, metrics, execution times, hardware and other details are available in the source code. Here, we summarize some of the details for convenience.
We used the implementation from the "rtdl" Python package (version 0.0.13).
Hy Axperparrmeters. We the rtdl. FTF PTTranssormar make basel inn method to create FT- Transsffore,, so most of hyperparameters is inherited from this method's ' signature, and the rest tuned shown in the corresponding table.
24

---

Published as a conference paper at ICLR 2024
Table 19: The hyperparameter tuning space for FT-Transformer
Parameter
Distribution
# blocks
UniformIni (1,
doken
UniformInt(166, 384]
Attention dropout rate
Uniform (0.0, 0.5)
FFN hidden dimension expansion rate
Uniform P//s, 8/s]
FFN dropout rate
Uniform (0.0, 0.5)
Residual dropout rate
{0.0, Uniform((0.00..]]
Learning rate
LogUniform (e-5,, 1e-3]
Weight decay
{0, Log 1e-4])
Tuning iterations
100
D.11 KNN
The implementation, tuning hyperparameters, evaluation hyperparameters, metrics, execution times, hardware and other details are available in the source code. Here, we summarize some of the details for convenience.
The features are preprocessed the same way as for DL models. The only hyperparameter the number of neighbors which tune in UniformInt(1, 128].
D.12 DNNR
The implementation, tuning hyperparameters, evaluation hyperparameters, metrics, execution times, hardware and other details are available in the source code. Here, we summarize some of the details for convenience.
We've used the official implementation but to evaluate DNNR on larger datasets with greater hyperparameters variability, we have rewritten parts the source code to make more efficient: enabling GPU usage, batched data processing, multiprocessing, where possible. Crucially, we leave DNNR in the source code. the underlying method unchanged. We provide our evaluate DNNR only regression problems. There is no support for classification problems,
Hyperparameters. We performed grid-search over the main DNNR hyperparameters on all datasets, falling back defaults (suggested by the authors) due to scaling issues on WE and MI.
Table 20: The hyperparameter grid used for DNNR. Here (A) [CA, HO); (B) (DI, BL, WE, MI). Notation: Nj number of features for the dataset.
Parameter
(Datasets) Parameter grid
Comment
# neighbors
(A,B) [1,2,3,.... .18]]
Learned scaling
(A,B) [No scaling, Trained scaling]
# neighbors used in scaling
(A,B) [8 NN, 2,3, 8,8, 16, 32, 64, 128]
8 B-Ny WE, MI
# epochs used in scaling
10
Cat. feature encoding
[one-hot, leave-onc-out]
# neighbors for derivative 1/
(A) LinSpace Ny, 18 N,, 20]
(B) LinSpace N,, 12 N,, 14]
https::/gi http:://gtthtt...oooooommm//////
25

---

Published as a conference paper at ICLR 2024
D.13 DKL
The implementation, tuning hyperparameters, evaluation hyperparameters, metrics, execution times, hardware and other details are available in the source code. Here, we summarize some of the details for convenience.
We used DKL implementation from GPyTorch (Gardner et al., 2018). We do not evaluate DKL on WE and MI datasets due to scaling issues (tuning alone takes day and 17 hours, compared to hours for TabR the medium DI dataset, for example). There is no support for classification problems, thus we evaluate DKL. only on regression problems.
Hyperparameters. As with MLP And perform tuning using the tree-structured Parzen Estimator algorithm from the Akiba et al. [1] library.
the same hidden dimension throughout the whole network.
Table 21: The hyperparameter tuning space for DKL
Parameter
Distribution
Kernel
rtbf,sm]
layers
UniformInt [1, 4]
Width (hidden size)
Uniformint (64, 768]
Dropout rate
{0.0, Uniform 0.5]]
Learning rate
LogUniform[le-5, 1e-2]
Weight decay
{0, LogUnifomm 61e-6, 1e-3]
Tuning iterations
100
D.14 ANP
While the original paper introducing ANP did not focus on the tabular data, conceptually, it is very relevant to prior work retrieval-basee tabular DL, so consider it one of the baselines.
The implementation, tuning hyperparameters, evaluation hyperparameters, metrics, execution times, hardware and other details are available in the source code. Here, we summarize some of the details for convenience.
We used the Pytorch implementation from unofficial repository" and modified it with respect to the official implementation from Kim et al. (2019). Specifically, we reimplemented Decoder class exactly it done in Kim et al. (2019) and changed binary cross-entropy loss with Gaussian negative log-likelihood loss Latent class since matches with the official implementation.
We do not evaluate ANP on the MI dataset due to scaling issues. Tuning alone on the smaller WE dataset took more than four days for 20(!) iterations (instead of 50-100 used for other algorithms). Also, there is no support for classification problems, thus we evaluate ANP only on regression problems.
We used 100 tuning iterations on CA and HO, 50 on DI, and 20 on BL and WE.
26

---

Published as a conference paper at ICLR 2024
Table 22: The hyperparameter tuning space for ANP
Parameter
Distribution
# decoder layers
(11
cross-attention layers
UniformInt [1,
self-attention layers
UniformIn(11
Width (hidden size) Learning rate
UniformInt (64, 384]
LogUniform (3e-5, 1e-3)
Weight decay
f0, LogUniformi 1c-6, 1c-4]}
D.15 NPT
We use the official NPT (Kossen al., 2021) implementation 8, We leave the model and training code unchanged and only adjust the datasets and their preprocessing according to our protocols.
We evaluate the Base configuration of the model and follow both NPT-Base architecture and optimization hyperparameters. We train NPT for 2000 epochs on CH, CA, AD, HO, 10000 epochs on OT, WE, MI. 15000 epochs on DI, BL, HI and 30000 epochs on CO. For all datasets that don't fit into the A100 80GB GPU, use batch size 4096 (as suggested in the NPT paper). We also decrease the hidden dim to 32 on WE and MI to avoid the OOM error.
Note that NPT is conceptually equivalent to other transformer- based con-aaraettiic tabular DL solutions: (Somepalli et al., 2021; Schäfl et al., 2022). All three methods use dot-product- based self-attention modules alternating between self-attention between object features and self-attention between objects (for the whole training dataset or its random subset).
D.16 SAINT
The implementation, tuning hyperparameters, evaluation hyperparameters, metrics, execution times, hardware and other details are available in the source code. Here, we summarize some of the details for convenience.
We use the official implementation of SAINT with important fix. Recall that, in SAINT, target object interacts with its context objects with intersample attention. In the official implementation of SAINT, context objects are taken from the same datasel part, as a target object: for training objects, context objects are taken from the training set, for validation objects - from the validation set, for test objects from the test set. This different from the approach described this paper, where context objects are always taken from the training set. Taking context objects from different dataset parts, as the official implementation of SAINT, may unwanted because of the following reasons:
1. model can have suboptimal validation and test performance because it is trained to operate when context objects are taken from the training set, but evaluated when context objects are taken from other dataset parts.
2. for a given validation/test object, the prediction depends on other validation/test objects. This not in line with other retrieval-brid -basd models, which may result in inconsistent comparisons. Also, in many real-world scenarios, during deployment/test time, input objects should be processed independently, which is not the case for the official implementation of SAINT:
For the above reasons, we slightly modify SAINT such that each individual sample attends only to itself and to context samples from the training set, both during training and evaluation. See the source code for details.
On small datasets (CH, CA, HO, AD, DI, OT, HI, BL) we fix the number of attention heads and performed hyperparameter tuiing using the tree-structured Parzen Estimator algorithm from the Akiba et al. (2019) library.
'https: / :/ thub. cow.///M.////MM/////// non www.mttri--rransoommeee Phttps:// /ttpss://gi thub. com/ somepago/ sainnt
27

---

Published as a conference paper at ICLR 2024
Table 23: The hyperparameter tuning space for SAINT
Parameter
Distribution
Depth
UniformInt
Width
Inif,rmInt(4,32.
Feed forward multiplier
Uniform [?/a, 8/a]
Attention dropout
Uniform [0, 0.5]
Feed forward dropout
Uniform [0, 0.5]
Learning rate
LogUniform (3-5,, 1c-3]
Weight decay
{0, LogUniform[1een le-4]}
On larger datasets (WE, CO, MI) we use slightly modified (for optimizing memory consumption) default configuration from the paper with following fixed hyperparameters:
depth 4
headsss= 8
dim=32
ffr_mult 4
attn_head_dim== 48
attn_dropout = 0.1
ff _f_rrppott> = 0.8
learning_rate 0.0001
wei gegcdaa 0.01
D.17 XGBOoST
The implementation, tuning hyperparameters, evaluation hyperparameters, metrics, execution times, hardware and other details are available in the source code. Here, we summarize some of the details for convenience.
In this work, we made our best to tune GBDT models as good as possible to make sure that the comparison is fair, and the conclusions are reliable. Compared prior work (Gorishniy et al., 2021; 2022), where GBDT is already extensively tuned, we doubled the number of tuning iterations, doubled the number of trees, increased the maximum depth and increased the number of early stopping rounds by 4x.
The following hyperparameters fixed and not tuned:
booster = "gbtree"
-_estimators 4000
tree_method = "gpu_ hist" pist"
early_stoppingg_roune mg_nd_ 200
We performed tuning using the tree-structured Parzen Estimator algorithm from the Akiba et al. (2019) library.
28

---

Published as a conference paper at ICLR 2024
Table 24: The hyperparameter tuning space for XGBoost
Parameter
Distribution
pollample__ytrees
Unilorm[005, 1.0]
gamma
{0.0, LogUniform 0.001, 100.0))
lambda
{0.0, LogUniform 001, 10.0]}
learning_rate
LogUniffor((00001. 1.0]
max_depth
UniformInt 3, 14]
min_child_weiqh weight
DogUniform((00000 100.0]
subsample
Uniform 0055, 1.0
# Tuning iterations
200
D.18 LIGHTGBM
The implementation, tuning hyperparameters, evaluation hyperparameters, metrics, execution times, hardware and other details are available in the source code. Here, we summarize some of the details for convenience.
In this work, we made our best to tune GBDT models as good as possible to make sure that the comparison fair, and the conclusions are reliable. Compared prior work (Gorishniy et al., 2021; 2022), where GBDT is already extensively tuned, we doubled the number of tuning iterations, doubled the number of trees, increased the maximum depth and increased the number of early stopping rounds by 4x.
The following hyperparameters are fixed and not tuned:
n_estimaton = 4000
rounds = 200
We performed tuning using the tree-structured Parzen Estimator algorithm from the Akiba al. (2019) library.
Table 25: The hyperparameter tuning space for LightGBM
Parameter
Distribution
Peature_fractton
Uniform 1.0]
lambda_11
(0.0, LogUniformO00 (0.1, 10.0]]
learning_rase
LogUniform(0.000 1.0]
num leaves
UniformInt(4 (4, 768]
min_sun_hessian__n__eeee
LogUniform/ [0.0001, 100.0]
bagging_fraction 200
Uniform(0.5, 1.0]
Tuning iterations
D.19 CATBOOST
The implementation, tuning hyperparameters, evaluation hyperparameters, metrics, execution times, hardware and other details are available in the source code. Here, summarize some of the details for convenience.
In this work, made best tune GBDT models as good possible to make sure that the comparison is fair, and the conclusions are reliable. Compared to prior work (Gorishniy et al., 2021; 2022), where GBDT is already extensively tuned, doubled the number of tuning iterations, doubled
29

---

Published as a conference paper at ICLR 2024
the number of trees, increased the maximum depth and increased the number of early stopping rounds by 4x.
The following hyperparameters fixed and not tuned:
n_estimators 4000
early_stopping___ rounds 200 od_pval = 0.001
We performed tuning using the tree-structured Parzen Estimator algorithm from the Akiba al. (2019) library.
Table 26: The hyperparameter tuning space for CatBoost
Parameter
Distribution
begpinggteemerature
Uniform (0.0, 1.0]
depth
UniformInt [3, 14]
12 ___leaf_ree
Uniform (0.1, 10.0]
deaf_estimacionn_teeattoooo
Uniform [1 10]
learning_rate
Log Uniuoommmm/ (0.001, 1.0]
Tuning iterations
200
E EXTENDED RESULTS WITH STANDARD DEVIATIONS
In this section, we provide the extended results with standard deviations for the main results reported in the main text. The results for the default benchmark are in the Table 27. The results for the benchmark from Grinsztajn et al. (2022) are in the Table 28.
Table 27: Extended results for the default benchmark. Results are grouped by datasets and span multiple pages below. Notation: 4 corresponds RMSE, corresponds to accuracy.
30

---

Published as a conference paper at ICLR 2024
CH
CA
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tuned Hyperparameters
KNN
0.837 + 0.000
KNN
0.588 + 0.000
DNNR
DNNR
0.430 0.000
DKL
DKL
0.521 0.055
ANP
ANP
0.472 = 0.007
NPT
0.858 1 0.003
NPT
0.474 0.003
SAINT
0.860 1 0.003
SAINT
0.468 0.005
MLP
0.854 + 0.003
MLP
0.499 0.004
MLP-PLR
0.860 + 0.002
0.860 + 0.001
MIP-PLR
0.476 + 0.004
0.470 + 0.001
TabR-S
0.860 + 0.002
0.862 + 0.002
TabR-S
0.403 0.002
0.396 0.000
TabR
0.862 = 0.002
0.865 + 0.001
TabR
0.400 0.003 0.391 0.002
CatBoost
0.858 0.002
0.859 0.001
CatBoost
0.429 0.001
0.4000
XGBoost
0.861 +: 0.002 0.861 +: 0.001
XGBoost
0.433 0.002 0.432 1: 0.001
LightGBM
0.860 0.001 0.860 1 0.000
LightGBM
0.435 0.002 0.434 0.001
Default hyperparameters
Default hyperparametons
CatBoost
0.860 +: 0.002 0.861 +: 0.001
CatBoost
0.433 0.001 0.432 1 0.001
XGBoost
0.855 + 0.000 0.856 + 0.000
XGBoost
0.471 0.000 0.471 4: 0.000
LightGBM
0.856 +: 0.000
0.856 + 0.000
LightGBM
0.449 0.000 0.449 +: 0.000
TabR-S
0.859 + 0.003 0.864 + 0.001
TabR-S
0.406 0.003 0.398 + 0.001
Tuned hyperparameteers (Tabll 2)
Tuned hyperparameteersss (Table
step-0
0.855 0.003 0.857 0.002
step-0
0.484 0.006 0.470 0.005
step-1
0.855 + 0.003
0.858 + 0.002
step-1
0.489 + 0.007 0.474 4: 0.005
step-2
0.860 0.002 0.862 0.003
step-2
0.418 0.002 0.411 0.000
step-3
0.859 1 0.002
0.862 1 0.002
step-3
0.408 0.003 0.399 1 0.002
HO 1
AD
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tuned Hyperparametees 0.834 0.000
KNN
3.744 = 0.000
KNN
DNNR
3.210 1 0.000
DNNR
DKL
3.423 + 0.393
DKL
ANP
3.162 0.028
ANP
NPT
3.175 + 0.032
NPT
0.853 0.010
SAINT
3.242 0.059
SAINT
0.860 0.002
MLP
3.112 = 0.036
MLP
0.853 0.001
MLP-PLR
3.056 0.021
0.019
MLP-PLR
0.870 0.002
0.873 0.001
TabR-S
3.067 + 0.040
2.996 0.027
TabR-S
0.865 0.002
0.868 1: 0.002
TabR
3.105 0.041
3.025 1.0110
TabR
0.870 0.001
0.872 0.001
CatBoost
3.117 + 0.013
3.106 + 0.002
CatBoost
0.871 0.001
0.872 + 0.001
XGBoost
3.177 + 0.010
3.164 + 0.007
XGBoost
0.872 0.001
0.872 0.000
LightGBM
3.177 = 0.009 3.167 0.005
LightGBM
0.871 0.001 0.872 + 0.000
Default hyperparameters
Default hyperparameters
CatBoost
3.122 + 0.011 3.108 0.002
CatBoost
0.873 + 0.001 0.874 + 0.001
XGBoost 3.368 0.000 3.368 0.000
XGBoost
0.871 0.000
0.871 + 0.000
LightGBM 3.222 = 0.000 3.222 + 0.000
LightGBM
0.869 0.000 0.869 + 0.000
TabR-S
3.093 0.060 2.971 0.017
TabR-S
0.858 0.001 0.859 0.000
Tuned hyperpurametees (Table 2)
Tuned hyperparameteers (Table 2)
step-0
3.234 + 0.053 3.144 0.034
step-0
0.857 0.002 0.858 + 0.000
step-1
3.205 + 0.056 3.104 0.043
step-1
0.857 0.002 0.860
0.000
2
3.153 + 0.031
3.117 0.012
step-2
0.858 0.002 0.862 + 0.001
step-3
3.158 + 0.017
3.117 0.006
step-3
0.863 0.002 0.866 1 0.001
31

---

Published as a conference paper at ICLR 2024
DI
OT
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tuned Hyperparameters
KNN
0.256 + 0.000
KNN
0.774 + 0.000
DNNR
0.145 + 0.000
DNNR
DKL
0.147 = 0.005
DKL
ANP
0.140 = 0.001
ANP
NPT
0.138 + 0.001
NPT
0.815 0.002
SAINT
0.137 + 0.002
SAINT
0.812 0.002
MLP
0.140 0.001
MLP
0.816 0.003
MLP-PLR
0.134 + 0.001
0.133 + 0.000
MIP-PLR
0.819 0.002
0.822 + 0.002
TabR-S
0.133 0.001
0.131 0.000
TabR-S
0.818 0.002
0.824 + 0.001
TabR
0.133 = 0.001
0.131 = 0.000
TabR
0.825 = 0.002
0.831 0.001
CatBoost
0.134 0.001
0.133 0.000
CatBoost
0.825 0.001
0.827
XGBoost
0.137 + 0.000 0.136 0.000
XGBoost
0.830 0.001
0.832 1: 0.001
LightGBM
0.136 0.000 0.136 0.000
LightGBM
0.830 0.001 0.832 0.001
Default hyperparamteerss
Default hyperparametons
CatBoost
0.133 + 0.000 0.132 0.000
CatBoost
0.820 0.001 0.822 1: 0.001
XGBoost
0.143 + 0.000 0.143 0.000
XGBoost
0.817 0.000 0.817 0.000
LightGBM
0.137 + 0.000
0.137 0.000
LightGBM
0.826 0.000 0.826 +: 0.000
TabR-S
0.133 + 0.001 0.131 0.000
TabR-S
0.816 0.002 0.824 + 0.000
Tuned hyperparametters (Table
Tuned hyperparameterrs (Table 2)
step-0
0.142 0.001 0.139 0.001
step-0
0.814 0.002 0.823 0.002
step-1
0.142 + 0.002 0.138 +: 0.000
step-1
0.814 + 0.002 0.824 4: 0.001
step-2
0.140 0.001 0.139 0.001
step-2
0.813 0.002 0.818 0.001
step-3
0.135 + 0.001
0.133 1 0.001
step-3
0.810 0.002 0.814 1: 0.001
HI
BL
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tuned Hyperparametees
KNN
0.665 = 0.000
KNN
0.712 0.000
DNNR
DNNR
0.704 0.000
DKL
DKL
0.699 0.001
ANP
ANP
0.705 0.005
NPT
0.721 + 0.003
NPT
0.692 + 0.001
SAINT
0.724 0.002
SAINT
0.693 0.001
MLP
0.719 = 0.002
MLP
0.697 = 0.001
MLP-PLR
0.729 0.002 0.735 0.000
MLP-PLR
0.687 0.000
0.684 108400.0 0.000
TabR-S
0.722 + 0.001 0.726 0.001
TabR-S
0.690 0.000
0.688 0.000
TabR
0.729 0.001
0.733 0.001
TabR
0.676 0.001
0.674
CatBoost
0.726 + 0.001 0.727 + 0.001
CatBoost
0.682 0.000
0.681
+ 0.000
XGBoost
0.725 + 0.002 0.726 0.001
XGBoost
0.681 0.000 0.680 0.000
LightGBM
0.726 0.001 0.726 0.001
LightGBM
0.680 0.000 0.679 + 0.000
Default hyperparameters
Default hyperparameters
CatBoost
0.725 + 0.001 0.726 0.001
CatBoost
0.685 0.000 0.684 + 0.000
XGBoost
0.716 0.000 0.716 0.000
XGBoost
0.683 0.000 0.683 + 0.000
LightGBM 0.720 = 0.000 0.720 + 0.000
LightGBM
0.681 0.000 0.681 + 0.000
TabR-S
0.719 0.002 0.724 0.000
TabR-S
0.691 0.000 0.688 0.000
Tuned hyperpurameters (Table 2)
Tuned hyperparameteer (Table 2)
step-0
0.719 + 0.002 0.727 + 0.000
step-0
0.699 0.001 0.694 + 0.001
step-1
0.719 - 0.002 0.724
0.001
step-1
0.698 0.001
0.693 0.001
2
0.720 + 0.002 0.723 + 0.001
step-2
0.692 0.001
0.690 + 0.000
step-3
0.722 + 0.002 0.724 + 0.000
step-3
0.692 0.001
0.688 0.000
32

---

Published as a conference paper at ICLR 2024
WE
CO
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tuned Hyperparameters
KNN
2.296 + 0.000
KNN
0.927 + 0.000
DNNR
1.913 0.000
DNNR
DKL
DKL
ANP
1.902 = 0.009
ANP
NPT
1.947 0.006
NPT
0.966 0.001
SAINT
1.933 + 0.028
SAINT
0.964 0.010
MLP
1.905 0.005
MLP
0.963 0.001
MLP-PLR
1.860 + 0.002
1.833 + 0.002
MIP-PLR
0.970 + 0.001
0.974 + 0.000
TabR-S
1.747 0.002
1.718 0.001
TabR-S
0.973 0.000
0.974 0.000
TabR
1.690 = 0.003
1.661 = 0.002
TabR
0.976 0.000
0.977 0,000
CatBoost
1.807 0.002
1.773 0.001
CatBoost
0.968 0.000
0.969 0.000
XGBoost
1.784 + 0.001
1.769 1 0.001
XGBoost
0.971 0.000
0.971 1 0.000
LightGBM
1.771 0.001 1.761 0.001
LightGBM
0.971 0.000 0.971 0.000
Default hyperpramsterr
Default hyperparametors
CatBoost
1.895 + 0.001 1.886 0.000
CatBoost
0.923 0.000 0.924 1: 0.000
XGBoost
1.920 + 0.000
1.920 0.000
XGBoost
0.966 0.000 0.966
0.000
LightGBM
1.845 + 0.003
1.817 0.001
LightGBM
0.884 0.016 0.899 +: 0.005
TabR-S
1.755 + 0.002 1.721 0.002
TabR-S
0.973 0.001 0.974 + 0.000
Tuned hyperparameter (Table 2)
Tuned hyperparameterss (Table 2)
step-0
1.903 0.004 1.835 0.004
step-0
0.957 0.002 0.965 0.001
step-1
1.906 + 0.003 1.845 + 0.001
step-1
0.960 + 0.002 0.967 + 0.001
step-2
1.804 0.003 1.754 0.001
step-2
0.972 0.000 0.973 0.000
step-3
1.814 + 0.003
1.765 1 0.001
step-3
0.975 + 0.001 0.976 1 0.000
MI
Method
Single model
Ensemble
Tuned Hyperparameters
KNN
0.764 = 0.000 0.765 0.000
DNNR
DKL
ANP
NPT
0.753 + 0.001
SAINT
0.763 0.007
MLP
0.748 = 0.000
MLP-PLR
0.744 0.000
0.743 0.000
TabR-S
0.750 + 0.001
0.749 1: 0.000
TabR
0.750 0.001
0.748 0.000
CatBoost
0.741 + 0.000
0.741 0.000
XGBoost
0.741 + 0.000 0.741 0.000
LightGBM
0.742 0.000 0.741 0.000
Default hyperparameters
CatBoost
0.745 + 0.000 0.744 0.000
XGBoost
0.750 0.000 0.750 0.000
LightGBM 0.747 = 0.000 0.744 + 0.000
TabR-S
0.757 0.001 0.752 0.001
Table 28: Extended results for Grinsztajn et al. (2022) benchmark. Results are grouped by datasets and span multiple pages below. Notation: corresponds RMSE, 1 corresponds to accuracy.
33

---

Published as a conference paper at ICLR 2024
Ailerons 4
Bike Sharing Demand +
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tuned Hyperpaaamttess
MLP
1.624 + 0.035 1.620 + 0.037
MLP
45.702 + 0.756 43.203 + 0.132
MLP-PLR
1.591 0.021
1.582 + 0.019
MLP-PLR
12.615 + 0.415 41.470 0.324
TabR-S
1.620 + 0.030
1.595 + 0.022
TabR-S
43.637 = 0.681
42.339 = 0.415
TabR
1.615 = 0.035
1.585 + 0.042
TabR
42.649 = 0.939
41.227 = 0.615
CatBoost
1.533 1 0.034
1.527 1 0.037
CatBoost
40.927 1 0.232
40.552 1 0.090
XGBoost
1.571 1: 0.041
1.565 1 0.040
XGBoost
42.766 1 0.126
42.606 1 0.039
LightGBM
1.581 + 0.038 1.577 + 0.040
LightGBM
42.503 + 0.190 42.342 0.149
Defanlt hyperparameters
Default hyperparametens
TabR-S
1,615 + 0.029 1.599 + 0.029
TabR-S
43,486 = 0.573 42.369 + 0.354
CatBoost
1.542 0.041 1.538 0.043
CatBoost
42.848 = 0.256 42. 02. 0.243
XGBoost
1.644 + 0.046
1.614 + 0.048
XGBoost
15.100 + 0.381
45.100 + 0.410
LightGBM
1.594 0.051 1.594 +: 0.053
LightGBM
13.089 + 0.103
43.089 0.111
Brazilian houses +
Higgs
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparamzters
Tuned Hyperparameters
MLP
0.049 = 0.018 0.046 0.021
MLP
0.723 = 0.002 0.725 + 0.001
MLP-PLR
0.043 + 0.019 0.040 0.022
MLP-PLR
0.728 0.001 0.730 +: 0.001
TabR-S
0.049 0.015
0.045 0.017
TabR-S
0.725 0.001 0.728 1: 0.000
TabR
0.045 1 0.016 0.041 +: 0.017
TabR
0.730 0.001 0.733 0.000
CatBoost
0.047 0.031 0.046 0.033
CatBoost
0.729 0.000 0.730 + 0.000
XGBoost
0.054 + 0.027 0.053 0.029
XGBoost
0.729 + 0.001 0.730 + 0.000
LightGBM
0.060 0.025 0.059 0.027
LightGBM
0.727 0.001 0.728 0.000
Default hyperparameters
Default hyperparameters
TabR-S
0.052 + 0.016 0.048 + 0.018
TabR-S
0.722 + 0.001 0.727 + 0.001
CatBoost
0.043 + 0.027 0.042 0.029
CatBoost
0.727 0.001 0.728 0.001
XGBoost
0.052 = 0.025
0.052 0.027
XGBoost
0.718 0.000 0.718 + 0.000
LightGBM
0.071 = 0.021 0.071 0.022
LightGBM
0.721 0.000 0.721 0.000
KDDCup09 upselling f
MagicTelescope f
Method
Single model
Ensemble
Method
Single model
Ensemble
Tined Hyperparametrrs
Tumed Hyperparameters
MLP
0.776 + 0.011 0.782 0.009
MLP
0.853 0.006 0.857 1: 0.004
MLP-PLR
0.797 + 0.009 0.802 + 0.010
MLP-PLR
0.860 + 0.007 0.863 + 0.007
TabR-S
0.784 0.014 0.786 0.017
TabR-S
0.868 0.006 0.873 + 0.004
TabR
0.791 + 0.012 0.803 + 0.008
TabR
0.864 0.005 0.868 1: 0.002
CatBoost
0.799 0.012 0.801 0.012
CatBoost
0.859 0.007 0.859 0.008
XGBoost
0.793 + 0.011 0.795 + 0.010
XGBoost
0.855 = 0.009 0.859 + 0.011
LightGBM
0.793 + 0.012 0.797 0.011
LightGBM
0.855 0.008 0.856 0.009
Detault hypeparamettes
Default hyperparametes
TabR-S
0.772 = 0.013 0.781 0.013
TabR-S
0.868 0.006 0.871 + 0.005
CatBoost
0.804 + 0.008 0.804
0.006
CatBoost
0.860 +: 0.007 0.860 +: 0.008
XGBoost
0.794 1 0.008 0.794 0.009
XGBoost
0.856 0.011 0.856 1 0.012
LightGBM
0.789 1 0.007 0.789 1 0.007
LightGBM
0.859 0.009
0.859 0.010
34

---

Published as a conference paper at ICLR 2024
Mercedes Benz Greener Manufacturing 4
MiamiHousing2016
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tuned Hyperparaeteers
MLP
8.383 + 0.854
8.336 + 0.888
MLP
0.161 + 0.003 0.157 + 0.003
MLP-PLR
8.383 + 0.851
8.336 0.888
MLP-PLR
0.150 0.002 0.147 0.002
TabR-S
8.351 = 0.815
8.269 = 0.840
TabR-S
0.142 0.002 0.139 + 0.002
TabR
8.319 = 0.819
8.244 = 0.844
TabR
0.139 0.002 0.136 0.002
CatBoost
8.163 0.819
8.155 + 0.844
CatBoost
0.142 0.002 0.141 1 0.003
XGBoost
8.218 1 0.817
8.209 + 0.846
XGBoost
0.144 0.003 0.143 0.003
LightGBM
8.208 0.823
8.162 0.857
LightGBM
0.146 0.002 0.145 0.003
Defnult hyperperameters
Defanlt hyperparameters
TabR-S
8,290 + 0.838 8.223 = 0.865
TabR-S
0.141 = 0.002 0.139 +: 0.002
CatBoost
8.167 0.825
8.164 0.848
CatBoost
0.142 0.003 0.141 0.003
XGBoost
8.371 + 0.787
8.371 + 0.810
XGBoost
0.160 0.003 0.160 + 0.003
LightGBM
8.280 + 0.845
8.280 0.869
LightGBM
0.152 0.004 0.152 0.004
MiniBooNE
OnlineNewsPPpullaity +
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparamsters
Tuned Hyperparameters
MLP
0.947 = 0.001 0.948 0.001
MLP
0.862 0.001 0.860 + 0.000
MLP-PLR
0.947 + 0.001 0.949 0.000
MLP-PLR
0.862 + 0.001 0.860 +: 0.000
TabR-S
0.949 + 0.001
0.950 0.000
TabR-S
0.868 0.001
0.863 1: 0.001
TabR
0.948 1 0.001
0.949 1 0.000
TabR
0.862 1 0.001
0.859 1 0.000
CatBoost
0.945 0.001
0.946 0.001
CatBoost
0.853 0.000
0.853 0.000
XGBoost
0.944 + 0.001
0.945 0.000
XGBoost
0.854 + 0.000 0.854 + 0.000
LightGBM
0.942 0.001 0.943 0.000
LightGBM
0.855 0.000 0.854
Default hyperparameters
Default hyperparameters
TabR-S
0.917 + 0.001 0.950 + 0.001
TabR-S
0.870 0.001 0.864 + 0.000
CatBoost
0.945 + 0.001 0.945 0.000
CatBoost
0.855 0.000 0.854
0.000
XGBoost
0.942 = 0.000
0.942 0.000
XGBoost
0.874 0.000 0.874 + 0.000
LightGBM
0.944 = 0.000 0.944 0.000
LightGBM
0.862 0.000 0.862 + 0.000
SGEMM GPU kernel performance L
analcatdata supreme
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tumed Hyperparameters
MLP
0.016 + 0.000 0.016 0.000
MLP
0.078 0.009 0.077 +: 0.010
MLP-PLR
0.015 + 0.000 0.015 + 0.000
MLP-PLR
0.079 = 0.008 0.077 + 0.008
TabR-S
0.017 0.001
0.016 0.000
TabR-S
0.080 0.007 0.076 + 0.005
TabR
0.015 + 0.000 0.015 + 0.000
TabR
0.081 = 0.009 0.075 1: 0.005
CatBoost
0.017 0.000
0.017 0.000
CatBoost
0.078 0.007 0.073 0.002
XGBoost
0.017 + 0.000 0.017 + 0.000
XGBoost
0.080 = 0.013 0.077 0.011
LightGBM
0.017 + 0.000 0.017 0.000
LightGBM
0.078 0.012 0.077 0.0011
hyperaarameters
Default hyperparameten
TabR-S
0.017 = 0.001 0.016 0.000
TabR-S
0.077 0.007 0.074 0.007
CatBoost
0.017 + 0.000 0.017
0.000
CatBoost
0.071 0.004 0.071 0.004
XGBoost
0.017 + 0.000 0.017 1.000
XGBoost
0.076 0.006 0.076 1 0.006
LightGBM
0.017 + 0.000 0.017 0.000
LightGBM
0.073 0.006 0.073 0.006
35

---

Published as a conference paper at ICLR 2024
bank-marketing
black friday 4
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tuned Hyperparaeeters
MLP
0.786 + 0.006 0.790 + 0.004
MLP
0.369 + 0.000 0.367 + 0.000
MLP-PLR
0.795 + 0.005 0.798 0.004
MLP-PLR
0.363 0.000 0.363 0.000
TabR-S
0.800 = 0.005 0.802 0.004
TabR-S
0.364 0.000 0.363 + 0.000
TabR
0.802 = 0.009 0.804 0.010
TabR
0.362 0.002 0.359 0.001
CatBoost
0.803 0.007 0.806 0.008
CatBoost
0.361 0.000
0.360 1 0.000
XGBoost
0.801 + 0.008 0.803 0.008
XGBoost
0.360 0.000 0.360 0.000
LightGBM
0.801 0.008 0.801 0.007
LightGBM
0.360 0.000 0.360
Default hyperperameters
Defanlt hyperparameters
TabR-S
0.800 + 0.006 0.801 + 0.005
TabR-S
0.364 = 0.000 0.363 + 0.000
CatBoost
0.803 0.009 0.803 0.009
CatBoost
0.361 0.000 0.361
0.000
XGBoost
0.800 + 0.009 0.800 0.009
XGBoost
0.362 0.000 0.362 + 0.000
LightGBM
0.803 + 0.004 0.803 0.004
LightGBM
0.361 0.000 0.361 0.000
california 4
compass f
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tuned Hyperparameters
MLP
0.149 = 0.002 0.146 0.001
MLP
0.768 = 0.005 0.776 + 0.006
MLP-PLR
0.138 1 0.001 0.135 0.000
MLP-PLR
0.783 0.007 0.796 + 0.006
TabR-S
0.124 1 0.001
0.121 0.000
TabR-S
0.863 0.003 0.870 1: 0.003
TabR
0.122 1 0.001 0.120 1 0.000
TabR
0.871 + 0.003 0.879 1 0.001
CatBoost
0.129 0.000
0.128 0.000
CatBoost
0.771 0.004 0.775 + 0.003
XGBoost
0.131 + 0.001 0.130 0.000
XGBoost
0.819 + 0.005 0.822 + 0.003
LightGBM
0.131 0.001 0.130 0.000
LightGBM
0.771 0.003 0.773 0.003
Default hyperparameters
Default hyperparameters
TabR-S
0.124 + 0.001 0.122 + 0.000
TabR-S
0.865 + 0.004 0.870 + 0.001
CatBoost
0.129 + 0.000 0.129 0.000
CatBoost
0.758 0.002 0.760 0.001
XGBoost
0.141 = 0.000
0.141 = 0.000
XGBoost
0.751 0.000 0.751 + 0.000
LightGBM
0.135 = 0.000 0.135 = 0.000
LightGBM
0.762 0.004 0.762 + 0.004
covertype f
cpu act |
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tumed Hyperprameterss
MLP
0.929 + 0.001 0.934 0.001
MLP
2.712 0.207 2.544 +: 0.052
MLP-PLR
0.944 + 0.002 0.950 + 0.001
MLP-PLR
2.270 - 0.048 2.214 + 0.059
TabR-S
0.953 0.000
0.954 0.000
TabR-S
2.298 0.053
2.223 0.050
TabR
0.957 + 0.000 0.958 + 0.000
TabR
2.128 = 0.078 2.063 1: 0.050
CatBoost
0.938 0.000 0.939 0.000
CatBoost
2.124 0.049 2.109 0.050
XGBoost
0.940 + 0.000 0.940 + 0.000
XGBoost
2.524 = 0.353 2.472 + 0.379
LightGBM
0.939 + 0.000 0.939 0.000
LightGBM
2.222 0.089 2.207 0.092
Detault hyperparamtess
Default hyperparameter
TabR-S
0.952 = 0.000 0.953 0.000
TabR-S
2.285 0.045 2.214 + 0.032
CatBoost
0.912 + 0.000 0.913 0.000
CatBoost
2.185 0.088 2.162 1 0.091
XGBoost
0.927 1 0.000 0.927 0.000
XGBoost
2.910 0.463 2.910 0.46
LightGBM
0.936 1 0.000 0.936 0.000
LightGBM
2.274 0.128 2.274 1..135
36

---

Published as a conference paper at ICLR 2024
credit f
diamonds
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tuned Hyperparameters
MLP
0.772 + 0.004 0.774 + 0.003
MLP
0.091 + 0.002
0.086 + 0.000
MLP-PLR
0.774 + 0.001 0.775 0.006
MLP-PLR
0.087 0.001
0.084 0.001
TabR-S
0.773 = 0.004 0.774 0.004
TabR-S
0.083 0.001
0.082 + 0.000
TabR
0.772 = 0.004 0.775 0.003
TabR
0.083 0.001
0.081 0.000
CatBoost
0.773 + 0.003 0.775 0.004
CatBoost
0.084 0.000
0.083 1 0.000
XGBoost
0.770 + 0.003 0.771 0.003
XGBoost
0.085 0.000
0.084 1 0.000
LightGBM
0.769 0.003 0.773 0.003
LightGBM
0.085 0.000
0.085 0.000
Default hyperperameters
Defanlt hyperparameters
TabR-S
0.772 + 0.005 0.774 + 0.005
TabR-S
0.084 = 0.001 0.082 + 0.001
CatBoost
0.771 0.005 0.773 0.002
CatBoost
0.084 0.000 0.084
0.000
XGBoost
0.772 + 0.002 0.772 + 0.002
XGBoost
0.088 0.000 0.088 + 0.000
LightGBM
0.771 0.003 0.771 0.003
LightGBM
0.086 0.000 0.086 0.000
electricity
elevators 4
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tuned Hyperparameters
MLP
0.832 = 0.004 0.841 0.002
MLP
0.005 0.000 0.005 + 0.000
MLP-PLR
0.841 + 0.004 0.849 0.000
MLP-PLR
0.002 0.000 0.002 + 0.000
TabR-S
0.924 + 0.003 0.929 0.001
TabR-S
0.005 0.000
0.005 1 0.000
TabR
0.937 1 0.002 0.942 0.000
TabR
0.002 1 0.000 0.002 0.000
CatBoost
0.880 0.002 0.882 0.001
CatBoost
0.002 0.000
0.002 0.000
XGBoost
0.890 + 0.001 0.891 0.001
XGBoost
0.002 + 0.000 0.002 + 0.000
LightGBM
0.887 0.001 0.887 0.001
LightGBM
0.002 0.000 0.002 0.000
Default hyperparameters
Default hyperparameters
TabR-S
0.887 + 0.004 0.893 + 0.002
TabR-S
0.005 + 0.000 0.005 + 0.000
CatBoost
0.875 + 0.001 0.877 0.000
CatBoost
0.002 0.000 0.002 0.000
XGBoost
0.882 = 0.000
0.882 0.000
XGBoost
0.002 0.000 0.002 + 0.000
LightGBM
0.890 = 0.000 0.890 0.000
LightGBM
0.002 0.000 0.002 0.000
fifa
house 16H
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparamterss
Tumed Hyperparameters
MLP
0.803 + 0.013 0.801 0.015
MLP
0.598 0.012 0.587 1: 0.004
MLP-PLR
0.794 + 0.011 0.792 + 0.012
MLP-PLR
0.594 = 0.003 0.589 + 0.002
TabR-S
0.790 0.012 0.786 0.012
TabR-S
0.608 0.016 0.590 + 0.006
TabR
0.791 + 0.014 0.787 +: 0.016
TabR
0.629 = 0.024 0.599 1: 0.000
CatBoost
0.783 0.012 0.782 0.011
CatBoost
0.599 0.005 0.596 0.003
XGBoost
0.780 + 0.011 0.780 + 0.011
XGBoost
0.591 = 0.007 0.585 + 0.004
LightGBM
0.781 + 0.012 0.779 0.012
LightGBM
0.575 0.002 0.573 +
Detault hyperparmmtees
Default hyperparameten
TabR-S
0.790 = 0.013 0.786 + 0.012
TabR-S
0.603 0.015 0.583 + 0.003
CatBoost
0.782 1 0.012 0.781 0.013
CatBoost
0.591 0.002 0.590 0.001
XGBoost
0.790 1 0.012 0.790 0.013
XGBoost
0.589 0.000 0.589 1..99-1.000 0.000
LightGBM
0.780 1 0.011 0.780 0.011
LightGBM
0.593 0.000
0.593 4.000
37

---

Published as a conference paper at ICLR 2024
house sales +
houses 4
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tuned Hyperparameters
MLP
0.181 + 0.001 0.178 + 0.000
MLP
0.233 + 0.002 0.227 + 0.001
MLP-PLR
0.169 + 0.001 0.168 0.000
MLP-PLR
0.228 0.002 0.224 0.000
TabR-S
0.169 = 0.001
0.166 0.000
TabR-S
0.199 0.001 0.196 + 0.000
TabR
0.164 0.001 0.161 = 0.000
TabR
0.201 0.002 0.197 + 0.000
CatBoost
0.167 + 0.000
0.167 0.000
CatBoost
0.216 0.001
0.214 1: 0.000
XGBoost
0.169 + 0.000 0.169 + 0.000
XGBoost
0.219 0.001 0.217 0.000
LightGBM
0.169 0.000 0.169 0.000
LightGBM
0.219 0.001 0.217 + 0.000
Default hyperperameteess
Defanlt hyperparameters
TabR-S
0.169 + 0.001 0.167 + 0.000
TabR-S
0.200 = 0.001 0.197 + 0.001
CatBoost
0.167 0.000 0.167
CatBoost
0.216 0.000 0.216 0.000
XGBoost
0.179 + 0.000 0.179 0.000
XGBoost
0.234 0.000 0.234 + 0.000
LightGBM
0.173 + 0.000 0.173 0.000
LightGBM
0.226 0.000 0.226 0.000
isolet 4
jannis f
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tuned Hyperparameters
MLP
2.223 = 0.189 2.037 0.106
MLP
0.785 = 0.003 0.787 + 0.002
MLP-PLR
2.224 + 0.156 2.030 0.103
MLP-PLR
0.799 0.003 0.804 +: 0.001
TabR-S
1.976 1 0.174
1.763 0.152
TabR-S
0.798 0.002 0.802 1: 0.002
TabR
1.992 1 0.181 1.748 0.143
TabR
0.805 0.002 0.811 1: 0.001
CatBoost
2.867 0.014
2.848 0.002
CatBoost
0.798 0.002 0.801 + 0.001
XGBoost
2.757 + 0.047 2.729 + 0.037
XGBoost
0.797 = 0.002 0.800 + 0.001
LightGBM
2.701 0.030 2.690 0.029
LightGBM
0.796 0.002 0.797
Default hyperparameters
Default hyperparameters
TabR-S
1.995 + 0.156 1.754 + 0.106
TabR-S
0.795 + 0.002 0.800 + 0.001
CatBoost
2.895 + 0.020 2.863 0.013
CatBoost
0.795 0.001 0.797 0.000
XGBoost
3.368 = 0.010
3.368 0.011
XGBoost
0.783 0.000 0.783 + 0.000
LightGBM
2.953 = 0.056 2.953 0.058
LightGBM
0.794 0.000 0.794 + 0.000
kdd ipums la 97-small f
medical charges
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tumed Hyperparameters
MLP
0.880 +: 0.007 0.880 0.006
MLP
0.082 0.000 0.081 +: 0.000
MLP-PLR
0.883 + 0.005 0.883 + 0.005
MLP-PLR
0.081 = 0.000 0.081 + 0.000
TabR-S
0.880 0.008 0.882 0.008
TabR-S
0.081 0.000
0.081 + 0.000
TabR
0.883 + 0.005 0.884 + 0.005
TabR
0.081 = 0.000
0.081 1: 0.000
CatBoost
0.879 0.009 0.880 0.010
CatBoost
0.082 0.000
0.082 0.000
XGBoost
0.883 + 0.009 0.883 + 0.008
XGBoost
0.082 0.000 0.082 + 0.000
LightGBM
0.879 + 0.007 0.880 0.007
LightGBM
0.082 0.000 0.082 0.000
Detault hyperparmmttssessss
Default hyperparameten
TabR-S
0.877 = 0.006 0.878 0.007
TabR-S
0.082 0.000 0.081 + 0.000
CatBoost
0.879 1 0.007 0.881 0.007
CatBoost
0.082 1 0.000 0.082 1: 0.000
XGBoost
0.883 1 0.010 0.883 0.011
XGBoost
0.084 0.000 0.084 1: 0.000
LightGBM
0.884 1 0.005 0.884 1 0.005
LightGBM
0.083 0.000 0.083 4 0.000
38

---

Published as a conference paper at ICLR 2024
pee-taxi-green-dee--e
particulate-matter-innn -at----matttttiieii matter-ukair-2017
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tuned Hyperparameters
MLP
0.397 + 0.001 0.391 + 0.001
MLP
0.377 + 0.001 0.374 + 0.000
MLP-PLR
0.368 + 0.002 0.364 0.000
MLP-PLR
0.367 0.001
0.366 0.000
TabR-S
0.358 = 0.022 0.338 0.003
TabR-S
0.361 0.000
0.359 0.000
TabR
0.372 = 0.009 0.350 0.003
TabR
0.360 0.000
0.358 0.000
CatBoost
0.365 1 0.001 0.363 0.000
CatBoost
0.365 0.000
0.364 0.000
XGBoost
0.379 + 0.000 0.379 0.000
XGBoost
0.364 0.000
0.364 0.000
LightGBM
0.369 0.000 0.368 0.000
LightGBM
0.364 0.000 0.363 0.000
Default hyperperameters
Defanlt hyperparameters
TabR-S
0.389 + 0.001 0.385 + 0.000
TabR-S
0.361 = 0.001 0.359 + 0.000
CatBoost
0.366 0.000 0.366 0.000
CatBoost
0.366 0.000 0.366
0.000
XGBoost
0.386 + 0.000 0.386 0.000
XGBoost
0.368 0.000 0.368 + 0.000
LightGBM
0.372 0.000 0.372 0.000
LightGBM
0.366 0.000 0.366 0.000
phoneme
pol
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparamaters
Tuned Hyperparameters
MLP
0.851 = 0.014 0.861 0.013
MLP
5.659 = 0.543 5.143 + 0.579
MLP-PLR
0.866 1 0.012 0.875 0.012
MLP-PLR
2.615 1 0.137 2.445 + 0.073
TabR-S
0.878 1 0.010 0.884 0.005
TabR-S
6.071 0.537
5.558 1 0.404
TabR
0.877 + 0.009 0.885 1 0.007
TabR
2.577 0.169 2.326 0.058
CatBoost
0.883 0.012 0.890 0.005
CatBoost
3.632 0.101
3.551
0.090
XGBoost
0.868 + 0.017 0.877 0.016
XGBoost
4.296 + 0.064
4.255 + 0.049
LightGBM
0.870 0.013 0.873 0.013
LightGBM
4.232 0.337 4.188 0.311
Default hyperparameters
Default hyperparameter
TabR-S
0.877 + 0.007 0.880 + 0.003
TabR-S
6.200 + 0.396 5.804 + 0.248
CatBoost
0.879 + 0.011 0.881 0.012
CatBoost
1.479 0.051
4.400 0.039
XGBoost
0.870 = 0.016 0.870 0.016
XGBoost
5.249 0.183 5.249 + 0.197
LightGBM
0.874 + 0.007 0.874 0.007
LightGBM
4.382 0.195 4.382 0.210
rl
road-safety Single model
Method
Single model
Ensemble
Method
Ensemble
Tuned Hyperparameters
Tumed Hyperparameters
MLP
0.671 + 0.013 0.677 0.013
MLP
0.786 0.001 0.789 1: 0.000
MLP-PLR
0.744 + 0.019 0.767 + 0.027
MLP-PLR
0.785 - 0.002 0.789 + 0.001
TabR-S
0.874 0.008 0.880 0.006
TabR-S
0.840 0.001
0.844 + 0.000
TabR
0.884 + 0.016 0.891 + 0.013
TabR
0.837 0.001 0.843 1: 0.000
CatBoost
0.790 0.007 0.793 0.005
CatBoost
0.801 0.001
0.802 0.000
XGBoost
0.797 + 0.012 0.799 + 0.012
XGBoost
0.810 = 0.002 0.813 + 0.000
LightGBM
0.781 + 0.010 0.787 0.007
LightGBM
0.798 0.001 0.800 0.000
hyperparameters
Default hyperparametes
TabR-S
0.871 + 0.008 0.876 + 0.007
TabR-S
0.791 0.003 0.796 + 0.003
CatBoost
0.785 1 0.010 0.790 0.004
CatBoost
0.792 0.001 0.793 1 0.000
XGBoost
0.775 + 0.003 0.775 0.003
XGBoost
0.796 0.000 0.796 1 0.000
LightGBM
0.778 1 0.003 0.778 1 0.003
LightGBM
0.803 0.000 0.803 4 0.000
39

---

Published as a conference paper at ICLR 2024
sulfur
superconduct
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparameters
Tuned Hyperprameetess
MLP
0.022 + 0.002 0.021 + 0.002
MLP
10.724 + 0.062
10.455 + 0.005
MLP-PLR
0.020 + 0.002 0.019 0.003
MLP-PLR
10.566 + 0.058
10.334 0.028
TabR-S
0.022 = 0.002 0.021 0.002
TabR-S
10.884 = 0.107
10.480 = 0.028
TabR
0.022 = 0.008 0.020 0.003
TabR
10.384 = 0.056
10.137 = 0.023
CatBoost
0.019 0.002 0.019 0.002
CatBoost
10.242 1 0.022
10.212 1 0.006
XGBoost
0.020 + 0.002 0.020 0.002
XGBoost
10.161 + 0.020
10.141 + 0.002
LightGBM
0.020 0.002 0.020 0.002
LightGBM
10. 16. + 0.012 10.155 0.005
Default hyperperameters
Default hyperparametens
TabR-S
0.021 + 0.003 0.021 + 0.002
TabR-S
10.812 = 0.110 10.423 + 0.046
CatBoost
0.019 0.002 0.019 0.003
CatBoost
10.263 = 0.028
10.222 0.006
XGBoost
0.022 + 0.002 0.022 0.002
XGBoost
10.736 + 0.000
10.736 + 0.000
LightGBM
0.021 0.001 0.021 0.001
LightGBM
10.471 + 0.000
10.471
0.000
visualizing soil
wine
Method
Single model
Ensemble
Method
Single model
Ensemble
Tuned Hyperparamzters
Tuned Hyperparameters
MLP
0.138 = 0.012 0.132 0.010
MLP
0.769 0.015 0.784 + 0.010
MLP-PLR
0.158 + 0.067 0.144 0.060
MLP-PLR
0.771 0.016 0.783 + 0.014
TabR-S
0.398 1 0.352 0.387 0.375
TabR-S
0.794 0.011
0.805 0.006
TabR
0.227 1 0.264 0.202 0.147
TabR
0.780 0.015 0.795 0.002
CatBoost
0.055 0.006 0.047 0.006
CatBoost
0.799 0.013 0.806 0.010
XGBoost
0.176 + 0.071 0.154 0.054
XGBoost
0.795 + 0.018 0.801 0.019
LightGBM
0.062 0.016 0.062 0.017
LightGBM
0.789 0.016 0.793
Default hyperparameters
Default hyperparameters
TabR-S
0.327 + 0.254 0.310 + 0.257
TabR-S
0.791 + 0.012 0.800 + 0.008
CatBoost
0.064 + 0.005 0.058 0.005
CatBoost
0.796 0.010 0.799 0.010
XGBoost
0.066 = 0.009 0.066 = 0.010
XGBoost
0.796 0.010 0.796 + 0.010
LightGBM
0.061 = 0.013 0.061 0.014
LightGBM
0.798 0.004 0.798 0.004
wine quality
year
Method
Single model
Ensemble
Method
Single model
Ensemble
Hyperparameters
Tumed Hyperparameters
MLP
0.672 + 0.015 0.659 0.016
MLP
8.964 0.018 8.901 1: 0.003
MLP-PLR
0.654 + 0.018 0.634 + 0.018
MLP-PLR
8.927 = 0.013 8.901 + 0.006
TabR-S
0.632 0.010 0.620 0.010
TabR-S
9.007 0.015
8.913 0.009
TabR
0.641 + 0.011 0.620 + 0.007
TabR
8.972 0.010 8.917 1: 0.003
CatBoost
0.609 0.013 0.606 0.014
CatBoost
9.037 0.007
9.005 0.003
XGBoost
0.604 + 0.013 0.602 - 0.014
XGBoost
9.031 = 0.003 9.024 + 0.001
LightGBM
0.613 + 0.014 0.612 0.014
LightGBM
9.020 0.002 9.013 0.001
Detault hyperaraaeeters
Default hyperparameten
TabR-S
0.628 = 0.015 0.614 0.015
TabR-S
9.067 0.022 8.893 + 0.008
CatBoost
0.628 1 0.012 0.626 0.012
CatBoost
9.073 0.008 9.046 0.001
XGBoost
0.648 1 0.008 0.648 0.008
XGBoost
9.376 0.000 9.376 1 0.000
LightGBM
0.641 + 0.011 0.641 1 0.012
LightGBM
9.214 0.000 9.214 4 0.000
40

---

Published as a conference paper at ICLR 2024
yprop 4 1
Method
Single model
Ensemble
Tuned Hyperparameters
MLP
0.027 + 0.001 0.027 + 0.001
MLP-PLR
0.027 + 0.001 0.027 0.001
TabR-S
0.027 = 0.000
0.027 0.001
TabR
0.027 = 0.000 0.027 = 0.000
CatBoost
0.027 0.000
0.027 1 0.001
XGBoost
0.027 + 0.001 0.027 1: 0.001
LightGBM
0.027 0.000 0.027 0.000
Default hyperparametees
TabR-S
0.027 = 0.001 0.027 + 0.001
CatBoost
0.027 0.000 0.027 0.000
XGBoost
0.027 + 0.001 0.027 + 0.001
LightGBM
0.027 0.000 0.027 0.000
41


Reviews:
[{"review_id": "7UJsyfPyd4", "reviewer": "Reviewer_aaV3", "summary": "This paper considers the problem of making predictions on tabular data. The authors propose a retrieval-augmented approach where a predictor takes the representation not of the table being predicted but also the representation of the nearest neighbors from a training dataset. The encoding representations and the predictors are training together and use straightforward architecture architectures. The main result is that a combination of the carefully crafted techniques outperforms GBDT on an ensemble of tasks. The training time is higher than GBDT but not unreasonable, and better compared to prior deep learning methods. The prediction times are better", "questions": "1. Inference time and compexity -- are the studies based on normalized inference time between models? If not, could you comment more? How does the inference complexity depend on the size of the table data?\n\n2. Could a different selection of datasets prove that the tabR is not superior to GBDT? In other words, are these datasets highly representative?\n\n3. Is it not surprising that Step-1 (adding context labels) did not help that much? One would guess that this is a big component of signal in retrieval augmentation.\n\n4. Not a question, but the methodology here reminds one of extreme classification and specifically this paper. https://arxiv.org/abs/2207.04452", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The results seem to be a significant advance over prior work in tabular data predictions. In particular, the first deep learning model to outperform GBDT on an ensemble of datasets.\n2. The experiments and analysis are quite extensive. Multiple datasets of different kinds of data, analysis of training and prediction times.\n3. Clear articulation of which techniques helped. the techniques are overall not too complex.", "weaknesses": "A comparison of the inference and query complexity between the methods is lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LGzAT6gNeL", "reviewer": "Reviewer_fJsx", "summary": "This work proposes a retrieval-augmented deep learning architecture for tabular regression/classification. The model passes $x$, the row to be classified/predicted, as well as additional retrieval context rows, through a learned encoder. TabR then retrieves the rows most similar to the encoded form of $x$, where similarity is defined as the Euclidean distance between the encoded versions of two rows, mapped through a linear layer. The top retrieval candidates and their respective labels are then sent through some more learned transformations before being aggregated and combined with the encoded form of the row to be classified/regressed. This combined embedding goes through more MLP layers to result in the output.\n\nThe paper goes through variants of the architecture and how each respective change impacts performance. It then compares against other deep learning-based models as well as gradient boosted decision trees. In both default-hyperparameter and tuned-hyperparameter settings, TabR performs well.", "questions": "See weaknesses. Also, what is $I_{cand}$? Is it all rows of the table that labels have been provided for? It's mentioned in page 3 that \"we use the same set of candidates for all input objects\" but what it the set of candidates exactly?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. The extensive amount of open-sourcing and experiment reproducibility is greatly appreciated.\n1. Strong results relative to both deep learning and boosted tree methods, and TabR-S's relatively strong performance relative to out-of-the-box boosted tree libraries suggests this isn't just excessive parameter tweaking and overfitting via architecture search.\n1. Easy to read, with key pieces of information generally emphasized appropriately.", "weaknesses": "1. Paper doesn't go into detail describing differences with prior deep learning-based tabular methods. What might explain the performance differences? Ex. \"prior work, where several layers with multi-head attention between objects and features are often used\" but was this what led to retrieval's low benefit in the past?\n1. Insufficient discussion of categorical variables. Is accuracy or training time particularly affected by their relative abundance relative to numerical features?\n1. The steps of Section 3.2 seem rather arbitrary. Some of the detail could be compressed to make room for more intuition why the final architecture makes more sense (content from A.1.1). Description of architectural changes that didn't work would also be very insightful.\n1. Paper describes training times in A.4, but I believe a summary of this is important enough to warrant inclusion in the main paper. Something like a mention of the geometric mean (over the datasets) of the ratio between TabR's training time to a gradient boosted methods, described in the conclusion, would be sufficient. While the ratio is likely >1, it is better to acknowledge this weakness than to hide it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CEBB2izG6I", "reviewer": "Reviewer_Ly5o", "summary": "The paper introduces TabR, a retrieval-augmented tabular deep learning model that outperforms gradient-boosted decision trees (GBDT) on various datasets. TabR incorporates a novel retrieval module that is similar to the attention mechanism, which helps the model achieve the best average performance among tabular deep learning models and is more efficient compared to prior retrieval-based models.", "questions": "1.  What's the reason for choosing m to be 96? How does m affect the performance of TabR?\n2.  What's the inference efficiency of TabR and how does it compare with other baselines (e.g., GBDT)?\n3.  Is TabR applicable to categorical features? It seems like the paper only considers continuous features.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. TabR demonstrates superior performance compared to GBDT and other retrieval-based tabular deep learning models on multiple datasets.\n2. The new similarity module in TabR has a reasonable intuitive motivation, allowing it to find and exploit natural hints in the data for better predictions.", "weaknesses": "1. Some aspects are not clear, see the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yVdQ7kKCcl", "reviewer": "Reviewer_EnVq", "summary": "The authors meticulously designed a supervised deep learning model for tabular data prediction, which operates in a retrieval-like manner. It outperformed tree-based models on middle-scale datasets, as well as other retrieval-based deep learning tabular learning models. To achieve this, they introduced a k-Nearest-Neighbors-like idea in model design.", "questions": "- In Section 3.1, you mentioned \"continuous (i.e., continuous) features.\" Could this be a typographical error?\n\n- I am curious if the L2 design is sensitive to uninformative features? You can offer some analysis or conduct experiments by adding some gaussian noise columns (uninformative features are commonly seen in tabular datasets) and observe the change of performances. Some transformation like logarithm may impact the results.\n\n- Some questions in weakness.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- As emphasized by the authors, their method has managed to outperform tree based models like xgboost on middle-scale datasets.\n\n- Overall, the presentation is clear, and the experiments are comprehensive. The details are clear and the model is highly reproducible.\n\n- This model is the best-performing retrieval based model.", "weaknesses": "- The motivations behind the module designs are not entirely clear. It appears that the authors made meticulous module (equation) optimization based on its performance on some datasets empirically. Then: \n\n(1) Why does employing the L2 distance, instand of the dot product, lead to improved performance (as shown in Eq. 3)? \n\n(2) Why is the T function required to use LinearWithoutBias? \n\n(3) We are uncertain about the robustness of the designed modules. If the dataset characteristics are changed, is it likely that the performance rankings will change significantly? The performances only on middle-sized datasets cannot show the robustness.\n\n...\n\nI suggest that providing a theoretical analysis or intuitive motivation would enhance the reader's understanding of those details.\n\n- Some sota DL approaches are not compared, such as T2G-Former (an improved version of FTT)[1], TabPFN [2], and TANGOS [3]. Especially, TabFPN is relatively similar to TabR. These papers are current SOTA, and may outperforms tree based models.\n\n[1] T2G-Former: Organizing tabular features into relation graphs promotes heterogeneous feature interaction\n\n[2] TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second\n\n[3] TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specialization\n\n- The major comparison lies among middle-scale datasets, accompanied with some results on few other datasets shown in Table 3. In scenarios involving sparse, medium, and dense data-distributed datasets (which typically occur in small, medium-sized, and large-sized datasets, respectively), I suppose that there exists a variance in the nearest neighbor retrieval pattern. Hence, conducting tests solely on medium-sized datasets may not suffice. Furthermore, the issue of inefficiency when dealing with large-scaled datasets appears to have hindered the authors from proving the method's effectiveness in large-scaled datasets.\n\n- The method proposed by the authors appears to have achieved slight performance advantages on certain datasets (although some SOTA are not compared). However, due to the lacks of explanation for the model details that are designed empirically, it seems unnecessary and risky to apply this method in real-world scenarios (for example, it's unclear whether L2 distance may fail when uninformative features are present; or, for instance, when a table has a feature with values [f_1, f_2, f_3, ..., f_n], and we take the logarithm of these values [log f_1, log f_2, log f_3, ..., log f_n] or their reciprocals, the method may perform poorly in such cases).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)