Revisiting Deep Learning Models for Tabular Data
Authors: Yury Gorishniy
Ivan Rubachev
Valentin Khrulkov
Artem Babenko
Venue: NeurIPS 2021 Poster (2021)
Keywords: tabular data
architecture
DNN

Abstract: The existing literature on deep learning for tabular data proposes a wide range of novel architectures and reports competitive results on various datasets. However, the proposed models are usually not properly compared to each other and existing works often use different benchmarks and experiment protocols. As a result, it is unclear for both researchers and practitioners what models perform best. Additionally, the field still lacks effective baselines, that is, the easy-to-use models that provide competitive performance across different problems.

In this work, we perform an overview of the main families of DL architectures for tabular data and raise the bar of baselines in tabular DL by identifying two simple and powerful deep architectures. The first one is a ResNet-like architecture which turns out to be a strong baseline that is often missing in prior works. The second model is our simple adaptation of the Transformer architecture for tabular data, which outperforms other solutions on most tasks. Both models are compared to many existing architectures on a diverse set of tasks under the same training and tuning protocols. We also compare the best DL models with Gradient Boosted Decision Trees and conclude that there is still no universally superior solution. The source code is available at https://github.com/yandex-research/rtdl.

Revisiting Deep Learning Models for Tabular Data
Yury Gorishniy
f8
Ivan Rubachey
Valentin Khrulkov?
Artem Babenko
Yandex, Russia
Moscow Institute Physics and Technology, Russia National Research University Higher School of Economics, Russia
Abstract
The existing literature on deep learning for tabular data proposes a wide range of novel architectures and reports competitive results various datasets. However, the proposed models are usually not properly compared to each other and existing works often different benchmarks and experiment protocols. As result, it is unclear for both researchers and practitioners what models perform best. Additionally, the field still lacks effective baselines, that is, the easy-to-use models that provide competitive performance across different problems.
In this work, we perform an overview of the main families of DL architectures for tabular data and raise the bar of baselines in tabular DL by identifying two simple and powerful deep architectures. The first one is ResNet-like architecture which turns out to be strong baseline that is often missing in prior works. The second model is our simple adaptation of the Transformer architecture for tabular data, which outperforms other solutions on most tasks. Both models are compared to many existing architectures tuning protocols. We also compare the best DL models with Gradient Boosted Decision Trees and conclude that there is still no universally superior solution. The source code is available at https: / //gi thub. www.:aadeo- resadee-reearcc/rttaa
diverse set of tasks under the same training and
Introduction
Due to the tremendous success deep learning on such data domains as images, audio and texts (Goodfellow al., 2016), there has been a lot of research interest to extend this success to problems with data stored in tabular format. In these problems, data points are represented as vectors of heterogeneous features, which is typical for industrial applications and ML competitions, where neural networks have a strong non-deep competitor in the form of GBDT (Chen and Guestrin, 2016; Ke al., 2017: Prokhorenkova deep learning for tabular data is appealing as it would allow constructing multi-modal pipelines for problems, where only one part of the input is tabular, and other parts include images, audio and other Di-feiendly data. Such pipelines can then be trained end-to-end by gradient optimization for all modalities. For these reasons, large number of DL solutions were recently proposed. and new models continue to emerge (Arik and Plister, 2020; Badirli et al., 2020; Hazimeh et al., 2020; Huang al., 2020; Klambauer et al., 2017: Popov al., 2020; Song et al., 2019; Wang al., 2017, 2020).
al., 2018). Along with potentially higher performance, using
Unfortunately, due to the lack of established benchmarks (such as ImageNet (Deng et al., 2009) for computer vision GLUE (Wang et al., 2019a) for NLP), existing papers different datasets for evaluation and proposed DL models are often not adequately compared to each other. Therefore, from the current literature, it is unclear what DL model generally performs better than others and whether GBDT is surpassed by DL models. Additionally, despite the large number of novel architectures, the field still lacks simple and reliable solutions that allow achieving competitive performance with moderate effort and provide stable performance across many tasks. In that regard, Multilayer
Cooresponndecce to: yura. www.grrisnniyyyratchh.....
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Perceptron (MLP) remains the main simple baseline for the field, however, it does not always represent a sigmificant challenge for other competitors.
The described problems impede the research process and make the observations from the papers not conclusive enough. Therefore, believe it is timely review the recent developments from the field and raise the bar of baselines in tabular DL. We start with a hypothesis that well-studied DL architecture blocks may be underexplored in the context of tabular data and may be used to design better baselines. Thus, we take inspiration from well-known battle-tested architectures from other fields and obtain two simple models for tabular data. The first one is a ResNet-like architecture (He et al.. 2015) and the second one is FT- "T-Transformer architecture (Vaswani al., 2017) for tabular data. Then, we compare these models with many existing solutions on diverse set of tasks under the same protocols of training and hyperparameters tuning. First, we reveal that none of the considered DL models can consistently outperform the ResNet-like model. Given its simplicity, it can serve as strong baseline for future work. Second, FT-Transformer demonstrates the best performance on most tasks and becomes new powerful solution for the field. Interestingly, FT-Transformer turns out to be more universal architecture for tabular data: it performs well on a wider range of tasks than the more "conventional" ResNet and other DL models. Finally, we compare the best DL models to GBDT and conclude that there is still no universally superior solution.
our simple adaptation of the Transformer
We summarize the contributions of our paper as follows:
1. We thoroughly evaluate the main models for tabular DL on diverse set of tasks to investigate their relative performance.
2. We demonstrate that simple ResNet-like architecture is an effective baseline for tabular DL, which was overlooked by existing literature. Given its simplicity, we recommend this baseline for comparison in future tabular DL works.
3. We introduce FT- Transsfffrr a simple adaptation of the Transformer architecture for tabular data that becomes a new powerful solution for the field. We observe that it is a more universal architecture: it performs well wider range of tasks than other DL models.
4. We reveal that there is still universally superior solution among GBDT and deep models.
2 Related work
The "shallow" state-of-the-art for problems with tabular data is currently ensembles of decision trees, such as GBDT (Gradient Boosting Decision Tree) (Friedman, 2001), which are typically the top-choice in various ML competitions. At the moment, there are several established GBDT libraries, such as XGBoost (Chen and Guestrin, 2016), LightGBM (Ke et al., 2017), CatBoost (Prokhorenkova al., 2018), which widely used by both ML researchers and practitioners. While these implementations vary in detail, on most of the tasks, their performances do not differ much (Prokhorenkova et al., 2018).
During several recent years, a large number of deep learning models for tabular data have been developed (Arik and Pfister, 2020; Badirli et al., 2020; Hazimeh et al., 2020; Huang et al., 2020; Klambauer et al., 2017; Popov et al., 2020; Song et al., 2019; Wang et al., 2017). Most of these models can be roughly categorized into three groups, which we briefly describe below.
Differentiable trees. The first group of models motivated by the strong performance of decision tree ensembles for tabular data. Since decision trees are not differentiable and do not allow gradient optimization, they cannot be used as a component for pipelines trained in the end-to-end fashion. To address this issue, several works (Hazimch et al., 2020; Kontschieder et al., 2015; Popov et al., 2020; Yang et al., 2018) propose to "smooth" decision functions in the internal tree nodes to make the overall tree function and tree routing differentiable. While the methods of this family can outperform GEDT on some tasks (Popov et al., 2020), in our experiments, they do not consistently outperform ResNet.
Attention-baser models. Due to the ubiquitous success of attention-based architectures for different domains (Dosovitskiy et al., 2021; Vaswani et al., 2017), several authors propose to employ attention- like modules for tabular DL. as well (Arik and Pfister, 2020; Huang et al., 2020; Song et al., 2019). In our experiments, we show that the properly tuned ResNet outperforms the existing attention-baat based

---

models. Nevertheless, we identify an effective way to apply the Transformer architecture (Vaswani al., 2017) to tabular data: the resulting architecture outperforms ResNet on most of the tasks.
Explicit modeling of multiplicative interactions. In the literature on recommender systems and click-through-rate prediction, several works criticize MLP since it is unsuitable for modeling mul- tiplicative interactions between features (Beutel et al.. 2018; Qin et al.. 2021; Wang et al., 2017). Inspired by this motivation, some works (Beutel al., 2018; Wang al., 2017, 2020) have proposed different ways to incorporate feature products into MLP. In our experiments, however, we do not find such methods to be superior properly tuned baselines.
The literature also proposes some other architectural designs (Badirli et al., 2020; Klambauer et al., 2017) that cannot be explicitly assigned any of the groups above. Overall, the community has developed a variety of models that are evaluated on different benchmarks and are rarely compared each other. Our work aims establish fair comparison of them and identify the solutions that consistently provide high performance.
Models for tabular data problems
In this section, describe the main deep architectures that we highlight in our work, as well as the existing solutions included in the comparison. Since we argue that the field needs strong easy-to-use baselines, we try to reuse well-established DL building blocks as much as possible when designing ResNet (section 3.2) and FT-Transformer (section 3.3). We hope this approach will result in conceptually familiar models that require less effort to achieve good performance. Additional discussion and technical details for all the models are provided in supplementary.
Notation. In this work, we consider supervised learning problems. D=[(a;, y:)) denotes
(n(mmm) and categorical [cal) features dataset, where (x= (nam) x; ((cat) E X represents numerical o:g
of an object and y; E Y denotes the corresponding object label. The total number features is denoted as k. The dataset split into three disjoint subsets: D Dtrain Deal U Dtest, where Dtrain is used for training, Deat is used for early stopping and hyperparameter tuning, and Drest used for the final evaluation. We consider three types of tasks: binary classification Y {0, 1). multiclass classification Y = {1,,... C}} and regression Y I.
3.1 MLP
We formalize the "MLP" architecture in Equation 1.
MLP(x: Linear (MLPBlock (... (MLPElock()))) (M.PLLocc((((MML)))cc(((( MLPBlock(z) = Dropout(ReLU(Linear([]]
(I)
3.2 ResNet
We aware of one attempt design ResNet-like baseline (Klambauer et al., 2017) where the reported results were not competitive. However, given ResNet's success story in computer vision (He al., 2015) and its recent achievements on NLP tasks (Sun and lyyer. 2021), we give a second try and construct simple variation of ResNet as described in Equation 2. The main building block is simplified compared to the original architecture, and there an almost clear path from the input to output which we find to be beneficial for the optimization. Overall, we expect this architecture to outperform MLP on tasks where deeper representations can be helpful.
ResNet(x) = Prediction (ResNetBBock (... (...BeeeeetBBock (Lnnear())))) ResNetBBlokk(m) pepoout(Linea(DDan (Droaarrroooouutellellliiieeeeee (Batccneooaaatthoooooooooao Prediction(r) = Linear (ReLU (BatchNorm (x)))
3

---

3.3 FT-Transformer
In this section, we introduce FT-Transformer (Feature Tokenizer Transformer) simple adaptation of the Transformer architecture (Vaswani et al., 2017) for the tabular domain. Figure demonstrates the main parts of FT-Transformer. In a nutshell, our model transforms all features (categorical and numerical) to embeddings and applies stack of Transformer layers to the em- beddings. Thus, every Transformer layer operates on the feature level of one object. We compare P1-Transformer to conceptually similar Autolnt in section 5.2.
To
T
[CLS]
CLS]
Predict
Feature Tokenizer
Transform ormer
Figure I: The FT-Transformer architecture. Firstly, Feature Tokenizer transforms features to embed- dings. The embeddings are then processed by the Transformer module and the final representation of the [CLS] token used for prediction.
Tj
2: (mrmm)
W
((num)
kdd
X
Forward
W (ead)
Norm
(oat)
b (cat)
F
A
k
Add
B
Multi-Head
Self Gelf-Atrention
(ooot)
(cut)
(ent)
d
Norm
02
T
Ti-1
(a)
Figure (a) Feature Tokenizer; in the example, there are three numerical and two categorical features; (b) One Transformer layer.
Feature Tokenizer, The Feature Tokenizer module (see Figure transforms the input features a to embeddings T' E Rxxd The embedding for given feature z is computed as follows:
T3 b fa( Rd
f : 12d.
where by the feature bias, l is implemented the element-wise multiplication with the vector (auum) E I and pfcat) implemented as the lookup table w/catt) R for categorical features. Overall:
T.(num) = bl (m)m)) + Izj
(num)
(nam)
E Rd
Tl(aa)
1/ (cat) + + +ef (
E |gd
T = stack (num)
(num)
7 (eat)
T (cat) k(ea))
E fxd
where e.' a one-hot vector for the corresponding categorical feature.
Transformer. At this stage, the embedding of the [CLS] token (or "classification token", or "output token", Devlin et al. (2019)) is appended T and I. Transformer layers F1, ...> Fz are applied:
To stack [CCSS] 2] T: FTTT1)).

---

We use the PreNorm variant for easier optimization (Wang et al., 2019b), see Figure 2. In the PreNorm setting, also found to be necessary remove the first normalization from the first Transformer layer to achieve good performance. See the original paper (Vaswani et al., 2017) for the background on Multi- Head Self-Attention (MHSA) and the Feed Forward module. See supplementary for details such as activations, placement of normalizations and dropout modules (Srivastava et al., 2014).
Prediction. The final representation of the [CLS] token is used for prediction:
ý Linear (BalaaayyaLLyyy )))
Limitations. FT-Transformer requires more resources (both hardware and time) for training than simple models such as ResNet and may not be easily scaled to datasets when the number of features is "too large" (it determined by the available hardware and time budget). Consequently, widespread usage of FT-Transformer for solving tabular data problems can lead to greater CO2 emissions produced by ML pipelines, since tabular data problems described problem lies in the quadratic complexity of the vanilla MHSA with respect to the number of features. However, the issue can be alleviated by using efficient approximations of MHSA (Tay et al.. 2020). Additionally, it is still possible to distill FT- Transfoffrrer into simpler architectures for better inference performance. We report training times and the used hardware supplementary.
ubiquitous. The main cause of the
3.4
Other models
In this section, we list the existing models designed specifically for tabular data that we include the comparison.
SNN (Klambauer et al., 2017). An MLP-like architecture with the SELU activation that enables training deeper models.
NODE (Popov et al., 2020). differentiable ensemble of oblivious decision trees.
TabNet (Arik and Pfister, 2020). A recurrent architecture that alternates dynamical reweigh- ing of features and conventional feed-forward modules.
GrowNet (Badirli et al., 2020). Gradient boosted weak MLPs. The official implementation supports only classification and regression problems.
DCN V2 (Wang al., 2020). Consists of an MLP-like module and the feature crossing module (a combination of linear layers and multiplications).
AutoInt (Song et al., 2019). Transforms features embeddings and applies series of attention-based transformations to the embeddings.
XGBoost (Chen and Guestrin, 2016). One of the most popular GBIT implementations.
CatBoost (Prokhorenkova et al., 2018). GBDT implementation that uses oblivious decision
trees (Lou and Obukhov, 2017) weak learners.
4 Experiments
In this section, we compare DL. models to each other as well as to GBDT. Note that in the main text, we report only the key results. In supplementary, we provide: (1) the results for all models on all datasets; (2) information on hardware; (3) training times for ResNet and FT-Traasformrr.
4.1 Scope of the comparison
In our work, focus the relative performance of different architectures and do not employ various model-agnostic DL practices, such as pretraining, additional loss functions, data augmentation, distillation, learning rate warmup, learning rate decay and many others. While these practices potentially improve the performance, our goal is to evaluate the impact of inductive biases imposed by the different model architectures.
4.2 Datasets
We use a diverse set of eleven public datasets (see supplementary for the detailed description). For each dataset, there is exactly one nain-validation-test split, so all algorithms use the same splits. The datasets include: California Housing (CA, real estate data, Kelley Pace and Barry (1997)), Adult (AD, income estimation, Kohavi (1996)), Helena (HE, anonymized dataset, Guyon et al. (2019)),
5

---

Jannis (JA, anonymized dataset, Guyon et al. (2019)), Higgs (HI, simulated physical particles, Baldi al. (2014); we use the version with 98K samples available the OpenML repository (Vanschoren et al., 40) ALOI (AI., images, Geusebroek et al. (2005)), Epsilon (EP, simulated physics experi- ments), Year (YE, audio features, Bertin-Mahieux et al. (2011)), Covertype (CO, forest characteristics, Blackard and Dean. (2000)), Yahoo (YA, search queries, Chapelle and Chang (2011), Microsoft (MI, search queries, Qin and Liu (2013)). We follow the pointwise approach to learning-to- ttoninngt--rank and treat ranking problems (Microsoft, Yahoo) as regression problems. The dataset properties are summarized Table I.
Table
Dataset properties. Notation: "RMSE" root-mean-square error, "Acc." accuracy.
CA
AD
HE
JA
HT
AI.
EP
YE
CO
YA
MI
#objects
20640
48842
65196
83733
98050
108000
500000
515345
581012
709877
1200192
#num. features
8
6
27
54
28
128
2000
90
54
699
136
#eat. features
0
8
0
metric
RMSE
Acc.
Acc.
Acc.
Acc.
Acc.
Acc.
RMSE
Acc.
RMSE
RMSE
#classes
2
100
2
1000
7
4.3 Implementation details
Data preprocessing. Data preprocessing is known to be vital for DL models. For each dataset, the same preprocessing was used for all deep models for fair comparison. By default, we used the quantile transformation from the Scikit-learn library (Pedregosa et al., 2011). We apply standard- ization (mean subtraction and scaling) to Helena and ALOI. The latter one represents image data, and standardization is common practice in computer vision. On the Epsilon dataset, we observed preprocessing to be detrimental to deep models' performance, so we use the raw features on this dataset. We apply standardization to regression targets for all algorithms.
Tuning. For every dataset, we carefully tune each model's hyperparameters. The best hyperparam- eters are the ones that perform best on the validation set, so the test set is never used for tuning. For most algorithms, we use the Optuna library (Akiba et al., 2019) run Bayesian optimization (the Pree-Structured Parzen Estimator algorithm), which is reported to be superior to random search (Turner et al., 2021). For the rest, we iterate over predefined sets of configurations recommended by corresponding papers. We provide parameter spaces and grids in supplementary. We set the budget for Optuna-bac based tuning in terms iterations and provide additional analysis on setting the budget in terms of time in supplementary.
Evaluation. For each tuned configuration, we run 15 experiments with different random seeds and report the performance on the test set. For some algorithms, we also report the performance of default configurations without hyperparameter tuning.
Ensembles. For each model, on each dataset, we obtain three ensembles by splitting the 15 single models into three disjoint groups of equal size and averaging predictions of single models within each group.
Neural networks. We minimize cross-entropy for classification problems and mean squared error for regression problems. For TabNet and GrowNet, we follow the original implementations and use the Adam optimizer (Kingma and Ba, 2017). For all other algorithms, use the AdamW optimizer (Loshchilov and Hutter, 2019). We do not apply learning rate schedules. For each dataset, we use predefined batch size for all algorithms unless special instructions on batch sizes given in the corresponding papers (see supplementary). We continue training until there are patience + 1 consecutive epochs without improvements the validation set; set patience = 16 for all algorithms.
Categorical features. For XGBoost, we use one-hot encoding. For CatBoost, we employ the built-in support for categorical features. For Neural Networks, we use embeddings of the same dimensionality for all categorical features.

---

Table Results for DL models. The metric values averaged over 15 random seeds are reported. See supplementary for standard deviations. For each dataset, top results are in bold. "Top" means "the gap between this result and the result with the best score is not statistically significant". For each dataset, ranks are calculated by sorting the reported scores; the "rank" column reports the average rank across all datasets. Notation: FT-T F1-Transformer,
RMSE, 1 ~ accuracy
CA 4
AD f
HE 1
JA
HI 1
AL f
EP 1
YE 1
CO 7
YA
MI
rank (std)
TabNet
0.510
0.850
0.378
0.723
0.719
0.954
0.8896
8.909
0.957
0.823
0.751
7.5 (2.0)
SNN
0.493
0.854
0.373
0.719
0.722
0.954
0.8975
8.895
0.961
0.761
0.751
6.4 (1.4)
AutoInt
0.474
0.859 0.372
0.721
0.725
0.945
0.8949
8.882
0.934
0.768
0.750
5.7 (2.3)
GrowNet
0.487
0.857
0.722
0.8970
8.827
0.765
0.751
5.7 (2.2)
MLP
0.499
0.852 0.383
0.719
0.723
0.954
0.8977
8.853
0.962
0.757
0.747
4.8 (1.9)
DCN2
0.484
0.853 0.385
0.716
0.723
0.955
0.8977
8.890
0.965
0.757
0.749
4.7 (2.0)
NODE
0.464
0.858 0.359
0.727
0.726
0.918
0.8958
8.784
0.958
0.753
0.745
3.9 (2.8)
ResNet
0.486
0.854
0.396
0.728
0.727
0.963
0.8969
8.846
0.964
0.757
0.748
3.3 (1.8)
FT-T
0.459
0.859 0.391 0.732
0.729
0.960
0.8982
8.855
0.970
0.756
0.746
1.8 (1.2)
4.4
Comparing DL models
Table 2 reports the results for deep architectures. The main takeaways:
MI.P still a good sanity check
ResNet turns out to be an effective baseline that none of the competitors can consistently outperform.
FT-Transformer performs best on most tasks and becomes new powerful solution for the field.
Tuning makes simple models such MLP and ResNet competitive, we recommend tuning baselines when possible. Luckily, today, it is more approachable with libraries such as Optuna (Akiba et al., 2019).
Among other models, NODE (Popov et al., 2020) is the only one that demonstrates high performance on several tasks. However, it is still inferior to ResNet on six datasets (Helena, Jannis, Higgs, ALOI, Epsilon, Covertype), while being a more complex solution. Moreover, is not a truly "single" model; in fact, it often contains significantly more parameters than ResNet and FT-Transformer and has an ensemble-like structure. We illustrate that by comparing ensembles in Table 3. The results indicate that FT-Transformer and ResNet benefit more from ensembling; in this regime, FT-Transformer outperforms NODE and the gap between ResNet and NODE is significantly reduced. Nevertheless, NODE remains a prominent solution among tree-based approaches.
Table 3: Results for ensembles of DL models with the highest ranks (see Table 2). For each model-dataset pair, the metric value averaged over three ensembles is reported. See supplementary for standard deviations. Depending on the dataset, the highest accuracy or the lowest RMSE is in bold. Due the limited precision, some different values are represented with the same figures. Notation: ~ RMSE, 1 accuracy.
CA 1 AD HE t
JA f
HIt
AL
EP
YE
CO1
MI 1
NODE
0.461 0.860 0.361
0.730
0.727
0.921
0.8970
8.716
0.965
0.750
0.744
ResNet
0.478
0.857
0.398
0.734
0.731
0.966
0.8976
8.770
0.967
0.751
0.745
Tranfsssfrrm 0.448 0. 0.860 0.398
0.739
0.731
0.967
0.8984
8.751
0.973
0.747
0.743
4.5
Comparing DL models and GBDT
In this section, our goal is to check whether DL models are conceptually ready to outperform GBDT. To this end, we compare the best possible metric values that one can achieve using GBDT or DL models, without taking speed and hardware requirements into account (undoubtedly, GBDT is more lightweight solution). We accomplish that by comparing ensembles instead of single models since
7

---

GBDT is essentially an ensembling technique and we expect that deep architectures will benefit more from ensembling (Fort al., 2020). We report the results in Table
Table 4: Results for ensembles of GBDT and the main DL models. For each model-dataset pair, the metric value averaged over three ensembles is reported. See supplementary for standard deviations. Notation follows Table 3.
CA 4 AD f HE t
JA t
HIt AL EP
YE CO1
YA
MI
Default hyperparameters
XGBoost
0.462 0.874
0.348 0.711 0.717 0.924 0.8799 9.192
0.964
0.761
0.751
CatBoost
0.428
0.873
0.386
0.724 0.728
0.948
0.8893
8.885
0.910
0.749
0.744
FT-Transformer
0.454
0.860
0.395
0.734
0.731 0.966 0.8969
8.727
0.973
0.747
0.742
Tuned hyperparameters
XGBoost
0,431 0.872
0.377 0.724 0.728
0.8861
8.819
0.969
0.732
0.742
CatBoost
0.423
0.874
0.388
0.727 0.729
0.8898
8.837
0.968
0.740
0.741
ResNet
0.478
0.857
0.398
0.731
0.731
0.966
0.8976
8.770
0.967
0.751
0.745
FT-Transformer
0.448
0.860
0.398
0.739
0.731 0.967 0.8984
8.751
0.973
0.747
0.743
Default hyperparameters. We start with the default configurations to check the "out-of-the-bon performance, which is an important practical scenario. The default FT-Transformer implies a configu- ration with all hyperparameters set to some specific values that we provide in supplementary. Table demonstrates that the ensemble of FT-Transformers mostly outperforms the ensembles of GBDT, which is not the case for only two datasets (California Housing, Adult). Interestingly, the ensemble of default FT-Transformers performs quite par with the ensembles of tuned FT-Transformers. The main takeaway: FT-Transformer allows building powerful ensembles out of the box.
Tuned hyperparameters. Once hyperparameters are properly tuned, GBDTs start dominating on some datasets (California Housing, Adult, Yahoo; see Table 4). In those cases, the gaps are significant enough to conclude that DL models do not universally outperform GBDT. Importantly, the fact that DL models outperform GBDT on most of the tasks does not mean that DL solutions are "better" in any sense. In fact, it only means that the constructed benchmark is slightly biased towards "DL-friendly" problems. Admittedly, GBDT remains an unsuitable solution to multiclass problems with a large number of classes. Depending on the number of classes, GBDT demonstrate unsatisfactory performance (Helena) or even be untunable due to extremely slow training (ALOI). The main takeaways:
there is still no universal solution among DL. models and GBDT
DL research efforts aimed at surpassing GBDT should focus datasets where GBDT outperforms state-of-the-an DL solutions. Note that including "DL-friendly" problems is still important to avoid degradation such problems.
4.6 An intriguing property of FT-Transformer
Table 4 tells one more important story. Namely, FT-Transformer delivers most of its advantage over the "conventional" DL model in the form of ResNet exactly on those problems where GBDT is superior to ResNer (California Housing, Adult, Covertype, Yahoo, Microsoft) while performing on par with ResNet on the remaining problems. In other words, FT- TT-anssffrmerr provides competitive performance on all tasks, while GBDT and ResNet perform well only on some subsets of the tasks. This observation may be the evidence that FT-Transformer is a more "universal" model for tabular data problems. We develop this intuition further in section 5.1. Note that the described phenomenon is not related to ensembling and is observed for single models too (see supplementary).
Analysis
5.1 When FT-Transformer is better than ResNet?
In this section, we make the first step towards understanding the difference in behavior between FT- Transforrrr and ResNet, which was first observed in section 4.6. To achieve that, we design
8

---

sequence of synthetic tasks where the difference in performance of the two models gradually changes from negligible dramatic. Namely, we generate and fix objects {xx)==1- perform the vraln-val-test split once and interpolate between two regression targets: JCBDT, which is supposed to be easier for GBDT and f'D., which expected to be easier for ResNet. Formally, for one object:
N, It),
y= a (CBDT()) + (f)
where fGBDT(x) is an average prediction of 30 randomly constructed decision trees, and for (a: ) is an MLP with three randomly initial- ized hidden layers. Both SGBDT and fDL are generated once, i.e. the same functions ap- plied to all objects (see supplementary for de- tails). The resulting targets standardized before training. The results are visualized in Figure 3. ResNet and FT-Transformer perform similarly well on the ResNet-friendlly tasks and outperform CatBoost on those tasks. However, the ResNet's relative performance drops signif- icantly when the target becomes more GBDT friendly. By contrast, FT-Transformer yields competitive performance across the whole range of tasks.
u2-
RMSE
0.1
03
ResNet
FT Trannsfformer
011
CatBoost
BCC
0.00
0.25
0.50
0.75
1.00
Figure Test RMSE averaged over five seeds (shadows represent std. dev.). One corresponds to one task; each task has the same set train, validation and test features, but different targets.
The conducted experiment reveals type of functions that are better approximated by FT-Transformer than by ResNet. Additionally,
the fact that these functions are based on decision trees correlates with the observations in section 4.6 and the results in Table where Fr-Transformer shows the most convincing improvements over ResNet exactly on those datasets where GBDT outperforms ResNet.
5.2 Ablation study
In this section, we test some design choices of FT-Transformer..
First, we compare FT- Transfsffrrer with AutoInt (Song et al., 2019), since it is the closest competitor in its spirit. AutoInt also converts all features to embeddings and applies self-attention on top of them. However, in its details, AutoInt significantly differs from FT-Trrasformmr: its embedding layer does include feature biases, its backbone significantly differs from the vanilla Transformer (Vaswani et al., 2017), and the inference mechanism does not use the [CLS] token.
Second, check whether feature biases in Feature Tokenizer essential for good performance.
We tune and evaluate FT-Transformer without feature biases following the same protocol as in section 4.3 and reuse the remaining numbers from Table 2. The results averaged over 15 runs reported Table and demonstrate both the superiority of the Transformer's backbone to that of AutoInt and the necessity of feature biases.
Table The results of the comparison between FT-Transformer and two attention- based alternatives: Autolnt and FT-Trrasformer without feature biases. Notation follows Table 2.
CA L
HE 1
JA 1
HI 1
AL
YE
L
CO1
MI J
AutoInt
0.474
0.372
0.721
0.725
0.945
8.882
0.934
0.750
FT- Transformer (wo feature biases)
0.470
0.381
0.724
0.727
0.958
0.751 8.855
8.843
0.964
FT- Transffoorrr
0.459
0.391
0.732
0.729
0.960
0.970
0.746

---

5.3 Obtaining feature importances from attention maps
In this section, we evaluate attention maps as a source of information on feature importances for FT-Transformer for given set of samples. For the i-th sample, we calculate the average attention map p: for the [CLS] token from Transformer's forward pass. Then, the obtained individual distributions are averaged into one distribution p that represents the feature importances:
P Σ
Σ by
Pr
Pinl-
nsamples
Thead x L
where Pist is the h-th head's attention map for the [CLS] token from the forward pass of the 1-th layer on the i-th sample. The main advantage of the described heuristic technique is its efficiency: it requires single forward for one sample.
In order to evaluate our approach, we compare it with Integrated Gradients (IG, Sundararajan et al. (2017)), general technique applicable to any differentiable model. We use permutation test (PT, Breiman (2001)) as a reasonable interpretable method that allows us to establish a constructive metric, namely, rank correlation. We all the methods the train and summarize results in Table 6. Interestingly, the proposed method yields reasonable feature importances and performs similarly to IG (note that this does not imply similarity to IG's feature importances). Given that IG can be orders of magnitude slower and the "baseline" in the form of PT requires feature + 1) forward passes (versus one for the proposed method), we conclude that the simple averaging of attention maps can be a good choice in terms of cost-effectiveness.
Table 6: Rank correlation (takes values in [-1, 1)) between permutation test's feature importances ranking and two alternative rankings: Attention Maps (AM) and Integrated Gradients (IG). Means and standard deviations over five runs are reported.
CA
HE
JA
HI
AL
YE
CO
MI
AM 0.81 (0.05) 0.77 (0.03) 0.78 (0.05) 0.91 (0.03) 0.84 (0.01) 0.92 (0.01) 0.84 (0.04) 0.86 (0.02)
IG 0.84 (0.08) 0.74 (0.03) 0.75 (0.04) 0.72 (0.03) 0.89 (0.01) 0.50 (0.03) 0.90 (0.02) 0.56 (0.02)
6 Conclusion
In this work, we have investigated the status quo in the field of deep learning for tabular data and improved the state of baselines in tabular DL. First, we have demonstrated that simple ResNet-like architecture can serve as an effective baseline. Second, we have proposed FT-Traanformer - simple adaptation of the Transformer architecture that outperforms other DL solutions on most of the tasks. We have also compared the new baselines with GBDT and demonstrated that GBDT still dominates on some tasks. The code and all the details of the study are open-soureed ! and we hope that our evaluation and two simple models (ResNet and FT-Transformer) will serve basis for further developments on tabular DI..
References
T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparameter optimization framework. In KDD, 2019.
S. O. Arik and T. Pfister. Tabnet: Attentive interpretable tabular learning. arXiv, 1908.07442v5 2020.
S. Badirli, X. Liu, Z. Xing, A. Bhowmik, K. Doan, and S. S. Keerthi. Gradient boosting neural networks: Grownet. arXiv, 2002.07971v2, 2020.
P. Baldi, P. Sadowski, and D. Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature Communications, 5, 2014.
10

---

T. Sartin-Mahieux, D. P. Ellis, B. Whitman, and P. Lamere. The million song dataset. In Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR 201 1), 2011.
A. Beutel, Covington, S. Jain, C. Xu, J. Li, V. Gatto, and E. H. Chi. Latent cross: Making use of context in recurrent recommender systems. In WSDM 2018: The Eleventh ACM International Conference on Web Search and Data Mining, 2018.
J. A. Blackard and D. J. Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and Electronics in Agriculture, 24(3):13 1303)111-11,,,111111111111111111 2000.
L. Breiman. Random forests. Machine Learning, 45(1):5-32, 2001. 2001.
O. Chapelle and Chang. Yahoo! learning to rank challenge overview. In Proceedings of the Learning to Rank Challenge, volume 14, 2011.
Chen and C. Guestrin, Xgboost: scalable tree boosting system. SIGKDD, 2016.
Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical
image database. In CVPR, 2009.
Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv, 1810.04805v2, 2019.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
S. Fort, Hu, and B. Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv, 1912.02757v,, 2020.
J. H. Friedman. Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5): 1989) 11-8-222,, 2001.
J. M. Geusebroek, G. J. Burghouts, and A. W. M. Smeulders. The amsterdam library of object images. Int. J. Comput. Vision, 610) 2005.
I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, 2016. http://wwa deeplearningbook. org.
I. Guyon, Sun-Hosoya, M. Boullé,
Escalante, S. Escalera, Liu, D. Jajetic, Ray, M. Saeed, M. Sebag, A. Statnikov, W. Tu, and E. Viegas. Analysis of the automl challenge series 2015-2018. In AutoML, Springer series on Challenges in Machine Learning, 2019.
H. Hazimeh, N. Ponomareva, P. Mol, Z. Tan, and R. Mazumder. The tree ensemble layer: Differen- tiability meets conditional computation. ICML, 2020.
K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. arXiv, 1512.03385v1, 2015.
X. Huang, A. Khetan, M. Cvitkovie, and Z. Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv, 2012.06678v1, 2020.
G. Ke, Q. Meng. T. Finley, T. Wang. W. Chen, W. Ma, Q. Ye, and T.-Y. Liu. Lightghm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30: 3146-3154, 2017.
R. Kelley Pace and R. Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3): 291-297, 1997.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv, 1412.6980v9, 2017.
G. Klambauer, T. Unterthiner, Mayr, and Hochreiter. Self-normalizing neural networks. In NIPS, 2017.

---

R. Kohavi. Scaling up the accuracy of naive- bayes classifiers: a decision-tree hybrid. In KDD, 1996.
P. Kontschieder, M. Fiterau, A. Criminisi, and S. Rota Bulo. Deep neural decision forests. In Proceedings of the IEEE international conference on computer vision, 2015.
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Y. Lou and M. Obukhov, Bdt: Gradient boosted decision tables for high accuracy and scoring efficiency. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning Python. Journal of Machine Learning Research, 12:2825 202822-28300 2011.
S. Popov, S. Morozov, and A. Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In ICLR, 2020.
L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin, Catboost: unbiased boosting with categorical features. In NeurlPS, 2018.
T. Qin and T. Liu, Introducing LETOR 4.0 datasets. arXiv, 1306.2597vl. 2013.
Z. Qin, L. Yan, H. Zhuang, Y. Tay, R. K. Pasumarthi, X. Wang, M. Bendersky, and M. Najork. Are neural rankers still outperformed by gradient boosted decision trees? In ICLR, 2021.
W. Song, C. Shi, Z. Xiao, Z. Duan, Y. Xu, M. Zhang, and J. Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In CIKM, 2019.
N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1): 1959-1958, 2014.
S. Sun and M. lyyer. Revisiting simple neural probabilistic language models. In NAACL, 2021.
M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In JCML, 2017.
Y. Tay, M. Dehghani, D. Bahri, and D. Metzler. Efficient transformers: A survey. arXiv, 2009.06732v1, 2020.
R. Turner, D. Eriksson, M. McCourt, J. Kiili, E. Laaksonen. Z. Xu, and I. Guyon. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. arXiv, https://axiiv.or//bs//11144 10tps//01...00011111 2021.
J. Vanschoren, J. N. Rijn, B. Bischl, and L. Torgo Openml: networked science in machine learning. arXiv, 1407.7722v1, 2014.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NIPS, 2017.
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, GLUE: A multi-task benchmark and analysis platform for natural language understanding. ICLR, 2019a.
Q. Wang, B. Li, T. Xiao, Zhu, C. Li, D. F. Wong, and Chao. Learning deep transformer models for machine translation. In ACL, 2019b.
R. Wang, B. Fu, G. Fu, and M. Wang. Deep & cross network for ad click predictions. In ADKDD, 2017.
R. Wang, Shivanna, Z. Cheng, S. Jain, Lin, L. Hong, and E. Chi. Den Improved deep & cross network and practical lessons for web-scale learning rank systems. arXiv, 13535v2, 2020.
Y. Yang, G. Morillo, and T. M. Hospedales. Deep neural decision trees. arXiv, 1806.06988v1, 2018.
12


Reviews:
[{"review_id": "tw3HqcdeGdn", "reviewer": "Reviewer_KYAo", "summary": "- The authors conduct extensive experiments on tabular data learning with various datasets and baselines.\n- The authors discover some important takeaways for tabular learning such as when we should use shallow or deep models.\n- The authors propose a novel tabular learning framework which shows the state-of-the-art performance in various datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Impactful paper\n- This paper tackles an important problem which has significant attention from both academia and industry.\n- Constructing standard experimental settings, datasets, metrics, and baselines in this field would be highly beneficial for future research in this area.\n- Also, it is important for practitioners which model works well for which condition. When we should use which model is a critical question for many practitioners.\n- The published codebase is clear and easy to understand and reproduce.\n\n2. ResNet\n- Have any previous works tried ResNet type of architecture for tabular data?\n- If yes, it would be good to cite those papers. \n- If not, this would be another contribution of this paper.\n\n3. Experimental details\n- How to treat the categorical variables? One-hot encoding? \n- Would be good to describe the number of feature ranges in the main manuscript.\n- Can we use \"patience\" as another hyper-parameter? How 16 is determined?\n\n4. Sanity check\n- It would be good if the authors can provide whether the implementation of each model can reproduce the results in each paper.\n- For instance, the authors can utilize at least one data per each baseline and check whether the results can be reproduced.\n- This would be important because it can be a sanity check whether the detailed implementation is accurate for those baselines.\n\n5. Etc\n- It would be good if the authors can clarify which dataset is regression, binary, and multi-class classification tasks in results tables.\n- I would like to know whether the performance of an FT-transformer is dependent on the input order of features.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors properly describe the limitations of the paper and baselines.\nHowever, it would be better if the authors can somehow tone down for the discovery and conclusions because those are only supported by the empirical results.", "review_text": "", "extra_scores": {"ethical_concerns": "There is no ethical concern.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "fdFXEeuwRFJ", "reviewer": "Reviewer_rxXz", "summary": "The paper provides a comparison of recently proposed deep learning (DL) models for tabular data against established gradient-boosting decision tree (GBDT) methods.\nThe experimental evaluation shows that there is no clear winner between DL and GDBT models.\n\nThe paper also evaluates an MLP with residual connections ('ResNet') and introduces a Transformer-based model (FT-Transformer) similar to AutoInt.\nThe proposed architectures (ResNet/FT-Transformer) are demonstrated to give highly competitive performance, with FT-Transformer often performing SOTA.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\nI agree with the authors that there is a need for a thorough review and comparison of 'deep learning methods on tabular data'.\nI like their overview of recent DL approaches in the introduction (and would have welcomed a more extensive overview).\nThe hyperparameter tuning of all involved models seems thorough at first impression.\nThe investigation of functional biases (S4.1) between GBDT and Neural Networks is an interesting idea.\n\n## Weaknesses\n\nUnfortunately, the weaknesses may outweigh the strengths for this submission.\nThe paper is uncomfortably split in two between (1) the comparison of established models and (2) the introduction of the FT-Transformer.\nI believe that both parts would benefit from additional work.\n\n### 1. Model Comparison\n\nFor a paper which lists the 'thorough' comparison of models on a 'wide range' of datasets as a key contribution,  the chosen selection of datasets is not adequate for a variety of reasons:\n1. Only one of the datasets has categorical features. All other datasets have exclusively numerical features.\n    * Categorical features are generally regarded as more challenging (especially for deep learning), so this omission may affect conclusions.\n    * For this one dataset, the authors do not employ one hot encoding, which may negatively affect performance for some models.\n2. 11 Datasets is definitely on the small side. (The SELU paper compares performance on 121 datasets.) \n3. All datasets are between 20k and 1M samples.\n    * There are a lot of UCI datasets with <20k samples. Why were they excluded? Surely these are relevant for such a general comparison?\n    * Why were datasets cut to <1M samples? E.g. the Higgs dataset (usually 11M samples) was cut in a non-standard way to <1M samples.\n\nFurther,\n\n4. one of the conclusions of the paper is that GBDT outperforms DL models on 'heterogeneous' but not on 'homogeneous' data.\n    * The separation into homogeneous and heterogeneous datasets performed by the authors *perfectly* aligns with this conclusion.\n    * However,  the specific reasons for how datasets where classified as homo-/heterogeneous is not explained in detail and feels rather arbitrary to me.\n        * The authors state that a dataset is heterogenous if it describes 'different physical properties' and is of 'diverse units of measurements'. As far as I can tell, the Higgs dataset should be heterogenous by this definition, but I would be happy to hear explanations from the author. (Higgs contains a bunch of particle physics properties. Why are these less homogeneous then the housing-related attributes of California housing?)\n        * For epsilon, I can't find any information online about the composition of its features, so why is it homogeneous then?\n    * The perfect separation between GBDT and DL performance on heterogeneous vs homogeneous datasets as well as the lack of justification makes me suspicious that it is not impossible that *the authors came up with the classification of these datasets into hetero-/homogeneous only **after** observing the results*.\n     * Also, no explanation of why performance of GDBT and DL differs on hetero-/homogeneous datasets is given.\n\n### 2. Experiments\n\n5.) For the Adult dataset, I have compared the reported performances in this paper to publically available results at https://www.openml.org/t/7592, which shows a large 0.1 difference between the top reported AUROC values. This might be indicative of problems with the experiment routine. I have not found such irregularities for a few other datasets that I've checked.\n\n6.) The paper makes claims of superiority of one method (e.g ResNets / FT-Transformer) over another across all observed datasets.\nWhy not performance in terms of average rank order across models directly, instead of eyeballing things?\nAdditionally, I would have preferred for the authors to also quantify variance across different splits (instead just over different model seeds), especially for datasets without established test sets.\n\n### 3. FT-Transformer\n\n7.) The paper is split weirdly between 'model comparison/literature review' and introducing the 'FT-Transformer' architecture. Although both parts are individually clear, the introduction of the FT-Transformer architecture comes up somewhat abruptly in the middle of the paper, after the reader has already seen results for the FT-Transformer, affecting clarity of the overall submission.\n\nI feel that making a big deal of the FT-Transformer as a contribution generally hurts this paper, because\n\n* It takes away space that could have been used for a more thorough comparison of architectures\n* The differences to AutoInt are small, despite what the authors claim. (Adding biases to the encoding sounds like a one line code change that seems to add a 0.01 improvement. Adding the mask tokens at input is more interesting but is not ablated.)\n\nHowever, both the ResNet-MLP and the FT-Transformer architecture are potentially interesting as a baselines for future work and maybe the authors could have spent more time discussing the tricks needed to make them work well. \n(Further, the conclusion that MLPs aren't horrible for tabular DL are in line with other recent work (https://arxiv.org/abs/2106.11189, previously at https://openreview.net/forum?id=2d34y5bRWxB).)\n\nOverall, I feel like the split between 'review comparison of prior work' and 'introduction of new models'  does not work well in this submission.\n\n### 4. Architecture Bias Experiments\n\nThe idea of randomly generating targets from GBDT/NN to study biases of the particular architectures is interesting.\nHowever, I feel that the idea is underdeveloped in this submission:\n * What's benefit of linearly interpolating targets? (In my mind, this doesn't add anything as only $\\alpha \\in [0, 1]$ has a clear meaning.)\n * Why not generate data from the transformer as well? \n * Why is XGBoost not included here? Why not extend this to all models in the spirit of the 'big comparison' paper?\n * Fig. 3 seems to suggest that CatBoost can accommodate all data while ResNets can't acccomodate CatBoost-data. How does this relate to the experiments (where ostensibly, ResNet > CatBoost for some datasets).\n * Can this be related to the statements about hetero-/homogeneous datasets?\n\n## Conclusion\n\nMaking a good literature review and model comparison paper is hard.\n\nThis paper is split between trying to provide that comparison and introducing the FT-Transformer.\nI am not convinced that the FT-Transformer contains enough information to warrant publication.\nThe comparison between models might be enough if appropriately extended.\nHowever, I feel that a significant number of the above comments need to be addressed first (mainly: more datasets, compare over different CV splits, show rank order, extend 4.1 to more methods).\n\n\n[edit] Score updated to a 5. See discussion below.\n[edit] Score updated to a 7. See discussion below.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper discusses limitations of the FT-Transformer in a paragraph in S. 3.6, where they identify the quadratic scaling in the number of features as the main limitation of the architecture.\nGiven that a main contribution of this paper is the comparison of architectures, a discussion of the limitations here would have been nice.\n\nI believe the authors have misunderstood the request for discussing 'potential negative societal impacts' of submissions: in the checklist, they again reference S. 3.6 where they discuss only scaling limitations and how to (potentially) alleviate them in future work. It is unclear to me how this relates to societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "_eIp9ZyLHZ", "reviewer": "Reviewer_wxKJ", "summary": "This paper\n1) conducts a careful comparison of many deep learning methods for tabular learning\n2) introduces a new baseline method for tabular learning (ResNet)\n3) introduces a new fancy architecture for tabular learning (FT-Transformer)", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very closely related to (and partially superceded by) [1],\nbut since [1] appeared on arXiv after the paper was submitted\nI will try to evaluate the current paper as though [1] did not exist.\n\nThe writing is clear and the experiments seem careful and thoughtful.\nContributions 1) and 2) are important and useful.\nUnfortunately, the new baseline 2) rather steals the thunder from the\nnew architecture 3).\nThe experiments don't make it clear that the new architecture is worthwhile.\nThe authors are very straightforward about the limitations of FT-transformer\n(and its much greater cost compared to ResNet), which I appreciate.\n\nHyperparameter tuning is a key issue for a comparison. \nGiven more time [or, more generally, computational resources] \nand more parameters (and enough data), DL models tend to improve.\nWere some of these methods given more time than others?\n(Including time to choose hyperparameters?)\nI'd like to have seen performance as a function of time, or after a fixed amount of time.\n\nAs a more narrow question, why Optuna? Does it matter?\n(Looking into the future, [1] uses BOHB [2]. Do you think swapping the hyperparameter tuner would affect your results?)\n\n* 185 \"in the ensembling mode\" -> define\n* 202 ALOI -> meaning of tilde is confusing, say \"The ALOI dataset has nearly 1000 classes\".\n* I found it easy to miss the definition of \"heterogeneous\". Maybe use a definition environment for it?\n* 240: FT-transformer is not sensitive to hyperparameters - I'd like to see more experiments demonstrating this, and looking at the quality of model as a function of training time (including hyperparameter optimization time)\n\n[1] Kadra, Arlind, et al. \"Regularization is all you Need: Simple Neural Nets can Excel on Tabular Data.\" arXiv preprint arXiv:2106.11189 (2021).\n[2] Falkner, Stefan, Aaron Klein, and Frank Hutter. \"BOHB: Robust and efficient hyperparameter optimization at scale.\" International Conference on Machine Learning. PMLR, 2018.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Fine.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "4B8ASoucG87", "reviewer": "Reviewer_RbMa", "summary": "This paper set up a standard benchmark for recent works on NN for tabular data, and GBDT, to check whether these works indeed outperform GBDT or not. Besides, two new models, ResNet-like MLP and transformer-based models are proposed, and seem can outperform previous works.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The standard benchmark is very helpful. With it, we can know which models are better. I believe this can boost the research for the tabular data learning domain. \n\n- The design of the FT-Transformer is simple and solid. Pre-ln is a good choice. The bias term in Tokenizer seems helpful.\n\nWeakness & questions:\n- As both xgboost and catboost are set to level-wise trees, it would be better to have the results of the leaf-wise tree, like LightGBM.\n\n- The selected benchmark datasets are quite messy. It will be better to categorize them by applications/scenarios. Besides, as tabular data is widely used in ads click prediction and recommenders systems, it would be better if the benchmark can cover these two types of tasks. \n\n- For the experiment results, it seems GBDT is not outperformed by NN in tabular data. I notice there are \"heterogeneous\" and \"non-heterogeneous\" datasets in this paper. Obviously, the heterogeneous dataset is more like the tabular data in the real-world (Although multi-modal data is more common, I think we should focus on heterogeneous tabular data, as NN is already very good in other data. As for high-dimensional multi-class data, it actually is not common in real-world applications with tabular data.). From the results, GBDT is still the best one in heterogeneous data. To avoid confusion, I think the author should make the table clear (even use two tables), let readers know there are two kinds of datasets. \n\n- Based on 3, I think the author should make the claim more carefully.  In my opinion, the heterogeneous dataset is more like tabular data. So the sentence like \"we show that the choice between GBDT and DL models highly depends on data and there is still no universally superior solution.\" looks ambiguous.\n\n- Do you have a plan to host a leaderboard, like GLUE in NLP?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "3MmFecT0g8", "reviewer": "Reviewer_hjQ3", "summary": "This paper investigates the performance of deep learning models on tabular data and compares it with traditional models such as gradient boosted decision trees. It shows that ResNet is good baseline for deep learning models on tabular data. It also proposes a new attention based architecture which surpasses all the deep learning methods mentioned in the paper.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The study is quite comprehensive covering a range of datasets\n- The paper analyzes multiple models from each category\n- The proposed transformer based model surpasses other models in most tasks and is good baseline for future tasks\n- All models are explained in detail in the supplementary materials for reproducibility\n- Training runtimes are also compared\n\nCons:\n- FT-Transformer, the proposed method, is more complex and takes longer to train on average than ResNet\n\nOverall, the paper is very useful and provides a great overview of the different approaches for tabular data. By proposing a new method which surpasses the approaches mentioned in the paper, it even adds to the existing body of baselines for future work in the area. The code is open sourced and the models are discussed in great detail in the Appendix. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes they have.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers have agreed on the positive aspects of the paper, and its important contributions for tabular deep learning. There were significant additions by the authors during the review process, particularly extra experimental results, method motivations, clarification of experimental setups and hyperparameter tuning. It is important to reflect these in the final version of the paper. Finally, as discussed, it would be an impactful addition to open-source the benchmarking framework. 
Decision: Accept (Poster)