TabM: Advancing tabular deep learning with parameter-efficient ensembling
Authors: Yury Gorishniy
Akim Kotelnikov
Artem Babenko
Venue: ICLR 2025 Poster (2025)
Keywords: tabular
tabular data
deep learning
architecture
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: Parameter-efficient ensembling has a massive positive impact on tabular MLPs, and TabM is a new SOTA architecture illustrating that.

Abstract: Deep learning architectures for supervised learning on tabular data range from simple multilayer perceptrons (MLP) to sophisticated Transformers and retrieval-augmented methods.
This study highlights a major, yet so far overlooked opportunity for substantially improving tabular MLPs; namely, parameter-efficient ensembling -- a paradigm for imitating an ensemble of models with just one model.
We start by describing TabM -- a simple model based on MLP and BatchEnsemble (an existing technique), improved with our custom modifications.
Then, we perform a large scale evaluation of tabular DL architectures on public benchmarks in terms of both task performance and efficiency, which renders the landscape of tabular DL in a new light.
In particular, we find that TabM outperforms prior tabular DL models, while the complexity of attention- and retrieval-based methods does not pay off.
Lastly, we conduct a detailed empirical analysis, that sheds some light on the high performance of TabM.
For example, we show that parameter-efficient ensembling is not an arbitrary trick, but rather a highly effective way to reduce overfitting and improve optimization dynamics of tabular MLPs.
Overall, our work brings an impactful technique to tabular DL, analyses its behaviour, and advances the performance-efficiency tradeoff with TabM -- a simple and powerful baseline for researchers and practitioners.

Published as a conference paper at ICLR 2025
TABM: ADVANCING TABULAR DEEP LEARNING WITH PARAMETER-EFHICIEN ENSEMBLING
Yury Gorishniy Yandex
Akim Kotelnikov HSE University, Yandex
Artem Babenko Yandex
ABSTRACT
Deep learning architectures for supervised learning on tabular data range from simple multilayer perceptrons (MLP) to sophisticated Transformers and retrieval- augmented methods. This study highlights a major, yet so far overlooked opportu- nity for designing substantially better MLP- based tabular architectures. Namely, our new model TabM relies on efficient ensembling, where one TabM efficiently imi- tates an ensemble of MLPs and produces multiple predictions per object. Compared to a traditional deep ensemble, in TabM, the underlying implicit ML.Ps are trained simultaneously, and (by default) share most of their parameters, which results in significantly better performance and efficiency. Using TabM as a new baseline, we perform large-scale evaluation of tabular DL architectures in terms of both task performance and efficiency, which renders the landscape of tabular DL in a new light. Generally, we show that MLPs, including TabM, form a line of stronger and more practical models compared to attention- and retrieval- find that TabM demonstrates the best perfor-
public benchmarks
based architectures. In particular, mance among tabular DL models. Then, we conduct an empirical analysis on the ensemble-like nature of TabM. We observe that the multiple predictions of TabM are weak individually, but powerful collectively. Overall, our work brings im- pactful technique to tabular DL and advances the performance-eficiene trade-off with TabM - a simple and powerful baseline for researchers and practitioners. The code is available at: https: / //ggttttuu com/ har//yanddxxxx research/tabme
INTRODUCTION
Supervised learning tabular data is ubiquitous machine learning (ML) scenario in wide range of industrial applications. Among classic non-deep-learning methods, the state-of-the-art solution for such tasks gradient-boosted decision trees (GBDT) (Prokhorenkova et al., 2018; Chen & Guestrin, 2016; Ke et al., 2017). Deep learning (DL) models for tabular data, in turn, are reportedly improving, and the most recent works claim to perform on par or even outperform GBDT on academic benchmarks (Hollmann et al., 2023; Chen et al., 2023b;a; Gorishniy et al., 2024).
However, from the practical perspective, it is unclear if tabular DL offers any obvious go-to baselines beyond simple architectures in the spirit of multilayer perceptron (MLP). First, the scale and consistency of performance improvements of new methods w.r.t. simple MLP-like baselines are not always explicitly analyzed in the literature. Thus, one has to infer those statistics from numerous per-dataset performance scores, which makes it hard to reason about the progress. At the same time, due to the extreme diversity of tabular datasets, consistency is especially valuable and pord-to-achieve property for a hypothetical go-to baseline. Second, efficiency-related properties, such training time, and especially inference throughput, sometimes receive less attention. While methods are usually equally affordable on small-to-medium datasets (e.g. <100K objects), their applicability to larger datasets remains uncertain. Third, some recent work generally suggests that the progress on academic benchmarks may not transfer that well to real-world tasks (Rubachev et al., 2024). With all the above in mind, in this work, we thoroughly evaluate existing tabular DL methods and find that MLP models do not yet offer a convincing replacement for MLPs.
At the same time, we identify previously overlooked path towards more powerful, reliable, and reasonably efficient tabular DL models. In nutshell, we find that the eeffiiete-efficientt approach to
"The corresponding author: yurygori shni y@gma com

---

Published as a conference paper at ICLR 2025
deep ensembling, where most weights are shared between ensemble members, allow one to make simple and strong tabular models out of plain MLPs. For example, MLP coupled with BatchEnsemble (Wen et al., 2020) models, such as FT- Fransfoofrrrer (Gorishniy et al., 2021), while being simpler and more efficient. This result alone suggests that efficient ensembling is low-hanging fruit for tabular DL..
long-cun bang-existing method- right away outperforms popular attention -ased
Our work builds on the above observations and offers TabM - a new powerful and practical model for researchers and practitioners. Drawing an informal parallel with GBDT (an ensemble of decision trees), TabM can also be viewed as a simple base model (MLP) combined with an ensembling-like technique, providing high performance and simple implementation the same time.
Main contributions. We summarize our main contributions as follows:
1. We present TabM a simple DL architecture for supervised learning on tabular data. TabM is based on MLP and parameter-efficien ensembling techniques closely related to BatchEnsemble (Wen et al., 2020). In particular, TabM produces Multiple predictions per object. TabM easily competes with GBDT and outperforms prior tabular DL models, while being more efficient than attention- and retrieval-based DL architectures.
2. We provide a fresh perspective on tabular DL models in large-scale evaluation along four dimensions: performance ranks, performance score distributions, training time, and inference throughput. One of our findings is that MLPs, including TabM, hit appealing performance- efficiency tradeoff, which is not the case for attention- and retrieval-basee models.
3. We show that the two key reasons for TabM's high performance are the collective training of the underlying implicit MLPs and the weight sharing. We also show that the multiple predictions of TabM are weak and overfitted individually, while their average is strong and generalizable.
2 RELATED WORK
recision-tree-basse models. Gradient-boosted decision trees (GBDT) (Chen & Guestrin, 2016; Ke al., 2017; Prokhorenkova et al., 2018) is strong and efficient bascline for tabular tasks. GBDT is a classic machine learning model, specifically, an ensemble of decision trees. Our model TabM is a deep learning model, specifically, a parameter-efficienn ensemble of MLPs.
Tabular deep learning architectures. A large number of deep learning architectures for tabular data have been proposed over the recent years. That includes attention- beased architectures (Song
et al., 2019; Gorishniy et al., 2021; Somepalli et al., 2021; Kossen et al., 2021; Yan et al., 2023),
retrieval- -uggeneee architectures (Somepalli et al., 2021; Kossen al., 2021; Gorishniy et al., 2024;
Ye et al., 2024), MLP-like models (Gorishniy et al., 2021; Klambauer et al., 2017; Wang et al., 2020)
and others (Arik& & Pfister, 2020; Popov et al., 2020; Chen et al., 2023b; Marton et al., 2024; Hollmann et al., 2023). Compared to prior work, the key difference of our model TabM is its computation flow, where one TabM imitates an ensemble of MLPs by producing multiple independently trained predictions. Prior attempts to bring ensemble- like elements tabular DL (Badirli et al., 2020; Popov et al., 2020) were not found promising (Gorishniy et al., 2021). 1). Also, being a simple feed-forward MLP-based model, TabM is significantly more efficient than some of the prior work. Compared to attention-base based models, TabM does not suffer from quadratic computational complexity w.r.t. the dataset dimensions. Compared to retrieval-based models, TabM is casily applicable to large datasets.
Improving tabular MLP-like models. Multiple recent studies achieved competitive performance with MLP-like architectures on tabular tasks by applying architectural modifications (Gorishniy et al., 2022), regularizations (Kadra et al., 2021; Jeffares et al., 2023a; Holzmüller et al., 2024), custom training techniques (Bahri al., 2021: Rubachev et al., 2022). Thus, it seems that tabular MLPs have good potential, but one has to deal with overfitting and optimization issues to reveal that potential. Our model TabM achieves high performance with MIP in a different way, namely, by using as the base backbone in a parameter-efficiee ensemble in the spirit of BatchEsnsemble (Wen et al., 2020). Our approach is orthogonal to the aforementioned training techniques and architectural advances.
Deep ensembles. In this paper, by deep ensemble, we imply multiple DL models of the same architecture trained independently (Jeffares et al., 2023b) for the same task under different random seeds (i.e. with different initializations, training batch sequences, etc.). The prediction of deep ensemble is the mean prediction of its members. Deep ensembles often significantly outperform single DI. models of the same architecture (Fort et al., 2020) and can excel in other tasks like uncertainty
2

---

Published as a conference paper at ICLR 2025
estimation dusto--distribution detection (Lakshminarayanan et al., 2017). It was observed that individual members of deep ensembles can learn to extract diverse information from the input, and the power of deep ensembles depends on this diversity (Allen-Zhu & Li, 2023). The main drawback of deep ensembles is the cost and inconvenience of training and using multiple models.
Parameter-efficiet deep "ensembles". To achieve the performance of deep ensembles at lower cost, multiple studies proposed architectures that imitate ensembles by producing multiple predictions with one model (Lee et al., 2015; Zhang al., 2020; Wen et al., 2020; Havasi et al., 2021; Antorán et al., 2020; Turkoglu et al., 2022). Such models can be viewed as "ensembles" where the implicit ensemble members share large amount of their weights. There are also non architectural approaches to efficient ensembling, e.g. FGE (Garipov et al., 2018). but we do not explore them, because we are interested specifically in architectural techniques. In this paper, we highlight parameter-efficien ensembling as an impactful paradigm for tabular DI.. In particular, we describe two simple variations of Batehensemble (Wen et al., 2020) that are highly effective for tabular ML.ss One variation uses a more efficient parametrization, and another one uses an improved initialization.
3 TABM
In this section, we present TabM - Tabular DL model that makes Multiple predictions.
3.1 PRELIMINARIES
Notation. We consider classification and regression tasks on tabular data. æ and y denote the features and a label, respectively, of one object from given dataset. A machine learning model takes x as input and produces ý as a prediction of y. N N and d E N respectively denote the "depth" (e.g. the number of blocks) and "width" (e.g. the size of the latent representation) of given neural network. dy E N is the output representation size (e.g. dy for regression tasks, and dy equals the number of classes for classification tasks).
Datasets. Our benchmark consists of 46 publicly available datasets used in prior work, including Grinsztajn et al. (2022); Gorishniy et al. (2024); Rubachev et al. (2024). The main properties of our benchmark are summarized in Table 1, and more details are provided in Appendix C.
Table 1: The overview of our benchmark. The "Split type" property explained in the text.
#Datasets
Train size
#Features
Task type
Split type
Min.
Q50
Mcan
Max.
Min.
Q50
Mean
Max.
#Regr.
#Classif.
Random
Domain-aware
46
1.8K
12K
76K
723K
20
108
986
28
18
37
9
Domain-aware splits. We pay extra attention to datasets with what we call "'dommin-wwar"" splits, including the eight datasets from the TabReD benchmark (Rubachev al., 2024) and the Microsoft dataset (Qin & Liu, 2013). For these datasets, their original real-world splits are available, e.g. time-aware splits in TabReD. Such datasets were shown be challenging for some methods because they naturally exhibit a certain degree of distribution shift between training and test parts (Rubachev et al., 2024). The random splits of the remaining 37 datasets are inherited from prior work.
Experiment setup. We use the setup from Gorishniy et al. (2024), and describe it in detail in subsection D.2. Most importantly, on each dataset, a given model undergoes hyperparameter tuning on the validation set, then the tuned model is trained from scratch under multiple random seeds, and the test metric averaged over the random seeds becomes the final score of the model on the dataset.
Metries. We use RMSE (the root mean square error) for regression tasks, and accuracy ROC-AUC for classification tasks depending on the dataset source. See subsection D.3 for details.
Also, throughout the paper, we often use the relative performance of models w.r.t. MI.P as the key
metric. This metric gives a unified perspective on all tasks and allows reasoning about the scale of improvements w.r.t. to simple bascline (MLP). Formally, a given dataset, the metric is defined score 1). 100%, where "score" is the metric of a given model, and "baseline" is the metric of MLP. In this computation, for regression tasks, we convert the raw metrics from RMSE to R² to better align the scales of classification and regression metrics.
as buseline
3

---

Published as a conference paper at ICLR 2025
3.2 A QUICK INTRODUCTION TO BATCHENSEMBLE.
For a given architecture, let's consider any linear layer I in it: (xx) = Wx + b, where E Rdi, W E R.dxd,, bb E Rdd, To simplify the notation, let di = d2 = d. In a traditional deep ensemble, the i-th member has its own set of weights W,, b, for this linear layer: 4(x,) W, + by, where r; is the object representation within the i-th member. By contrast, in BatchEnsemble, this linear layer is either (1) fully shared between all members, (2) mostly shared: (x(4)) = 8; @ (W @ xx)) bj, where @ is the elementwise multiplication, W E RRdd is shared between all members, and ri, 8,, b; E IR are not shared between the members. This is equivalent to defining the i-th weight matrix as W; - W o (s:r?), ). To ensure diversity of the ensemble members, r; and s; of all members are ini- tialized randomly with 11. All other layers are fully shared between the members of BatchEnsemble.
The described parametrization allows packing all ensemble members in one model that simultancously takes objects as input, and applies all implicit members in parallel, without explicitly materializing each member. This is achieved by replacing one or more linear layers of the original neural network with their BatchEnsemble versions: [ = ((X O R)W) @ S + B, where X E R stores object representations (one per member), and R, S, B E Rd store the non-shared weights (r,, 8,, b,) of the members, as shown at the lower left part of Figure
Terminology. In this paper, we call ri. 81, b; R. S and B adupters, and the implicit members of parameter-efficient emsembles (e.g. BatchEnsemble)) - -implicit submodels or simply submodels. Overhead to the model size. With BatchEnsemble, adding new ensemble member means adding only one row to each of the matrices R, S. and B, which results in 3d new parameters per layer. For typical values of d, this is a negligible overhead to the original layer size d² + d.
Overhead to the runtime. Thanks to the modern hardware, the large number of shared weights and the parallel execution of the lo forward passes, the runtime overhead of Ensembbe can be (significantly) lower than xk (Wen et al., 2020). Intuitively, if the original workload underutilizes the hardware, there are more chances pay less than xk overhead.
3.3 ARCHITECTURE
TabM is one model representing an ensemble of l MLPs. Contrary to conventional deep ensembles, in TabM, the & MLPs are trained in parallel and share most of their weights by default, which leads to better performance and efficiency. We present multiple variants of TabM that differ in their weight-sharing strategies, where TabM and TabMmini are the most effective variants, and TabMpacked is a conceptually important variant potentially useful in some cases. We obtain our models in several steps, starting from essential baselines. We always use the ensemble size k = 32 and analyze this hyperparameter in subsection 5.3. In subsection 1, we explain that using MLP the base model is crucial because of its excellent efficiency.
MLP. We define MLP as a sequence of N simple blocks followed by a linear prediction head: MLP(x)= = Linear(Block,)) (Block (Block (z))), where (1) = Dropout(ReLUg RRenarLnnear((2)))
MLPx* k = MIP + Deep Ensemble. We denote the traditional deep ensemble of k independently trained MLPs as MLPxk To clarify, this means tuning hyperparameters of one MLP, then inddpen- dently training l tuned MLPs under different random seeds, and then averaging their predictions. The performance of MLP** is reported in Figure 2. Notably, the results are already better and more stable than those of FT-Transformer (Gorishniy et al., 2021) - the popular attention-based baseline.
Although the described approach is somewhat default way to implement optimized for the task performance of the ensemble. First, for each of the & MLPs, the training is stopped based on the individual validation score, which is optimal for each individual MLP, but can be suboptimal for their ensemble. Second, the hyperparameters are also tuned for one MLP without knowing about the subsequent ensembling. All TabM variants are free from these issues.
ensemble, it is not
As the first step towards better and more efficient en- sembles of MLPs, we implement & MLPs as one large model using Tacked-Ensembll (Laurent et al., 2023). This results in TabMpacked illustrated in Figure 1. As an architecture, TabMpacked is equivalent to MLPx* and stores i independent MLPs without any weight sharing. However, the critical difference is that TabM processes inputs in parallel, which means that one training step of TabM consists of k parallel training steps of the individual MI.Ps. This allows monitoring the
Tab.Mpacked = MLP +
4

---

Published as a conference paper at ICLR 2025
performance of the ensemble during the training and stopping the training when it is optimal for the whole ensemble, not for individual MLPs. As a consequence, this also allows tuning hyperparameters for packd for model. As shown in Figure 2, TabMpackdd delivers significantly better performance compared MLPx* Efficiency-wise, for typical depth and width of MLPs, the runtime overhead of TabMp is noticeably less than x due to the parallel execution of the & forward passes on the modern hardware. Nevertheless, x overhead of TacMpacked to the model size motivates further exploration.
Tab. MLP + Batchensemble. To reduce the size of TTbbMackedd we now turn to weight sharing between the MIPs, and naively apply BatchEnsemble (Wen et al., 2020) instead of Packed- Ensemble, as described subsection 3.2. This gives us TabMa preliminary version TabM. In fact, the architecture (but not the initialization) of TabMuuvv is already equivalent to that of TabM, so Figure is applicable. Interestingly, Figure reports higher performance of Tabv compared to TabMpacked Thus, constraining the ensemble with weight sharing turns out to be highly effective regularization on tabular tasks. The alternatives to BatchEnsemble are discussed in subsection A.I.
Mean
Tost
Bxdxd
Shared
Not shared
Linear,
Train
oW,
+
ReLJU
Drop
Backbonee
Heads
X
Tabi
Packadk Euseemble
x
On
@W
+.0
RoLD
Drop
Or
@W
ReLU
Drop
@R
0.5
++B
XN
OR
XN
Tabx
TabMatat
Figure 1: (Upper left) A high-level illustration of TabM. One TabM represents an ensemble of & MLPs processing inputs parallel. The remaining parts of the figure are three different parametrizations of the & MLP backbones. (Upper right) TabMpacked consists of & fully independent MLPs. (Lower left) TabM is obtained by injecting three non-shared adapters R. S, B each of the N linear layers of MLP (" the initialization differs from Wen et al. (2020)). (Lower right) TabMania is obtained by keeping only the very first adapter R of TabM and removing the remaining 3N -1 adapters. (Details) Input transformations such as one-hot-ent mne-hoi-encoding feature embeddings (Gorishniy et al., 2022) are omitted for simplicity. Drop denotes dropout (Srivastava et al., 2014).
Relative improvement MLP
8%
MLP
6%
Altention
MLP (Oms)
4%
Meaa
2%
0%
-2%
MLP
FI-T
MLPx31
TahMppecked TabMasiee
TabMosa
TahMaini
TabM
TabM
TabM)
TabMiss
0000000
3.66 2.45
1.00
292244112
2202445
Figure 2: The performance of models described in subsection 3.3 on 46 datasets from Table I; plus several baselines on the left. For given model, one dot on jitter plot describes the performance score of the 46 datasets. The box plots describe the percentiles of the jitter plots: the boxes describe the 25th, 50th, and 75th percentiles, and the whiskers describe the 10th and 90th percentiles. Outliers clipped. The numbers the bottom are the mean and standard deviations over the jitter plots. For each model, hyperparameters tuned. "Model denotes ensemble of models.
5

---

Published as a conference paper at ICLR 2025
Tab.aain = MLP + MiniEnsemble. By construction, the just discussed TabMaaive (illustrated as "TabM" in Figure 1) has 3N adapters: R. S and B in each of the N blocks. Let's consider the very first adapter, i.c. the first adapter R in the first linear layer. Informally, its role can be described as mapping the l inputs living in the same representation space to k: different representation spaces before the tabular features this adapter is critical. First, we remove it from TabMnaive and keep the remaining 3/V - -1 adapters untouched, which gives us TabMaad with worse performance, as shown in Figure Then, we do the opposite: we keep only the very first adapter of aave and remove the remaining 3.V 1 adapters, which gives us TabMmini - the minimal version of TabM. TabMaini illustrated in Figure 1, where we call the described approach "MiniiEsemble". Figure 2 shows that TabM performs even slightly better than TabMm despite having only one adapter instead of 3N adapters.
mixed with @W for the first time. A simple experiment reveals that
TabM MLP + BatchEnsemble + Better initialization. The just obtained results motivate the next step. We go back to the architecture of TabM with all 3N adapters, but initialize all multiplicative adapters A and S, except for the very first one, deterministically with 1. As such, at initialization, the deterministically initialized adapters have no effect, and the model behaves like mabMmini but these adapters free add more expressivity during training. This gives TabM, illustrated in Figure 1. Figure shows that TabM is the best variation so far.
Hyperparameters. Compared MLP, the only new hyperparameter of TabM is k - the number of implicit submodels. We heuristically set k = 32 and do not tune this value. We analyze the influence of l: in subsection 5.3. We also share additional observations on the learning rate in subsection A.3.
Limitations and practical considerations are commented in subsection A.4.
3.4 IMPORTANT PRACTICAL MODIFICATIONS OF TABM
Shared training batches. Recall that the order of training objects usually varies between ensemble members, because of the random shuffling with different seeds. For TabM, in terms of Figure 1, that corresponds to X storing & different training objects { We observed that reusing the training batches between the TabM's 's submodels results in only minor performance loss average (depending on a dataset), as illustrated with TabM in Figure 2. practice, due to the simpler implementation and better efficiency, sharing training batches can be reasonable starting point.
t f~ Non-linear feature embeddings. In Figure 2, TabM mini denotes TabMaini with non-linear feature embeddings from (Gorishniy et al., 2022), which demonstrates the high utility of feature embeddings for TabM. Specifically, we embeddings (see subsection D.8 for details).
slightly modified version of the piecewise-
xN ~ Deep ensemble. In Figure 2, TabMix mini denotes an ensemble of five independent TabM models, showing that TabM itself can benefit from the conventional deep ensembling.
mini
3.5
SUMMARY
The story behind TabM shows that technical details of how to construct and train an ensemble have a major impact on task performance. Most importantly, we highlight simultaneous training of the (implicit) ensemble members and weight sharing between them. The former is responsible for the ensemble-aware stopping of the training, and the latter apparently serves as a form of regularization.
EVALUATING TABULAR DEEP LEARNING ARCHITECTURES
Now, we perform an empirical comparison of many tabular models, including TabM.
4.1 BASELINES
In the main text, we use the following baselines: MLP (defined in subsection 3.3), FT-Transformer denoted as "FT-T" (the attention-bused model from Gorishniy et al. (2021)), SAINT (the attention- and retrieval-based model from Somepalli et al. (2021)), T2G-Former denoted as "T2G" (the attention- based model from Yan et al. (2023)), ExcelFormer denoted as "Excel" (the cicettio-bbsse model from Chen et al. (2023a)), TabR (the retrieval-based model from Gorishniy al. (2024)), MoA

---

Published as a conference paper at ICLR 2025
denoted "MNCA" (the retrieval-baseel model from Ye et al. (2024)) and GBDT, including XGBoost (Chen & Guestrin, 2016), LightGBM (Ke et al., 2017) and CatBoost (Prokhorenkova et al., 2018).
The models with non-linear feature embeddings from Gorishniy al. (2022) are marked with f or depending on the embedding type (see subsection D.8 for details on feature embeddings):
MLP! and TabM mini use a modified version of the piecewise-linear embeddings.
TabR', MNCA*, and MLP* (also known as MLP-PLR) use various periodic embeddings.
More baselines are evaluated in Appendix B. Implementation details are provided in Appendix D.
4.2 TASK PERFORMANCE
We evaluate all models following the protocol announced in subsection 3.1 and report the results in Figure 3 (see also the critical difference diagram in Figure 9). We make the following observations:
1. The performance ranks render TabM as the top-tier DL model.
2. The middle and right parts of Figure provide fresh perspective on the per-dataset metrics. TabM holds leadership among the DL models. Meanwhile, many DL methods turn out to be no better or even worse than MLP on non-negligible number of datasets, which shows them as less reliable solutions, and changes the ranking, especially on the domain-aware splits (right).
3. One important characteristic of model is the weakest part of its performance profile (c.g. the 10th or 25th percentiles in the middle plot) since it shows how reliable the model is on "inconvenient" datasets. From that perspective, MLPf seems be decent practical option between the plain MLP and TabM, especially given its simplicity and efficiency compared to retrieval-based alternatives, such as TabR and MCdernNCA.
Summary. TabM confidently demonstrates the best performance among tabular DL models, and can serve reliable go-to DI. baseline. This is not the case for attention- and retrieval-based models. Overall, MLP-like models, including TabM, form a representative set of tabular DL baselines.
Preformation ranlss datress
Performatiio soures
Pa Reformatcc scoess
dataurte wodeon
fatuwee doralt
Monn
MLP-
Ex
TabR
Excel
5.2 2.
MLP
SAINT
SAINT
SAINT
Excd"
(00
FI-T
2.
FI-T
TabR
5 00
T2G
TabR
MNCA
MNCA
2.0
T2G
MLP
TabR
8.
MLPP
FT-T
MLP
2.
MNCA
MNCAF
GBM
34 2:0
LgliGMM
T2G
XGooo
TebRt
CatBoost
Cooso
XGBosst
MLP1
MNCAR
TubM
LightBMM
Tohlt
232 2
CalBoost
TibM
TabM
2.2
MNCAF
XGBood
TalMd
1.
Butanval
TubM
TubM!
miai
-25
055
21%
6.55
6.%
0%
10%
Baak
Jaaatio
ME.1"
Utdattio
Figure 3: The task performance tabular models on the 46 datasets from Table 1. (Left) The mean and standard deviations of the performance ranks over all datasets summarize the head-to-head comparison between the models on all datasets. (Middle & Right) The relative performance w.r.t. the plain multilayer perceptron (MLP) allows reasoning about the scale and consistency of improvements over this simple baseline. One dot of a jitter plot corresponds to the performance of a model on one of the 46 datasets. The box plots visualize the 10th, 25th, 50th, 75th, and 90th percentiles of the jitter plots. Outliers are clipped. The separation in random and domain- aware dataset splits is explained in subsection 3.1. (*Evaluated under the common protocol without data augmentations)
7

---

Published as a conference paper at ICLR 2025
4.3 EFFICIENCY
Now, we evaluate tabular models in terms of training and inference efficiency, which becomes serious reality check for some of the methods. We benchmark exactly those hyperparameter configurations of models that are presented in Figure 3 (see subsection B.3 for the motivation).
TabM mini & TabM mini" Additionally, in this section, we mark with the asterisk (*) the versions of TabM enhanced with two Efficiency-relatee plugins available out-of-the-box in PyTorch (Paszke et al., 2019): the automatic mixed precision (AMP) and torch. compil. ile (Ansel et al., 2024). The purpose of those TabM variants to showcase the potential of the modern hardware and software for a powerful tabular DI. model, and they should not be directly compared to other DL models. However, the implementation simplicity of TabM plays an important role, because it facilitates the seamless integration of the aforementioned Py plugins.
Training time. We focus on training times on larger datasets, because on small datasets, all methods become almost equally affordable, regardless of the formal relative difference. Nevertheless, in Figure 10, we provide measurements on small datasets as well. The left side of Figure reveals that TabM offers practical training times. By contrast, the long training times of attention- and retrieval-based models become one more limitation of these methods.
Inference throughput. The right side of Figure 4 tells essentially the same story the left side. In subsection B.3, we also report the inference throughput on GPU with large batch sizes.
Applicability to large datasets. In Table 2, we report metries on two large datasets. As expected, attention- and retrieval-base models struggle, yielding extremely long training times, or being simply inapplicable without additional effort. See subsection D.4 for implementation details.
Parameter count, Most tabular networks are overall compact. This, in particular, applies to TabM, because its size is by design comparable MLP. We report model sizes in subsection B.3.
Summary. Simple MLPs are the fastest DL models, with TabM being the runner-up. The attention- and bebsee models are significantly slower. Overall, MLP-like models, including TabM, form a representative set of practical and accessible tabular DL baselines.
Training time dattaeets with 10EK objects NVITAA
Inference tharoughppt with batch size 1 Device CPU lunel 7-7800N, siegle threed
ML.P
MLF GEIFT
MLP
ore 0o
Reteesal
Auarika
XGBooet
dog
XGBoost
MLP (Ches)
MLP
90%
o
TabM
Mean
TabM
TabM
TabM!
TabM
Taha
Tabi
FT-T-co
orga
MNCA
T20- T2G-a
MNCA2
Tabk
TabR
MNCA
MNCA 1
PT-T
SAINT
SAINT
Q
10s
2m
10mo
1030
2000
3000
1000
5000
6000
Time
Objjeta per
Figure 4: Training times (left) and inference throughput (right) of the models from Figure 3. One dot represents a measurement on one dataset. TabM mini is the optimized TabM mini (see subsection 4.3).
Table 2: RMSE (upper rows) and training times (lower rows) on two large datasets. The best values are in bold. The meaning of model colors follows Figure 3.
#Objects
#Features
XGBoost
MLP
TabM mini
TabM mini
FT-T
TabR
0.1601
0.1592
0.1583
0.1582
0.1594
Maps Routing
6.5M
986
OOM
28m
15m
2h
13.5h
45.5h
1.4234
1.4842
1.4090
1.4122
1.4409
Weather
13M
103
00M
10m
15m
1.3h
3.3h
13.5h
8

---

Published as a conference paper at ICLR 2025
5 ANALYSIS
5.1 PERFORMANCE AND TRAINING DYNAMICS OF THE INDIVIDUAL SUBMODELS
Recall that the prediction of TabM defined as the mean prediction of its implicit submodels that share most of their weights. In this section, we take a closer look at these submodels.
For the next experiment, we intentionally simplify the setup as described in detail in subsection D.5. Most importantly, all models have the same depth 3 and width 512, and are trained without early stopping, i.e. the training goes beyond the optimal epochs. We use TabM from Figure with 32 denoted as Ta We use TabM mini (i.e. essentially one plain MLP) as a natural baseline for the submodels of k=12 because each the 32 submodels has the architecture of TabM mini
mini
mini
We visualize the training profiles on four diverse datasets (two classification and two regression problems of different sizes) in Figure 5. As a reminder, the mean the & individual losses is what explicitly optimized during the training of TabMaini. the loss of the collective mean prediction corresponds to how TabMmini makes predictions on inference, and TabM 1 just baseline.
mini
Clarrn
House
Oua
Microsuft
(3)
6.4
10
1.0
6.2-
0.6
6.0
150
500
300
303
310
600
200
200
Li0)
200
Fpech
Fjarch
Rouch
Figrid
1.2
12
0.5r
Tost Look
1.0
1.0
0.5
02 00
85
0.2
Tisin
Figure 5: The training profiles of TabM mini and TabM as described in subsection 5.1. (Upper) The training curves. = [] represents the mean individual loss over the 32 submodels. (Lower) Same as the first row, but in the train-test coordinates: each dot represents some epoch from the first row, and the training generally goes from left to right. This allows reasoning about overfitting by comparing test loss values for given train loss value.
32
In the upper row of Figure 5, the collective mean prediction of the submodels is superior to their individual predictions in terms of both training and test losses. After the initial epochs, the training loss of the baseline MLP is lower than that of the collective and individual predictions.
In the lower row of Figure 5, we see a stark contrast between the individual and collective performance of the submodels. Compared to the baseline MLP, the submodels look overfitted individually, while their collective prediction exhibits substantially better generalization. This result is strict evidence of non-trivial diversity of the submodels: without that, their collective test performance would be similar to their individual test performance. Additionally, we report the performance of the Best submodel of TabM across many datasets under the name TabM B in Figure 6. As such, individually, even the best submodel of TabM is no better than a simple MI.P
Summary. TabM draws its power from the collective prediction of weak, but diverse submodels.
5.2 SELECTING SUBMODELS AFTER TRAINING
The design of TabM allows selecting only subset of submodels after training based on any criteria, simply by pruning extra prediction heads and the corresponding rows of the adapter matrices. To showcase this mechanics, after the training, the best collective performance on the validation set, and denote this "pruned" TabM as TabM[G]. The performance reported in Figure 6 shows that TabM[G] is slightly behind the vanilla TabM. On average over 46 datasets, the greedy submodel selection results in 8.8 + 6.6 submodels out of the initial k- 32, which can result in faster inference. See subsection D.6 for implementation details.
Greedily construct a subset of TabM's submodels with

---

Published as a conference paper at ICLR 2025
Relative inprovement over MLP
8%
MLP
1.0%
6%
MLP [Oun)
Rellution imprevvment
4%
0.0%
2%
-1.0%%
0%
-2.0%
-2%
MLP CALE
TabM 233
TabM B
TabM[G 2.2 2.05
4
52
128
-306-188
Figure 6: The performance on the 46 datasets from Table 1. TB and TabM [G] are de- scribed in subsection 5.1 and subsection 5.2.
Figure 7: The average performance of TabM with layers of the width d across 17 datasets as a function k.
5.3 HOw DOES THE PERFORMANCE OF TABM DEPEND ON k-?
To answer the question the title, we consider TabM with n layers of the size and different values of k. and report the average performance over multiple datasets in Figure 7 (the implementation details are provided subsection D.7). The solid curves correspond to n = 3, and the dark green correspond to d 512. Our main observations TabM is (i.e. when n and d increase), the more submodels it can accommodate effectively. For example, note how the solid curves corresponding different d diverge too high values of & can be detrimental. Perhaps, weight sharing limits the number of submodels that can productively "coexist" in one network, despite the presence of non- noarsh adapters. Third, too narrow (d = 64) or too shallow (n 1) configurations of TabM can lead to suboptimal performance, least the scope of middle-to-large datasets considered this work.
follows. First, it seems that the "larger"
and 4. Second,
5.4 PARAMETER-EFFICHEE ENSEMBLING REDUCES THE NUMBER OF DEAD NEURONS
Here, show empirically that the design of TabM naturally leads to higher utilization of the backbone's weights. Even without technical definitions, this sounds intuitive, since TabM has to implement & (diverse) computations using the amount of weights close to that of one MI.P
Let's consider TabM as illustrated in Figure 1. By design, each the shared neurons of TabM used times per forward pass, where "neuron" refers the combination of the linear transformation and the subsequent nonlinearity (e.g. ReLU). By contrast, in plain MLP (or in TabM with k 1), each neuron is used only once per forward pass. Thus, technically, a neuron TabMmini has more chances to be activated, which overall may lead to lower portion of dead neurons in TabMmiai compared MLP (a dead neuron is a neuron that never activates, and thus has no impact on the prediction). Using the experiment setup from subsection 5.1, we compute the portion of dead neurons TabM using its best validation checkpoint. On average across 46 datasets, for k: k = 32, get 0.29 + 0.17 and 0.14 + 0.09 portion of dead neurons, respectively, which is in line with the described intuition. Technically, on a given dataset, this metric is computed as the percentage of neurons that never activate fixed set of 2048 training objects.
and
CONCLUSION & FUTURE WORK
In this work, have demonstrated that tabular multilayer perceptrons (MLPs) greatly benefit from parameter- efficceet ensembling. Using this insight, we have developed TabM - simple MLP. based model with state-of-the-art performance. In large-scale comparison with many tabular DL models, we have demonstrated that TabM is ready to serve as a new powerful and efficient tabular DL. baseline. Along the way, we highlighted the important technical details behind TabM and discussed the individual performance of the implicit submodels underlying TabM.
One idea for future work is bring the power of (parameter-)eeficceent ensembles to other, non-tabular, domains with optimization-related challenges and, ideally, lightweight base models. Another idea is evaluate TabM for uncertainty estimation and out-of-distribution (OOD) detection on tabular data, which is inspired by works like Lakshminarayanan et al. (2017).
10

---

Published as a conference paper at ICLR 2025
Reproducibility statement. The code is provided in the following repository: link. It contains the implementation of TabM, hyperparameter tuning scripts, evaluation scripts, configuration files with hyperparameters (the TOML files in the exp/ directory), and the report files with the main metrics (the ISON files in the exp/ directory). In the paper, the model is described in section 3, and the implementation details provided in Appendix D.
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In KDD, 2019. 18
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. In ICLR, 2023. 3
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky,
Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary Vito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, C. K. Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Shunting Zhang, Michael Suo, Phil Tillet, Xu Zhao, Eikan Wang. Keren Zhou, Richard Zou, Xiaodong Wang, Ajit Mathews, William Wen, Gregory Chanan, Peng Wu, and Soumith Chintala. Pytorch Faster machine learning through dynamic python bytecode transformation and graph compilation. In ASPLOS, 2024. 8
Javier Antorán, James Urquhart Allingham, and José Miguel Hermández-Lobato. Depth uncertainty in neural networks. In NeurIPS, 2020. 3
Serean O. Arik and Tomas Pfister. 1908.07442v5, 2020. 2
TabNet: Attentive interpretable tabular learning.
arXiv,
Sarkhan Badirli, Xuanqing Liu, Zhengming Xing, Avradeep Bhowmik, Khoa Doan, and Sathiya Keerthi. Gradient boosting neural networks: GrowNet. arXiv, 2002.0797lv2, 2020. 2
Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. SCARF: Self-supervised contrastive learning using random feature corruption. In ICLR, 2021.
Jintai Chen, Jiahuan Yan, Danny Ziyi Chen, and Jian Wu. ExcelFormer: neural network surpassing gbdts on tabular data. arXiv, 2301.02819v1. 2023a. 6, 20, 24, 25
Kuan- Yu Chen, Ping-Han Chiang, Hsin-Rung Chou, Ting- Ting-Wei Wei Chen, and Tien-Hao Chang. Trompt: Towards a better deep neural network for tabular data. In ICML, 2023b. 1, 2, 15
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In SIGKDD, 2016. 1, 7
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspec- tive. arXiv, 1912.02757v2, 2020.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin. Dmitry P. Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In NeurlPS, 2018.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. NewrlPS, 2021. 2, 4, 15, 23, 25
Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. In NewrlPS, 2022. 2, 5, 6, 7, 14, 15, 20, 21, 23
Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. TabR: Tabular deep learning meets nearest neighbors. In ICLR, 2024. 1, 2, 3, 6, 17, 18, 19, 20, 22, 23, 24, 25
11

---

Published as a conference paper at ICLR 2025
Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In NeurIPS, the "Datasets and Benchmarks' track, 2022. 3, 17, 18, 23, 24, 29
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshmi- narayanan, Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In ICLR, 2021. 3, 14
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: transformer that solves small tabular classification problems in a second. In ICLR, 2023. 1, 2, 15
David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned mlps and boosted trees on tabular data. arXiv, 2407 1407.0449lv1, 2024.
Alan Jeffares, Tennison Liu, Jonathan Crabbé, Fergus Imrie, and Mihaela van der Schaar. TANGOS: Regularizing tabular neural networks through gradient orthogonalization and specialization. In ICI.R, 2023a. 2
Alan Jeffares, Tennison Liu, Jonathan Crabbé, and Mihaela van der Schaar. Joint training of deep ensembles fails due to learner collusion. In NeurIPS, 2023b.
Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. In NeurIPS, 2021. 2
Guolin Kc, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tin-Yan Liu. LightGBM: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 50:3146-31543 2017. 2,
Myung Jun Kim, Léo Grinsztajn, and Gaël Varoquaux. CARTE: pretraining and transfer for tabular learning, arXiv, abs/2402..6675VV 2024. 16
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. NIPS, 2017. 2, 15
Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Tom Rainforth, and Yarin Gal. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. In NeurIPS,
2021. 2
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, 2017. 3, 10, 14
Olivier Laurent, Adrien Lafage, Enzo Tartaglione, Geoffrey Daniel, Jean-Mare Martinez, Andrei Bursuc, and Gianni Franchi. Packed ensembles for efficient uncertainty estimation. In ICLR, 2023,
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why M heads are better than one: Training diverse ensemble of deep networks, arXiv, abs/1511.06314 14, 2015. 3, 14
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 18
Sascha Marton, Stefan Lüdtke, Christian Bartelt, and Heiner Stuckenschmidt, GRANDE: Gradient- based decision tree ensembles for tabular data. In ICLR, 2024. 2
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Z. Yang, Zachary Detito, Martin Raison. Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high- performance deep learning library. NeurIPS, 2019.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Couraapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-288 28330, 2011. 18
12

---

Published as a conference paper at ICLR 2025
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In ICLR, 2020. 2
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey
Gulin. CatBoost: unbiased boosting with categorical features. In NeurIPS, 2018. 1,
Tao Qin and Tie- Yan Liu. Introducing LETOR 4.0 datasets. arXiv, 1306.2597v1, 2013. 3
Ivan Rubachev, Artem Alekberov, Yury Gorishniy, and Artem Babenko. Revisiting pretraining objectives for tabular deep learning. arXiv, 2207.03208v 2022. 2
Ivan Rubachev, Nikolay Kartashev, Yury Gorishniy, and Artem Babenko. TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks. arXiv, 2406. 18066193800 19380v4, 2024. 1, 3, 17, 18, 19, 20, 22, 23, 25, 36
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. SAINT: improved neural networks for tabular data via row attention and contrastive pre-training. arXiv, 2106.01342v1, 2021. 2, 6, 25
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In CIKM, 2019.
2, 15, 26
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15()):1229-155, 2014. 5
Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mip architecture for vision. In NeurlPS, 2021. 15
Mehmet Ozgur Turkoglu, Alexander Becker, Hüseyin Anil Gündüz, Mina Rezaei, Bernd Bischl, Rodrigo Caye Daudt, Stefano D' Aronco, Jan D. Wegner, and Konrad Schindler. Film-ensemble: Probabilistic deep learning via feature-wise linear modulation. In NeurIPS 2022, 2022. 3, 14, 15
Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed H. Chi. Den v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. arXiv, 2008. 13535v2, 2020. 2, 15
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In ICLR, 2020. 2. 4. 5, 14. 15
Jiahuan Yan, Jintai Chen, Yixuan Wu, Danny Z. Chen, and Jian Wu. T2G-FORMER: organizing
tabular features into relation graphs promotes heterogeneous feature interaction. In AAAI, 2023. 2, 6, 23, 24
Han-Jia Ye, Huai-Hong Yin, and De- Ch-Chunn Zhan. Modern neighborhood components analysis: deep tabular baseline two decades later. arXiv, 2407 .000700355vl. 2024. 2, 7, 20, 23
Shaofeng Zhang, Meng Liu, and Junchi Yan. The diversified ensemble neural network. In NewrIPS.
2020. 3
13

---

Published as a conference paper at ICLR 2025
A ADDITIONAL DISCUSSION ON TABM
A.1 MOTIVATION
Why BatchEnsemble? Among relatively case-to-use "efficient ensembling" methods, beyond BatchEnsemble, there are examples such as dropout ensembles (Lakshminarayanall et al., 2017), naive multi-head architectures, TreeNet (Lee et al., 2015). However, in the literature, they were consistently outperformed by more advanced methods, including BatchEnsemble (Wen et al., 2020), MIMO (Havasi et al., 2021), FiLM-Ensemble (Turkoglu et al., 2022).
Among advanced methods, BatchEnsemble seems to be one of the simplest and most flexible options. For example, FiLM--nnemblee (Turkoglu et al., 2022) requires normalization layers to be presented in the original architecture, which is not always the case for tabular MLPs. MIMO (Havasi et al., 2021), in turn, imposes additional limitations compared to Batchbns First, requires concatenating (not stacking, as with Batchensemble) all k input representations, which increases the input size of the first linear layer. With the relatively high number of submodels k = 32 used in our paper, this can be issue on datasets with large number of features, especially when feature embeddings (Gorishniy et al., 2022) are used. For example, for k = 32, the number features m = 1000, and the feature embedding size = 32, the input size approaches first linear layer of MLP. Second, with BatchEnsemble, it is easy to explicitly materialize, analyze, and prune individual submodels. By contrast, in MIMO, all submodels are implicitly entangled within one MLP, and there is no easy way to access individual submodels.
million resulting in extremely large
Why MLPs? Despite the applicability of BatchEnsemble (Wen al., 2020) to almost any architecture, we focus specifically on MLPs. The key reason is efficiency. First, to achieve high performance, throughout the paper, we use the relatively large number of submodels k 32. However, the desired less-than- x runtime overhead of BatchEnsemble typically happens only when the original model underutilizes the power of parallel computations of given hardware. This will not be the case for sieention-basedd models on datasets with a large number of features, as well as for retrieval based models on datasets with a large number of objects. Second, as we show in subsection 4.3, attention- and retrieval-bann -ased models are already slow as-is. By contrast, MLPs are exceptionally efficient, to the extent that slowing them down even by an order of magnitude will still result in practical models.
Also, generally speaking, the definition of MLP suggested in subsection 3.3 and used in TabM is not special, and more advanced MLP-like backbones can be used. However, in preliminary experiments, we did not observe the benefits of more advanced backbones. Perhaps, small technical differences between backbones become less impactful in the context of parameter-efficieet ensembling, at least in the scope of hiddle-to-large-ee
datasets.
A.2 TABM WITH FEATURE EMBEDDINGS
Notation. In this paper, we use t to mark TabM variants with the piecewise-ling embeddings (e.g. TabM mini TabM etc.).
Implementation details. In fact, there are no changes in the usage of feature embeddings compared to plain MLPs: feature embeddings are applied, and the result is flattened, before being passed to the backbones in terms of Figure For example, if dataset has m continuous features and all of them are embedded, the very first adapter R will have the shape k x mdd, where de is the feature embedding size. For TabM mini and TabM', we initialize the first multiplicative adapter R of the first linear layer from the standard normal distribution (0, 1). The remaining details are best understood from the source code.
Efficiency. When feature embeddings are used, the simplified batching strategy from subsec- tion 3.4 allows for more efficient implementation, when the feature embeddings are applied to the original bat ch size size objects, and the result is simply cloned k times (compared to embedding k x ba batch... a i ze objects with the original batching strategy).
A.3 HYPERPARAMETERS
We noticed that the typical optimal learning rate for TabM is higher than for MLP (note that, on each dataset, the batch size is the same for all DI. models). We hypothesize that the reason is the
14

---

Published as a conference paper at ICLR 2025
effectively larger batch size for TabM because of how the training batches constructed (even if the simplified batching strategy from subsection 3,4 is used).
A.4 LIMITATIONS AND PRACTICAL CONSIDERATIONS
TabM does introduce any new limitations compared to BatchEnsemble (Wen al., 2020).
Nevertheless, we note the following:
The MLP backbone used TabM is one of the simplest possible, and generally, more advanced backbones can be used. That said, some backbones may require additional care when used in TabM. For example, we did not explore backbones with normalization layers. For such layers, is possible to allocate non-shared trainable affine transformations for each implicit submodel by adding one multiplicative and one additive adapter after the normalization layer (i.e. like in FiL.M-Ensebble (Turkoglu et al., 2022)). Additional experiments are required to find the best strategy.
For ensemble-like like models, such TabM, the notion of "the final object embedding" changes: now, is not a single vector, but a set of & vectors. exactly one object embedding is required, then additional experiments may be needed find the best way combine embeddings into one. The presence of multiple object embeddings can also be important for scenarios when TabM is used for solving more than one task, in particular when pretrained as a generic feature extractor and then reused for other tasks. The main practical guideline is that the prediction branches should not interact with each other (e.g. through attention, pooling, etc.) and should always be trained separately.
B EXTENDED RESULTS
This section complements section 4.
B. ADDITIONAL BASELINES
In addition to the models from subsection 4.1, we consider the following baselines:
MLP-PLR Gorishniy et al. (2022), that is, an MLP with periodic embeddings.
ResNet (Gorishniy et al., 2021)
SNN (Klambauer al., 2017)
DCNv2 (Wang et al., 2020)
AutoInt (Song al., 2019)
MLP-Mixer is our adaptation of Tolstikhin et al. (2021) for tabular data.
Trompt (Chen et al., 2023b) (our reimplementation, since there is official implementation)
We also evaluated TabPFN (Hollmann et al., 2023), where possible. The results for this model are available only in Appendix because this model is by design not applicable regression tasks, which is a considerable number of our datasets. Overall, TabPFN specializes in small datasets, In line with that, the performance of TabPFN on our benchmark was not competitive.
B.2 TASK PERFORMANCE
Figure is different version of Figure 3 with additional basclines. Overall, none of the additional baselines affect our main story.
Figure is the critical difference diagram (CDD) computed over exactly the same results that were used for building Figure
15

---

Published as a conference paper at ICLR 2025
Performance
Performance sconos
Performanc scones
Os 27 duuedd wilh rediia 194
Ou Si dataado vala cestess splt
Ox. deteeii with Oesssreeere spla
Secoad by tae casin nass
Seetid by tan ranes soer
Sceted by tan rasn scer
Mau
DON2
TXN2
TabR
SNN
SNN
SAINT
MLP
Autolut
DCN2
Autolat
Exod"*
SNN
RmsNtt
MLP
Autolut
Exoek
3.334
BesNer
MLP-MMisr
MLP-X Maxer
493
SAINT
Excel
SAINT
FI-T
Tabit
FT-T
18:87
TabR
MNCA
Tomps
3.6
MLP-Miser Miser
Trompt
T2G
Trompr
MLP
T2G
ResNet
MLP
MLP
FT-T
M
MLP
MNCA
MLP
55.4 24
MNCA
T2G
GBM
Ligh
Catloost
XGBooot
TabR
MLP
XCBonst
MLPi
25
TabM
LightGBM
MNCAF
MLR
Cat Boust
TuhM-
ToRR-
201
Bensooa
MNCA
XGBoost
TabM
1.6
MEP
TabM
TabM
Figure 8: An extended comparison of tabular models as in Figure 3. Note that the ranks (left) are computed only over the 37 datasets with random splits because ResNet, Autolnt, and MLP-Mixer were evaluated only on one 1 out of 9 datasets with domain-aware splits.
10
12
TabM
MLP
TabM
Escel!
TabM
SAINT
TabRt
FT-T
CarRoosr
T2G
MNCA
'[xbit
XGDoost
MNCA
LightGBM
Figure 9: Critical difference diagram. The computation method is taken from the Kim et al. (2024).
B.3 EFFICIENCY
This section complements subsection 4.3.
Additional results.
Figure 10 complements Figure by providing the training times on smaller datasets and the inference throughput on GPU with large batch sizes.
Table 3 provide the number of trainable parameters for some of the models from Figure 3.
Motivation for the benchmark setup. Comparing models under all possible kinds of budgets (task performance, the number of parameters, training time, etc.) on all possible hardware (GPU, CPU, etc.) with all possible batch sizes is rather infeasible. As such, we set a narrow goal of providing high-level intuition on the efficiency in a transparent setting. Thus, benchmarking the transparently obtained tuned hyperparameter configurations works well for our goal. Yet, this choice also has a limitation: the hyperparameter tuning process is not aware the efficiency budget, so it can prefer much heavier configurations even if they lead to tiny performance improvements, which will negatively affect efficiency without a good reason. Overall, we hope that the large number of datasets compensates for potentially imperfect per-dataset measurements.
16

---

Published as a conference paper at ICLR 2025
Motivation for the two setups for measuring inference throughput.
The setup on the right side of Figure 4 simulates the online per-object predictions. The setup the right side of Figure 10 simulates the offline batched computations.
Training
with 100KK objects
Inverence thronghiuut with maxiuun batch size Besier GPU NVIDIA 205071
Deetarr GPU NVIII AIM
XGDoost
Peatt (1
MLF. GRarT
Amarioo
Reteosal
MLP
MTP (Ohes)
XGBonet
MNCA
Meas
TabM
MNCA
MNCA
ee Quatp ozp
MNCA
TabM
60 e (oqgho
FT-T
TabB
c00g) cQ 4
TahM
TabR'
TabR-
FT-T
o o 2 00n
TabR
Abpeous?
T2G
o. 0
T2G
8gll 0
SAINT
SAINT
10s
2m
15m
1h
3
Object per socond (1)
Figure 10: (Leff) Training time on datasets with less than 100K objects. (Right) Inference throughput on GPU with maximum possible batch size (i.e. the batch size depends on a model).
Table 3: Mean number of parameters with std. dev, for 7 different tuned models across all 46 datasets.
TabM
MLP
FF-T
T2G
TabR
ModeraNCA
SAINT 175.4M 565.4M
1.4M 1.3
1.0M # 1.0
1.2M = 1.2M
2.1M # 166
858K + 1.4M
1.0M + 1.1M
C DATASETS
In total, we use 46 datasets:
1. 38 datasets are taken from Gorishniy et al. (2024), which includes:
(a) 28 datasets from Grinsztajn et al. (2022). See the original paper for the precise dataset information.
(b) 10 datasets from other sources. Their properties are provided in Table 4.
2. 8 datasets from the TabReD benchmark (Rubachev al., 2024). Their properties are provided in Table 5.
In fact, the aforementioned 38 datasets from Gorishniy al. (2024) is only subset of the datasets used in Gorishniy et al. (2024), Namely, we did not include the following of the remaining datasets:
The datasets that, according to Rubachev et al. (2024), have incorrect splits and/or label leakage, including: SGEMM.GPU Bet.GGGPUeeeeeeeeeeeeeeee perfoaneerf sulfur, visualizing -soil, and the weather forecast- ing dataset (it is replaced by the correct weather forecasting dataset from TabReD (Rubachev et al., 2024)).
Bike-Sharing Deraaid,gDemand,, compass, electricity,
rl from (Grinsztajn et al., 2022). We observed abnormal results on these datasets. This is an anonymous dataset, which made the investigation impossible, so we removed this dataset to avoid confusion.
yprop_4.1 from (Grinsztajn et al., 2022). Strictly speaking, this dataset was omitted due to mistake side. For future work, note that the typical performance gaps this dataset have low absolute values in terms of RMSE. Perhaps, R² may a more appropriate metric for this dataset.
17

---

Published as a conference paper at ICLR 2025
Table Properties of those datasets from Gorishniy et al. (2024) that are not part of Grinsztajn et al. (2022) or TabReD Rubachev et al. (2024). "# Num", "# Bin", and "# Cat" denote the number of numerical, binary, and categorical features, respectively. The table taken from (Gorishniy et al., 2024).
Name
Train
Validation
# Test
Num
Bin Cat
Task type
Batch size
Churn Modelling
6400
1600
20000
3
Binclass 128
California Housing
13 209
3300
4128
Regression 256
House 16H
14 581
3646
4557
16
0
Regression 256
Adult
26 048
6513
16281
6
1
8
Binclass 256
Diamond
34 521
8631
10
6
0
Regression 512
Otto Group Products
39 601
9901
12376
93
0
Multiclass 512
Higgs Small
62 751
15 15688
19610
0
Binclass 512
Black Friday
106 764
26
33365
1
Regression 512
Covertype
371 8/17
92262
116.203
10
4
Multiclass 1024
Microsoft
723
235 259
241 521
131
5
0
Regression 1024
Table Properties of the datasets from the TabReD benchmark (Rubachev et al., 2024). "# Num", *# Bin",, and "# Cat" denote the number of numerical, binary, and categorical features, respectively,
Name
Train
# Validation
Test # Num
Bin # Cat
Task type Batch size
Sberbank Housing
18 847
827
4 647
365
17
Regression 256
Ecom Offers
109
24261
26 455
Binclass 1024
Maps Routing
160
59975
59 951
984
0
2
Regression 1024
Homesite Insurance
224320
20 20138
16 295
253
23
23
Binclass 1024
Cooking Time
227 087
51 251
41 648
186
3
3
Regression 1024
Homecredit Default
267 645
58018
56 001
612
2
82
Binclass 1024
Delivery ETA
479415
34174
36 927
Regression 1024
Weather
340596
42359
40 840
100
3
Regression 1024
D IMPLEMENTATION DETAILS
D.1 HARDWARE
Most of the experiments were conducted on a single NVIDIA A100 GPU. rare exceptions, we used machine with single NVIDIA 2080 Ti GPU and Intel(R) Core(TM) i7-7800X CPU @ 3.50GHz.
D.2 EXPERIMENT SETUP
We mostly follow the experiment setup from Gorishniy et al. (2024). As such, some of the text below copied from (Gorishniy al., 2024).
Data preprocessing, For each dataset, for all DL-based solutions, the same preprocessing was used for fair comparison. For numerical features, by default, we used a slightly modified version of the quantile normalization from the Scikit-learn package (Pedregosa et al., 2011) (see the source code), with rare exceptions when it turned out to be detrimental (for such datasets, we used the standard normalization or no normalization). For categorical features, we used one-hot encoding. Binary features (i.e. the ones that take only two distinct values) are mapped to {0, 1} without any further preprocessing. We completely follow Rubachev et al. (2024) on Table 5 datasets.
Training neural networks. For DL-based algorithms, we minimize cross-entropy for classification problems and mean squared error for regression problems. We use the AdamW optimizer (Loshchilov Hutter, 2019). We do not apply lcarning rate schedules. We do not use data augmentations. We apply global gradient clipping 1.0. For each dataset, used predefined dataset- batch size. We continue training until there are pat ience consecutive epochs without improvements on the validation set; patiencs
16 for the DL models.
Hyperparameter tuning. In most cases, hyperparameter tuning is performed with the TPE sampler (typically, 50-100 iterations) from the Optuna package (Akiba al., 2019). Hyperparameter tuning
18

---

Published as a conference paper at ICLR 2025
spaces for most models are provided in individual sections below (example for TabM: subsection D.9). We follow Rubachev et al. (2024) and use 25 iterations on some datasets from Table
Evaluation. On given dataset, for given model, the tuned hyperparameters are evaluated under multiple (in most cases, 15) random seeds. The mean test metric and its standard deviation over these random seeds are then used to compare algorithms described in subsection D.3.
D.3 METRICS
We use Root Mean Squared Error for regression tasks, ROC-AUC for classification datasets from Table 5 (following Rubachev et al. (2024)), and accuracy for the rest of datasets (following Gorishniy et al. (2024)). We also tried computing ROC-AUC for all classification datasets, but did not observe any significant changes (see Figure 1 1), and its standard deviation are obtained by training a given model with tuned hyperparameters from scratch on given dataset under 15 different random seeds.
stuck to prior work. By default, the mean test score
How we compute ranks. Our method of computing ranks used in Figure does not count small improvements as wins, hence the reduced range of ranks compared to other studies. Intuitively, our ranks can be considered as "tiers".
Recall that, on given dataset, the performance of given model A is expressed with the mean Amean and the standard deviation Asd of the performance score computed after the evaluation under multiple random seeds. Assuming the higher score the better, we define that the model better than the model B if: Amem Asd > Bmean- In other words, model is considered better if it has a better mean score and the margin is larger than the standard deviation.
On a given dataset, when there are many models, we sort them in descending score order. Starting from the best model (with a rank equal to 1) we iterate over models and assign the rank all models that are no worse than the best model according to the above rule. The first model in descending order that worse than the best model assigned rank and becomes the new reference model. We continue the process until all models ranked. Ranks computed independently for each dataset.
D.4 IMPLEMENTATION DETAILS OF SUBSECTION 4.3
Applicability to large datasets. The two datasets used in Table are full versions of the "Weather" and "Maps Routing" datasets from the TabReD benchmark Rubachev et al. (2024). Their smaller versions with subsampled training set were already included in Table and were used when building Figure 3. The validation and test sets are the same for the small and large versions of these datasets, so the task metrics are comparable between the two versions. When running models on the large versions of the datasets, experiment can be seen as a quick assessment of the applicability of several tabular DL to large datasets without strong focus the task performance. All models, except for FT-Transformer, were evaluated under 3 random seeds. FT- Fr-ntonsfrrmer was evaluated under random seed.
reused the hyperparameters tuned for their small versions. Thus, this
D.5 IMPLEMENTATION DETAILS OF SUBSECTION 5.1
Experiment setup. This paragraph complements the description of the experiment setup in subsec- tion 5.1. Namely, in addition to what is mentioned in the main text:
Dropout and weight decay turned off.
To get representative training profiles for all models, the learning rates are tuned separately for TabM*=! mini and TabMk=32 on validation sets using the usual metrics (i.e. RMSE accuracy) the guidance. The grid for learning rate tuning was: numpy. logspace (numpy. 1og10 5), numpy. 1og10 (5e-3). num=25).
mini
D.6 IMPLEMENTATION DETAILS OF SUBSECTION 5.2
T. Here, clarify the implementation details for TabM[G] described in subsection 5.2. G is obtained from a trained TabM by greedily selecting submodels from TabM starting from the best and stopping when two conditions simultaneously true for the first time: (1) adding
19

---

Published as a conference paper at ICLR 2025
Pesformanne
Performance scures
Pestoctaatiee
15 dituart citt reaties ejll
V-drranne wrth doess encenuple
Sasted by the came rack
Setaal by the sase soee
Scrted by Uhe time wane
Mon
DCN2
DON2 + -
+
SNN
SNN
SAINT
Autolat
Autelnt
0
DCN2
MLP
MLP
SNN
ResNet
Exndl
Autolat
Dooel
ReaNet
40
ML.P-Miser
MLP-MMxer
SAINT
Fxod
SAINT
12.33
FT-T-
TahR
FT-T
53 Su
TabR
MNCA
Thompt
1::
MIP--Miwt-
Thomps
T2G
T2G
MLP
TabR
Trompt
BesNet
MLPI
FT-T
MNCA
MLP
MNCA
MLP
MNCA-
T2G
Xt
23
TabR
CaBoost
Lg
GBM
MNCA
22421
XGBonet
MLP
Cot
23
TeaM
Ligh
TabM
MBP GRM?
CatBontt
TobM
TabR
Amsetion Syotioal
MNCA
0
XGBood
Tahh
MIPP (Okz))
Tab
TabM
01
155
55
166
395
10%
Boak (4)
Balutid Legerrmannt ovet M827
Idactt asproweaci our NLP
Figure 11: Same as Figure but ROC-AUC used the metric for all classification datasets. The two multiclass datasets presented in our benchmark are not taken into account.
any new submodel does not improve the validation metric of the collective prediction; (2) the current validation metric is already better than that of the initial model with all it submodels. To clarify, during the greedy selection, the i-th submodel is considered to be better than the j-th submodel if adding the i-th submodel to the aggregated prediction leads to better validation metries (i.e. it is not the same as adding the submodel in the order of their individual validation metrics).
D.7 IMPLEMENTATION DETAILS OF SUBSECTION 5.3
Figure 7 shows the mean percentage improvements (see subsection D.3) over MLP across 17 datasets: all datasets except for Covertype from Table 4, and all datasets from TabReD (Rubachev et al., 2024). We have used the dropout rate 0.1 and tuned the learning rate separately for each value of k. The score on each dataset averaged over 5 seeds.
D.8 NON-LINEAR EMBEDDINGS FOR CONTINUOUS FEATURES
Notation, We use the notation based on i and $ only for brevity. Any other unambiguous notation can be used in future work.
Updated piecewise-linear embeddings. We use slightly different implementation of the piecewise- linear embeddings compared to Gorishniy et al. (2022). Architecture-wise, our implementation corresponds to the "Q-L." and "T-L." variations from Table in Gorishniy et al. (2022) (we use the quantile-based bins for simplicity). In practice, our implementation is significantly faster and uses different parametrization and initialization. See the source code for details.
Other models. Since it is not feasible to test all combinations of backbones and embeddings, for baselines, we stick to the embeddings used in the original papers (applies to TabR (Gorishniy et al., 2024), ExcelFormer (Chen et al., 2023a) and ModernNCA (Ye et al., 2024)). For all models with feature embeddings (including TabM, MLP, TabR, ModernNCA, ExcelFormer), the embeddings- related details are commented in the corresponding sections below.
20

---

Published as a conference paper at ICLR 2025
D.9 TABM
Feature embeddings. TabM
mini and TabM are the versions of TabM with non-linear feature
embeddings. TabM mini and TabM use the updated piccewise-linear feature embeddings mentioned in subsection D.8.
Table 6 provides the hyperparameter tuning spaces for TabM and TabMmini- Table provides the hyperparameter tuning spaces for TabM and TabM
Table 6: The hyperparameter tuning space for TabM and TabM Here, (B) (Covertype, Mi- crosoft, Table 5) and (A) contains all other datasets.
Parameter
Distribution or Value
k
32
layers
UniformInt(L [1
Width (hidden size)
Uniformint 1024]
Dropout rate
(0.0, Uniform(0.0) 0.5]
Learning rate
LogUniform| [1e-4, 5e-3]
Weight decay
(0, LogUniform( le-1]}
Tuning iterations
(A) 100 (B) 50
Table 7: The hyperparameter tuning space for TabM mini and TabM Here, (B) {Covertype, Microsoft, Table 5} and (A) contains all other datasets.
Parameter
Distribution Value
32
layers
UniformInt( [L, 4]
Width (hidden size)
UniformInt [64, 1024]
Dropout rate
{0.0, Uniform[0.0m 0.5)
# PLE bins
UniformInt [8, 32]
Learning rate
LogUniform [5e-5, 3e-3]
Weight decay
{0, Log Loggnnffrrmmleee1eeel 1e-1]
Tuning iterations
(A) 100 (B) 50
D.10 MLP
Feature embeddings. MIP and MLP are the versions of MLP with non-linear feature embeddings. MI.t uses the updated piecewise-linear embeddings mentioned in subsection D.8. MLP (also known MLP-PLR) uses the periodic embeddings (Gorishniy al., 2022). Technically, it is the Python package. We tested
Periodi cEmbeddings class from the rt stdl.nnm.embedde two variations: with Fite=False and lite=True, In the paper, only the former one reported, but in the source code, the results for both are available.
Table 8, Table Table 10 provide the hyperparameter tuning spaces for MLP, MLP and MLP! respectively.
D.11 TABR
Feature embeddings. TabR is the version of TabR with non-linear feature em- beddings. TabR uses the periodic embeddings (Gorishniy et al., 2022), specifi- cally, PeriodicEmbeddingg (lite=True) from the rtdl num embeddings Python package most datasets. On the datasets from Table 5, TabR uses the
21

---

Published as a conference paper at ICLR 2025
Table 8: The hyperparameter tuning space for MLP.
Parameter
Distribution
# layers
UniformInt[11 6]
Width (hidden size)
Uniformlnt [64, 1024]
Dropout rate
{0.0, Uniform 0.5]
Learning rate
LogUniformm3e0 (3-55, 1e-3]
Weight decay
{0, Log MoogUnfformm[ee-1... 1e-1]]
Tuning iterations
100
Table 9: The hyperparameter tuning space for MLP
Parameter
Distribution
# layers
1niformlnt(1,55 5]
Width (hidden size)
Uniformint [64, 1024]
Dropout rate
{0.0, Uniform[0.0, 0.5]
Learning rate
LogUniformm3eer (3e-5, 1e-3]
Weight decay
{0, LogUniformm(e--,, le-1]}
d_embedding
UniformInt|/s (8, 32]
bins
UniformIni [2, 128)
# Tuning iterations
100
Table 10: The hyperparameter tuning space for MT.Pi.
Parameter
Distribution
# layers
[11
Width (hidden size)
Uniformint (64, 1024]
Dropout rate
(0.0, Uniform[0.0, 0.5]
Learning rate
LogUniformm33e- 1e-3]
Weight decay
(0, LogUnooforo[[e-444 le-1]}
n. frequencies d_embedding
UniformInt|[1 [16, 96]
UniformInt [16, 32]
frequency scale
LogUniform 1e-2, lel]
# Tuning iterations
100
Periodi cEmbeddings (lite=True) embeddings the Sberbank Housing and Ecom Offers datasets, and LenearReLUEbeedcinge on the rest (to fit the computations into the GPU memory, following the original TabR paper).
Since we follow the training and evaluation protocols from Gorishniy et al. (2024), and TabR was proposed in Gorishniy et al. (2024), simply reuse the results for TabR. More details can be found in Appendix. from Gorishniy et al. (2024). When tuning TabR on the datasets from Table 5, we have used 25 tuning iterations and the same tuning space for TabR from Rubachev et al. (2024).
22

---

Published as a conference paper at ICLR 2025
D.12 FT-TRANSFORMER
We used the implementation from the setdd.revisitingg inn.indellssssssmmmmmoooom...eel Python package. The results on datasets from Table were copied from Rubachev al. (2024), because the experiment setups compatible.
Table 11: The hyperparameter tuning space for FT-Transformer Gorishniy al. (2021). Here, (B) (Covertype, Microsoft) and (A) contains all other datasets (except Table 5).
Parameter
Distribution Value
# blocks
UniformInt [1,
4okenn
UniformInt [16, 384]
Attention dropout rate
Uniform (0.0, 0.5]
FFN hidden dimension expansion rate
Uniform [2/3, 8/a]
FFN dropout rate
Uniform (0.0, 0.5)
Residual dropout rate
{0.0, 10.00 0.2]}
Learning rate
LogUniform (3-55, 1e-3]
Weight decay
{0, Log LogUniform([e-4, le-1]]
# Tuning iterations
(A) 100 (B) 50
D.13 MODERNNCA
Feature embeddings. We adapted the official implementation of Ye et al. (2024). We used periodic embeddings Gorishniy et al. (2022) (specifically, Perico cEmbeddinge (lite=True) from the rtdl_num, wem_nm_emmmt emmeddinng Python package) for ModernNCA! and no embeddings for Modern- NCA. Table 12 and Table 13 provides hyperparameter tuning spaces for each ModemnNA and ModernNCAF.
Table 12: The hyperparameter tuning space for ModernNCA. Here, (C) = [Table 5), (B) = [Covertype. Microsoft] and (A) contains all other datasets.
Parameter
Distribution
# blocks
UniformInt [0, 2]
datock
UniformInt [64, 1024]
dim
Uniformint [64, 1024]
Dropout rate
Uniform (0.0, 0.5]
Sample rate
Uniform (0.05, 0.6]
Learning rate
Log Uniform (e-on,mmmmee,,.1 1e-1]
Weight decay
{0, Log MoogUniformm[eee (1e6 1e-3]}
Tuning iterations
(A) 100 (B, C) 50
D.14
T2G-FORMES
We adapted the implementation and hyperparameters of Yan et al. (2023) from the official repository
Table 14 provides hyperparameter tuning space.
D.15 SAINT
We completely adapted hyperparameters and protocol from Gorishniy et al. (2024) to evaluate SAINT on Grinsztajn et al. (2022) benchmark. Results on datasets from Table were directly taken from
23

---

Published as a conference paper at ICLR 2025
Table 13: The hyperparameter tuning space for ModemNCAF. Here, (C) {Table 5}, (B) {Covertype, Microsoft) and (A) contains all other datasets.
Parameter
Distribution
# blocks
UniformInt [0, 2]
datock
UniformInt [64, 1024]
dim
UniformInt(64, 1024]
Dropout rate
Uniform 0.5]
Sample rate
Uniform Un.5orm(0.05,, 0.6]
Learning rate
LogUniform[le-5, 1e-1]
Weight decay frequencies
(0, (nn 1e-3)
Uniformint [16, 96]
d embedding frequency init scale
UniformInt [16, 32]
Log Loguniform|pp (0.01, 10]
# Tuning iterations
(A) 100 (B, C) 50
Table 14: The hyperparameter tuning space for T2G-Former Yan et al. (2023). Here, (C) {Table 5}. (B) = [Covertype, Microsoft) and (A) contains all other datasets. Also, we used 50 tuning iterations on some datasets from Grinsztajn al. (2022).
Parameter
Distribution or Value
blocks
(A) UniformInt(3, 4] (B, C) (1iformInt(1,3]
dtoken
UniformInt/64 (64,
Attention dropout rate
Uniform 0.5]
FFN hidden dimension expansion rate
(A, Un/a 8/3] (C) 4/3
FFN dropout rate
Uniform[0.0, 0.5]
Residual dropout rate
(0.0, Uniform (0.0, 0.2]]
Learning rate
LogUniform (3-5, 1e-3)
Col. Learning rate
LogUniform (5-33, 5e-2)
Weight decay
{0, Log [1e-6, 1e-1]}
Tuning iterations
(A) 100 (B) (C) 25
Gorishniy et al. (2024). Additional details can be found in Appendix. D from Gorishniy et al. (2024). We have used a default configuration on big datasets due to the very high cost of tuning (see Table 15).
D.16 EXCELFORMER
Feature embeddings. ExcelFormer (Chen et al., 2023a) uses custom non-linear feature embeddings based GLU-style activation, the original paper for details.
We adapted the implementation and hyperparameters of Chen et al. (2023a) from the official reposi- tory?. For fair comparison with other models, we did not the augmentation techniques from the paper in our experiments. See Table 16.
D.17 CATBOOST, XGBOOST AND LIGHTGBM
Since our setup is directly taken from Gorishniy et al. (2024), we simply reused their results for GBDTs from the official repository Importantly, in series of preliminary experiments,
24

---

Published as a conference paper at ICLR 2025
Table 15: et al. (2024).
The default hyperparameters for SAINT (Somepalli et al., 2021) on datasets from Rubachev
Parameter
Value
depth
2
dtoken
32
Rheads
4
dhead
8
Attention dropout rate
0.1
FFN hidden dimension expansion rate
1
FFN dropout rate
0.8
Learning rate
le-4
Weight decay
le-2
Table 16: The hyperparameter tuning space for Excelformer Chen et al. (2023a). Here, (D) (Homecredit, Maps Routing), (C) (Table 5 w/o (D)), (B) {Covertype, Microsoft) and (A) contains all other datasets.
Parameter
Distribution or Value
# blocks
(A, B) (niformInt 5] (C) UniformInt[2, (D) UniformInt(1,3) (A, B) {32, 64, 128, 256) (C) {16, 32, 64) (D) 16,
dioken
Theads
(A,B) (4, 8, 16,32) (C) (4,8, 16) (D)
Attention dropout rate
0.3
FFN dropout rate
0.0
Residual dropout rate
Uniform 0.0, 0.5]
Learning rate
LogUniform 3e-5, 1e-3]
Weight decay
{0, LogUnnooormmieee 1e-1]}
Tuning iterations
(A) 100 (B) 50 (C, D) 25
confirmed that those results reproducible in our instance their setup. The details can be found in Appendix.D D from Gorishniy et al. (2024). Results on datasets from Table were copied from the paper (Rubachev al., 2024).
D.18 AUTOINT
We used an implementation from Gorishniy et al. (2021) which is an adapted official implementation'.
1.18.1 TABPEN
Since TabPFN accepts only less than 10K training samples we use different subsamples of size 10K for different random seeds. Also, TabPFN not applicable to regressions and datasets with more than 100 features.
25

---

Published as a conference paper at ICLR 2025
Table 17: The hyperparameter tuning space for AutoInt (Song al., 2019). Here, (B) {Covertype. Microsoft) and (A) contains all other datasets.
Parameter
Distribution
# blocks
UniformInt(1, 6]
dioken
UniformInt(8, 64]
Theads
Attention dropout rate
{0, Uniformm0.00 [0.0, 0.5]
Embedding dropout rate
(0, Uniform (0.0. 0.5]
Learning rate
LogUniformm3ee 1e-3]
Weight decay
(0, [eg le-1]}
Tuning iterations
(A) 100 (B) 50
E PER-DATASET RESULTS WITH STANDARD DEVIATIONS
Table 18: Extended results for the main benchmark. Results are grouped by datasets. One ensemble consists of five models trained independently under different random seeds.
churn f
california J
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
0.8553 0.0029
0.8582
0.0008
MLP
0.4948 0.0058 0.4880 + 0.0022
TabPFN
0.8624
0.0008
TabPFN
ResNet
0.85-15 0.0044
0.8565
0.0035
ResNet
0.4915 0.0031
0.4862 + 0.0017
DCN2
0.8567 + 0.0020
0.8570
+ 0.0017
DCN2
0.4971 + 0.0122
0.4779 + 0.0022
SNN
0.8506 +: 0.0051
0.8533 1 0.0033
SNN
0.5033 1: 0.0075
0.4933 1 0.0035
Trompt
0.8600 nan
Trompt
0.4579 nan
Autolnt
0.8607 0.0047
0.8622 0.0003
Autolnt
0.4682 0.0063
0.4490 + 0.0028
MLP-Mixen
0.8592 0.0036
0.8630 0.0005
HLP-Mixe
0.4746 0.0056
0.4509 0.0029
Excel'
0.8618 0.0023
0.8625 nan
Excel
0.4544 0.0048
0.4350
wan
SAINT
0.8603 0.0029
SAINT
0.4680 0.0048
FT-T
0.8593 0.0028
0.8598
0.0025
FT-T
0.4635 + 0.0048
0.4515 + 0.0016
T2G
0.8613 + 0.0015
T2G
0.4640 1: 0.0100
0.4462 1 mATI
MLPt-lite
0.8624 0.0010
0.8638 0.0012
MLP!-lite
0.4652 0.0045
0.4549 + 0.0006
MLP!
0.8624 0.0026
0.8640
1: 0.0010
MLP
0.4597 0.0058
0.4482 1 0.0026
MLP
0.8580 0.0028
0.8605
0.0018
MLP
0.4530 1: 0.0029
0.4491 1: 0.0010
XCBoost
0.8605 +. 0.0022
0.8608
0.001 0.0013
XCBoost
0.4327 1: 0.0016
0.4316 1 0.0007
LightGBM
0.8600 + 0.0008
0.8600
0.000
LightGBM
0.4352 + 0.0019
0.4339
+ 0.0008
Boost
0,8582 0.0017
0.8588
+ 0.0008
CatBoost
0.4294 + 0.0012
0.4265 0.0003
TabR
0.8599 0.0025
0.8620
- 0.0023
TabR
0.4030 0.0023
0.3964 0.0013
TabR
0.8625 1 0.0021
TabR
0.3998 1: 0.0033
MNCA
0.8595 + 0.0028
0.8615 + 0.0013
MNCA
0.4239 + 0.0012
0.4231 + 0.0005
MNCA
0.8606 0.0032
0.8607
0.0008
MNCA
0.4142 0.0031
0.4071 1: 0.0029
TabM
0.8613 0.0025
0.8615
0.0005
TabM
0.4509 0.0032
0.4490 + 0.0018
TabM
0.8605 0.0016
0.8612 0.0008
TabM
0.4414 0.0012
0.4002 + 0.0001
TabM[G]
0.8609 0.0024
TabM
D.4413 0.0020
TabMpiol
0.8633 0.0018
0.8638
0.0012
TabMain
0.4479 0.0022
0.4461
0.0011
T mini
0.8606 + 0.0023
0.8630 + 0.0030
TabM
Tabiiinn mini
0.4275 1 0.0024
0.4244 1 0.0006
26

---

Published as a conference paper at ICLR 2025
house 4
adult
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
3.1117 0.0294 3.0706 0.0140
MLP
0.8540 0.0018
0.8559 + 0.0011
TabPFN
TabPFN
ResNet
3.1143 + 0.0258
3.0706 + 0.0098
ResNet
0.8554 + 0.0011
0.8562 + 0.0006
DCN2
3.3327 + 0.0878
3.1303 + 0.0410
DCN2
0.8582 + 0.0011
0.8593 + 0.0002
SNN
3.2176 0.0376
3.1320 1 0.0155
SNN
0.8582 0.0009
0.8603 1 0.0012
Trompt
3.0638 nan
Trompt
0.8590 man
AutoInt
3.2157 0.0436
3.1261 0.0095
Autoint
0.8592 0.0016
0.8612 + 0.0004
MLP-Mixer
3.1871 0.0519
3.0184 0.0086
MLP- Mixer
0.8598 0.0013
0.8617 1 0.0002
Excel'
3.2460 0.0685
3.1097 + non
Excel"
0.8613 + 0.0024
0.8641 + nan
SAINT
3.2424 + 0.0595
SAINT
0.8601 + 0.0019
FT-T
3.1823 + 0.0460
3.0974 0.0334
FT-T
0.8588 + 0.0015
0.8608 + 0.0011
T2G
3.1613 0.0320
3.0982
nan
T2G
0.8601 0.0011
0.8622 + nan
MLP1-lite
3.0633 0.0248
3.0170 0.0070
MLPi-lite
0.8693 + 0.0007
0.8702 + 0.0006
MLP
3.0775 0.0336
3.0268 0.0170
MLP
0.8694 0.0011
0.8704 0.0008
MLP
3.0999 +1 0.0351
3.0401 1 0.0071
MLP
0.8603 1: 0.0009
0.8616 1 0.0006
XGBoost
3.1773 + 0.0102
3.1644 + 0.0068
XGBoost
0.8720 + 0.0006
0.8723 + 0.0002
LBg
3.1774 :: 0.0087
3, 3.1672
0.0050
LightGBM
0.8713 0,0007
0.8721 + 0.0004
CaoBoost
3.1172 0.0125
3.1058 0.0022
CatBoost
0.8714 0.0012
0.8723 0.0007
TabR
3.0667 0.0403
2.9958 0.0270
TabR
0.8646 0.0022
0.8680 + 0.0019
TabR
3.1048 1: 0.0410
TabR*
0.8699 0.0011
MNCA
3.0884 0.0286
3.0538
0.0072
MNCA
0.8677 0.0018
0.8696 + 0.0003
MNCA
3.0701 + 0.0388
3.0149 + 0.0308
MNCA
0.8717 + 0.0008
0.8742 + 0.0006
TabM
3.0002 0.0182
2.9796
0.0024
TabM
0.8582 0.0011
0.8588 0.0003
TabM
3.0038 0.0097
2.9906 0.0026
TabM
0.8575 0.0008
0.8583 + 0.0004
GabM[G]
3.0082 0.0184
TabM[G]
0.8572 0.0010
mab
3.0394 0.0139
3.0206 + 0.0128
TabMain
0.8598 + 0.0011
0.8604 + 0.0000
TabM mini
2.9976 + 0.0196
2.9854 + 0.0076
TabM mini
0.8700 1 0.0007
0.8701 1 0.0003
diamond 1
otto f
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
0.1404 + 0.0012 0.1362 + 0.0003
MLP
0.8175 + 0.0022
0.8222 + 0.0007
TabPFN
TabPEN
0.7408 + 0.0028
ResNet
0.1396 0.0029
0.1361 0.0011
ResNet
0.8174 0.0021
0.8198 + 0.0006
DCN2
0.1420 0.0032
0.1374
0.0020
DCN2
0.8064 0.0021
0.8208
0.0023
SNN
0.1473 0.0057
0.1424
0.0008
SNN
0.8087 0.0020
0.8156
+ 0.0013
Trompt
0.1391 + nan
Trompt
0.8093 + nan
AutoInt
0.1392 + 0.0014
0.1361 + 0.0004
Autoint
0.8050 + 0.0084
0.8111 + 0.0020
MLP- Mixer
0.1400 + 0.0025
0.1378 0.0008
MLP-Mixen Mixer
0.8092 0.0040
0.8136 + 0.0010
Excel'
0.1766 0.0023
0.1712 + nan
Excel"
0.8102 0.0022
0.8220 + nan
SAINT
0.1369 0.0019
SAINT
0.8119 0.0018
FT-T
0.1376 0.0013
0.1360 0.0002
FT-T
0.8133 0.0033
0.8221 + 0.0013
T2G
0.1372 0.0011
0.1346
T2G
0.8161 0.0019
+ 0.8272
MLP
0.1342 0.0008
0.1325 0.0004
MLP2-lite
0.8190 + 0.0021
0.8271 + 0.0015
MLP!
0.1337 + 0.0010
0.1317 + 0.0003
MLP:
0.8189 +: 0.0015 0.8253 + 0.0000
MLP
0.1323 0.0010 0.1301 0.0005
MLP1
0.8205 0.0021 0.8290 0.0006
XGBoost
0.1368 0.0004
0.1363 0.0001
XGBoost
0.8297 0.0011
0.8316 + 0.0008
LightGBM
0.1359 0.0002
0.1358 1 0.0001
LightGBM
0.8302 0.0009
0.8316 1 0.0013
Boost
0.1335 1. 0.0006
0.1327 1 0.0004
CatBoost
0.8250 1: 0.0013
0.8268 1 0.0002
TabR
0.1327 + 0.0010
0.1311
+ 0.0005
TabR
0.8179 + 0.0022
0.8236
+ 0.0009
TabR:
0.1333 0.0013
TabR:
0.8246 + 0.0018
MNCA
0.1370 + 0.0018
0.1348 + 0.0005
MNCA
0.8275 + 0.0012
0.8313 0.0006
MNCA
0.1327 0.0012
0.1315 + 0.0006
MNCA
0.8265 0.0015
0.8304 + 0.0006
Tab
0.1342 + 0.0017
0.1327 + 0.0004
TabM
0.8268 + 0.0014
0,8300 + 0.0007
TabM
0.1310 + 0.0007
0.1307 0.0002
TabM
0.8275 + 0.0014
0.8284 + 0.0005
G
0.1309 + 0.0008
TabM[G]
0.8254 + 0.0022
-
TabMmiul
0.1323 0.0007
0.1317 0.0002
TabMpini
0.8282
0.0014
0.8299 + 0.0005
TabM
0.1315 0.0006
0.1312 0.0001
TabM
0.8342 0.0012
0.8356 4: 0.0004
27

---

Published as a conference paper at ICLR 2025
higgs-small
black-fridas
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
0.7180 0.0027 0.7192 0.0005
MLP
0.6955 0.0004
0.6942 + 0.0002
TabPFN
0.6727 1: 0.0034
TabPFN
ResNet
0.7256 + 0.0020
0.7307 + 0.0001
ResNet
0.6929 + 0.0008
0.6907 + 0.0002
DCN2
0.7164 + 0.0030
0.7237 + 0.0011
DCN2
0.6968 + 0.0013
0.6936 + 0.0007
SNN
0.71422 0.0024
0.7171 110711:0.000 0.0020
SNN
0.6996 0.0013
0.6978 1 0.0004
Trompt
0.7262 nan
Trompt
0.6983 nan
AutoInt
0.7240 0.0028
0.7287 0.008
Autoint
0.6994 0.0082
0.6927 + 0.0021
MLP-Mixes
0.7248 0.0023
0.7334
MLP- Mixer
0.6905 0.0021
0.6851 + 0.0011
Excel'
0.7262 0.0017 0.7329 non
Excel"
0.6947 + 0.0016
0.6908 + nan
SAINT
0.7236 + 0.0019
SAINT
0.6934 + 0.0009
FT-T
0.7281 0.0016
0.0004
FT-T
0.6987 + 0.0192
0.6879 + 0.0023
T2G
0.7352 0.0037 0.7400
nan
T2G
0.6887 0.0046
0.6832 + nan
MLP1-lite
0.7260 + 0.0017
+..3344
MLPi-lite
0.6849 + 0.0006
0.6824 + 0.0002
MLP
0.7261 0.0010 0.7270 1 0.0003
MLP
0.6857 0.0004
0.6838 0.0002
MLP
0.7210 +: 0.0016 0.7252 0.0005
MLP
0.6836 1: 0.0006
0.6812 1 0.0002
XGBoost
0.7246 + 0.0015 0 0.0013
XGBoost
0.6806 0.0001
0.6805 + 0.0000
LBg
0.7256 + 0.0009 0.7263 0.0000
LightGBM
0.6799 0,0003
0.6795 + 0.0001
Cooso
0.7260 0.0011 0.7273 0.0010
CatBoost
0.6822 0.0003
0.6813 0.0002
TabR
0.7223 + 0.0010
0.7257 0.0008
TabR
0.6899 0.0004
0.6883 + 0.0002
TabR
0.7294 1 0.0014
TabR*
0.6761 +: 0.0009
MNCA
0.7263 0.0023
0.7292
0.0006
MNCA
0.6893 0.0004
0.6883 + 0.0000
MNCA
0.7300 + 0.0020
0.7348
0.0008
MNCA
0.6885 + 0.0007
0.6863 + 0.0003
TabM
0.7383 0.0028
0.7409 = 0.0010
TabM
0.6875 0.0015
0.6866 0.0003
TabM
0.7394
0.0018
0.7409
0.0008
TabM
0.6869 0.0004
0.6865 + 0.0001
TabM[G]
0.7392
0.0016
TabM[G]
0.6865 0.0005
mab
0.7338
0.0011
0.7345 + 0.0008
TabMmin
0.6863 + 0.0006
0.6856 + 0.0003
TabM mini
0.7361
0.0011
0.7383
0.0008
TabM mini
0.6781 +. 0.0004
0.6773 + 0.0001
covtype2 f
microsoft + Single model
Method
Single model
Ensemble
Method
Ensemble
MLP
0.9630 + 0.0012
0.9664 + 0.0004
MLP
0.7475 + 0.0003 0.7460 + 0.0003
TabPFN
0.7606 + 0.0022
TabPFN
ResNet
0.9638 0.0005
0.9685
0.0003
ResNet
0.7472 0.0004
0.7452 + 0.0004
DCN2
0.9622 0.0019
0.9673 0.0011
DCN2
0.7499 0.0003
0.7477 + 0.0001
SNN
0.9636 0.0010
0.9677 0.0002
SNN
0.7488 0.0004
0.7470 + 0.0001
Trompt
0.9286 + nan
Trompt
0.7476 + nan
AutoInt
0.9614 + 0.0016
0.9696 + 0.0005
Autoint
0.7482 + 0.0005
0.7455 + 0.0002
MLP-Mixee Mixer
0.9663 + 0.0019
0.9699 0.0014
MLP-M Mixer
0.7436 + 0.0001 0.7482 0.0008
Excel'
0,9606 0.0018
0.9670 nan
Excel"
0.7479 0.0007 0.7442 + nan
SAINT
0.9669 + 0.0010
SAINT
0.7625 0.0066
FT-T
0.9698 0.0008
0.9731
0.0006
FT-T
0.7460 0.0007
0.7422 + 0.0004
T2G
0.9668 0.0008
0.9708 nan
T2G
0.7460 0.0006
+ 0.7427
MLP
0.9690 0.0008
0.9721 0.0006
MLP2-lite
0.7446 0.0002
0.7434 + 0.0002
MLP:
0.9713 +: 0.0006
0.9758 + 0.0000
MLP:
0.7444 +: 0.0003
0.7429 + 0.0001
MLP
0.9697 0.0008
0.9721 0.0005
MLP
0.7465 0.0005
0.7448 0.0001
XGBoost
0.9710 0.0002
0.9713 0.0000
XGBoost
0.7413 0.0001
0.7410 0.0000
LightGBM
0.9709 0.0003
LightGBM
0.7417 1:: 0.0001
0.7413 1 0.0000
Boost
0.9670 1. 0.0003
0.9680 + 0.0002
CatBoost
0.7412 J: 0.0001
0.7406 1. 0.0000
TabR
0.9737 + 0.0005
0.9745 + 0.0006
TabR
0.7503 + 0,0006
0.7485
+ 0.0002
TabR:
0.9752 0.0003
TabR:
0.7501 + 0.0005
MNCA
0.9724 + 0.0003
0.9729 + 0.0001
MNCA
0.7458 + 0.0003
0.7448 +:0488+10 0.0002
MNCA
0.9747 0.0002
0.9747 + 0.0002
MNCA
0.7460 0.0008
0.7435 + 0.0004
Tab
0.9712 4: 0.0008
0.9729 0.0003
TabM
0.7434 =:: 0.0003
0.7424 + 0.0001
TabM
0.9735 + 0.0004
0.9743 +09743+0.00 0.0001
TabM
0.7432 + 0.0004
0.7426 + 0.0001
G
0.9730 + 0.0005
TabM[G]
0.7432 + 0.0004
TabMmiul
0.9710 0.0007
0.9727 0.0002
TabMpini
0.7436
0.0002
0.7430 + 0.0002
TabM
0.9755 0.0003
0.9762 + 0.0001
TabM!
0.7423
0.0002
0.7416 0.0001
28

---

Published as a conference paper at ICLR 2025
Table 19: Extended results for Grinsztajn et al. (2022) benchmark. Results grouped by datasets. One ensemble consists of five models trained independently with different random seeds.
wine
phoneme 1
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
0.7778 + 0.0153 0.7907 + 0,0117
MLP
0.8525 0.0126
0.8635 + 0.0099
TabPFN
0.7908 0.0063
TabPIN
0.8684 1 0.0050
ResNet
0.7710 + 0.0137
0.7839 0.0083
ResNet
0.8456 + 0.0121
0.8501 + 0.0066
DCN2
0.7492 = 0.0147 0.7764 + 0.0095
DCN2
0.8342 + 0.0151
0.8543 + 0.0118
SNN
0.7818 1. 0.0143
0.7994 1 0.0097
SNN
0.8596 1 0.0124
0.8687 1 0.0080
Trompt
0.7818 + 0.0081
Trompt
0.8465 + 0.0205
Autolnt
0.7745 + 0.0144
Autolnt
0.8623 0,0138
0,8754 0.0095
ULP-Mixes
0.7769 0.0149
HLP--Mixe
0.8629 0.0123
0.8757 1: 0.0095
Excel'
0.7631 0.0171
0.7765
Excel
0.8551 0.0092
0.8711 1 0.0081
SAINT
0.7684 0.0144
SAINT
0.8657 + 0.0130
FT-T
0.7755 0.0133
0.7894
FT-T
0.8667 + 0.0127
0.8795 + 0.0093
T2G
0.7733 T 0.0118
0.7933
T2G
0.8672 1: 0.0166
0.8765 1 0.0141
MLP4-lite
0.7803 0.0157
0.7944
MLP4-lite
0.8742 0.0120
0.8861 + 0.0071
MLP!
0.7733 + 0.0185
0.7856
MLP!
0.8757 + 0.0118
0.8856 + 0.0065
MLP
0.7814 0.0132 0.7919
0.009
MLP
0.8647 1: 0.0098 0.8761 1: 0.0076
XGBoost
0.7949 + 0.0178
0.0186
XGBoost
0.8682 + 0.0174 0.8771 = 0.0156
LightGBM
0.7890 + 0.0160
LightGBM
0.8702 + 0,0129
0.8733 + 0.0126
CatBoost
0.7994 1 0.0131
0,0098
CalBoost
0.8827 0,0117
0.8897
+ 0.0055
TabR
0.7936 + 0.0114
0.8055
+
TabR
0.8781 + 0.0096
0.8840
- 0.0054
TabR
0.7804 +: 0.0148
TabR
0.8772 1: 0.0087
MNCA
0.7911 + 0.0135
MNCA
0.8835 + 0,0079
0,8861 + 0.0057
MNCA
0.7867 0.0113
0.7933
MNCA
0.8828 + 0.0082
0.8925 + 0.0056
TabM
0.7961 + 0.0136
0.8011 +.008
TabM
0.8701 +: 0.0167
0.8766 + 0.0128
TabM
0.7941 0.0124
0.7985
+ 0.0139
TabM
0.8831 0.0121
0.8880 + 0.0108
TabM[G]
0.7879 + 0.0161
Ta]M[G]
0.8762 0.0144
TabM
0.7937 0.0103 0.7890 0.0130
TabMmini
0.8803 0.0098
0.8842
0.0067
TabM mini
0.7839 + 0.0169 0.7917 + 0.0144
TabM mini mini
0.8780 + 0.0119 0.8817 + 0.0101
analcatdata supreme
Mercedes Benz Greener Manufacturing J
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
0.0782 0.0081 0.0766 1 0.0090
MLP
8.3045 1 0.8708
8.2682 1 0.8992
TabPFN
TabPFN
ResNet
0,0852 + 0.0076 0.0823 + 0,0078
ResNet
8.4434 + 0. 0979
8.3178 + 0.8482
DCN2
0.0811 0.0137 0.0759 1 0.0086
DCN2
8.3540 0.8314
8.3021 1 0.8579
SNN
0.0826 0.0096 0.0779
0.0098
SNN
8.2718 0.8152
8.2236 + 0.8479
Trompt
0.0782 0.0095
Trompt
8.3409 0.9840
AutoInt
0.0783 0.0078 0.0768 + 0.0083
AutoInt
8,4001 +: 0.9256 8.3237 +: 0.9658
MLP-Mixer
0.0770 1. 0.0082
0.0759
0.0088 0.0081
MLP-Minn
8.2860 1: 0.8656
8.2398 1 0.9023
Excel'
0.0796 + 0.0101
Excel
8.2244 + 0.8514
8.1918 + 0.9387
SAINT
0.0773 + 0.0078
SAINT
8.3556 + 0.9566
FT-T
0.0787 1 0.0086
0.0775 0.009
FT-T
8.2252 1 0.8617
8.1616 + 0.8834
T2G
0.0775 0.0081
0.0763
0.0084
T2G
8.2120 0.8485
8.1651 +: 0.9339
MLP*-lite
0.0798 + 0.0088
0.0769 + 0.0092
MLPt-lite
8.3045 + 0.8708
8.2682 + 0.8992
MLP
0.0786 0.0073
0.0720 + 0.0053
MLP:
8.3045 + 0.8708
8.2682 + 0.8992
MLP!
0.0774 11: 0.0064
0.0759 0.0063
MLP!
8.3045 1 0.8708 8.2682 + 0.8992
XGBoost
0.0801 0.0126
0.0774
XGBoost
8.2177 + 0.8175
8.2092 - 0.8458
LightGBM
0.0778 0.0115
0.0767
1 0.0110
LightGBM
8,2078 0.8231
8.1618 0.8566
Boost
0.0780 0.0067 0.0734
CatBoost
8.1629 0.8193
8.1554 + 0.8439
TabR.
0,0803 0.0066
TabR
8,3506 0.8149
8.2694
+ 0.8399
TabR
0.0807 + 0.0088
TabR
8.3187 + 0.818
MNCA
0.0809 0.0072
0.0784 0.006
MNCA
8,2557 0.8602
8.1771 0.8710
0.0825 + 0.0090
0.0793
4 0.0072
+
8.2557 + 0.8602
8.1771 + 0.8710
TabM
0.0777 0.0099
0.0769
0.0105
TabM
8.1995 + 0.9130 8.2215 + 0.8910
TabM
0.0786 + 0.0055
0.0781 +
TabM
8.2052 + 0.9043
8.1965 + 0.9306
GabM[G]
0.0808 + 0.0063
TabM[G]
8.2235 0.8867
TabM
0.0773 + 0.0077
0.0763 0.007
TabM
8.2075 1 0.9185
8.89 + 0.9412
TabM mini
0.0764 0.0071
0.0749 0.0076
TabMt mini
8,2075 + 0.9185
8.86 + 0.9442
29

---

Published as a conference paper at ICLR 2025
KDDCupp9_upsellne
kdd.ipums Ja..mmsJa.977-mllll
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
0.7759 0.0137 0.7806 0.0125
MLP
0.8828 0.0061
0.8845 + 0.0055
TabPFN
TabPFN
0.8578 1: 0.0046
ResNet
0.7811 + 0.0124
0.7861 + 0.0109
ResNet
0.8823 + 0.0070
0.8821 + 0.0060
DCN2
0.7850 + 0.0161
0.7884 + 0.0135
DCN2
0.8770 + 0.0072
0.8824 + 0.0068
SNN
0.7884 + 0.0122
0.7940
1 0.0116
SNN
0.8722 0.0093
0.8733
1 0.0083
Trompt
0.7994 0.0055
Trompt
0.88-17 0.0070
AutoInt
0.8004 0.0075
0.8037
Autoint
0.8808 0.0083
0.8830 + 0.0081
MLP MLP-Mixer
0.7979 0.0105
0.0094
MLP- Mixer
0.8762 0.0100
0.8770 + 0.0088
Excel'
0.7903 0.0074
0.7939
0.0099
Excel"
0.8803 + 0.0054
0.8823 + 0.0071
SAINT
0.7942 + 0.0112
SAINT
0.8837 + 0.0055
FT-T
0.7957 + 0.0127
0.7960
0.0139 0.0139
FT-T
0.8795 + 0.0077
0.8792
0.0062
T2G
0.8037 0.0100
0.7988
0.0084 0.0084
T2G
0.8833 0.0054
0.8841 - 0.0062
MLP1-lite
0.7962 0.0093
0.0105
MLPi-lite
0.8765 + 0.0108
0.8765 0.010s
MLP
0.8005 0.0097
0.8032
1 0.0117
MLP
0.8816 0.0057
0.8818 1 0.0048
MLP
0.7925 +: 0.0123 0.7963
0.0089
MLP
0.8757 1: 0.0101
0.8756 1 0.0104
XGBoost
0.7930 0.0108
0.0102
XGBoost
0.8825 + 0.0089
0.8835 + 0.0085
LBg
0.7932 + 0.0119
LightGBM
0.8792 0,0075
0.8802
= 0.0067
CaoBoost
0.7992 0.0117
0.0121
CatBoost
0.8793 0.0088
0.8803
0.0100
TabR
0.7838 0.0136
0.7859
TabR
0.8798 0.0081
0.8819
0.0078
TabR
0.7908 1 0.0123
TabR*
0.8831 0.0050
MNCA
0.7939 0.0097
0.7989
0.0115
MNCA
0.8819 0.0054
0.8832 + 0.0048
MNCA
0.7960 + 0.0131
0.8008
0.0100 0.0110
+
MNCA
0.8837 + 0.0062
0.8860 + 0.0059
TabM
0.8002 0.0103
0.8021
0.0074
TabM
0.8845 0,0063
0.8848 0.0070
TabM
0.8024 0.0111
0.8054 0.0123
TabM
0.8823 0.0079
0.8825 + 0.0071
TabM[G]
0.7988
0.0118
TabM[G]
0.8818 0.0082
mab
0.7971
0.0117
0.7982 0.0107
TabMain
0.8784 + 0.0123
0.8786 + 0.0133
TabM mini
0.8024 1 0.0075
0.8035 0.0088
TabM mini
0.8779 + 0.0094
0.8784 1 0.0108
wine-quality $ Single model
isolet +
Method
Ensemble
Method
Single model
Ensemble
MLP
0.6707 + 0.0178 0.6530 + 0.0152
MLP
2.2744 + 0.2203 2.0018 + 0.1111
TabPFN
TabPEN
ResNet
0.6687 0.0166
0.6543 0.0170
ResNet
2.2077 0.2248
1.9206 + 0.1478
DCN2
0.7010 0.0171
0.6699
0.0139
DCN2
2.2449 0.1579
2.0176 + 0.0770
SNN
0.6604 0.0174
0.6245 0.0140
SNN
2.4269 0.2382
2.1142 + 0.1262
Trompt
0.6605 + 0.0153
Trompt
2.6219 + 0.0315
AutoInt
0.6840 + 0.0126
0.0446
Autolnt
2.6130 + 0.1658 2.3308 + 0.1088
MLP-Mixes Mixer
0.6672 + 0.0263
0.6291
MLP-Mise
2.0915 + 0.1159 2.3344 + 0.2073
Excel'
0.6881 0.0182
0.0199
Excel"
2.8691 0.0882
2.5989
+ 0.0664
SAINT
0.6797 0.0161
SAINT
2.7696 0.0200
FT-T
0.6787 0.0149 0.0564
FT-T
2.4879 0.2524
2.1501 + 0.1506
T2G
0.6783 0.0170
0.023
T2G
2.2867 0.2189
1.9179 + 0.1530
MLP
0.6569 0.0167
0.012
MLP1-lite
2.2719 0,1006
2.1026 = 0.1088
MLP:
0.6532 + 0.0133
0.0140
MLP:
2.1832 +: 0.1124 2.0775 + 0.0805
MLP
0.6721 0.0180 0.046
MLPt
2.0979 0.1779 1.9283 0.1334
XGBoost
0.6039 0.0134
0.6025
XGBoost
2.7567 0.0470
2.7294 + 0.0366
LightGBM
0.6135 0.0138
0.0144
LightGBM
2.7005 1: 0.0296
2.6903 1: 0.0290
Boost
0.6088 1. 0.0132
0.6060
+: 0.0137 0.0133
CatBoost
2.8847 1: 0.0227
2.8574
1 0.0148
TabR
0,6315 + 0.0097
0.6197
116197 0.009
TabR
1.9760 + 0,1738
1.7627
+ 0.1520
TabR:
0.6412 0.0105
TabR:
1.9919 0.1813
MNCA
0.6154 + 0.0083
1.6088
+
MNCA
1.7905 + 0.1594
1.6205 0.1676
MNCA
0.6099 + 0.0144
0.6028
MNCA
1.8912 0.1851
1.7147 + 0.1348
TabM
0.6169 +: 0.0123
0.6131 0.0126
TabM
1.8831 + 0.1194
+ 0.1088 1.8230
1.8578
TabM
0.6328 + 0.0172
0.6297 0.0180
TabM
1.8433 + 0.1196
+ 0.1197
G
0.6369 + 0.0179
TabM[G]
1.9091 + 0.1345
TabMmiul
0.6314 0.0142
0.6272 0.0146
TabMpini
1.9421 0.0971
1.9013 0.0813
TabM
0.6294 0.0120
0.6241 0.0118
TabM
1.7799 + 0.0859
1.7560 + 0.0795
30

---

Published as a conference paper at ICLR 2025
epu_act L
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
2.6814 + 0.2291
2.4953 0.1150
MLP
0.7860 0.0057
0.7887 + 0.0052
TabPFN
TabPFN
0.7891 1: 0.0091
ResNet
2.3933 + 0.0641
2.3005 + 0.0397
ResNet
0.7921 + 0.0076
0.7932 + 0.0066
DCN2
2.7868 + 0.1999
2.4884 + 0.0327
DCN2
0.7859 + 0.0068
0.7917 + 0.0078
SNN
2.5811 0.1480
2.3863
0.0324
SNN
0.7836 0.0074
0.7882
1: 0.0054
Trompt
2.2133 0.0221
Trompt
0.7975 0.0080
AutoInt
2.2537 0.0536
2.1708
0.0349
Autoint
0.7917 0.0071
0.7956
0.0058
MLP-Mixes
2.3079 0.0829
2.1831
+ 0.0470
MLP- Mixer
0.8001 1 0.0048 0.7954 0.0059
Excel'
2.3094 + 0.2401
2.1411 0.0767
Excel"
0.7957 + 0.0090
0.7985
+ 0.0106
SAINT
2.2781 + 0.0630
SAINT
0.7953 + 0.0058
FT-T
2.2394 + 0.0508
2.1494
0.0268
FT-T
0.7918 + 01.0066
0.7951 + 0.0071
T2G
2.2111 0.0413
2.1330
0.0316
T2G
0.7918 0.0058
0.7955 0.0047
MLP1-lite
2.2730 + 0.0457
2.1899 + 0.0419
MLPi-lite
0.7947 0.0101
0.7977 0.0117
MLP
2.2671 0.0383 2.1940 0.0433
MLP
0.7988 0.0092
0.8024 0.0093
MLP
2.3309 +. 0.0719 2.2516 1 0.0574
MLP
0.7981 1: 0.0065
0.8008 0.0057
XGBoost
2.5237 + 0.3530
2.4723 + 0.3789
XGBoost
0.8013 0.0081
0.8030 0.0076
LBg
2.2223 + 0.0894
2.2067 0.0916
LightGBM
0.8006 0,0078
0.8013
0.0072
CaoBoost
2.1239 0.0489 2.1092 0.0499
CatBoost
0.8026 0.0068
0.8056
0.0082
TabR
2.2980 0.0529 2.2228 0.0501
TabR
0.7995 0.0054
0.8015
0.0037
TabR
2.1278 0.0783
TabR*
0.8023 0.0088
MNCA
2.2603 0.0479
2.2339
0.0508
MNCA
0.7961 0.0065
0.8003 + 0.0077
MNCA
2.2105 + 0.0483
2.1396 + 0.0474
MNCA
0.7977 + 0.0081
0.8010 + 0.0084
TabM
2.1940 0.0523
2.1677
0.0487
TabM
0.7908 0.0068
0.7915 0.0068
TabM
2.1402 0.0588
2.1265
0.0580
TabM
0.7944 0.0060
0.7944 0.0052
GabM[G]
2.1549 0.0626
TabM[G]
0.7935
0.0064
mab
2.1638
0.0420
2.1508 + 0.0416
TabMain
0.7941 + 0.0055
0.7943 + 0.0045
TabM mini
2.1391 0.0542
2.1221 + 0.0570
TabM mini
0.7989
0.0086
0.8002 + 0.0074
Brazilian, houses
MagicTelescope f Single model
Method
Single model
Ensemble
Method
Ensemble
MLP
0.0473 + 0.0179 0.0440 + 0.0207
MLP
0.8539 + 0.0060
0.8566 + 0.0061
TabPFN
TabPEN
0.8579 + 0.006-
ResNet
0.0505 0.0181
0.0458 0.0207
ResNet
0.8589 0.0068
0.8651 + 0.0049
DCN2
0.0477 0.0172
0.0427 0.0207
DCN2
0.8432 0.0074
0.8490 + 0.0046
SNN
0.0630 0.0162
0.0556 0.0175
SNN
0.8536 0.0052
0.8567 + 0.0047
Trompt
0.0404 + 0.0266
Trompt
0.8605 + 0..0022
AutoInt
0.0470 + 0.0192
+.0437
Autoint
0.8522 + 0.0056
0.8560 + 0.0034
MLP-Mixee Mixer
0.0513 0.0234
0.021
MLP-Mixen Mixer
0.8571 + 0.0080
0.8621 + 0.0044
Excel'
0.0450 0.0156
0.0199
Excel"
0.8480 0,0090
0.8543 + 0.0075
SAINT
0.0479 0.0205
SAINT
0.8595 0.0060
FT-T
0.0438 0.0181
0.0412
FT-T
0.8588 0.0046
0.8643 0.0037
T2G
0.0468 0.0165
0.0436
T2G
0.8553 0.0055
0.8595
0.0051
MLP
0.0426 0.0180
0.0397
MLP2-lite
0.8591 0.0061
0.8626 = 0.0044
MLP!
0.0437 +: 0.0203
4.0407
MLP:
0.8575 + 0.0056
0.8605 + 0.0051
MLP
0.0421 0.0209
MLPł
0.8593 0.0054
0.8621 0.0037
XGBoost
0.05411 0.0270
0.0555
XGBoost
0.8550 0.0094
0.8589 0.0110
LightGBM
0.0603 0.0249
0.05599
LightGBM
0.8547 0.0085
0.8556 1 0.0086
Boost
0.0468 1. 0.0312
0.0456
CatBoost
0.8586 1 0.0070
0.8588
1 0.0077
TabR
0.0490 + 0.0152
0.0174
TabR
0.8682 + 0.0058
0.8729
+ 0.0038
TabR:
0.0451 0.0163
TabR:
0.8641 + 0.0052
MNCA
0.0527 + 0.0157
0.0509
MNCA
0.8602 + 0.0061
0.8628 + 0.0041
MNCA
0.0553 0.0192
0.0511
MNCA
0.8622 0.0085
0.8681 + 0.0064
Tab
0.0443 +: 0.0213
0.0431
TabM
0.8607 +: 0.0058
0.8622 + 0.0050
TabM
0.0417 + 0.0208
0.0413 0.0222
TabM
0.8622 + 0.0049
0.8631 + 0.0046
G
0.0424 + 0.0201
TabM[G]
0.8600 + 0.0055
TabMmiul
0.0433 0.0232
0.0428 0.0247
TabMpini
0.8606 0.0055
0.8618 + 0.0049
TabM
0.0416 0.0215
4.02330
TabM!
0.8644 + 0.0088
0.8673 + 0.0075
31

---

Published as a conference paper at ICLR 2025
Ailerons
MiamiHoussing0166
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
0.0002 0.0000 0.0002 0.0000
MLP
0.1614 + 0.0033
0.1574 + 0.0043
TabPFN
TabPFN
ResNet
0.0002 + 0.0000
0.0002 + 0.0000
ResNet
0.1548 + 0.0030
0.1511 + 0.0027
DCN2
0.0002 + 0.0000
0.0002 + 0.0000
DCN2
0.1683 + 0.0099
0.1575 + 0.0047
SNN
0.0002 1 0.0000
0.0002 1 0.0000
SNN
0.1618 0.0029
0.1557 1 0.0021
Trompt
0.0002 0.0000
Trompt
0.1478 0.0028
AutoInt
0.0002 0.0000
0.0002 0.0000
Autoint
0.1537 0.0035
0.1478 0.0027
MLP MLP-Mixer
0.0002 0.0000
0.0002 0.0000
MLP- Mixer
0.1527 0.0037
0.1479 + 0.0033
Excel'
0.0002 0.0000
0.0002 + 0.0000
Excel"
0.1519 + 0.0038
0.1442 + 0.0022
SAINT
0.0002 + 0.0000
SAINT
0.1507 + 0.0022
FT-T
0.0002 + 0.0000
0.0002 + 0.0000
FT-T
0.1514 + 01.029
0.1462 + 0.0031
T2G
0.0002 0.0000
0.0002
0.0000
T2G
0.1523 0.0023
0.1478 = 0.0024
MLP1-lite
0.0002 0.0000
0.0002
0.0000
MLPi-lite
0.1514 0.0025
0.1479 0.0017
MLP
0.0002 0.0000
0.0002 1 0.0000
MLP
0.1512 0.0019
0.1470 0.0024
MLP
0.0002 1 0.0000
0.0002 1:02211000 0,0000
MLP
0.1461 1: 0.0015
0.1433 0.0022
XGBoost
0.0002 0.0000
0.0002 0.0000
XGBoost
0.1440 + 0.0029
0.1434 0.0029
Lig
0.0002 + 0.0000
0.0002
0,0000
LightGBM
0.1461 0.0025
0.1455 + 0.0030
CaoBoost
0.0002 0.0000
0.0002
0.0000
CatBoost
0.1417 0.0021
0.1408 0.0026
TabR
0.0002 0.0000
0.0002
0.0000
TabR
0.1417 0.0025
0.1390 0.0020
TabR
0.0002 1 0.0000
TabR
0.1392 0.0023
MNCA
0.0002 0.0000
0.0002
0.0000
MNCA
0.1503 0.0040
0.1477 + 0.0032
MNCA
0.0002 + 0.0000
0.0002
:: 0.0000
MNCA
0.1475 + 0.0031
0.1438 + 0.0024
TabM
0.0002 0.0000
0.0002
0.0000
TabM
0.1483 0.0030
0.1465
0.0029
TabM
0.0002 0.0000
0.0002
0.0000
TabM
0.1478 0.0012
0.1471 + 0.0011
TabM[G]
0.0002 0.0000
TabM[G]
0.1482 0.0012
mab
0.0002
0.0000
0.0002
0.0000
TabMain
0.1481 + 0.0021
0.1471 + 0.0020
TabM mini
0.0002 1 0.0000 0.0002 1 0.0000
TabM mini
0.1408 0.0019
0.1399 + 0.0018
OnlineNewsaPoullarig + Single model
credit 1
Method
Ensemble
Method
Single model
Ensemble
MLP
0.8643 + 0.0007 0.8632 + 0.0005
MLP
0.7735 + 0.0042
0.7729 + 0.0047
TabPFN
TabPEN
0.7636 + 0.0015
ResNet
0.8665 0.0011
0.8639 0.0000
ResNet
0.7721 0.0033
0.7738 + 0.0027
DCN2
0.8714
0.0013
0.8648 0.0004
DCN2
0.7703 0.0034
0.7746
+ 0.0026
SNN
0.8692 0.0015
0.8665
0.0005
SNN
0.7712 0.0045
0.7716
+ 0.0059
Trompt
0.8623 + nan
Trompt
0.7740 + 0..0066
AutoInt
0.8636 + 0.0022
0.8596 + 0.0008
Autoint
0.7737 + 0.0050
0.7765 + 0.0058
MLP- Mixer
0.8615 + 0.0008
0.8598 0.0004
MLP-Mixen Mixer
0.7768 + 0.0059 0.7748 + 0.0038
Excel'
0,8605 0.0024
0.8556 nan
Excel"
0.7724 0.0038
0.7740 + 0.0069
SAINT
0.8600 0.0007
SAINT
0.7739 0.0052
FT-T
0.8629 0.0019
0.8603 0.0000
FT-T
0.7745 0.0041
0.7767 + 0.0040
T2G
0.8632 0.0009
0.8572
nan
T2G
0.7744 + 0.0046
0.7762 + 0.0057
MLP
0.8604 + 0.0009
0.8591 = 0.0004
MLP2-lite
0.7749 0.0055
0.7767 + 0.0075
MLP:
0.8591 + 0.0001
0.8585 + 0.0001
MLP:
0.7731 +: 0.0034 0.7741 + 0.0043
MLP
0.8585 0.0003
0.8581 0.0001
MLP1
0.7758 0.0040 0.7772 0.0055
XGBoost
0.85-15 0.0002
0.8543
0.0000
XGBoost
0.7698 0.0027 0.7706 + 0.0029
LightGBM
0.85-46 0.0002
0.8544 +: 0.0000
LightGBM
0.7686 0.0028
0.7726 1 0.0034
Boost
0.8532 1. 0.0003
0.8527 1115271100 0.0001
CatBoost
0.7734 1 0.0035
0.7752 1 0.0038
TabR
0.8677 + 0.0013
0.8633 0.0009
TabR
0.7730 + 0.0043
0.7740 0.0040
TabR:
0.8624 0.0011
TabR:
0.7723 0.0037
MNCA
0.8651 + 0.0003
0.8650 + 0.0002
MNCA
0.7739 + 0.0032
0.7757 + 0.0026
MNCA
0.8644 0.0010
0.8624 0.0006
MNCA
0.7734 0.0045
0.7754 + 0.0040
TabM
0.8584 + 0.0003
0.8581
1558110.000 0.0001
TabM
0.7755 + 0.0049 0.7751 +: 0.0042
TabM
0.8579 + 0.0003
0.0001 0.8575
TabM
0.7760 + 0.0043
0.7771
+ 0.0044
G
0.8579 + 0.0004
TabM[G]
0.7754 + 0.0045
TabMmiul
0,8588
0.0004
0.8581
0.0003
TabMpini
0.7752 0.0047
0.7754 0.0048
TabM
0.8563 0.0004
0.8558 + 0.0002
TabM
0.7761 +: 0.0033
0.7760 4: 0.0028
32

---

Published as a conference paper at ICLR 2025
elevators 4
fifa
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
0.0020 0.0001 0.0019 0.0000
MLP
0.8038 0.0124
0.8011 + 0.0143
TabPFN
TabPFN
ResNet
0.0019 + 0.0000
0.0019 + 0.0000
ResNet
0.8025 + 0.0140
0.7985 + 0.0149
DCN2
0.0019 + 0.0000
0.0019 + 0.0000
DCN2
0.8046 + 0.0135
0.7993 + 0.0129
SNN
0.0020 0.0001
0.0019 1.0000 0.0000
SNN
0.8074 0.0140
0.8031 1 0.0147
Trompt
0.0018 0.0000
Trompt
0.7880 0.0180
AutoInt
0.0019 0.0000
0.0018
Autoint
0.7923 0.0128
0.7886 + 0.0127
MLP MLP-Mixer
0.0019 0.0000
0.0018
MLP- Mixer
0.7936 0.0119
0.7903 + 0.0133
Excel'
0.0019 0.0000
0.0018
Excel"
0.7909 + 0.0111
0.7862 + 0.0161
SAINT
0.0018 + 0.0000
SAINT
0.7901 + 00.0188
FT-T
0.0019 + 0.0000
0.0018 01,000
FT-T
0.7928 + 0..032
0.7888 + 0.0130
T2G
0.0019 + 0.0000
0.0018 0.0000
T2G
0.7928 0.0139
0.7904 0.0183
MLP1-lite
0.0019 + 0.0000
0.0018
MLPi-lite
0.7940 0.0118
0.7898 0.0141
MLP
0.0018 0.0000
0.0018
0.0000
MLP
0.7907 0.0092
0.7870
0.0096
MLP
0.0018 1. 0.0000
0.0018
0.0000
MLP
0.7806 1 0.0104
0.7800 118000100 0.0114
XGBoost
0.0020 + 0.0000
0.0020
0.0000
XGBoost
0.7800 0.0108
0.7795 + 0.0114
Lig
0.0020 0.0000
0.0020
= 0,0000
LightGBM
0.7806 0,0120
0.7787 = 0.0122
CaoBo
0.0020 0.0000
0.0019
0.0000
CatBoost
0.7835 0.0116
0.7817
0.0114
TabR
0.0049 0.0000
0.0000
TabR
0.7902 0.0119
0.7863
+ 0.0120
TabR
0.0019 1: 0.0001
TabR
0.7914 0.0136
MNCA
0.0019 0.0000
0.0019 0.0000
MNCA
0.7967 0.0138
0.7933 0.01
MNCA
0.0018 + 0.0000
0.0018 0.0000
MNCA
0.7909 + 0.0107
0.7866
+
TabM
0.0019 0.0000
0.0018
0.0000
TabM
0.7974 + 0.0144
0.7954
TabM
0.0018 0.0000
0.0018 0.0000
TabM
0.7953 0.0135
TabM[G]
0.0018 0.0000
TabM[G]
0.7948 0.0135
mab
0.0018 0.0000
0.0018 0.0000
TabMain
0.7938 + 0.0156
0.7920 0.0176
TabM mini
0.0018 1 0.0000
0.0018 0.0000
TabM mini
0.7771 + 0.0107 0.7761 1 0.0117
house_sales +
medical.charge J Single model
Method
Single model
Ensemble
Method
Ensemble
MLP
0.1790 + 0.0009 0.1763 + 0.0003
MLP
0.0816 + 0.0001
0.0814 + 0.0000
TabPFN
TabPEN
ResNet
0.1755 0.0014
0.1738 0.0006
ResNet
0.0824 0.0003
0.0817 + 0.0001
DCN2
0.1862 0.0032
0.1778 0.0015
DCN2
0.0818 0.0003
0.0815 + 0.0001
SNN
0.1800) 0.0008
0.1770
0.0004
SNN
0.0827 0.0006
0.0817 + 0.0001
Trompt
0.1667 + nan
Trompt
0.0812 + nan
AutoInt
0.1700 + 0.0014
0.1670 + 0.0008
Autoint
0.0822 + 0.0007
0.0814 + 0.0001
MLP- Mixer
0.1704 + 0.0007
0.1690 0.0005
MLP-Mixen Mixer
0.0814 + 0.0002
0.0811 + 0.0000
Excel'
0.1713 0.0010
0.1668 nan
Excel"
0.0817 0,0004
0.0813 + man
SAINT
0.1713 0.0015
SAINT
0.0814 0.0002
FT-T
0.1690 0.0010
0.1659 0.0004
FT-T
0.0814 0.0002
0.0812 + 0.0000
T2G
0.1689 0.0010
0.1664
nan
T2G
0.0813 0.0002
+ 0.0811
MLP
0.1699 0.0008
0.1687 + 0.0007
MLP2-lite
0.0812 0.0002
0.0810 + 0.0000
MLP:
0.1690 + 0.0005
0.1676 0.0003
MLP:
0.0812 + 0.0001
0.0809 + 0.0001
MLP
0.1687 0.0004
0.1681 0.0001
MLP1
0.0812 0.0000
0.0811 0.0000
XGBoost
0.1694 0.0003
0.1689 0.0001
XGBoost
0.0825 0.0001
0.0825 + 0.0000
LightGBM
0.1692 0.0004
0.1686 0.0001
LightGBM
0.0820 1: 0.0000
0.0820 +: 0.0000
Boost
0.1669 1. 0.0001
0.1667 1 0.0000
CatBoost
0.0816 1 0.0000
0.0815 1 0.0000
TabR
0.1689 + 0.0009
0.1657
0.0003
TabR
0.0815 + 0.0002
0.0812 + 0.0000
TabR:
0.1636 0.0009
TabR:
0.0811 + 0.0001
MNCA
0.1737 + 0.0013
MNCA
0.0811 + 0.0001
0.0810 + 0.0000
MNCA
0.1694 + 0.0007
0.1670
0.0003
MNCA
0.0809 0.0000
0.0808 + 0.0000
Tab
0.1692 + 0.0011
0.1680
0.000 0.1662 0.0002
TabM
0.0813 + 0.0001
0.0812 + 0.0000
TabM
0.1666 + 0.0003
TabM
0.0812 + 0.0000
0.0812 + 0.0000
G
0.1667 + 0.0003
TabM[G]
0.0812 + 0.0000
TabMminl
0.1673
0.0004
0.1668 + 0.0001
TabMpini
0.0813 0.0000
0.0813 + 0.0000
TabM
0.1652 0.0003
0.1644 0.0001
TabM!
0.0811 + 0.0001
0.0811 0.0000
33

---

Published as a conference paper at ICLR 2025
pol
superconduct.
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
5.5244 0.5768
4.9945 0.5923
MLP
10.8740 + 0.0868
10.4118 + 0.0429
TabPFN
TabPFN
ResNet
6.3739 + 0.6286
5.8181 + 0.6054
ResNet
10.7711 + 0.1454
10.3195 + 0.0168
DCN2
6.5374 + 0.9479
5.1814 + 0.7775
DCN2
10.8108 + 0.0957
10.4342 + 0.0179
SNN
6.1816 0.7366
5.5959 1 0.8243
SNN
10.8562 1 0.1300
10.3342 1 0.0509
Trompt
3.2337 0.0605
Trompt
10.4142 + nan
AutoInt
3.3295 0.3379
2.7999 0.1776
Autoint
11.0019 0.1391
10.4469 + 0.0521
MLP-Mixes
3.2011 0.2921 2.8698
0.2577
MLP- Mixer
10.7502 11: 0.0800
10.3281 1 0.0450
Excel'
3.0682 0.2389
2.5816
Excel"
1.
10.4094 nan
SAINT
2.7203 + 0.1858
SAINT
10.7807 + 0.1077
FT-T
2.6974 + 0.1666
0.0724
FT-T
10.8256 116256+0000 0.1692
10.3391 0.0794 10.3017 nan
T2G
2.9539 0.1994
2.6282 0.0730
T2G
10.8310 +
MLP1-lite
2.8239 + 0.2173
2.5266
MLPi-lite
10.5058 0.0758
10.2322 + 0.0463
MLP
2.5452 0.1221
2.3700 0.0867
MLP
10.5061 11: 0.0330
10.2440 0.0127
MLP
2.4958 +. 0.1292
2.3651 1 0.1223
MLP
10.7220 1 0.0757
10.3758 1 0.0606
XGBoost
1.2963 0.0644
4.2548
0.0488
XGBoost
10.1610 + 0.0201
10.1413
0.0025
LBg
4.2320 0.3369
4.1880 0,3110
LightGBM
10.1634 + 0.0118
1052
0.0050
CaoBo
3.6320 0.1006 3.5505
0.0896
CatBoost
10.2422 1 0.0222
10.2116 1 0.0058
TabR
6.0708 0.5368
5.5578
0.4036
TabR
10.8842 + 0.1073
10.4800
0.0280
TabR
2.5770 0.1689
TabR*
10.3835 1 0.0562
MNCA
5.7878 0.4884
5.3773
0.5463
MNCA
10.4419 + 0.0640
10.2926 + 0.0261
MNCA
2.9083 + 0.1364
2.6717
0.0530
MNCA
10.5651 + 0.0616
10.3155 + 0.0253
TabM
3.3595 0.4017
3.2130
0.3979
TabM
10.3379 + 0.0338
10.1943
0.0291
TabM
3.0198 0.29755
2.9595
0.3107
TabM
10.2628 + 0.0275
10.2300 + 0.0108
GabM[G]
3.0358 0.3077
TabM[G]
10.2572 + 0.0463
Tab
3.1351 0.1952
TabMain
10.2472 0.0000
0.0208
10.2094 + 0.0057
TabM mini
2.2808 0.0343
2.2383 0.0111
TabM mini
10.1326 1 0.0186
10.0866 1 0.0070
jannis f
MiniBooNE f
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
0.7840 + 0.0018
0.7872 + 0.0007
MLP
0.9480 + 0.0007
0.9498 + 0.0001
TabPFN
0.7419 + 0.0018
TabPEN
0.9266 + 0.0012
ResNet
0.7923 0.0024
0.7958
0.0010
ResNet
0.9488 0.0011
0.9504 + 0.0005
DCN2
0.7712 0.0029
0.7825 0.0009
DCN2
0.9433 0.0011
0.9470
0.0010
SNN
0.7818 0.0025
0.7859
0.0011
SNN
0.9476 0.0013
0.9491
1 0.0010
Trompt
0.8027 + nan
Trompt
0.9473 + nan
AutoInt
0.7933 + 0.0018
0.7983 + 0.0013
Autolnt
0.9447 + 0.0014
0.9473 + 0.0010
MLP-Mixes Mixer
0.7927 0.0025
0.0012
MLP-Mixen Mixer
0.9483 + 0.0002 0.9416 0.0014
Excel'
0.7954 0.0015
0.8021 nan
Excel"
0.9430 0.0015
0.9451 + nan
SAINT
0.7971 0.0028
SAINT
0.9471 0.0009
FT-T
0.7940 0.0028
0.0900
FT-T
0.9467 0.0014
0.9486 + 0.0010
T2G
0.7998 0.0024
0.8052
T2G
0.9475 0.0014
+ 0.9508
MLP
0.7923 + 0.0018
0.7945
MLP2-lite
0.9466 0.0009
0.9478 + 0.0004
MLP!
0.7941 + 0.0017
0.7967 0.0011
MLP:
0.9473 +: 0.0010
0.9193 + 0.0001
MLP
0.7891 0.0013 0.7900
0.0006
MLP1
0.9482 0.0008
0.9492 0.0001
XGBoost
0.7967 0.0019 0.7998
0.0007
XGBoost
0.9436 0.0006
0.9452 + 0.0003
LightGBM
0.7956 0.0017
0.7968
1 0.0005
LightGBM
0.9422 1: 0.0009
0.9427 +: 0.0003
Boost
0.7985 1. 0.0018
1 0.0012
CatBoost
0.9453 1: 0.0008
0.9459
1. 0.0005
TabR
0.7983 + 0.0022
0.8023 0.0018
TabR
0.9487 + 0,0008
0.9500 + 0.0002
TabR:
0.8051 0.0023
TabR:
0.9475 + 0.0007
MNCA
0.7993 + 0.0019
0.8042 0.0013
MNCA
0.9488 + 0.0010
0.9505 105005+0000 0.0001
MNCA
0.8068 0.0021
0.8128
0.0007
MNCA
0.9493 0.0012
0.9501 + 0.0008
Tab
0.8066 + 0.0015
0.8075
4.0004 190755000 0.0004
TabM
0.9500 +: 0.0005
0.9505 + 0.0002
TabM
0.8080 + 0.0019
0.0017 0.8102
TabM
0.9503 + 0.0006
0.9501
+ 0.0002
G
0.8064 + 0.0018
TabM[
0.9496 + 0.0010
TabMmiul
0.8053
0.0012
0.8066
0.0001
TabMpini
0.9495 + 0.0005
0.9500 0.0002
TabM
0.8078 0.0008
0.8086 + 0.0005
TabM!
0.9490 0.0004
0.9492 0.0002
34

---

Published as a conference paper at ICLR 2025
nye-taxi-green-de--DD
uutittrr-uuai-220177
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
0.3951 0.0009 0.3921 0.0003
MLP
0.3759 0.0004
0.3729 + 0.0003
TabPFN
TabPFN
ResNet
0.3899 + 0.0016
0.3873 + 0.0009
ResNet
0.3743 + 0.0007
0.3718 + 0.0005
DCN2
0.3919 + 0.0009 0.3889 + 0.0003
DCN2
0.3738 + 0.0004 0.3759 + 0.0012
SNN
0.3933 1 0.0013 0.3899 1 0.0004
SNN
0.3790 0.0007
0.3744 +: 0.0002
Trompt
0.3979 nan
Trompt
0.3700 nan
AutoInt
0.4084 0.0256
0.3967 0.005
Autoint.
0.3723 0.0011
0.3692 + 0.0010
MLP-Mixee
0.3914 0.0026
0.3861 0.0013
MLP- Mixer
0.3741 0.0010
0.3698 + 0.0004
Excel'
0.3969 0.0036
0.3897 nan
Excel"
0.3699 + 0.0014
0.3652 + nan
SAINT
0.3905 + 0.0013
SAINT
0.3704 + 0.0014
FT-T
0.3937 + 0.0064
0:3889 0.0018
FT-T
0.3735 + 0.0012
0.3686 + 0.0004
T2G
0.3908 0.0045
0.3858
nan
T2G
0.3631 + nan 0.3676 0.0024
MLP1-lite
0.3812 + 0.0018
0.3761 + 0.0016
MLPi-lite
0.3665 + 0.0008
0.3642 + 0.0003
MLP
0.3795 0.0016
0.3733 0.0013
MLP
0.3657 0.0007 0.3629 0.0002
MLP
0.3680 T: 0.0006
0.3653
1 0.0005
MLP
0.3649 1: 0.0011
0.3637 1 0.0008
XGBoost
0.3792 + 0.0002
0.3787 0.0000
XGBoost
0.3641 + 0.0001
0.3640 + 0.0000
Lig
0.3688 0.0002
0.3684
0,0000
LightGBM
0.3637 0,0001
0.3635
+ 0.0000
CaoBo
0.3644 0.0005
0.3632 0.0003
CatBoost
0.3647 0.0004
0.3637 0.0002
TabR
0.3577 0.0222
0.3380
0.0027
TabR
0.3613 0.0005
0.3590
0.0002
TabR
0.3725 1 0.0091
TabR
0.3596 0.0004
MNCA
0.3728 0.0012
0.3720 0.0010
MNCA
0.3670 0.0004
0.3649 + 0.0002
MNCA
0.3536 + 0.0052
0.3407 + 0.0009
MNCA
0.3646 + 0.0001
0.3643 + 0.0000
TabM
0.3866 0.0006
0.3855 0.0003
TabM
0.3686 0.0006
0.3679 0.0003
TabM
0.3849
0.0005
0.3843 0.0002
TabM
0.3671 0.0007
0.3665 + 0.0002
TabM[G]
0.3848
0.0005
TabM[G]
0.3667 0.0009
mab
0.3853
0.0005
0.3845 + 0.0003
TabMain
0.3664 + 0.0006
0.3655 + 0.0002
TabM mini
0.3485 1 0.0038
0.3448
0.0020
TabM mini
0.3593 0.0004
0.3589 + 0.0000
road-safety f
year 1
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
0.7857 + 0.0019
0.7873 + 0.0004
MLP
8.9628 + 0.0232
8.8931 = 0.0066
TabPFN
0.7338 + 0.0032
TabPEN
ResNet
0.7875 0.0007
0.7898 0.0008
ResNet
8.9658 0.0239
8.8755 + 0.0066
DCN2
0.7781 0.0014
0.7823 0.0012
DCN2
9.2761 0.0401
9.0640
0.0156
SNN
0.78417 0.0010
0.7865
0.0002
SNN
9.0054 0.0256
8.9351 + 0.0073
Trompt
0.7804 + nan
Trompt
8.9707 + nan
AutoInt
0.7826 + 0.0030
0.7883 + 0.0011 0.0013
Autolnt
9.0430 + 0.0280
8.9619 + 0.0092
MLP-Mixee Mixer
0.7878 +:: 0.0032
0.7919 0.0015
MLP-M Mixer
8.9589 0.0182
8.9086 + 0.0177
Excel'
0.7864 + 0.0053
0.7907 nan
Excel"
9.0395 0.0266
8.9551 + nan
SAINT
0.7584 0.0584
SAINT
9.0248 0.0225
FT-T
0.7907 0.0012
0.7943 0.0007
FT-T
9.0005 0.0215
8.9360 + 0.0013
T2G
0.7912 0.0026
0.7961
nan
T2G
8.9775 0.0138
+ 8.8979
MLP
0.7867 0.0018
0 0.0002
MLP2-lite
8,9355 0.0103
8.9063 + 0.0030
MLP:
0.7853 + 0.0014 0.7881 + 0.0007
MLP:
8.9455 + 0.0173
8.9083 + 0.0046
MLP
0.7899 0.0009 0.7935 0.0003
MLP1
8.9379 0.0206
8.8753 0.0038
XGBoost
0.8101 0.0017 0.8129 0.0004
XGBoost
9.0307 0.0028
9.0245 + 0.0015
LightGBM
0.7982 0.0012
0.7996 + 0.0005
LightGBM
9.0200 1: 0.0025
9.0128 1. 0.0015
Boost
0.8012 1. 0.0009
0.8022 0.0002
CatBoost
9.0370 1 0.0073
9.0054
1: 0.0028
TabR
0.8403 + 0.0014
0.8441 0.0005
TabR
9.0069 + 0.0152
8.9132
+ 0.0088
TabR:
0.8374 0.0013
TabR:
8.9721 + 0.0105
MNCA
0.8080 + 0.0013
0.8121 + 0.0006
MNCA
8.9476 + 0.0152
8.8977 0.0037
MNCA
0.8232 0.0017
0.8287 0.0008
MNCA
8.8973 0.0082
8.8550 + 0.0031
Tab
0.7946 + 0.0013
0.7961
0.0005
TabM
8.8701 +: 0.0110
8,8517 + 0.0022
TabM
0.7958 + 0.0011
0.7968
0.0004
TabM
8.8705 + 0.0043
8.8642 + 0.0028
GabM[G]
0.7954 + 0.0016
TabM[
8.8723 + 0.0080
TabMmiul
0.7933
0.0030
0.7970 0.0006
TabMpini
8.9164 + 0.0089
8.9021 + 0.0036
TabM
0.7999 0.0023
0.8059 + 0.0012
TabM
8.8737 + 0.0119
8.8561 4: 0.0054
35

---

Published as a conference paper at ICLR 2025
Table 20: Extended results for TabReD Rubachev et al. (2024) benchmark. Results grouped by
datasets. One ensemble consists of five models trained independently under different random seeds.
sberbank sberbank-housing L
ecom-offers
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
0.2529 0.0078 0.2474 + 0.0052
MLP
0.5989 0.0017 0.5995 + 0.0011
TabPFN
TabPIN
ResNet
ResNet
DCN2
0.2616 + 0.0049 0.2506 + 0.0015
DCN2
0.5996 + 0.0043 0.6039 + 0.0028
SNN
0.2671 + 0.0140
0.2555 1 0.0033
SNN
0.5912 1 0.0056 0.5961 1 0.0033
Trompt
0.2509 + non
Trompt
0.5803 + nan
Autolnt
Autolnt
MLP-MMB Mixer
Excel'
0.2533 0.0046
0.2485 nan
Excel"
0.5759 0.0066
0.5759 nan
SAINT
0.2467 0.0019
SAINT
0.5812 0.0098
FT-T
0.2440 0.0038
0.2367 + 0,0010
FT-T
0.5775 + 0.0063
0.5817 + 0.0021
T2G
0.2416 11 0.0025
0.2343 1 TAn
T2G
0.5791 1 0.0056
0.5824 1 NMT
MLP4-lite
0.2528 0.0055
0.2503 0.0029
MLP4-lite
0.5800 0.0029
0.5819 + 0.0011
MLP!
0.2412 + 0.0031 0.2355 + 0.0006
MLP!
0.5846 + 0.0048 0.5872 + 0.0018
MLP
0.2383 0.0032 0.2327 0.0009
MLP
0.5949 +: 0.0013 0.5953 +: 0.0006
XGBoost
0.2419 = 0.0012
0.0000
XGBoost
0.5763 + 0.0072
0.5917 = 0.0035
LightGBM
0.2468 + 0.0009
0.2467 + 0.0002
LightGBM
0.5758 + 0.0006
0.5758 + 0.0003
CatBoost
0.2482 + 0.0034
0,0016
CatBoost
0.5596 0.0068
0.5067
+ 0.0011
TabR
0.2820 0.0323
0.2603 - 0.0048
TabR
0.5943 0.0019
0.5977
+ 0.0009
TabR
0.25-12 11: 0.0101
TabR
0.5762 1: 0.0052
MNCA
0.2593 + 0.0053 0,2520 + 0.0032
MNCA
0.5765 + 0.0087
0.5820 + 0.0047
MNCA
0.2448 0.0039
0.2404 + 0.0025
MNCA
0.5758 0.0050
0.5796 + 0.0009
TabM
0.2469 1 0.0035 0.2440 0.0026
TabM
0.5948 0.0006
0.5952 1 0.0004
TabM
0.2439 0.0021
0.2428 0.0006
TabM
0.5941 0.0003
0.5941 + 0.0000
GabM[G]
0.2436 0.0027
Ta
0.5970
0.0010
TabMmini
0.2433
0.0017
0.2422 0.0004
TabMmini
0.5942
0.0003
0.5943 0.0001
TabM mini
0.2334 + 0.0018 0.2324 + 0.0009
TabM mini mini
0.5910 + 0.0012 0.5913 + 0.0002
maps-routing Single model
Nomesite-insuranc
Method
Ensemble
Method
Single model
Ensemble
MLP
0.1625 0.0001 0.1621 + 0.0000
MLP
0.9506 0.0005 0.9514 1 0.0001
TabPFN
TabPFN
ResNet
ResNet
DCN2
0.1656 + 0.0004 0.1636 0.0001
DCN2
0.9398
0.0053 0.9432 + 0.0018 0.0013 0.9484 4: 0.0007 0.9588 nan
SNN
0.1634
0.0002 0.1625 0.0000
SNN
0.9473
Trompt
0.1624 non
Trompt
AutoInt
AutoInt.
MLP-Mixer
MLP-M Mixer
Excel'
0.1621 +: nan 0.1634 + nan
0.1628 + 0.0001
Excel
0.9622 + 0.0004 0.9635 + man
SAINT
SAINT
0.9613 + nan
FT-T
0.1625 0.0003
0.1619 0.0001
FT-T
0.9622 0.0006
0.9633 +: 0.0001
T2G
0.1616 0.0001
0.1608
nan
T2G
0.9624 0.0006
0.9637 1: nan
MLP*-lite
0.1618 + 0.0002
0.1613 + 0.0000
MLP
0.9609 + 0.0009
0.9626 + 0.0003
MLP
0.1613 0.1618 0.0002
0.0001
MLP:
0.9617 0.0004
0.9630 + 0.0002
MLP!
0.1620 11: 0.0002
0.1614 0.0000
MLP!
0.9582 1: 0.0014
0.9599 1 0.0002
XGBoost
0.1616 0.0001
0.0004
XGBoost
0.9601 + 0.0002
0.9602 + 0.0000
LightGBM
0.1618 0.0000
0.1616
LightGBM
0.9603 0.0002
0.9604 0.0001
Boost
0.1619 0.0001 0.0000
CatBoost
0.9606 0.0003
0.9609
+ 0.0001
TabR
0.1639 0.0003
0.1622
TabR
0.9487 + 0.0014
0.9505
+ 0.0001
TabR
0.1622 + 0.0002
TabR
0.9556 + 0.0021
MNCA
0.1625 0.0001
0. 0.0001
MNCA
0.9514 0.0038
0.9522 0.0027
0.1627 + 0.0002
0.1623
0.0001
0.9620 + 0.0006
0.9635 + 0.0002
TabM
0.1612 4: 0.0001
+ 0.0000
TabM
0.9641 + 0.0004
0.9641 + 0.0003
TabM
0.1612 + 0.0001
0.1610 0.0001
TabM
0.9640 + 0.0002
0.9642 + 0.0001
Tab
0.1611 0.0001
TabM[G]
0.9641 + 0.0003
TabM
0.1612 1 0.0001
0.1610
TabM
0.9643 1 0.0003
0.9645 0.0001
TabM mini
0.1610 + 0.0001
0.1609 + 0.0000
T mini
0.9631 + 0.0003
0.9634 + 0.0001
36

---

Published as a conference paper at ICLR 2025
cooking-time
homecredit-defalt
Method
Single model
Ensemble
Method
Single model
Ensemble
MLP
0.4828 0.0002 0.4822 0.0000
MLP
0.8538 0.0014 0.8566 + 0.0005
TabPFN
TabPFN
ResNet
ResNet
DCN2
0.4834 + 0.0003 0.4822 + 0.0001
DCN2
0.8549 + 0.0002 0.8541 0.0016 0.8569 1 0.0010
0.8471 + 0.0019
SNN
0.4835 0.0006
0.4818 0.0002
SNN
Trompt
0.4809 nan
Trompt
0.8355 nan
AutoInt
Autoint
MLP-Mixer
MLP- ALP--Mixer
Excel'
0,4821 + 0.0005 0.4808 + nan
Excel"
0.8513 + 0.0024 0.8564 +
SAINT
0.48440 + nan
SAINT
0.8377 + nan
FT-T
0.4820 + 0.0008
0.4813 + 0.0005
FT-T
0.8571 + 0..023
0.8611 + 0.0013
T2G
0.4809 0.0008
0.4797 nan
T2G
0.8597 0.0007
0.8629 + nan
MLP1-lite
0.4811 0.0004
0.4805 + 0.0001
MLPi-lite
0.8598 + 0.0009
0.8607 = 0.0003
MLP
0.4809 0.0006
0.4804 0.0003
MLP
0.8572 0.0011
0.8590 0.0003
MLP
0,4812 1: 0.0004
0.4807 1: 0.0002
MLP
0.8568 1: 0.0039
0.8614 1 0.0014
XGBoost
0.4823
0.0001
0.4821
0.0000
XGBoost
0.8670 + 0.0005
0.8674 + 0.0001
Lig
0.4826
0.0001
0.4825
0,0001
LightGBM
0.8664 0.0004
0.8667 + 0.0000
CaoBoost
0.4823
0.0001
0.4820 0.0001
CatBoost
0.8627 YEATI
TabR
0.4828
0.0008
0.4814 0.0004
TabR
0.8501 0.0027
0.8548
0.0003
TabR
0.4818 +: 0.0006
TabR*
0.8541 0.0021
MNCA
0.4825 0.0004
0,4819
0.0003
MNCA
0.8531 0.0018
0.8569 + 0.0004
MNCA
0.4818 + 0.0005
0.4809
0.0003
MNCA
0.8541 +: 0.0033
0.8606 + 0.0024
TabM
0.4803 0.0006
0.4797 0.0003
TabM
0.8583 0,0010
0.8599 0.0006
TabM
0.4804 0.0002
0.4802
0.0000
TabM
0.8599 0.0010
0.8607 + 0.0002
GabM[G]
0.4800
0.0002
TabM[G]
0.8588 0.0013
mab
0.4803
0.0001
0.4801 + 0.0001
TabMain
0.8605 + 0.0010
0.8614 + 0.0007 0.0007
TabM mini
0.4804 0.0001
0.4803
0.0000
TabM mini
0.8635 0.0008
0.8646 1 0.0004
delivery-eta J Single model
weather + Single model
Method
Ensemble
Method
Ensemble
MLP
0.5493 + 0.0007 0.5478 + 0.0006
MLP
1.5378 + 0.0054 1.5111 + 0.0029
TabPFN
TabPEN
ResNet
ResNet
DCN2
0.5516 0.0014
0.5495 0.0004
DCN2
1.5606 0.0057
1.5292 + 0.0028 1.5013 + 0.0034
SNN
0.0008 0.5519 + nan
0.5495
0.5479 0.0001
SNN
0.0085 1.5187 + nan
1.5280
Trompt
Trompt
AutoInt
Autoint
MLP-Min Mixer
MLP-Mise -Mxee
Excel'
0.5552 0.0030
0.5524 nan
Excel"
1.5131 0.0022
1.4707 + nan
SAINT
0.5528 nan
SAINT
1.5097 0.0045
FT-T
0.5542
0.0026
0.5523 0.0018
FT-T
1.5104 + 0.0097
1.4719 + 0.0040
T2G
0.5527 0.0016
0.5512
T2G
1.4849 0.0087
1.4513 +
MLP
0.5521 0.0014
0.5512 0.0005
MLP2-lite
1.5170 0.0040
1.4953 + 0.0023
MLP!
0.5535 + 0.0019
0.5526 0.0009
MLP:
1.5139 +: 0.0031
1.4978 + 0.0020
MLP
0.5521 0.0019
0.5511 0.0007
MLPt
1.5162 0.0020
1.5066 0.0008
XGBoost
0.5468 0.0002
+
XGBoost
1.4671 + 0.0006
1.4629
+ 0.0002
LightGBM
0.5468 0.0001
0.5465
0.0000
LightGBM
1.4625 1 0.0008
1.4581 1 0.0003
Boost
0.5465 1. 0.0001
4 0.0000
CatBoost
1.4688 1: 0.0019
TabR
0.5514 + 0.0024
0.5480 0.0005
TabR
1.4666 + 0.0039
1.4547 + 0.0008
TabR:
0.5520 0.0015
TabR:
1.4458 + 0.0018
MNCA
0.5498 + 0.0007
0.5488 0.000
MNCA
1.5062 + 0.0054
1.4822 + 0.0013
MNCA
0.5494 0.0006 0.5507 0.0013
MNCA
1.5008 0.0034
1.4782 + 0.0011
Tab
0.5510 +: 0.0015
1.0000000000
TabM
1.4786 +: 0.0039
1.4715 + 0.0020
TabM
0.5494 + 0.0004
0.5492 0.0001
TabM
1.4722 + 0.0024
1.4675 + 0.0009
G
0.5509 + 0.0003
TabM[G]
1.4728 + 0.0022
TabMmiul
0.5497
0.0007
0.5495
0.0003
TabMpini
1.4716 0.0016
1.4669 + 0.0010
TabM
0.5510 0.0019
0.5502 + 0.0000
TabM
1.4651 +: 0.0020
1.4581 + 0.0016
37


Reviews:
[{"review_id": "GXHewfO2S6", "reviewer": "Reviewer_CrVx", "summary": "The authors propose TabM, a variation of an MLP with $k$ heads that uses a shared backbone, and only certain aspects of the architecture in the beginning  and in the end are specialized. The heads are trained independently. The work is compared with prior DL methods (plain and attention architectures), retrieval-based methods, and tree-based methods on a diverse collection of classification and regression tasks. Based on the results, the proposed method manages to outperform all baselines with tuned hyperparameters.", "questions": "- Table 2, for the MLP (Maps Routing dataset) the time unit is missing. I am guessing it is 10 minutes.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method achieves better performance compared to the other methods with tuned hyperparameters. \n- The set of considered baselines is extensive.\n- Extensive results on classification/regression datasets. Additionally, results are provided on well-known benchmarks in the domain.\n- The authors provide extensive ablations of the different components of the method. The authors additionally provide a time comparison of the different methods.", "weaknesses": "- The work is difficult to read, the captions of Figure 2 and Figure 3 are long and blend with the core manuscript text, making it hard to keep track of the manuscript flow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7w4U1ukAL0", "reviewer": "Reviewer_4UNA", "summary": "This paper come up with an efficient ensembling method for tabular deep learning — which integrate BatchEnsemble with multilayer perceptron. This method exhibits great improvements in their benchmark.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- [Major] This method is simple yet effective.\n- [Major] The paper is easy to read.", "weaknesses": "- [Medium] The proposed method is not much novel. Note that this concern is not the main reason that I chose to reject this paper. If other concerns listed can be addressed (e.g., more technical contributions, comprehensive empirical study for depper understandings), I think this paper is still very useful.\n- [Major] More experiments are needed to fully understand the behavior of TabM. In the abstract, the authors mentioned “we show that parameter-efficient ensembling is not an arbitrary trick, but rather a highly effective way to reduce overfitting and improve optimization dynamics of tabular MLPs.” However, in the paper, there is no discussion why TabM helps with reducing overfitting and optimization dynamics. Of course the experiments proves this, but it seems like a general benefits of ensemble models. At least, some explanation supported by experiments is needed to fully understand this method — especially this method is already very simple and not much novel, understanding this better is important to have sufficient technical contribution.\n- [Major] Baselines missing. This method is an ensemble method. At least we could compare it to other naive ensemble method. For instance, how much speed up it provides compared to normal ensemble; how robust it compared to normal ensemble; etc.\n- [Major] Datasets missing. Yes this paper test on many datasets — but the more important thing is the diversity. For instance, how about high-dimensional datasets and large datasets (I know it is on Table 2, but only evaluated on two datasets and three baselines, moreover, only TabMmini has been evaluated). I suggest the authors to use TabZilla hard benchmark, and then evaluate the method’s capability under different circumstances. These kind of experiments can make the findings more interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eKQOGYTuzH", "reviewer": "Reviewer_For5", "summary": "This paper proposes TabM, a deep learning model for tabular data based on MLP enhanced with BatchEnsemble, along with custom modifications designed to improve the model's efficiency and performance. The authors position TabM as an efficient alternative to more complex architectures like Transformer-based models, claiming it achieves superior performance without their added computational cost. The paper also provides a detailed analysis, suggesting that parameter-efficient ensembling can effectively address overfitting and improve optimization dynamics in tabular MLPs.", "questions": "1. In Table 2 (p.10), the last column of the MLP row lists \"10.\" Could the authors confirm if it is measured in minutes, as there’s no unit mentioned?\n2. In line 1027, the authors state, *For numerical features, by default, we used a slightly modified version of the quantile normalization from the Scikit-learn package, with rare exceptions when it turned out to be detrimental.* Could the authors clarify how they define “detrimental” in this context and provide further explanation?\n3. Could the authors provide more detailed diversity metrics for the predictions of the 32 base models?\n4. Line 431 notes, *Interestingly, the best prediction head of TabM performs no better than the plain MLP.* Would an ensemble of 32 independently and undertrained MLPs yield similarly strong performance?\n5. The ranking method used is unconventional. Is this ranking method novel to this paper? Could the authors also provide results for a more traditional average rank metric to facilitate comparison, as it would be interesting to see how these two measures differ?\n6. One advantage of using deep learning for tabular data is the ability to generate embeddings that are useful for downstream tasks and multimodal learning. How does TabM determine the representation of each sample? Is it based on concatenation, averaging across base models, or some other method? Additionally, could the authors provide a comparison of TabM’s embeddings with those from other deep learning approaches on specific tasks to showcase their effectiveness?\n\n---\nLastly, if the authors can address the concerns noted in the weaknesses and questions, I will consider raising my score accordingly.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-structured and clearly written, making the methods, results, and analysis easy to understand.\n2. TabM demonstrates competitive performance, positioning it as an efficient alternative for tabular data modeling.\n3. By introducing BatchEnsemble techniques into tabular data modeling, the authors have adapted and enhanced a previously underutilized approach in this domain.\n4. The paper provides a thorough analysis of each component's impact on model performance, contributing valuable insights.", "weaknesses": "1. Although the authors mention similar methods, a more detailed comparison with other related approaches for tabular data, such as Trompt, would be helpful. Despite structural differences, both methods share commonalities in prediction, such as averaging multiple head (cycle) outputs and summing losses across heads (cycles). \n2. Comparisons with standard ensemble methods are insufficient. Incorporating results from methods like model soup could help position TabM's performance.\n3. Dataset scope is somewhat limited, particularly for classification tasks. For a more comprehensive evaluation, the authors might consider incorporating the datasets from Tabzilla, which could provide richer classification benchmarks and enable a more detailed analysis, such as comparing performance across binary, multiclass, and regression tasks or evaluating model performance on datasets of varying sizes.\n\n[1] Kuan-Yu Chen, Ping-Han Chiang, Hsin-Rung Chou, Ting-Wei Chen, Tien-Hao Chang: Trompt: Towards a Better Deep Neural Network for Tabular Data. In ICML\n\n[2] Duncan C. McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C., Ganesh Ramakrishnan, Micah Goldblum, Colin White: When Do Neural Nets Outperform Boosted Trees on Tabular Data? In NeurIPS", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "89EnZUCQv6", "reviewer": "Reviewer_aGcY", "summary": "The paper applies the BatchEnsemble technique to MLPs for tabular data, and investigates several modifications. The results show an improvement over MLPs and several deep baseline models and GBRT on a broach benchmark of 50 datasets from the literature. The experiments show that in the first adapter layer in particular is extremely critical, and results in the majority of gains.", "questions": "- Are number of layers for MLP and TabM tuned as hyper-parameters? What are typical values for each?\n\n- Why is k not included in the hyper-parameter search for TabM?\n\n- How do you ensure diversity for the MLPs in the ensemble in Figure 6?\n\n- How do you initialize the last layers in TabM to ensure diversity?\n\n- Is it possible that TabM_mini outperforms the real ensemble because of more diversity in the initialization?\n\n- Is it correct that when doing the full ensemble, adding an adapter into each model would lead to an equivalent ensemble? I was wondering why this ablation was not performed, but I assume random initialization of weights for each member of the ensemble makes the adapters irrelevant.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper discusses the adoption of the BatchNorm architecture for the tabular setting, and describes several interesting ablations.\nThe empirical evaluation is broad and in-depth and well presented.\nThe empirical results are quite strong against a broad variety of baselines.\nThe paper is well written.", "weaknesses": "- The technical novelty of the paper is somewhat small; however, this is made up for by the in-depth analysis with somewhat surprising results and the good performance of the proposed model.\n- The paper doesn't describe the relationship to dropout, even though dropout was original describes as an efficient ensemble approach. Given the simple nature of TabM_mini, there seem to be some obvious parallels between drop-out and TabM_mini that I think are worth discussing. In particular, TabM_mini actually contains a dropout layer, and it would be interesting to evaluate how the two forms of ensembling complement each other.\n- The comparison doesn't include TabPFN, an extremely strong baseline. McElfresh showed that even subsampling data to at most 3000 datapoints, TabPFN outperforms most other models.\n- Given the focus on improving MLP performance, a comparison with regularization cocktails and \"Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data\" might be relevant but not essential.\n- Given the somewhat surprising result about the importance of the initial adapter, it would be great to have more ablations on the different components. In particular, it's unclear in how far the initial adapter and last layer are coupled. How does the performance change if the best performing last layer is used for all the initial adapters? I.e. are the initial layer and last layer co-adapted or do they work independently? \n\n## Minor comments\nFigure 2 cuts off points at 8%. It seems there's a lot of points on exactly that line, which seems a bit suspicious. Are these all the outliers that are clipped to the limits of the figure? It would be great to indicate outliers in the plot.\n\nTable 2 is far below the mention and maybe should be moved up.\n\nTable 2, MLP on the Maps dataset is missing units in duration.\n\nThe title of section 5.3 should read \"How does the performance of TabM depend on k?\"\n\nLine 257: \"requires a special care\" should be \"requires special care\"\n\nLine 160: multiplication with 100% is not mathematically meaningful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)