TabM: Advancing tabular deep learning with parameter-efficient ensembling

## Abstract

Abstract Deep learning architectures for supervised learning on tabular data range from simple multilayer perceptrons (MLP) to sophisticated Transformers and retrieval-augmented methods.
This study highlights a major, yet so far overlooked opportunity for substantially improving tabular MLPs: namely, parameter-efficient ensembling
— a paradigm for implementing an ensemble of models as one model producing multiple predictions.
We start by developing TabM — a simple model based on MLP and our variations of BatchEnsemble (an existing technique).
Then, we perform a large-scale evaluation of tabular DL architectures on public benchmarks in terms of both task performance and efficiency, which renders the landscape of tabular DL in a new light.
Generally, we show that MLPs, including TabM , form a line of stronger and more practical models compared to attention- and retrieval-based architectures.
In particular, we find that TabM demonstrates the best performance among tabular DL models.
Lastly, we conduct an empirical analysis on the ensemble-like nature of TabM .
For example, we observe that the multiple predictions of TabM are weak individually, but powerful collectively.
Overall, our work brings an impactful technique to tabular DL, analyses its behaviour, and advances the performance-efficiency trade-off with TabM — a simple and powerful baseline for researchers and practitioners.
The code is available at: https://github.com/yandex-research/tabm .

## 1 Introduction

Supervised learning on tabular data is a ubiquitous machine learning (ML) scenario in a wide range of industrial applications.
Among classic non-deep-learning methods, the state-of-the-art solution for such tasks is gradient-boosted decision trees (GBDT) (Prokhorenkova et al., ; Chen & Guestrin, ; Ke et al., ).
Deep learning (DL) models for tabular data, in turn, are reportedly improving, and the most recent works claim to perform on par or even outperform GBDT on academic benchmarks (Hollmann et al., ; Chen et al., ; ; Gorishniy et al., ).

However, from the practical perspective, it is unclear if tabular DL offers any obvious go-to baselines beyond simple architectures in the spirit of a multilayer perceptron (MLP).
First, the scale and consistency of performance improvements of new methods w.r.t. simple MLP-like baselines are not always explicitly analyzed in the literature.
Thus, one has to infer those statistics from numerous per-dataset performance scores, which makes it hard to reason about the progress.
At the same time, due to the extreme diversity of tabular datasets, consistency is an especially valuable and hard-to-achieve property for a hypothetical go-to baseline.
Second, efficiency-related properties, such as training time, and especially inference throughput, sometimes receive less attention.
While methods are usually equally affordable on small-to-medium datasets (e.g. $<$100K objects), their applicability to larger datasets remains uncertain.
Third, some recent work generally suggests that the progress on academic benchmarks may not transfer that well to real-world tasks (Rubachev et al., ).
With all the above in mind, in this work, we thoroughly evaluate existing tabular DL methods and find that non-MLP models do not yet offer a convincing replacement for MLPs.

At the same time, we identify a previously overlooked path towards more powerful, reliable and reasonably efficient tabular DL models.
In a nutshell, we find that the parameter-efficient approach to deep ensembling, where most weights are shared between ensemble members, allows making simple and strong tabular models out of plain MLPs.
For example, MLP coupled with BatchEnsemble (Wen et al., ) — a long-existing method — right away outperforms popular attention-based models, such as FT-Transformer (Gorishniy et al., ), while being simpler and more efficient.
This result alone suggests that the parameter-efficient ensembling is a low-hanging fruit for tabular DL.

Our work builds on the above observations, and offers TabM — a new powerful and practical model for researchers and practitioners.
Drawing an informal parallel with GBDT (an ensemble of decision trees), TabM can also be viewed as a simple base model (MLP) combined with an ensembling-like technique, providing high performance and simple implementation at the same time.

Main contributions.
We summarize our main contributions as follows:

- 1.
We present TabM — a simple DL architecture for supervised learning on tabular data.
TabM is based on MLP and parameter-efficient ensembling techniques closely related to BatchEnsemble (Wen et al., ).
In particular, TabM produces Multiple predictions per object.
TabM easily competes with GBDT and outperforms prior tabular DL models, while being more efficient than attention- and retrieval-based DL architectures.
- 2.
We provide a fresh perspective on tabular DL models in a large-scale evaluation along four dimensions: performance ranks, performance score distributions, training time and inference throughput.
One of our findings is that MLPs, including TabM, hit an appealing performance-efficiency tradeoff, which is not the case for attention- and retrieval-based models.
- 3.
Empirically, we show that the multiple predictions of TabM are weak and overfitted individually, while their average is strong and generalizable.
The training gradients of TabM, in turn, can be viewed as an “ensemble” of diverse gradients coming from the multiple predictions.

## 2 Related work

Decision-tree-based models.
Gradient-boosted decision trees (GBDT) (Chen & Guestrin, ; Ke et al., ; Prokhorenkova et al., ) is a strong and efficient baseline for tabular tasks.
GBDT is a classic machine learning model, specifically, an ensemble of decision trees.
Our model TabM is a deep learning model, specifically, a parameter-efficient ensemble of MLPs.

Tabular deep learning architectures.
A large number of deep learning architectures for tabular data has been proposed over the recent years.
That includes attention-based architectures (Song et al., ; Gorishniy et al., ; Somepalli et al., ; Kossen et al., ; Yan et al., ), retrieval-augmented architectures (Somepalli et al., ; Kossen et al., ; Gorishniy et al., ; Ye et al., ), MLP-like models (Gorishniy et al., ; Klambauer et al., ; Wang et al., ) and others (Arik & Pfister, ; Popov et al., ; Chen et al., ; Marton et al., ; Hollmann et al., ).
Compared to prior work, the key difference of our model TabM is its computation flow, where one TabM imitates an ensemble of MLPs by producing multiple independently trained predictions.
Prior attempts to bring ensemble-like elements to tabular DL (Badirli et al., ; Popov et al., ) were not found promising (Gorishniy et al., ).
Also, being a simple feed-forward MLP-based model, TabM is significantly more efficient than some of the prior work.
Compared to attention-based models, TabM does not suffer from quadratic computational complexity w.r.t. the dataset dimensions.
Compared to retrieval-based models, TabM is easily applicable to large datasets.

Improving tabular MLP-like models.
Multiple recent studies achieved competitive performance with MLP-like architectures on tabular tasks by applying architectural modifications (Gorishniy et al., ), regularizations (Kadra et al., ; Jeffares et al., ; Holzmüller et al., ), custom training techniques (Bahri et al., ; Rubachev et al., ).
Thus, it seems that tabular MLPs have good potential, but one has to deal with overfitting and optimization issues to reveal that potential.
Our model TabM achieves high performance with MLP in a different way, namely, by using it as the base backbone in a parameter-efficient ensemble in the spirit of BatchEsnsemble (Wen et al., ).
Our approach is orthogonal to the aforementioned training techniques and architectural advances.

Deep ensembles.
In this paper, by a deep ensemble, we imply multiple DL models of the same architecture trained independently (Jeffares et al., ) for the same task under different random seeds (i.e. with different initializations, training batch sequences, etc.).
The prediction of a deep ensemble is the mean prediction of its members.
Deep ensembles often significantly outperform single DL models of the same architecture (Fort et al., ), and can excel in other tasks like uncertainty estimation or out-of-distribution detection (Lakshminarayanan et al., ).
It was observed that individual members of deep ensembles can learn to extract diverse information from the input, and the power of deep ensembles depends on this diversity (Allen-Zhu & Li, ).
The main drawback of deep ensembles is the cost and inconvenience of training and using multiple models.

Parameter-efficient deep “ensembles”.
To achieve the performance of deep ensembles at a lower cost, multiple studies proposed architectures that imitate ensembles by producing multiple predictions with one model (Lee et al., ; Zhang et al., ; Wen et al., ; Havasi et al., ; Antorán et al., ; Turkoglu et al., ).
Such models can be viewed as “ensembles” where the implicit ensemble members share a large amount of their weights.
There are also non-architectural approaches to efficient ensembling, e.g. FGE (Garipov et al., ), but we do not explore them, because we are interested specifically in architectural techniques.
In this paper, we highlight parameter-efficient ensembling as an impactful paradigm for tabular DL.
In particular, we describe two simple variations of BatchEnsemble (Wen et al., ) that are highly effective for tabular MLPs.
One variation uses a more efficient parametrization, and another one uses an improved initialization.

## 3 TabM

In this section, we present TabM — a Tabular DL model that makes Multiple predictions.

### 3.1 Preliminaries

Notation.
We consider classification and regression tasks on tabular data.
$x$ and $y$ denote the features and a label, respectively, of one object from a given dataset.
A machine learning model takes $x$ as input and produces $\hat{y}$ as a prediction of $y$.
$N\in\mathbb{N}$ and $d\in\mathbb{N}$ respectively denote the “depth” (e.g. the number of blocks) and “width” (e.g. the size of the latent representation) of a given neural network.
$d_{y}\in\mathbb{N}$ is the output representation size (e.g. $d_{y}=1$ for regression tasks, and $d_{y}$ equals the number of classes for classification tasks).

Datasets.
Our benchmark consists of 46 publicly available datasets used in prior work, including Grinsztajn et al. (); Gorishniy et al. (); Rubachev et al. ().
The main properties of our benchmark are summarized in Table 1, and more details are provided in Appendix C.

**Table 1: The overview of our benchmark. The “Split type” property is explained in the text.**
| #Datasets | Train size | #Features | Task type | Split type |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Min. | Q50 | Mean | Max. | Min. | Q50 | Mean | Max. | #Regr. | #Classif. | Random | Domain-aware |
| 46 | 1.8K | 12K | 76K | 723K | 3 | 20 | 108 | 986 | 28 | 18 | 37 | 9 |

Domain-aware splits.
We pay extra attention to datasets with what we call “domain-aware” splits, including the eight datasets from Rubachev et al. () and the Microsoft dataset (Qin & Liu, ).
For these datasets, their original real-world splits are available, for example, time-aware splits as in Rubachev et al. ().
Such datasets were shown to be challenging for some methods, because they naturally exhibit a certain degree of distribution shift between training and test parts (Rubachev et al., ).
The random splits of the remaining 37 datasets are inherited from prior work.

Experiment setup.
We use the setup from Gorishniy et al. (), and describe it in detail in subsection D.2.
Most importantly, on each dataset, a given model undergoes hyperparameter tuning on the validation set, then the tuned model is trained from scratch under multiple random seeds, and the test metric averaged over the random seeds becomes the final score of the model on the dataset.

Metrics.
We use RMSE (the root mean square error) for regression tasks, and accuracy or ROC-AUC for classification tasks depending on the dataset source.
See subsection D.3 for details.

Also, throughout the paper, we often use the relative performance of models w.r.t. MLP as the key metric.
This metric gives a unified perspective on all tasks and allows reasoning about the scale of improvements w.r.t. to a simple baseline (MLP).
Formally, on a given dataset, the metric is defined as $\left(\frac{\text{score}}{\text{baseline}}-1\right)\cdot 100\%$, where “score” is the metric of a given model, and “baseline” is the metric of MLP.
In this computation, for regression tasks, we convert the raw metrics from RMSE to $R^{2}$ to better align the scales of classification and regression metrics.

### 3.2 A quick introduction to BatchEnsemble.

For a given architecture, let’s consider any linear layer $l$ in it: $l(x)=Wx+b$, where $x\in\mathbb{R}^{d_{1}}$, $W\in\mathbb{R}^{d_{2}\times d_{1}}$, $b\in\mathbb{R}^{d_{2}}$.
To simplify the notation, let $d_{1}=d_{2}=d$.
In a traditional deep ensemble, the $i$-th member has its own set of weights $W_{i},b_{i}$ for this linear layer: $l_{i}(x_{i})=W_{i}x_{i}+b_{i}$, where $x_{i}$ is the object representation within the $i$-th member.
By contrast, in BatchEnsemble, this linear layer is either (1) fully shared between all members, or (2) mostly shared: $l_{i}(x_{i})=s_{i}\odot(W(r_{i}\odot x_{i}))+b_{i}$, where $\odot$ is the elementwise multiplication, $W\in\mathbb{R}^{d\times d}$ is shared between all members, and $r_{i},s_{i},b_{i}\in\mathbb{R}^{d}$ are not shared between the members.
This is equivalent to defining the $i$-th weight matrix as $W_{i}=W\odot(r_{i}s_{i}^{T})$.
To ensure diversity of the ensemble members, $r_{i}$ and $s_{i}$ of all members are initialized randomly with $\pm 1$.
All other layers are fully shared between the members of BatchEnsemble.

The described parametrization allows packing all ensemble members in one model that simultaneously takes $k$ copies of the object as input, and applies all $k$ implicit members in parallel, without explicitly materializing each member.
This is achieved by replacing one or more linear layers of the original neural network with their BatchEnsemble versions:
$l_{\text{BE}}(X)=((X\odot R)W)\odot S+B$,
where $X\in\mathbb{R}^{k\times d}$ stores $k$ representations of the same input object (one per member), and $R,S,B\in\mathbb{R}^{d}$ store the non-shared weights ($r_{i}$, $s_{i}$, $b_{i}$) of the members, as shown at the lower left part of Figure 1.

Overhead to the model size.
With BatchEnsemble, adding a new ensemble member means adding only one row to each of the matrices $R$, $S$ and $B$, which results in $3d$ new parameters per layer.
For typical values of $d$, this is a negligible overhead to the original layer size $d^{2}+d$.
Overhead to the runtime.
Thanks to the modern hardware, the large number of shared weights and the parallel execution of the $k$ forward passes, the runtime overhead of BatchEnsemble can be (significantly) lower than $\times k$ (Wen et al., ).
Intuitively, if the original workload underutilizes the hardware, there are more chances to pay less than $\times k$ overhead.

Terminology.
In this paper, we call $r_{i}$, $s_{i}$, $b_{i}$, $R$, $S$ and $B$ adapters, and the implicit members of parameter-efficient emsembles (e.g. BatchEnsemble) — implicit submodels or simply submodels.

### 3.3 TabM mini subscript TabM mini \mbox{\text{TabM}}_{\text{mini}} TabM start_POSTSUBSCRIPT mini end_POSTSUBSCRIPT & TabM

Our models $\mbox{\text{TabM}}_{\text{mini}}$ and TabM are based on a multilayer perceptron (MLP) and parameter-efficient ensembling methods, with a strong connection to BatchEnsemble (Wen et al., ), introduced in subsection 3.2.
In subsection A.1, we explain that we use specifically BatchEnsemble as the baseline efficient ensembling method because of its good balance between performance and ease of use, while using MLP as the base model is crucial because of its excellent efficiency.
We obtain our models in several steps, starting from essential baselines.
We always use the ensemble size $k=32$ and analyze this hyperparameter in subsection 5.3.

MLP.
We define MLP as a sequence of $N$ simple blocks followed by a linear prediction head:
$\text{MLP}(x)=\text{Linear}(\text{Block}_{N}(\ldots(\text{Block}_{1}(x)))$, where $\text{Block}_{i}(x)=\text{Dropout}(\text{ReLU}(\text{Linear}((x)))$.

$\text{MLP}^{\times k}$ = MLP + Deep Ensemble.
We denote the traditional deep ensemble of $k$ independently trained MLPs as $\text{MLP}^{\times k}$.
This method is illustrated in Figure 1, and its performance is reported in Figure 2 (the hyperparameter tuning is performed for one MLP, after which the tuned MLP is ensembled).
Interestingly, the results are already better and more stable than those of FT-Transformer (Gorishniy et al., ) — the popular attention-based baseline.
And, given the significantly better efficiency of MLPs (as will be shown later), $\text{MLP}^{\times k}$ may actually be no less practical than FT-Transformer, especially with additional techniques like Packed-Ensembles (Laurent et al., ).
That said, we continue towards more efficient approaches.

TabMnaive = MLP + BatchEnsemble.
Now, instead of the deep ensemble, we naively apply BatchEnsemble to the backbone of MLP, while keeping the prediction heads separate.
This gives us TabMnaive — a preliminary suboptimal version of TabM.
In fact, the architecture (but not the initialization) of TabMnaive is already equivalent to that of TabM, so Figure 1 is applicable.
The performance of TabMnaive shown in Figure 2 is important for two reasons.
First, TabMnaive — the efficient version of $\text{MLP}^{\times k}$ — is noticeably better than $\text{MLP}^{\times k}$ itself, which is intriguing.
We are not aware of similar results for BatchEnsemble in general, and share some thoughts on this phenomenon in subsection A.2.
Second, TabMnaive right away outperforms FT-Transformer, which demonstrates the great potential of parameter-efficient ensembling for MLPs.
This motivates further exploration.

Figure: Figure 1: (Upper left) A template for implementing an ensemble of $k$ MLPs. The remaining parts of the figure are three different parametrizations of the $k$ MLP backbones, all described in subsection 3.3. In all cases, each of the $k$ MLP backbones independently processes its own copy of the input object. (Upper right) $\text{MLP}^{\times k}$ is a traditional deep ensemble of $k$ fully independent MLPs. (Lower left) TabM is obtained by injecting three non-shared adapters $R$, $S$, $B$ in each of the $N$ linear layers of one MLP (^∗ the initialization differs from Wen et al. ()). (Lower right) $\mbox{\text{TabM}}_{\text{mini}}$ is obtained by keeping only the very first adapter $R$ of TabM and removing the remaining $3N-1$ adapters. Thus, $\mbox{\text{TabM}}_{\text{mini}}$ applies the same shared MLP to $k$ object representations, with only two non-shared elements ensuring diversity of predictions: the randomly initialized multiplicative adapter $R$ and the $k$ prediction heads. (Details) Input transformations such as one-hot-encoding, feature embeddings (Gorishniy et al., ) and others are omitted for simplicity. In practice, they are applied (and the result is flattened) before the Clone module. Drop denotes dropout (Srivastava et al., ).
Refer to caption: x1.png

Figure: Figure 2: The performance of models described in subsection 3.3 on 46 datasets from Table 1; plus several baselines on the left. For a given model, one dot on a jitter plot describes the performance score on one of the 46 datasets. The box plots describe the percentiles of the jitter plots: the boxes describe the 25th, 50th and 75th percentiles, and the whiskers describe the 10th and 90th percentiles. Outliers are clipped. The numbers at the bottom are the mean and standard deviations over the jitter plots. For each model, hyperparameters are tuned. “$\text{Model}^{\times k}$” denotes an ensemble of $k$ models.
Refer to caption: x2.png

$\mbox{\text{TabM}}_{\text{mini}}$ = MLP + MiniEnsemble.
By construction, the just discussed TabMnaive (illustrated as “TabM” in Figure 1) has $3N$ adapters: $R$, $S$ and $B$ in each of the $N$ blocks.
Among the $3N$ adapters, the first adapter $R$ in the very first linear layer is responsible for transforming the $k$ equal copies of the input (packed as $X$ in Figure 1) to $k$ different representations before the tabular features are mixed with $@W$ for the first time.
A simple experiment reveals that this adapter is critical.
First, we remove it from TabMnaive and keep the remaining $3N-1$ adapters untouched, which gives us TabMbad with worse performance, as shown in Figure 2.
Then, we do the opposite: we keep only the very first adapter of TabMnaive and remove the remaining $3N-1$ adapters, which gives us $\mbox{\text{TabM}}_{\text{mini}}$ — the minimal version of TabM.
$\mbox{\text{TabM}}_{\text{mini}}$ is illustrated in Figure 1, where we call the described approach “MiniEnsemble”.
Perhaps, surprisingly, but Figure 2 shows that $\mbox{\text{TabM}}_{\text{mini}}$ performs better than TabMnaive, despite having only one adapter instead of $3N$ adapters.

TabM = MLP + BatchEnsemble + Better initialization.
The just obtained results motivate the next step.
We go back to the architecture of TabMnaive with all $3N$ adapters, but initialize all multiplicative adapters $R$ and $S$, except for the very first one, deterministically with $1$.
As such, at initialization, the deterministically initialized adapters have no effect, and the model behaves like $\mbox{\text{TabM}}_{\text{mini}}$, but these adapters are free to add more expressivity during training.
This gives us TabM, illustrated in Figure 1.
Figure 2 shows that TabM is the best variation so far.

$\mbox{\text{TabM}}_{\text{mini}}^{\dagger}$ & $\mbox{\text{TabM}}^{\dagger}$.
Non-linear feature embeddings (Gorishniy et al., ) are known to boost the performance of many tabular models, especially of MLPs.
We denote $\mbox{\text{TabM}}_{\text{mini}}$ and TabM with non-linear feature embeddings as $\mbox{\text{TabM}}_{\text{mini}}^{\dagger}$ and $\mbox{\text{TabM}}^{\dagger}$, respectively.
By default, we recommend using the piecewise-linear embeddings (Gorishniy et al., ).
In subsection A.3, we provide additional implementation details, such as slightly different initialization.
Figure 2 shows that, $\mbox{\text{TabM}}_{\text{mini}}^{\dagger}$ is competitive with $\mbox{\text{TabM}}^{\dagger}$, so we will use $\mbox{\text{TabM}}_{\text{mini}}^{\dagger}$ for simplicity.

Intuition.
To give additional intuition on TabM, we make the following observations:

- •
Setting $k=1$ makes TabM identical to one plain MLP.
- •
Increasing $k$ by one adds a negligible number of new parameters to TabM.
- •
TabM, viewed as a single model, can benefit from the deep ensembling, see $\mbox{\text{TabM}}_{\text{mini}}^{\dagger\times 5}$ in Figure 2.
- •
In Transformer-like (Vaswani et al., ) and Mixer-like (Tolstikhin et al., ) models:
(a) the latent representation shape is $m\times d$, where $m$ is the number of tabular features, and $d$ is the embedding size; (b) the $m$ embeddings are mixed with each other in attention or linear layers, and (c) per-embedding transformations (the FFN layers) are the same for all embeddings.
By contrast, in TabM: (a) the shape is only $k\times d$, (b) the $k$ embeddings never interact with each other, and (c) per-embedding transformations contain embedding-specific weights (adapters).

Hyperparameters.
Compared to MLP, the only new hyperparameter of TabM is $k$ — the number of implicit submodels.
We heuristically set $k=32$ and do not tune this value.
We analyze the influence of $k$ in subsection 5.3.
We also noticed that the average optimal learning rate for TabM is higher than for MLP, which is explained in subsection A.4.

Limitations and practical considerations are commented in subsection A.5.

Next steps.
The performance of TabM in Figure 2 renders it as a highly promising model.
This motivates a full-fledged empirical comparison against prior tabular models (section 4) and detailed analysis of TabM’s behaviour (section 5).

## 4 Evaluating tabular deep learning architectures

Now, we perform an empirical comparison of many tabular models, including TabM introduced in section 3.
The implementation details of the models are provided in Appendix D.

### 4.1 Baselines

In the main text, we use the following baselines:
MLP (the classic multilayer perceptron), FT-Transformer denoted as “FT-T” (the attention-based model from Gorishniy et al. ()), SAINT (the attention- and retrieval- based model from Somepalli et al. ()), T2G-Former denoted as “T2G” (the attention-based model from Yan et al. ()), ExcelFormer denoted as “Excel” (the attention-based model from Chen et al. ()), TabR (the retrieval-based model from Gorishniy et al. ()), ModernNCA denoted as “MNCA” (the retrieval-based model from Ye et al. ()) and three GBDT implementations: XGBoost (Chen & Guestrin, ), LightGBM (Ke et al., ) and CatBoost (Prokhorenkova et al., ).

The models with non-linear feature embeddings from Gorishniy et al. () are marked with $\dagger$ or $\ddagger$ depending on the embedding type (see subsection D.8 for details on feature embeddings):

- •
$\text{MLP}^{\dagger}$ and $\mbox{\text{TabM}}_{\text{mini}}^{\dagger}$ use a modified version of the piecewise-linear embeddings.
- •
$\text{TabR}^{\ddagger}$, $\text{MNCA}^{\ddagger}$, and $\text{MLP}^{\ddagger}$ (also known as MLP-PLR) use various periodic embeddings.

We provide results for more baselines in Appendix B.

Figure: Figure 3: The task performance of tabular models on the 46 datasets from Table 1. (Left) The mean and standard deviations of the performance ranks over all datasets summarize the head-to-head comparison between the models on all datasets. (Middle & Right) The relative performance w.r.t. the plain multilayer perceptron (MLP) allows reasoning about the scale and consistency of improvements over this simple baseline. One dot of a jitter plot corresponds to the performance of a model on one of the 46 datasets. The box plots visualize the 10th, 25th, 50th, 75th and 90th percentiles of the jitter plots. Outliers are clipped. The separation in random and domain-aware dataset splits is explained in subsection 3.1.
Refer to caption: x3.png

Figure: Figure 4: Training times (left) and inference throughput (right) of the models from Figure 3. One dot represents a measurement on one dataset. $\mbox{\text{TabM}}_{\text{mini}}^{\dagger*}$ is the optimized $\mbox{\text{TabM}}_{\text{mini}}^{\dagger}$ (see subsection 4.3).
Refer to caption: x4.png

**Table 2: RMSE (upper rows) and training times (lower rows) on two large datasets. The best values are in bold. The meaning of model colors follows Figure 3.**
|  | #Objects | #Features | $\mathrm{XGBoost}$ | $\mathrm{MLP}$ | $\mbox{\text{TabM}}_{\text{mini}}^{\dagger*}$ | $\mbox{\text{TabM}}_{\text{mini}}^{\dagger}$ | $\mathrm{FT}\text{-}\mathrm{T}$ | $\mathrm{TabR}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Maps Routing | $6.5$M | $986$ | $0.1601$ | $0.1592$ | $0.1583$ | $\mathbf{0.1582}$ | $0.1594$ | OOM |
| $28$m | $\mathbf{15}$m | $2$h | $13.5$h | $45.5$h |  |  |  |  |
| Weather | $13$M | $103$ | $1.4234$ | $1.4842$ | $\mathbf{1.4090}$ | $\mathbf{1.4112}$ | $1.4409$ | OOM |
| $\mathbf{10}$m | $15$m | $1.3$h | $3.3$h | $13.5$h |  |  |  |  |

### 4.2 Task performance

We evaluate all models following the protocol announced in subsection 3.1, and report the results in Figure 3 (see also the critical difference diagram in Figure 9).
We make the following observations:

- 1.
The performance ranks render TabM as the top-tier DL model.
- 2.
The middle and right parts of Figure 3 provide a fresh perspective on the per-dataset metrics.
TabM holds its leadership among the DL models.
Meanwhile, many DL methods turn out to be no better or even worse than MLP on a non-negligible number of datasets, which shows them as less reliable solutions, and changes the ranking, especially on the domain-aware splits (right).
- 3.
One important characteristic of a model is the weakest part of its performance profile (e.g. the 10th or 25th percentiles in the middle plot), since it shows how reliable the model is on “inconvenient” datasets.
From that perspective, MLP^† seems to be a decent practical option between the plain MLP and TabM, especially given its simplicity and efficiency compared to retrieval-based alternatives, such as TabR and ModernNCA.

Summary. TabM confidently demonstrates the best performance among tabular DL models, and can serve as a reliable go-to DL baseline.
This is not the case for attention- and retrieval- based models.
Overall, MLP-like models, including TabM, form a representative set of tabular DL baselines.

### 4.3 Efficiency

Now, we evaluate tabular models in terms of training and inference efficiency, which becomes a serious reality check for some of the methods.
We benchmark exactly those hyperparameter configurations of models that are presented in Figure 3 (see subsection B.3 for the motivation).

$\mbox{\text{TabM}}_{\text{mini}}^{\dagger*}$.
Additionally, we include $\mbox{\text{TabM}}_{\text{mini}}^{\dagger*}$, which is $\mbox{\text{TabM}}_{\text{mini}}^{\dagger}$ enhanced with two efficiency-related plugins available out-of-the-box in PyTorch (Paszke et al., ): the automatic mixed precision (AMP) and torch.compile (Ansel et al., ).
The purpose of $\mbox{\text{TabM}}_{\text{mini}}^{\dagger*}$ is to showcase the potential of the modern hardware and software for a powerful tabular DL model, and it should not be directly compared to other DL models.
However, the implementation simplicity of TabM plays an important role, because it facilitates the seamless integration of the aforementioned PyTorch plugins.

Training time.
We focus on training times on larger datasets, because on small datasets, all methods become almost equally affordable, regardless of the formal relative difference.
Nevertheless, in Figure 10, we provide measurements on small datasets as well.
The left side of Figure 4 reveals that TabM offers practical training times.
By contrast, the long training times of attention- and retrieval-based models become one more limitation of these methods.

Inference throughput.
The right side of Figure 4 tells essentially the same story as the left side.
In subsection B.3, we also report the inference throughput on GPU with large batch sizes.

Applicability to large datasets.
In Table 2, we report metrics on two large datasets.
As expected, attention- and retrieval-based models struggle, yielding extremely long training times, or being simply inapplicable without additional effort.
See subsection D.4 for implementation details.

Parameter count.
Most tabular networks are overall compact.
This, in particular, applies to TabM, because its size is by design comparable to MLP.
We report model sizes in subsection B.3.

Summary.
Simple MLPs are the fastest DL models, with TabM being the runner-up.
The attention- and retrieval-based models are significantly slower.
Overall, MLP-like models, including TabM, form a representative set of practical and accessible tabular DL baselines.

## 5 Analysis

### 5.1 Performance and training dynamics of the individual submodels

Figure: Figure 5: The training profiles of $\mbox{\text{TabM}}_{\text{mini}}^{k=32}$ and $\mbox{\text{TabM}}_{\text{mini}}^{k=1}$ as described in subsection 5.1. (Upper) The training curves. $k=32[i]$ represents the mean individual loss over the $32$ submodels. (Middle) Same as the first row, but in the train-test coordinates: each dot represents some epoch from the first row, the training generally goes from left to right. This allows reasoning about overfitting by comparing test loss values for a given train loss value. (Lower) The mean pairwise cosine similarity between the $k$ individual gradients of $\mbox{\text{TabM}}_{\text{mini}}^{k=32}$ with the default initialization (green) and two suboptimal initializations described in subsection 5.1. Formally: $\frac{2}{n\cdot k(k-1)}\sum_{l,i,j(i<j)}\cos{(g_{i}^{l},g_{j}^{l})}$, where $g_{i}^{l}$ is the gradient of the $i$-th submodel induced by the $l$-th of the $n=1000$ training objects. See subsection D.5 for details. The legends contain the test scores if early stopping was used.
Refer to caption: x6.png

Recall that the prediction of TabM is defined as the mean prediction of its $k$ implicit submodels.
These submodels share almost all of their weights, and are trained simultaneously.
In this section, we take a closer look at the individual performance and training dynamics of these submodels.

For the next experiment, we intentionally simplify the setup as described in detail in subsection D.5.
Most importantly, all models have the same depth $3$ and width $512$, and are trained without early stopping, i.e. the training goes beyond the optimal epochs.
We use $\mbox{\text{TabM}}_{\text{mini}}$ from Figure 1 with $k=32$ denoted as $\mbox{\text{TabM}}_{\text{mini}}^{k=32}$.
We use $\mbox{\text{TabM}}_{\text{mini}}^{k=1}$ (i.e. essentially one plain MLP) as a natural baseline for the submodels of $\mbox{\text{TabM}}_{\text{mini}}^{k=32}$, because each of the $32$ submodels has the architecture of $\mbox{\text{TabM}}_{\text{mini}}^{k=1}$.

We visualize the training profiles on four diverse datasets (two classification and two regression problems of different sizes) in Figure 5.
As a reminder, the mean of the $k$ individual losses is what is explicitly optimized during the training of $\mbox{\text{TabM}}_{\text{mini}}$, the loss of the collective mean prediction corresponds to how $\mbox{\text{TabM}}_{\text{mini}}$ makes predictions on inference, and $\mbox{\text{TabM}}_{\text{mini}}^{k=1}$ is just a baseline.

In the upper row of Figure 5, the collective mean prediction of the submodels is superior to their individual predictions in terms of both training and test losses.
After the initial epochs, the training loss of the baseline MLP is lower than that of the collective and individual predictions.

In the middle row of Figure 5, we see a stark contrast between the individual and collective performance of the submodels.
Compared to the baseline MLP, the submodels look overfitted individually, while their collective prediction exhibits substantially better generalization.
This result is strict evidence of a non-trivial diversity of the submodels: without that, their collective test performance would be similar to their individual test performance.
Additionally, we report the performance of the Best submodel of TabM over many datasets under the name $\text{\mbox{\text{TabM}}}[\text{B}]$ in Figure 6.
As such, individually, even the best submodel of TabM is no better than a simple MLP.

The lower row of Figure 5 analyzes the gradient structure of $\mbox{\text{TabM}}_{\text{mini}}^{k=32}$.
As a reminder, due to the simultaneous training and the weight sharing between the $k=32$ submodels, most weights of $\mbox{\text{TabM}}_{\text{mini}}$ receive the mean of the $k$ gradients per object on each training step.
The green line in the lower row of Figure 5 shows a near zero cosine similarity between these $k$ gradients.
This may explain the higher training loss of $\mbox{\text{TabM}}_{\text{mini}}^{k=32}$ compared to $\mbox{\text{TabM}}_{\text{mini}}^{k=1}$ in the first row if Figure 5: perhaps, the weight sharing combined with the diverse gradients prevents $\mbox{\text{TabM}}_{\text{mini}}$ from (over)optimizing for the training task.

In the same lower row of Figure 5, we run an ablation study on the two sources of submodel diversity in $\mbox{\text{TabM}}_{\text{mini}}$: the random initializations in the adapter $R$ and in the $k$ prediction heads.
When all rows of $R$ (i.e. $r_{i}$ in terms of Figure 1) receive the same initialization, while the $k$ prediction heads are initialized completely randomly (the orange line), the submodel gradients are correlated, and the task performance is poor.
By contrast, the issue is less pronounced when the $k$ prediction heads receive the same initialization, and the initialization of $R$ is completely random (the purple line), though it also can hurt the performance.
Thus, the first adapter seems to be a more impactful source of gradient diversity.
Overall, we see the gradient diversity as an experimental metric requiring more exploration.

Summary.
TabM draws its power from the collective prediction of weak, but diverse submodels.

### 5.2 Selecting submodels after training

The design of TabM allows selecting only a subset of submodels after training based on any criteria, simply by pruning extra prediction heads and the corresponding rows of the adapter matrices.
To showcase this mechanics, after the training, we Greedily construct a subset of TabM’s submodels with the best collective performance on the validation set, and denote this “pruned” TabM as $\text{\mbox{\text{TabM}}}[\text{G}]$.
The performance reported in Figure 6 shows that $\text{\mbox{\text{TabM}}}[\text{G}]$ is slightly behind the vanilla TabM.
On average over 46 datasets, the greedy submodel selection results in $8.8\pm 6.6$ submodels out of the initial $k=32$, which can result in faster inference.
See subsection D.6 for implementation details.

### 5.3 How the performance of TabM depends on k 𝑘 k italic_k ?

Figure: Figure 6: The performance on the 46 datasets from Table 1. $\text{\mbox{\text{TabM}}}[\text{B}]$ and $\text{\mbox{\text{TabM}}}[\text{G}]$ are described in subsection 5.1 and subsection 5.2.

Figure: Figure 7: The average performance of TabM and $\mbox{\text{TabM}}_{\text{mini}}$ over $9$ datasets with different values of $k$.

To answer the question in the title, we take TabM with the number of layers $3$ and the width $512$, tune the learning rate for each $k$ separately, and report the performance in Figure 7.
Based on the figure, and the results in section 4, we suggest that $k=32$ used throughout the paper is a reasonable default value with a good balance between performance and efficiency.
Also, from Figure 7, it seems that TabM accommodates large numbers of submodels more easily than $\mbox{\text{TabM}}_{\text{mini}}$.
Perhaps, the larger number of submodel adapters in TabM provides the important additional weight capacity to fit more submodels in one model of a given size.
The implementation details are available in subsection D.7.

## 6 Conclusion & Future work

In this work, we have demonstrated that tabular multilayer perceptrons (MLPs) greatly benefit from parameter-efficient ensembling.
Using this insight, we have developed TabM — a simple MLP-based model with state-of-the-art performance.
In a large-scale comparison with many tabular DL models, we have demonstrated that TabM is ready to serve as a new powerful and efficient tabular DL baseline.
Finally, we have analyzed the properties of the implicit submodels underlying TabM.

One idea for future work is to bring the power of (parameter-)efficient ensembles to other, non-tabular, domains with optimization-related challenges and, ideally, lightweight base models.
Another idea is to evaluate TabM for uncertainty estimation and out-of-distribution (OOD) detection on tabular data, which is inspired by works like Lakshminarayanan et al. ().

Reproducibility statement.
The code is provided in the following repository: [link](https://github.com/yandex-research/tabm).
It contains the implementation of TabM, hyperparameter tuning scripts, evaluation scripts, configuration files with hyperparameters (the TOML files in the exp/ directory) and the report files with the main metrics (the JSON files in the exp/ directory).
In the paper, the model is described in section 3, and the implementation details are provided in Appendix D.

## Appendix A Additional discussion on TabM

### A.1 Motivation

Why BatchEnsemble?
Among relatively ease-to-use “efficient ensembling” methods, beyond BatchEnsemble, there are examples such as dropout ensembles (Lakshminarayanan et al., ), naive multi-head architectures, TreeNet (Lee et al., ).
However, in the literature, they were consistently outperformed by more advanced methods, including BatchEnsemble (Wen et al., ), MIMO (Havasi et al., ), FiLM-Ensemble (Turkoglu et al., ).

Among advanced methods, BatchEnsemble seems to be one of the simplest and most flexible options.
For example, FiLM-Ensemble (Turkoglu et al., ) requires normalization layers to be presented in the original architecture, which is not always the case for tabular MLPs.
MIMO (Havasi et al., ), in turn, imposes additional limitations compared to BatchEnsemble.
First, it requires concatenating (not stacking, as with BatchEnsemble) all $k$ input representations, which increases the input size of the first linear layer.
With the relatively high number of submodels $k=32$ used in our paper, this can be an issue on datasets with a large number of features, and especially when feature embeddings (Gorishniy et al., ) are used.
For example, for $k=32$, the number of features $m=1000$ and the feature embedding size $l=32$, the input size approaches one million resulting in an extremely large first linear layer of MLP.
Second, with BatchEnsemble, it is easy to explicitly materialize, analyze and prune individual submodels.
By contrast, in MIMO, all submodels are implicitly entangled within one MLP, and there is no easy way to access individual submodels.

Why MLPs?
Despite the applicability of BatchEnsemble (Wen et al., ) to almost any architecture, we focus specifically on MLPs.
The key reason is efficiency.
First, to achieve high performance, throughout the paper, we use the relatively large number of submodels $k=32$.
However, the desired less-than-$\times k$ runtime overhead of BatchEnsemble typically happens only when the original model underutilizes the power of parallel computations of a given hardware.
This will not be the case for attention-based models on datasets with a large number of features, as well as for retrieval-based models on datasets with a large number of objects.
Second, as we show in subsection 4.3, attention- and retrieval-based models are already slow as-is.
By contrast, MLPs are exceptionally efficient, to the extent that slowing them down even by an order of magnitude will still result in practical models.

Also, generally speaking, the definition of MLP suggested in subsection 3.3 and used in TabM is not special, and more advanced MLP-like backbones can be used.
However, in preliminary experiments, we did not observe the benefits of more advanced backbones.
Perhaps, small technical differences between backbones become less impactful in the context of parameter-efficient ensembling, at least in the scope of middle-to-large-sized datasets.

### A.2 Why TabM outperforms a full-fledged deep ensemble?

As shown in Figure 2, TabMnaive, TabM and $\mbox{\text{TabM}}_{\text{mini}}$ are all superior to $\text{MLP}^{\times k}$ — the full-fledged ensemble of $k$ MLPs.
Moreover, the performance gap is significant.
To the best of our knowledge, this result is rather not expected, because the literature on efficient ensembles is usually focused on catching up with deep ensembles, not on outperforming them, let alone significantly outperforming them.
Plus, after the training, TabM can be explicitly materialized as a traditional ensemble of $k$ MLPs.
With that in mind, we highlight three hypotheses for the superior performance of TabM compared to $\text{MLP}^{\times k}$.
The first one relies on the analysis from subsection 5.1, and the other two are more general.

First, we highlight the combination of the weight sharing and gradient diversity (observed in the third row of Figure 5) in TabM.
Perhaps, the regularization power of the weakly aligned individual gradients cannot be recovered by technical details such as hyperparameter tuning, training protocol, etc.
In this case, we can say that MLP ensembles seem to win from this regularization when training on tabular data.
This, in turn, can be related to the common belief that optimization on tabular data is challenging, especially for MLPs.
Overall, it is unclear if the phenomenon will generalize to other models and/or domains.

Second, the hyperparameter tuning for $\text{MLP}^{\times k}$ may be suboptimal.
Recall that, in Figure 2, we tune hyperparameters of one ensemble-unaware MLP, and then train the tuned MLP from scratch $k$ times to obtain $\text{MLP}^{\times k}$.
Perhaps, if the hyperparameters are tuned directly for $\text{MLP}^{\times k}$, then more powerful (and exotic) configurations can be found.
However, that would make the tuning $\times k$ more expensive.

Third, the parallel training (and, in particular, early stopping) of the implicit submodels in TabM may be important.
The direct analogy for $\text{MLP}^{\times k}$ would be to train all $k$ members in parallel on the same training batches.
In Figure 2, all members $\text{MLP}^{\times k}$ are trained independently under different random seeds (and, in particular, over different training batch sequences).

### A.3 TabM with feature embeddings

Here, we provide additional implementation details for $\mbox{\text{TabM}}_{\text{mini}}^{\dagger}$ and $\mbox{\text{TabM}}^{\dagger}$ introduced in subsection 3.3.
In fact, there are no changes in the usage of feature embeddings compared to plain MLPs.

Technically, feature embeddings are applied, and the result is flattened, before the Clone module in terms of Figure 1.
For example, if a dataset has $m$ continuous features and all of them are embedded, the very first adapter $R$ will have the shape $k\times md_{e}$, where $d_{e}$ is the feature embedding size.
For $\mbox{\text{TabM}}_{\text{mini}}^{\dagger}$ and $\mbox{\text{TabM}}^{\dagger}$, we initialize the first multiplicative adapter $R$ of the first linear layer from the standard normal distribution $\mathcal{N}(0,1)$.
The remaining details are best understood from the source code.

### A.4 Hyperparameters

As mentioned in the main text, we noticed that the typical optimal learning rate for TabM is higher than for MLP
(we share the tuned hyperparameters on all datasets in the repository, so the precise comparison is possible).
We hypothesize that the reason is the effectively larger batch size because of the Clone operation (see Figure 1), which, in particular, leads to the $k$ mostly orthogonal gradients per object (see subsection 5.1) on each training step.
The latter leads to lower gradient norms after averaging the $k$ gradients, which can be related to the need for larger learning rates.

### A.5 Limitations and practical considerations

TabM does not introduce any new limitations compared to BatchEnsemble (Wen et al., ).
Nevertheless, we note the following:

- •
The MLP backbone used in TabM is one of the simplest possible, and generally, more advanced backbones can be used.
That said, some backbones may require additional care when used in TabM.
For example, we did not explore backbones with normalization layers.
For such layers, it is possible to allocate non-shared trainable affine transformations for each implicit submodel by adding one multiplicative and one additive adapter after the normalization layer (i.e. like in FiLM-Ensemble (Turkoglu et al., )).
Additional experiments are required to find the best strategy.
- •
Arguably the key limitation is that BatchEnsemble-like techniques are not “local”, but instead affect the whole model starting from the first modified layer.
Namely, when the computation flow hits the first modified layer, the $k$ prediction branches are created, and the rest of the network will have to make $k$ times more computations.
This can be easily affordable for small base models, but may be less affordable for heavy base models.
- •
For ensemble-like models, such as TabM, the notion of “the final object embedding“ changes: now, it is not a single vector, but a set of $k$ vectors.
This can be important for scenarios when TabM is used for solving more than one task, in particular, when it is pretrained as a generic feature extractor and then reused for other tasks.
The main practical guideline is that the $k$ prediction branches should never interact with each other (e.g. through attention, pooling, etc.) and should always be trained separately.

## Appendix B Extended results

This section complements section 4.

### B.1 Additional baselines

In addition to subsection 4.1, we consider the following models:

- •
MLP-PLR Gorishniy et al. (), that is, an MLP with periodic embeddings.
- •
ResNet (Gorishniy et al., )
- •
SNN (Klambauer et al., )
- •
DCNv2 (Wang et al., )
- •
AutoInt (Song et al., )
- •
MLP-Mixer is our adaptation of Tolstikhin et al. () for tabular data.
- •
Trompt (Chen et al., ) (our reimplementation, since there is no official implementation)

We also evaluated TabPFN (Hollmann et al., ), where possible.
The results for this model are available only in Appendix E, because this model is by design not applicable to regression tasks, which is a considerable number of our datasets.
Overall, TabPFN specializes on small datasets.
In line with that, the performance of TabPFN on our benchmark was not competitive.

### B.2 Task performance

Figure 8 is a different version of Figure 3 with additional baselines.
Overall, none of the additional baselines affects our main story.

Figure 9 is the critical difference diagram (CDD) computed over exactly the same results that were used for building Figure 3.

Figure: Figure 8: An extended comparison of tabular models as in Figure 3. Note that the ranks (left) are computed only over the 37 datasets with random splits, because ResNet, AutoInt and MLP-Mixer were evaluated only on one $1$ out of $9$ datasets with domain-aware splits.
Refer to caption: x11.png

Figure: Figure 9: Critical difference diagram. The computation method is taken from the Kim et al. ().
Refer to caption: x12.png

### B.3 Efficiency

This section complements subsection 4.3.

Additional results.

Figure 10 complements Figure 4 by providing the training times on smaller datasets and the inference throughput on GPU with large batch sizes.

Table 3 provide the number of trainable parameters for some of the models from Figure 3.

Motivation for the benchmark setup.
Comparing models under all possible kinds of budgets (task performance, the number of parameters, training time, etc.) on all possible hardware (GPU, CPU, etc.) with all possible batch sizes is rather infeasible.
As such, we set a narrow goal of providing a high-level intuition on the efficiency in a transparent setting.
Thus, benchmarking the transparently obtained tuned hyperparameter configurations works well for our goal.
Yet, this choice also has a limitation: the hyperparameter tuning process is not aware of the efficiency budget, so it can prefer much heavier configurations even if they lead to tiny performance improvements, which will negatively affect efficiency without a good reason.
Overall, we hope that the large number of datasets compensates for potentially imperfect per-dataset measurements.

Motivation for the two setups for measuring inference throughput.

- •
The setup in the right side of Figure 4 simulates the online per-object predictions.
- •
The setup in the right side of Figure 10 simulates the offline batched computations.

Figure: Figure 10: (Left) Training time on datasets with less than 100K objects. (Right) Inference throughput on GPU with maximum possible batch size (i.e. the batch size depends of a model).
Refer to caption: x13.png

**Table 3: Mean number of parameters with std. dev. for 7 different tuned models across all 46 datasets.**
| TabM | MLP | FT-T | T2G | TabR | ModernNCA | SAINT |
| --- | --- | --- | --- | --- | --- | --- |
| $1.4M\pm 1.3M$ | $1.0M\pm 1.0M$ | $1.2M\pm 1.2M$ | $2.1M\pm 1.6M$ | $858K\pm 1.4M$ | $1.0M\pm 1.1M$ | $175.4M\pm 565.4M$ |

## Appendix C Datasets

In total, we use 46 datasets:

- 1.
$38$ datasets are taken from Gorishniy et al. (), which includes:
(a)
$28$ datasets from Grinsztajn et al. ().
See the original paper for the precise dataset information.
(b)
$10$ datasets from other sources.
Their properties are provided in Table 4.
- 2.
$8$ datasets are taken from Rubachev et al. ().
Their properties are provided in Table 5.

In fact, the aforementioned $38$ datasets from Gorishniy et al. () is only a subset of the datasets used in Gorishniy et al. ().
Namely, we did not include the following of the remaining datasets:

- •
The datasets that, according to Rubachev et al. (), have incorrect splits and/or label leakage, including:
$\mathrm{Bike_Sharing_Demand}$,
$\mathrm{compass}$,
$\mathrm{electricity}$,
$\mathrm{SGEMM_GPU_kernel_performance}$,
$\mathrm{sulfur}$,
$\mathrm{visualizing_soil}$,
and the weather forecasting dataset (it is replaced by the correct weather forecasting dataset from Rubachev et al. ()).
- •
$\mathrm{rl}$ from (Grinsztajn et al., ).
We observed abnormal results on these datasets.
This is an anonymous dataset, which made the investigation impossible, so we removed this dataset to avoid confusion.
- •
$\mathrm{yprop_4_1}$ from (Grinsztajn et al., ).
Strictly speaking, this dataset was omitted due to a mistake on our side.
For future work, we note that the typical performance gaps on this dataset have low absolute values in terms of RMSE.
Perhaps, $R^{2}$ may be a more appropriate metric for this dataset.

**Table 4: Properties of those datasets from Gorishniy et al. () that are not part of Grinsztajn et al. () or Rubachev et al. (). “# Num”, “# Bin”, and “# Cat” denote the number of numerical, binary, and categorical features, respectively. The table is taken from (Gorishniy et al., ).**
| Name | # Train | # Validation | # Test | # Num | # Bin | # Cat | Task type | Batch size |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Churn Modelling | $6\,400$ | $1\,600$ | $2\,000$ | $10$ | $3$ | $1$ | Binclass | 128 |
| California Housing | $13\,209$ | $3\,303$ | $4\,128$ | $8$ | 0 0 | 0 0 | Regression | 256 |
| House 16H | $14\,581$ | $3\,646$ | $4\,557$ | $16$ | 0 0 | 0 0 | Regression | 256 |
| Adult | $26\,048$ | $6\,513$ | $16\,281$ | $6$ | $1$ | $8$ | Binclass | 256 |
| Diamond | $34\,521$ | $8\,631$ | $10\,788$ | $6$ | 0 0 | $3$ | Regression | 512 |
| Otto Group Products | $39\,601$ | $9\,901$ | $12\,376$ | $93$ | 0 0 | 0 0 | Multiclass | 512 |
| Higgs Small | $62\,751$ | $15\,688$ | $19\,610$ | $28$ | 0 0 | 0 0 | Binclass | 512 |
| Black Friday | $106\,764$ | $26\,692$ | $33\,365$ | $4$ | $1$ | $4$ | Regression | 512 |
| Covertype | $371\,847$ | $92\,962$ | $116\,203$ | $15$ | $4$ | $1$ | Multiclass | 1024 |
| Microsoft | $723\,412$ | $235\,259$ | $241\,521$ | $131$ | $5$ | 0 0 | Regression | 1024 |

**Table 5: Properties of the datasets from the TabReD benchmark (Rubachev et al., ). “# Num”, “# Bin”, and “# Cat” denote the number of numerical, binary, and categorical features, respectively.**
| Name | # Train | # Validation | # Test | # Num | # Bin | # Cat | Task type | Batch size |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Sberbank Housing | $18\,847$ | $4\,827$ | $4\,647$ | $365$ | $17$ | $10$ | Regression | 256 |
| Ecom Offers | $109\,341$ | $24\,261$ | $26\,455$ | $113$ | $6$ | 0 0 | Binclass | 1024 |
| Maps Routing | $160\,019$ | $59\,975$ | $59\,951$ | $984$ | 0 0 | $2$ | Regression | 1024 |
| Homesite Insurance | $224\,320$ | $20\,138$ | $16\,295$ | $253$ | $23$ | $23$ | Binclass | 1024 |
| Cooking Time | $227\,087$ | $51\,251$ | $41\,648$ | $186$ | $3$ | $3$ | Regression | 1024 |
| Homecredit Default | $267\,645$ | $58\,018$ | $56\,001$ | $612$ | $2$ | $82$ | Binclass | 1024 |
| Delivery ETA | $279\,415$ | $34\,174$ | $36\,927$ | $221$ | $1$ | $1$ | Regression | 1024 |
| Weather | $106\,764$ | $42\,359$ | $40\,840$ | $100$ | $3$ | 0 0 | Regression | 1024 |

## Appendix D Implementation details

### D.1 Hardware

Most of the experiments were conducted on a single NVIDIA A100 GPU.
In rare exceptions, we used a machine with a single NVIDIA 2080 Ti GPU and Intel(R) Core(TM) i7-7800X CPU @ 3.50GHz.

### D.2 Experiment setup

We mostly follow the experiment setup from Gorishniy et al. ().
As such, some of the text below is copied from (Gorishniy et al., ).

Data preprocessing.
For each dataset, for all DL-based solutions, the same preprocessing was used for fair comparison.
For numerical features, by default, we used a slightly modified version of the quantile normalization from the Scikit-learn package (Pedregosa et al., ) (see the source code), with rare exceptions when it turned out to be detrimental (for such datasets, we used the standard normalization or no normalization).
For categorical features, we used one-hot encoding.
Binary features (i.e. the ones that take only two distinct values) are mapped to $\{0,1\}$ without any further preprocessing. We completely follow Rubachev et al. () on Table 5 datasets.

Training neural networks.
For DL-based algorithms, we minimize cross-entropy for classification problems and mean squared error for regression problems.
We use the AdamW optimizer (Loshchilov & Hutter, ).
We do not apply learning rate schedules.
We do not use data augmentations.
We apply global gradient clipping to $1.0$.
For each dataset, we used a predefined dataset-specific batch size.
We continue training until there are patience consecutive epochs without improvements on the validation set; we set $\texttt{patience}=16$ for the DL models.

Hyperparameter tuning.
In most cases, hyperparameter tuning is performed with the TPE sampler (typically, 50-100 iterations) from the Optuna package (Akiba et al., ).
Hyperparameter tuning spaces for most models are provided in individual sections below (example for TabM: subsection D.9). We follow Rubachev et al. () and use $25$ iterations on some datasets from Table 5.

Evaluation.
On a given dataset, for a given model, the tuned hyperparameters are evaluated under multiple (in most cases, $15$) random seeds.
The mean test metric and its standard deviation over these random seeds are then used to compare algorithms as described in subsection D.3.

### D.3 Metrics

We use Root Mean Squared Error for regression tasks, ROC-AUC for classification datasets from Table 5 (following Rubachev et al. ()), and accuracy for the rest of datasets (following Gorishniy et al. ()).
We also tried computing ROC-AUC for all classification datasets, but did not observe any significant changes (see Figure 11), so we stuck to prior work.
By default, the mean test score and its standard deviation are obtained by training a given model with tuned hyperparameters from scratch on a given dataset under 15 different random seeds.

How we compute ranks.
Our method of computing ranks does not count small improvements as wins, hence the reduced range of ranks compared to other studies.
Intuitively, our ranks can be considered as “tiers”.

Assume the higher the score the better and $\texttt{mean_score_Ref}>\texttt{mean_score_A}$. Then reference Model_Ref and Model_A are equal (have the same rank) if $(\texttt{mean_score_Ref}-\texttt{mean_score_A})\leq\texttt{std_Ref}$. To assign ranks, we sort models in descending score order. Starting from the best model (with rank equal to 1) we iterate over models and assign first rank to all models that are equal to the best model according to the mentioned definition. The first model in descending order that is not equal to the best model is assigned with second rank and becomes a new reference model. We continue the process until all models are ranked. Ranks are computed independently for each dataset.

### D.4 Implementation details of subsection 4.3

Applicability to large datasets.
The two datasets used in Table 2 are the full versions of the “Weather” and “Maps Routing” datasets from Rubachev et al. ().
Their smaller versions with subsampled training set were already included in Table 1, and were used when building Figure 3.
The validation and test sets are the same for the small and large versions of these datasets, so the task metrics are comparable between the two versions.
When running models on the large versions of the datasets, we reused the hyperparameters tuned for their small versions.
Thus, this experiment can be seen as a quick assessment of the applicability of several tabular DL to large datasets, without a strong focus on the task performance.
All models, except for FT-Transformer, were evaluated under $3$ random seeds.
FT-Transformer was evaluated under $1$ random seed.

Figure: Figure 11: Same as Figure 3, but ROC-AUC is used as the metric for all classification datasets. The two multiclass datasets presented in our benchmark are not taken into account.
Refer to caption: x15.png

### D.5 Implementation details of subsection 5.1

Experiment setup.
This paragraph complements the description of the experiment setup in subsection 5.1.
Namely, in addition to what is mentioned in the main text:

- •
Dropout and weight decay are turned off.
- •
To get representative training profiles for all models, the learning rates are tuned separately for $\mbox{\text{TabM}}_{\text{mini}}^{k=1}$ and $\mbox{\text{TabM}}_{\text{mini}}^{k=32}$ on validation sets using the usual metrics (i.e. RMSE or accuracy) as the guidance.
The grid for learning rate tuning was: numpy.logspace(numpy.log10(1e-5), numpy.log10(5e-3), num=25).
- •
The adapter $R$ in $\mbox{\text{TabM}}_{\text{mini}}$ is initialized from $\mathrm{N}(0,1)$ instead of $\pm 1$.
We observed that $\pm 1$ resulted in occasional minor jumps in some of the curves shown in Figure 5.
The number and magnitude of the jumps were low and had no effect on the story.
However, in line with all other changes to the experiment setup in this section, we adjusted the initialization to avoid the jumps, to ensure that the training runs were not disturbed by side-effects of unknown nature.

Cosine similarity between the $k$ gradients.
This paragraph complements the story about the third row of Figure 5.
The reported cosine similarity between the $k$ gradients was computed as follows.
Before the training, we randomly select $n=1000$ training objects — these reference objects stay the same during the whole training run.
During the training, at the start of every epoch, we calculate the metric according to the formula in the caption of Figure 5:

- 1.
First, for each of the $n$ reference objects, we compute the $k$ individual gradients, and omit the gradient components related to the weights of the prediction heads.
Thus, the size of one gradient equals the number of all parameters, minus the number of parameters in one prediction head.
- 2.
Then for each of the $n$ reference objects, we compute the $\frac{k(k-1)}{2}$ pairwise cosine similarities between the $k$ individual gradients.
- 3.
Finally, we average the cosine similarities over all pairs of gradients over all reference objects, which gives the formula in the caption of Figure 5.

Limitations.
Continuing the story about the gradient diversity, we note that in high-dimensional spaces, the near-zero values of cosine similarity can be hard to interpret, which is a potential limitation of our analysis.
That said, a non-trivial positive (negative) value is rather a decent indicator of a non-trivial positive (negative) correlation between vectors.

### D.6 Implementation details of subsection 5.2

$\text{\mbox{\text{TabM}}}[\text{G}]$.
Here, we clarify the implementation details for $\text{\mbox{\text{TabM}}}[\text{G}]$ described in subsection 5.2.
$\text{\mbox{\text{TabM}}}[\text{G}]$ is obtained from a trained TabM by greedily selecting submodels from TabM starting from the best one and stopping when two conditions are simultaneously true for the first time: (1) adding any new submodel does not improve the validation metric of the collective prediction; (2) the current validation metric is already better than that of the initial model with all $k$ submodels.
To clarify, during the greedy selection, the $i$-th submodel is considered to be better than the $j$-th submodel if adding the $i$-th submodel to the aggregated prediction leads to better validation metrics (i.e. it is not the same as adding the submodel in the order of their individual validation metrics).

### D.7 Implementation details of subsection 5.3

Figure 7 shows the mean percentage improvements (see subsection D.3) over MLP across 9 datasets from Table 4 (without Covertype).
We have used a $\mbox{\text{TabM}}_{\text{mini}}$  with $3$ hidden layers of the width $d=512$, the dropout rate 0.1 and tuned learning rate for different $k$.
The score on each dataset is averaged over $5$ seeds.

### D.8 Non-linear embeddings for continuous features

Notation.
We use the notation based on $\dagger$ and $\ddagger$ only for brevity.
Any other unambiguous notation can be used in future work.

Updated piecewise-linear embeddings.
We use a slightly different implementation of the piecewise-linear embeddings compared to Gorishniy et al. ().
Architecture-wise, our implementation corresponds to the “Q-L” and “T-L” variations from Table 2 in Gorishniy et al. () (we use the quantile-based bins for simplicity).
In practice, our implementation is significantly faster, and uses a different parametrization and initialization.
See the source code for details.

TabM.
For TabM, we recommend the aforementioned updated piecewise-linear embeddings as the default embedding architecture.
This corresponds to the $\mbox{\text{TabM}}_{\text{mini}}^{\dagger}$ and $\mbox{\text{TabM}}^{\dagger}$ models.

Other models.
Since it is not feasible to test all combinations of backbones and embeddings, for baselines, we stick to the embeddings used in the original papers (applies to TabR (Gorishniy et al., ), ExcelFormer (Chen et al., ) and ModernNCA (Ye et al., )).
For all models with feature embeddings (including TabM, MLP, TabR, ModernNCA, ExcelFormer),
the embeddings-related details are commented in the corresponding sections below.

### D.9 TabM

Feature embeddings.
$\mbox{\text{TabM}}_{\text{mini}}^{\dagger}$ and $\mbox{\text{TabM}}^{\dagger}$ are the versions of TabM with non-linear feature embeddings.
$\mbox{\text{TabM}}_{\text{mini}}^{\dagger}$ and $\mbox{\text{TabM}}^{\dagger}$ use the updated piecewise-linear feature embeddings mentioned in subsection D.8.

Table 6 provides the hyperparameter tuning spaces for TabM and $\mbox{\text{TabM}}_{\text{mini}}$.
Table 7 provides the hyperparameter tuning spaces for $\mbox{\text{TabM}}^{\dagger}$ and $\mbox{\text{TabM}}_{\text{mini}}^{\dagger}$.

**Table 6: The hyperparameter tuning space for TabM and $\mbox{\text{TabM}}_{\text{mini}}$. Here, (B) = {Covertype, Microsoft, Table 5} and (A) contains all other datasets.**
| Parameter | Distribution or Value |
| --- | --- |
| $k$ | $32$ |
| # layers | $\mathrm{UniformInt}[1,5]$ |
| Width (hidden size) | $\mathrm{UniformInt}[64,1024]$ |
| Dropout rate | $\{0.0,\mathrm{Uniform}[0.0,0.5]\}$ |
| Learning rate | $\mathrm{LogUniform}[1e\text{-}4,5e\text{-}3]$ |
| Weight decay | $\{0,\mathrm{LogUniform}[1e\text{-}4,1e\text{-}1]\}$ |
| # Tuning iterations | (A) 100 (B) 50 |

**Table 7: The hyperparameter tuning space for $\mbox{\text{TabM}}_{\text{mini}}^{\dagger}$ and $\mbox{\text{TabM}}^{\dagger}$. Here, (B) = {Covertype, Microsoft, Table 5} and (A) contains all other datasets.**
| Parameter | Distribution or Value |
| --- | --- |
| $k$ | $32$ |
| # layers | $\mathrm{UniformInt}[1,4]$ |
| Width (hidden size) | $\mathrm{UniformInt}[64,1024]$ |
| Dropout rate | $\{0.0,\mathrm{Uniform}[0.0,0.5]\}$ |
| # PLE bins | $\mathrm{UniformInt}[8,32]$ |
| Learning rate | $\mathrm{LogUniform}[5e\text{-}5,3e\text{-}3]$ |
| Weight decay | $\{0,\mathrm{LogUniform}[1e\text{-}4,1e\text{-}1]\}$ |
| # Tuning iterations | (A) 100 (B) 50 |

### D.10 MLP

Feature embeddings.
MLP^$\dagger$ and MLP^$\ddagger$ are the versions of MLP with non-linear feature embeddings.
MLP^$\dagger$ uses the updated piecewise-linear embeddings mentioned in subsection D.8.
MLP^$\ddagger$ (also known as MLP-PLR) uses the periodic embeddings (Gorishniy et al., ).
Tehcnically, it is the PeriodicEmbeddings class from the rtdl_num_embeddings Python package.
We tested two variations: with lite=False and lite=True.
In the paper, only the former one is reported, but in the source code, the results for both are available.

Table 8, Table 9, Table 10 provide the hyperparameter tuning spaces for MLP, MLP^$\dagger$ and MLP^$\ddagger$, respectively.

**Table 8: The hyperparameter tuning space for MLP.**
| Parameter | Distribution |
| --- | --- |
| # layers | $\mathrm{UniformInt}[1,6]$ |
| Width (hidden size) | $\mathrm{UniformInt}[64,1024]$ |
| Dropout rate | $\{0.0,\mathrm{Uniform}[0.0,0.5]\}$ |
| Learning rate | $\mathrm{LogUniform}[3e\text{-}5,1e\text{-}3]$ |
| Weight decay | $\{0,\mathrm{LogUniform}[1e\text{-}4,1e\text{-}1]\}$ |
| # Tuning iterations | 100 |

**Table 9: The hyperparameter tuning space for $\mathrm{MLP}^{\dagger}$.**
| Parameter | Distribution |
| --- | --- |
| # layers | $\mathrm{UniformInt}[1,5]$ |
| Width (hidden size) | $\mathrm{UniformInt}[64,1024]$ |
| Dropout rate | $\{0.0,\mathrm{Uniform}[0.0,0.5]\}$ |
| Learning rate | $\mathrm{LogUniform}[3e\text{-}5,1e\text{-}3]$ |
| Weight decay | $\{0,\mathrm{LogUniform}[1e\text{-}4,1e\text{-}1]\}$ |
| d_embedding | $\mathrm{UniformInt}[8,32]$ |
| n_bins | $\mathrm{UniformInt}[2,128]$ |
| # Tuning iterations | 100 |

**Table 10: The hyperparameter tuning space for $\mathrm{MLP}^{\ddagger}$.**
| Parameter | Distribution |
| --- | --- |
| # layers | $\mathrm{UniformInt}[1,5]$ |
| Width (hidden size) | $\mathrm{UniformInt}[64,1024]$ |
| Dropout rate | $\{0.0,\mathrm{Uniform}[0.0,0.5]\}$ |
| Learning rate | $\mathrm{LogUniform}[3e\text{-}5,1e\text{-}3]$ |
| Weight decay | $\{0,\mathrm{LogUniform}[1e\text{-}4,1e\text{-}1]\}$ |
| n_frequencies | $\mathrm{UniformInt}[16,96]$ |
| d_embedding | $\mathrm{UniformInt}[16,32]$ |
| frequency_init_scale | $\mathrm{LogUniform}[1e\text{-}2,1e\text{1}]$ |
| # Tuning iterations | 100 |

### D.11 TabR

Feature embeddings.
TabR^$\ddagger$ is the version of TabR with non-linear feature embeddings.
TabR^$\ddagger$ uses the periodic embeddings (Gorishniy et al., ), specifically, PeriodicEmbeddings(lite=True) from the rtdl_num_embeddings Python package on most datasets.
On the datasets from Table 5, TabR^$\ddagger$ uses the PeriodicEmbeddings(lite=True) embeddings on the Sberbank Housing and Ecom Offers datasets, and LinearReLUEmbeddings on the rest (to fit the computations into the GPU memory, following the original TabR paper).

Since we follow the training and evaluation protocols from Gorishniy et al. (), and TabR was proposed in Gorishniy et al. (), we simply reuse the results for TabR.
More details can be found in Appendix.D from Gorishniy et al. ().
When tuning TabR^$\ddagger$ on the datasets from Table 5, we have used $25$ tuning iterations and the same tuning space as for TabR from Rubachev et al. ().

### D.12 FT-Transformer

We used the implementation from the ”rtdl_revisiting_models” Python package. The results on datasets from Table 5 were copied from Rubachev et al. (), because the experiment setups are compatible.

**Table 11: The hyperparameter tuning space for FT-Transformer Gorishniy et al. (). Here, (B) = {Covertype, Microsoft} and (A) contains all other datasets (except Table 5).**
| Parameter | Distribution or Value |
| --- | --- |
| # blocks | $\mathrm{UniformInt}[1,4]$ |
| $d_{token}$ | $\mathrm{UniformInt}[16,384]$ |
| Attention dropout rate | $\mathrm{Uniform}[0.0,0.5]$ |
| FFN hidden dimension expansion rate | $\mathrm{Uniform}[\nicefrac{{2}}{{3}},\nicefrac{{8}}{{3}}]$ |
| FFN dropout rate | $\mathrm{Uniform}[0.0,0.5]$ |
| Residual dropout rate | $\{0.0,\mathrm{Uniform}[0.0,0.2]\}$ |
| Learning rate | $\mathrm{LogUniform}[3e\text{-}5,1e\text{-}3]$ |
| Weight decay | $\{0,\mathrm{LogUniform}[1e\text{-}4,1e\text{-}1]\}$ |
| # Tuning iterations | (A) 100 (B) 50 |

### D.13 ModernNCA

Feature embeddings.
We adapted an official implementation of Ye et al. (). We have used periodic embeddings Gorishniy et al. () (specifically, PeriodicEmbeddings(lite=True) from the rtdl_num_embeddings Python package) for ModernNCA^$\ddagger$ and no embeddings for ModernNCA. Table 12 and Table 13 provides hyperparameter tuning spaces for each ModernNCA and ModernNCA^$\ddagger$.

**Table 12: The hyperparameter tuning space for ModernNCA. Here, (C) = {Table 5}, (B) = {Covertype, Microsoft} and (A) contains all other datasets.**
| Parameter | Distribution |
| --- | --- |
| # blocks | $\mathrm{UniformInt}[0,2]$ |
| $d_{block}$ | $\mathrm{UniformInt}[64,1024]$ |
| dim | $\mathrm{UniformInt}[64,1024]$ |
| Dropout rate | $\mathrm{Uniform}[0.0,0.5]$ |
| Sample rate | $\mathrm{Uniform}[0.05,0.6]$ |
| Learning rate | $\mathrm{LogUniform}[1e\text{-}5,1e\text{-}1]$ |
| Weight decay | $\{0,\mathrm{LogUniform}[1e\text{-}6,1e\text{-}3]\}$ |
| # Tuning iterations | (A) 100 (B, C) 50 |

**Table 13: The hyperparameter tuning space for ModernNCA^$\ddagger$. Here, (C) = {Table 5}, (B) = {Covertype, Microsoft} and (A) contains all other datasets.**
| Parameter | Distribution |
| --- | --- |
| # blocks | $\mathrm{UniformInt}[0,2]$ |
| $d_{block}$ | $\mathrm{UniformInt}[64,1024]$ |
| dim | $\mathrm{UniformInt}[64,1024]$ |
| Dropout rate | $\mathrm{Uniform}[0.0,0.5]$ |
| Sample rate | $\mathrm{Uniform}[0.05,0.6]$ |
| Learning rate | $\mathrm{LogUniform}[1e\text{-}5,1e\text{-}1]$ |
| Weight decay | $\{0,\mathrm{LogUniform}[1e\text{-}6,1e\text{-}3]\}$ |
| n_frequencies | $\mathrm{UniformInt}[16,96]$ |
| d_embedding | $\mathrm{UniformInt}[16,32]$ |
| frequency_init_scale | $\mathrm{LogUniform}[0.01,10]$ |
| # Tuning iterations | (A) 100 (B, C) 50 |

### D.14 T2G-Former

We adapted the implementation and hyperparameters of Yan et al. () from the official repository(^1^11https://github.com/jyansir/t2g-former). Table 14 provides hyperparameter tuning space.

**Table 14: The hyperparameter tuning space for T2G-Former Yan et al. (). Here, (C) = {Table 5}, (B) = {Covertype, Microsoft} and (A) contains all other datasets. Also, we used 50 tuning iterations for some datasets from Grinsztajn et al. ().**
| Parameter | Distribution or Value |
| --- | --- |
| # blocks | (A) $\mathrm{UniformInt}[3,4]$ (B, C) $\mathrm{UniformInt}[1,3]$ |
| $d_{token}$ | $\mathrm{UniformInt}[64,512]$ |
| Attention dropout rate | $\mathrm{Uniform}[0.0,0.5]$ |
| FFN hidden dimension expansion rate | (A, B) $\mathrm{Uniform}[\nicefrac{{2}}{{3}},\nicefrac{{8}}{{3}}]$ (C) $4/3$ |
| FFN dropout rate | $\mathrm{Uniform}[0.0,0.5]$ |
| Residual dropout rate | $\{0.0,\mathrm{Uniform}[0.0,0.2]\}$ |
| Learning rate | $\mathrm{LogUniform}[3e\text{-}5,1e\text{-}3]$ |
| Col. Learning rate | $\mathrm{LogUniform}[5e\text{-}3,5e\text{-}2]$ |
| Weight decay | $\{0,\mathrm{LogUniform}[1e\text{-}6,1e\text{-}1]\}$ |
| # Tuning iterations | (A) 100 (B) 50 (C) 25 |

### D.15 SAINT

We completely adapted hyperparameters and protocol from Gorishniy et al. () to evaluate SAINT on Grinsztajn et al. () benchmark. Results on datasets from Table 4 were directly taken from Gorishniy et al. (). Additional details can be found in Appendix.D from Gorishniy et al. (). We have used a default configuration on big datasets due to very high cost of tuning (see Table 15).

**Table 15: The default hyperparameters for SAINT (Somepalli et al., ) on datasets from Rubachev et al. ().**
| Parameter | Value |
| --- | --- |
| depth | $2$ |
| $d_{token}$ | $32$ |
| $n_{heads}$ | $4$ |
| $d_{head}$ | $8$ |
| Attention dropout rate | $0.1$ |
| FFN hidden dimension expansion rate | $1$ |
| FFN dropout rate | $0.8$ |
| Learning rate | $1e\text{-}4$ |
| Weight decay | $1e\text{-}2$ |

### D.16 Excelformer

Feature embeddings.
ExcelFormer (Chen et al., ) uses custom non-linear feature embeddings based on a GLU-style activation, see the original paper for details.

We adapted the implementation and hyperparameters of Chen et al. () from the official repository(^2^22https://github.com/WhatAShot/ExcelFormer).
For fair comparison with other models, we did not use the augmentation techniques from the paper in our expirements. See Table 16.

**Table 16: The hyperparameter tuning space for Excelformer Chen et al. (). Here, (D) = {Homecredit, Maps Routing}, (C) = {Table 5 w/o (D)}, (B) = {Covertype, Microsoft} and (A) contains all other datasets.**
| Parameter | Distribution or Value |
| --- | --- |
| # blocks | (A, B) $\mathrm{UniformInt}[2,5]$ (C) $\mathrm{UniformInt}[2,4]$ (D) $\mathrm{UniformInt}[1,3]$ |
| $d_{token}$ | (A, B) $\{32,64,128,256\}$ (C) $\{16,32,64\}$ (D) $\{4,8,16,32\}$ |
| $n_{heads}$ | (A,B) $\{4,8,16,32\}$ (C) $\{4,8,16\}$ (D) $4$ |
| Attention dropout rate | $0.3$ |
| FFN dropout rate | $0.0$ |
| Residual dropout rate | $\mathrm{Uniform}[0.0,0.5]$ |
| Learning rate | $\mathrm{LogUniform}[3e\text{-}5,1e\text{-}3]$ |
| Weight decay | $\{0,\mathrm{LogUniform}[1e\text{-}4,1e\text{-}1]\}$ |
| # Tuning iterations | (A) 100 (B) 50 (C, D) 25 |

### D.17 CatBoost, XGBoost and LightGBM

Since our setup is directly taken from Gorishniy et al. (), we simply reused their results for GBDTs from the official repository(^3^33https://github.com/yandex-research/tabular-dl-tabr).
Importantly, in a series of preliminary experiments, we confirmed that those results are reproducible in our instance of their setup.
The details can be found in Appendix.D from Gorishniy et al. (). Results on datasets from Table 5 were copied from the paper (Rubachev et al., ).

### D.18 AutoInt

We used an implementation from Gorishniy et al. () which is an adapted official implementation(^4^44https://github.com/shichence/AutoInt).

**Table 17: The hyperparameter tuning space for AutoInt (Song et al., ). Here, (B) = {Covertype, Microsoft} and (A) contains all other datasets.**
| Parameter | Distribution |
| --- | --- |
| # blocks | $\mathrm{UniformInt}[1,6]$ |
| $d_{token}$ | $\mathrm{UniformInt}[8,64]$ |
| $n_{heads}$ | 2 |
| Attention dropout rate | $\{0,\mathrm{Uniform}[0.0,0.5]\}$ |
| Embedding dropout rate | $\{0,\mathrm{Uniform}[0.0,0.5]\}$ |
| Learning rate | $\mathrm{LogUniform}[3e\text{-}5,1e\text{-}3]$ |
| Weight decay | $\{0,\mathrm{LogUniform}[1e\text{-}4,1e\text{-}1]\}$ |
| # Tuning iterations | (A) 100 (B) 50 |

#### D.18.1 TabPFN

Since TabPFN accepts only less than 10K training samples we use different subsamples of size 10K for different random seeds. Also, TabPFN is not applicable to regressions and datasets with more than $100$ features.

## Appendix E Per-dataset results with standard deviations

**Table 18: Extended results for the main benchmark. Results are grouped by datasets.**
| churn ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.8553\pm 0.0029$<br>$0.8582\pm 0.0008$<br>$\mathrm{TabPFN}$<br>–<br>$0.8624\pm 0.0008$<br>$\mathrm{ResNet}$<br>$0.8545\pm 0.0044$<br>$0.8565\pm 0.0035$<br>$\mathrm{DCN2}$<br>$0.8567\pm 0.0020$<br>$0.8570\pm 0.0017$<br>$\mathrm{SNN}$<br>$0.8506\pm 0.0051$<br>$0.8533\pm 0.0033$<br>$\mathrm{Trompt}$<br>$0.8600\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.8607\pm 0.0047$<br>$0.8622\pm 0.0003$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.8592\pm 0.0036$<br>$0.8630\pm 0.0005$<br>$\mathrm{Excel}$<br>$0.8618\pm 0.0023$<br>$0.8625\pm nan$<br>$\mathrm{SAINT}$<br>$0.8603\pm 0.0029$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.8593\pm 0.0028$<br>$0.8598\pm 0.0025$<br>$\mathrm{T2G}$<br>$0.8613\pm 0.0015$<br>–<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.8624\pm 0.0010$<br>$0.8638\pm 0.0012$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.8624\pm 0.0026$<br>$0.8640\pm 0.0010$<br>$\mathrm{MLP^{\dagger}}$<br>$0.8580\pm 0.0028$<br>$0.8605\pm 0.0018$<br>$\mathrm{XGBoost}$<br>$0.8605\pm 0.0022$<br>$0.8608\pm 0.0013$<br>$\mathrm{LightGBM}$<br>$0.8600\pm 0.0008$<br>$0.8600\pm 0.0000$<br>$\mathrm{CatBoost}$<br>$0.8582\pm 0.0017$<br>$0.8588\pm 0.0008$<br>$\mathrm{TabR}$<br>$0.8599\pm 0.0025$<br>$0.8620\pm 0.0023$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.8625\pm 0.0021$<br>–<br>$\mathrm{MNCA}$<br>$0.8595\pm 0.0028$<br>$0.8615\pm 0.0013$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.8606\pm 0.0032$<br>$0.8607\pm 0.0008$<br>TabM<br>$0.8613\pm 0.0025$<br>$0.8615\pm 0.0005$<br>$\mathrm{TabM[G]}$<br>$0.8611\pm 0.0018$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.8625\pm 0.0025$<br>$0.8638\pm 0.0021$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.8608\pm 0.0019$<br>$0.8592\pm 0.0003$ | california ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.4948\pm 0.0058$<br>$0.4880\pm 0.0022$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.4915\pm 0.0031$<br>$0.4862\pm 0.0017$<br>$\mathrm{DCN2}$<br>$0.4971\pm 0.0122$<br>$0.4779\pm 0.0022$<br>$\mathrm{SNN}$<br>$0.5033\pm 0.0075$<br>$0.4933\pm 0.0035$<br>$\mathrm{Trompt}$<br>$0.4579\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.4682\pm 0.0063$<br>$0.4490\pm 0.0028$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.4746\pm 0.0056$<br>$0.4509\pm 0.0029$<br>$\mathrm{Excel}$<br>$0.4544\pm 0.0048$<br>$0.4350\pm nan$<br>$\mathrm{SAINT}$<br>$0.4680\pm 0.0048$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.4635\pm 0.0048$<br>$0.4515\pm 0.0016$<br>$\mathrm{T2G}$<br>$0.4640\pm 0.0100$<br>$0.4462\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.4652\pm 0.0045$<br>$0.4549\pm 0.0006$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.4597\pm 0.0058$<br>$0.4482\pm 0.0026$<br>$\mathrm{MLP^{\dagger}}$<br>$0.4530\pm 0.0029$<br>$0.4491\pm 0.0010$<br>$\mathrm{XGBoost}$<br>$0.4327\pm 0.0016$<br>$0.4316\pm 0.0007$<br>$\mathrm{LightGBM}$<br>$0.4352\pm 0.0019$<br>$0.4339\pm 0.0008$<br>$\mathrm{CatBoost}$<br>$0.4294\pm 0.0012$<br>$0.4265\pm 0.0003$<br>$\mathrm{TabR}$<br>$0.4030\pm 0.0023$<br>$0.3964\pm 0.0013$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.3998\pm 0.0033$<br>–<br>$\mathrm{MNCA}$<br>$0.4239\pm 0.0012$<br>$0.4231\pm 0.0005$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.4142\pm 0.0031$<br>$0.4071\pm 0.0029$<br>TabM<br>$0.4509\pm 0.0032$<br>$0.4490\pm 0.0018$<br>$\mathrm{TabM[G]}$<br>$0.4507\pm 0.0027$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.4476\pm 0.0036$<br>$0.4425\pm 0.0009$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.4314\pm 0.0036$<br>$0.4261\pm 0.0019$ |
| --- | --- |
| house ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$3.1117\pm 0.0294$<br>$3.0706\pm 0.0140$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$3.1143\pm 0.0258$<br>$3.0706\pm 0.0098$<br>$\mathrm{DCN2}$<br>$3.3327\pm 0.0878$<br>$3.1303\pm 0.0410$<br>$\mathrm{SNN}$<br>$3.2176\pm 0.0376$<br>$3.1320\pm 0.0155$<br>$\mathrm{Trompt}$<br>$3.0638\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$3.2157\pm 0.0436$<br>$3.1261\pm 0.0095$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$3.1871\pm 0.0519$<br>$3.0184\pm 0.0086$<br>$\mathrm{Excel}$<br>$3.2460\pm 0.0685$<br>$3.1097\pm nan$<br>$\mathrm{SAINT}$<br>$3.2424\pm 0.0595$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$3.1823\pm 0.0460$<br>$3.0974\pm 0.0334$<br>$\mathrm{T2G}$<br>$3.1613\pm 0.0320$<br>$3.0982\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$3.0633\pm 0.0248$<br>$3.0170\pm 0.0070$<br>$\mathrm{MLP^{\ddagger}}$<br>$3.0775\pm 0.0336$<br>$3.0268\pm 0.0170$<br>$\mathrm{MLP^{\dagger}}$<br>$3.0999\pm 0.0351$<br>$3.0401\pm 0.0071$<br>$\mathrm{XGBoost}$<br>$3.1773\pm 0.0102$<br>$3.1644\pm 0.0068$<br>$\mathrm{LightGBM}$<br>$3.1774\pm 0.0087$<br>$3.1672\pm 0.0050$<br>$\mathrm{CatBoost}$<br>$3.1172\pm 0.0125$<br>$3.1058\pm 0.0022$<br>$\mathrm{TabR}$<br>$3.0667\pm 0.0403$<br>$2.9958\pm 0.0270$<br>$\mathrm{TabR^{\ddagger}}$<br>$3.1048\pm 0.0410$<br>–<br>$\mathrm{MNCA}$<br>$3.0884\pm 0.0286$<br>$3.0538\pm 0.0072$<br>$\mathrm{MNCA^{\ddagger}}$<br>$3.0704\pm 0.0388$<br>$3.0149\pm 0.0308$<br>TabM<br>$3.0002\pm 0.0182$<br>$2.9796\pm 0.0024$<br>$\mathrm{TabM[G]}$<br>$3.0156\pm 0.0231$<br>–<br>$\mathrm{TabM_{mini}}$<br>$3.0496\pm 0.0225$<br>$3.0225\pm 0.0077$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$2.9902\pm 0.0271$<br>$2.9648\pm 0.0035$ | adult ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.8540\pm 0.0018$<br>$0.8559\pm 0.0011$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.8554\pm 0.0011$<br>$0.8562\pm 0.0006$<br>$\mathrm{DCN2}$<br>$0.8582\pm 0.0011$<br>$0.8593\pm 0.0002$<br>$\mathrm{SNN}$<br>$0.8582\pm 0.0009$<br>$0.8603\pm 0.0012$<br>$\mathrm{Trompt}$<br>$0.8590\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.8592\pm 0.0016$<br>$0.8612\pm 0.0004$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.8598\pm 0.0013$<br>$0.8617\pm 0.0002$<br>$\mathrm{Excel}$<br>$0.8613\pm 0.0024$<br>$0.8641\pm nan$<br>$\mathrm{SAINT}$<br>$0.8601\pm 0.0019$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.8588\pm 0.0015$<br>$0.8608\pm 0.0011$<br>$\mathrm{T2G}$<br>$0.8601\pm 0.0011$<br>$0.8622\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.8693\pm 0.0007$<br>$0.8702\pm 0.0006$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.8694\pm 0.0011$<br>$0.8704\pm 0.0008$<br>$\mathrm{MLP^{\dagger}}$<br>$0.8603\pm 0.0009$<br>$0.8616\pm 0.0006$<br>$\mathrm{XGBoost}$<br>$0.8720\pm 0.0006$<br>$0.8723\pm 0.0002$<br>$\mathrm{LightGBM}$<br>$0.8713\pm 0.0007$<br>$0.8721\pm 0.0004$<br>$\mathrm{CatBoost}$<br>$0.8714\pm 0.0012$<br>$0.8723\pm 0.0007$<br>$\mathrm{TabR}$<br>$0.8646\pm 0.0022$<br>$0.8680\pm 0.0019$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.8699\pm 0.0011$<br>–<br>$\mathrm{MNCA}$<br>$0.8677\pm 0.0018$<br>$0.8696\pm 0.0003$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.8717\pm 0.0008$<br>$0.8742\pm 0.0006$<br>TabM<br>$0.8582\pm 0.0011$<br>$0.8588\pm 0.0003$<br>$\mathrm{TabM[G]}$<br>$0.8577\pm 0.0009$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.8584\pm 0.0010$<br>$0.8591\pm 0.0005$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.8679\pm 0.0017$<br>$0.8690\pm 0.0005$ |
| diamond ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.1404\pm 0.0012$<br>$0.1362\pm 0.0003$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.1396\pm 0.0029$<br>$0.1361\pm 0.0011$<br>$\mathrm{DCN2}$<br>$0.1420\pm 0.0032$<br>$0.1374\pm 0.0020$<br>$\mathrm{SNN}$<br>$0.1473\pm 0.0057$<br>$0.1424\pm 0.0008$<br>$\mathrm{Trompt}$<br>$0.1391\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.1392\pm 0.0014$<br>$0.1361\pm 0.0004$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.1400\pm 0.0025$<br>$0.1378\pm 0.0008$<br>$\mathrm{Excel}$<br>$0.1766\pm 0.0023$<br>$0.1712\pm nan$<br>$\mathrm{SAINT}$<br>$0.1369\pm 0.0019$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.1376\pm 0.0013$<br>$0.1360\pm 0.0002$<br>$\mathrm{T2G}$<br>$0.1372\pm 0.0011$<br>$0.1346\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.1342\pm 0.0008$<br>$0.1325\pm 0.0004$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.1337\pm 0.0010$<br>$0.1317\pm 0.0003$<br>$\mathrm{MLP^{\dagger}}$<br>$0.1323\pm 0.0010$<br>$0.1301\pm 0.0005$<br>$\mathrm{XGBoost}$<br>$0.1368\pm 0.0004$<br>$0.1363\pm 0.0001$<br>$\mathrm{LightGBM}$<br>$0.1359\pm 0.0002$<br>$0.1358\pm 0.0001$<br>$\mathrm{CatBoost}$<br>$0.1335\pm 0.0006$<br>$0.1327\pm 0.0004$<br>$\mathrm{TabR}$<br>$0.1327\pm 0.0010$<br>$0.1311\pm 0.0005$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.1333\pm 0.0013$<br>–<br>$\mathrm{MNCA}$<br>$0.1370\pm 0.0018$<br>$0.1348\pm 0.0005$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.1327\pm 0.0012$<br>$0.1315\pm 0.0006$<br>TabM<br>$0.1342\pm 0.0017$<br>$0.1327\pm 0.0004$<br>$\mathrm{TabM[G]}$<br>$0.1340\pm 0.0014$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.1367\pm 0.0015$<br>$0.1352\pm 0.0008$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.1320\pm 0.0010$<br>$0.1307\pm 0.0005$ | otto ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.8175\pm 0.0022$<br>$0.8222\pm 0.0007$<br>$\mathrm{TabPFN}$<br>–<br>$0.7408\pm 0.0028$<br>$\mathrm{ResNet}$<br>$0.8174\pm 0.0021$<br>$0.8198\pm 0.0006$<br>$\mathrm{DCN2}$<br>$0.8064\pm 0.0021$<br>$0.8208\pm 0.0023$<br>$\mathrm{SNN}$<br>$0.8087\pm 0.0020$<br>$0.8156\pm 0.0013$<br>$\mathrm{Trompt}$<br>$0.8093\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.8050\pm 0.0034$<br>$0.8111\pm 0.0020$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.8092\pm 0.0040$<br>$0.8136\pm 0.0010$<br>$\mathrm{Excel}$<br>$0.8102\pm 0.0022$<br>$0.8220\pm nan$<br>$\mathrm{SAINT}$<br>$0.8119\pm 0.0018$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.8133\pm 0.0033$<br>$0.8221\pm 0.0013$<br>$\mathrm{T2G}$<br>$0.8161\pm 0.0019$<br>$0.8272\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.8190\pm 0.0021$<br>$0.8271\pm 0.0015$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.8189\pm 0.0015$<br>$0.8253\pm 0.0000$<br>$\mathrm{MLP^{\dagger}}$<br>$0.8205\pm 0.0021$<br>$0.8290\pm 0.0006$<br>$\mathrm{XGBoost}$<br>$0.8297\pm 0.0011$<br>$0.8316\pm 0.0008$<br>$\mathrm{LightGBM}$<br>$0.8302\pm 0.0009$<br>$0.8316\pm 0.0013$<br>$\mathrm{CatBoost}$<br>$0.8250\pm 0.0013$<br>$0.8268\pm 0.0002$<br>$\mathrm{TabR}$<br>$0.8179\pm 0.0022$<br>$0.8236\pm 0.0009$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.8246\pm 0.0018$<br>–<br>$\mathrm{MNCA}$<br>$0.8275\pm 0.0012$<br>$0.8313\pm 0.0006$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.8265\pm 0.0015$<br>$0.8304\pm 0.0006$<br>TabM<br>$0.8268\pm 0.0014$<br>$0.8300\pm 0.0007$<br>$\mathrm{TabM[G]}$<br>$0.8204\pm 0.0025$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.8267\pm 0.0011$<br>$0.8298\pm 0.0008$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.8342\pm 0.0014$<br>$0.8365\pm 0.0005$ |
| higgs-small ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.7180\pm 0.0027$<br>$0.7192\pm 0.0005$<br>$\mathrm{TabPFN}$<br>–<br>$0.6727\pm 0.0034$<br>$\mathrm{ResNet}$<br>$0.7256\pm 0.0020$<br>$0.7307\pm 0.0001$<br>$\mathrm{DCN2}$<br>$0.7164\pm 0.0030$<br>$0.7237\pm 0.0011$<br>$\mathrm{SNN}$<br>$0.7142\pm 0.0024$<br>$0.7171\pm 0.0020$<br>$\mathrm{Trompt}$<br>$0.7262\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.7240\pm 0.0028$<br>$0.7287\pm 0.0008$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.7248\pm 0.0023$<br>$0.7334\pm 0.0007$<br>$\mathrm{Excel}$<br>$0.7262\pm 0.0017$<br>$0.7329\pm nan$<br>$\mathrm{SAINT}$<br>$0.7236\pm 0.0019$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.7281\pm 0.0016$<br>$0.7334\pm 0.0013$<br>$\mathrm{T2G}$<br>$0.7352\pm 0.0037$<br>$0.7400\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.7260\pm 0.0017$<br>$0.7304\pm 0.0008$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.7261\pm 0.0010$<br>$0.7270\pm 0.0003$<br>$\mathrm{MLP^{\dagger}}$<br>$0.7210\pm 0.0016$<br>$0.7252\pm 0.0005$<br>$\mathrm{XGBoost}$<br>$0.7246\pm 0.0015$<br>$0.7264\pm 0.0013$<br>$\mathrm{LightGBM}$<br>$0.7256\pm 0.0009$<br>$0.7263\pm 0.0007$<br>$\mathrm{CatBoost}$<br>$0.7260\pm 0.0011$<br>$0.7273\pm 0.0010$<br>$\mathrm{TabR}$<br>$0.7223\pm 0.0010$<br>$0.7257\pm 0.0008$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.7294\pm 0.0014$<br>–<br>$\mathrm{MNCA}$<br>$0.7263\pm 0.0023$<br>$0.7292\pm 0.0006$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.7300\pm 0.0020$<br>$0.7348\pm 0.0008$<br>TabM<br>$0.7383\pm 0.0028$<br>$0.7409\pm 0.0010$<br>$\mathrm{TabM[G]}$<br>$0.7372\pm 0.0021$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.7344\pm 0.0016$<br>$0.7366\pm 0.0012$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.7348\pm 0.0017$<br>$0.7379\pm 0.0006$ | black-friday ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.6955\pm 0.0004$<br>$0.6942\pm 0.0002$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.6929\pm 0.0008$<br>$0.6907\pm 0.0002$<br>$\mathrm{DCN2}$<br>$0.6968\pm 0.0013$<br>$0.6936\pm 0.0007$<br>$\mathrm{SNN}$<br>$0.6996\pm 0.0013$<br>$0.6978\pm 0.0004$<br>$\mathrm{Trompt}$<br>$0.6983\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.6994\pm 0.0082$<br>$0.6927\pm 0.0021$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.6905\pm 0.0021$<br>$0.6851\pm 0.0011$<br>$\mathrm{Excel}$<br>$0.6947\pm 0.0016$<br>$0.6908\pm nan$<br>$\mathrm{SAINT}$<br>$0.6934\pm 0.0009$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.6987\pm 0.0192$<br>$0.6879\pm 0.0023$<br>$\mathrm{T2G}$<br>$0.6887\pm 0.0046$<br>$0.6832\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.6849\pm 0.0006$<br>$0.6824\pm 0.0002$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.6857\pm 0.0004$<br>$0.6838\pm 0.0002$<br>$\mathrm{MLP^{\dagger}}$<br>$0.6836\pm 0.0006$<br>$0.6812\pm 0.0002$<br>$\mathrm{XGBoost}$<br>$0.6806\pm 0.0001$<br>$0.6805\pm 0.0000$<br>$\mathrm{LightGBM}$<br>$0.6799\pm 0.0003$<br>$0.6795\pm 0.0001$<br>$\mathrm{CatBoost}$<br>$0.6822\pm 0.0003$<br>$0.6813\pm 0.0002$<br>$\mathrm{TabR}$<br>$0.6899\pm 0.0004$<br>$0.6883\pm 0.0002$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.6761\pm 0.0009$<br>–<br>$\mathrm{MNCA}$<br>$0.6893\pm 0.0004$<br>$0.6883\pm 0.0000$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.6885\pm 0.0007$<br>$0.6863\pm 0.0003$<br>TabM<br>$0.6875\pm 0.0015$<br>$0.6866\pm 0.0003$<br>$\mathrm{TabM[G]}$<br>$0.6870\pm 0.0014$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.6865\pm 0.0016$<br>$0.6843\pm 0.0005$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.6807\pm 0.0013$<br>$0.6783\pm 0.0009$ |
| covtype2 ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.9630\pm 0.0012$<br>$0.9664\pm 0.0004$<br>$\mathrm{TabPFN}$<br>–<br>$0.7606\pm 0.0022$<br>$\mathrm{ResNet}$<br>$0.9638\pm 0.0005$<br>$0.9685\pm 0.0003$<br>$\mathrm{DCN2}$<br>$0.9622\pm 0.0019$<br>$0.9673\pm 0.0011$<br>$\mathrm{SNN}$<br>$0.9636\pm 0.0010$<br>$0.9677\pm 0.0002$<br>$\mathrm{Trompt}$<br>$0.9286\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.9614\pm 0.0016$<br>$0.9696\pm 0.0005$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.9663\pm 0.0019$<br>$0.9699\pm 0.0014$<br>$\mathrm{Excel}$<br>$0.9606\pm 0.0018$<br>$0.9670\pm nan$<br>$\mathrm{SAINT}$<br>$0.9669\pm 0.0010$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.9698\pm 0.0008$<br>$0.9731\pm 0.0006$<br>$\mathrm{T2G}$<br>$0.9668\pm 0.0008$<br>$0.9708\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.9690\pm 0.0008$<br>$0.9721\pm 0.0006$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.9713\pm 0.0006$<br>$0.9758\pm 0.0000$<br>$\mathrm{MLP^{\dagger}}$<br>$0.9697\pm 0.0008$<br>$0.9721\pm 0.0005$<br>$\mathrm{XGBoost}$<br>$0.9710\pm 0.0002$<br>$0.9713\pm 0.0000$<br>$\mathrm{LightGBM}$<br>$0.9709\pm 0.0003$<br>–<br>$\mathrm{CatBoost}$<br>$0.9670\pm 0.0003$<br>$0.9680\pm 0.0002$<br>$\mathrm{TabR}$<br>$0.9737\pm 0.0005$<br>$0.9745\pm 0.0006$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.9752\pm 0.0003$<br>–<br>$\mathrm{MNCA}$<br>$0.9724\pm 0.0003$<br>$0.9729\pm 0.0001$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.9747\pm 0.0002$<br>$0.9747\pm 0.0002$<br>TabM<br>$0.9712\pm 0.0008$<br>$0.9729\pm 0.0003$<br>$\mathrm{TabM[G]}$<br>$0.9707\pm 0.0008$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.9693\pm 0.0008$<br>$0.9713\pm 0.0001$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.9740\pm 0.0006$<br>$0.9754\pm 0.0001$ | microsoft ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.7475\pm 0.0003$<br>$0.7460\pm 0.0003$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.7472\pm 0.0004$<br>$0.7452\pm 0.0004$<br>$\mathrm{DCN2}$<br>$0.7499\pm 0.0003$<br>$0.7477\pm 0.0001$<br>$\mathrm{SNN}$<br>$0.7488\pm 0.0004$<br>$0.7470\pm 0.0001$<br>$\mathrm{Trompt}$<br>$0.7476\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.7482\pm 0.0005$<br>$0.7455\pm 0.0002$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.7482\pm 0.0008$<br>$0.7436\pm 0.0001$<br>$\mathrm{Excel}$<br>$0.7479\pm 0.0007$<br>$0.7442\pm nan$<br>$\mathrm{SAINT}$<br>$0.7625\pm 0.0066$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.7460\pm 0.0007$<br>$0.7422\pm 0.0004$<br>$\mathrm{T2G}$<br>$0.7460\pm 0.0006$<br>$0.7427\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.7446\pm 0.0002$<br>$0.7434\pm 0.0002$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.7444\pm 0.0003$<br>$0.7429\pm 0.0001$<br>$\mathrm{MLP^{\dagger}}$<br>$0.7465\pm 0.0005$<br>$0.7448\pm 0.0001$<br>$\mathrm{XGBoost}$<br>$0.7413\pm 0.0001$<br>$0.7410\pm 0.0000$<br>$\mathrm{LightGBM}$<br>$0.7417\pm 0.0001$<br>$0.7413\pm 0.0000$<br>$\mathrm{CatBoost}$<br>$0.7412\pm 0.0001$<br>$0.7406\pm 0.0000$<br>$\mathrm{TabR}$<br>$0.7503\pm 0.0006$<br>$0.7485\pm 0.0002$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.7501\pm 0.0005$<br>–<br>$\mathrm{MNCA}$<br>$0.7458\pm 0.0003$<br>$0.7448\pm 0.0002$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.7460\pm 0.0008$<br>$0.7435\pm 0.0004$<br>$\mathrm{TabM}$<br>$0.7434\pm 0.0003$<br>$0.7424\pm 0.0001$<br>$\mathrm{TabM[G]}$<br>$0.7435\pm 0.0003$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.7444\pm 0.0003$<br>$0.7431\pm 0.0002$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.7427\pm 0.0002$<br>$0.7416\pm 0.0002$ |

**Table 19: Extended results for the main benchmark. Results are grouped by datasets.**
| wine ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.7778\pm 0.0153$<br>$0.7907\pm 0.0117$<br>$\mathrm{TabPFN}$<br>–<br>$0.7908\pm 0.0063$<br>$\mathrm{ResNet}$<br>$0.7710\pm 0.0137$<br>$0.7839\pm 0.0083$<br>$\mathrm{DCN2}$<br>$0.7492\pm 0.0147$<br>$0.7764\pm 0.0095$<br>$\mathrm{SNN}$<br>$0.7818\pm 0.0143$<br>$0.7994\pm 0.0097$<br>$\mathrm{Trompt}$<br>$0.7818\pm 0.0081$<br>–<br>$\mathrm{AutoInt}$<br>$0.7745\pm 0.0144$<br>$0.7909\pm 0.0160$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.7769\pm 0.0149$<br>$0.7950\pm 0.0087$<br>$\mathrm{Excel}$<br>$0.7631\pm 0.0171$<br>$0.7765\pm 0.0121$<br>$\mathrm{SAINT}$<br>$0.7684\pm 0.0144$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.7755\pm 0.0133$<br>$0.7894\pm 0.0083$<br>$\mathrm{T2G}$<br>$0.7733\pm 0.0118$<br>$0.7933\pm 0.0137$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.7803\pm 0.0157$<br>$0.7964\pm 0.0146$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.7733\pm 0.0185$<br>$0.7856\pm 0.0160$<br>$\mathrm{MLP^{\dagger}}$<br>$0.7814\pm 0.0132$<br>$0.7919\pm 0.0098$<br>$\mathrm{XGBoost}$<br>$0.7949\pm 0.0178$<br>$0.8010\pm 0.0186$<br>$\mathrm{LightGBM}$<br>$0.7890\pm 0.0160$<br>$0.7929\pm 0.0106$<br>$\mathrm{CatBoost}$<br>$0.7994\pm 0.0131$<br>$0.8057\pm 0.0098$<br>$\mathrm{TabR}$<br>$0.7936\pm 0.0114$<br>$0.8055\pm 0.0057$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.7804\pm 0.0148$<br>–<br>$\mathrm{MNCA}$<br>$0.7911\pm 0.0135$<br>$0.8005\pm 0.0121$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.7867\pm 0.0113$<br>$0.7953\pm 0.0114$<br>$\mathrm{TabM}$<br>$0.7961\pm 0.0136$<br>$0.8011\pm 0.0084$<br>$\mathrm{TabM[G]}$<br>$0.7855\pm 0.0164$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.7904\pm 0.0123$<br>$0.7986\pm 0.0055$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.7886\pm 0.0167$<br>$0.7963\pm 0.0113$ | phoneme ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.8525\pm 0.0126$<br>$0.8635\pm 0.0099$<br>$\mathrm{TabPFN}$<br>–<br>$0.8684\pm 0.0050$<br>$\mathrm{ResNet}$<br>$0.8456\pm 0.0121$<br>$0.8504\pm 0.0066$<br>$\mathrm{DCN2}$<br>$0.8342\pm 0.0151$<br>$0.8543\pm 0.0118$<br>$\mathrm{SNN}$<br>$0.8596\pm 0.0124$<br>$0.8687\pm 0.0080$<br>$\mathrm{Trompt}$<br>$0.8465\pm 0.0205$<br>–<br>$\mathrm{AutoInt}$<br>$0.8623\pm 0.0138$<br>$0.8754\pm 0.0095$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.8629\pm 0.0123$<br>$0.8757\pm 0.0095$<br>$\mathrm{Excel}$<br>$0.8551\pm 0.0092$<br>$0.8711\pm 0.0081$<br>$\mathrm{SAINT}$<br>$0.8657\pm 0.0130$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.8667\pm 0.0127$<br>$0.8795\pm 0.0093$<br>$\mathrm{T2G}$<br>$0.8672\pm 0.0166$<br>$0.8765\pm 0.0141$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.8742\pm 0.0120$<br>$0.8861\pm 0.0071$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.8757\pm 0.0118$<br>$0.8856\pm 0.0065$<br>$\mathrm{MLP^{\dagger}}$<br>$0.8647\pm 0.0098$<br>$0.8761\pm 0.0076$<br>$\mathrm{XGBoost}$<br>$0.8682\pm 0.0174$<br>$0.8771\pm 0.0156$<br>$\mathrm{LightGBM}$<br>$0.8702\pm 0.0129$<br>$0.8733\pm 0.0126$<br>$\mathrm{CatBoost}$<br>$0.8827\pm 0.0117$<br>$0.8897\pm 0.0055$<br>$\mathrm{TabR}$<br>$0.8781\pm 0.0096$<br>$0.8840\pm 0.0054$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.8772\pm 0.0087$<br>–<br>$\mathrm{MNCA}$<br>$0.8835\pm 0.0079$<br>$0.8861\pm 0.0057$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.8828\pm 0.0082$<br>$0.8925\pm 0.0056$<br>$\mathrm{TabM}$<br>$0.8701\pm 0.0167$<br>$0.8766\pm 0.0128$<br>$\mathrm{TabM[G]}$<br>$0.8668\pm 0.0180$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.8686\pm 0.0153$<br>$0.8758\pm 0.0091$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.8790\pm 0.0098$<br>$0.8885\pm 0.0056$ |
| --- | --- |
| analcatdata_supreme ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.0782\pm 0.0081$<br>$0.0766\pm 0.0090$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.0852\pm 0.0076$<br>$0.0823\pm 0.0078$<br>$\mathrm{DCN2}$<br>$0.0811\pm 0.0137$<br>$0.0759\pm 0.0086$<br>$\mathrm{SNN}$<br>$0.0826\pm 0.0096$<br>$0.0779\pm 0.0098$<br>$\mathrm{Trompt}$<br>$0.0782\pm 0.0095$<br>–<br>$\mathrm{AutoInt}$<br>$0.0783\pm 0.0078$<br>$0.0768\pm 0.0083$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.0770\pm 0.0082$<br>$0.0759\pm 0.0081$<br>$\mathrm{Excel}$<br>$0.0796\pm 0.0101$<br>$0.0776\pm 0.0101$<br>$\mathrm{SAINT}$<br>$0.0773\pm 0.0078$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.0787\pm 0.0086$<br>$0.0775\pm 0.0091$<br>$\mathrm{T2G}$<br>$0.0775\pm 0.0081$<br>$0.0763\pm 0.0084$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.0798\pm 0.0088$<br>$0.0769\pm 0.0092$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.0786\pm 0.0073$<br>$0.0720\pm 0.0053$<br>$\mathrm{MLP^{\dagger}}$<br>$0.0774\pm 0.0064$<br>$0.0759\pm 0.0063$<br>$\mathrm{XGBoost}$<br>$0.0801\pm 0.0126$<br>$0.0774\pm 0.0107$<br>$\mathrm{LightGBM}$<br>$0.0778\pm 0.0115$<br>$0.0767\pm 0.0110$<br>$\mathrm{CatBoost}$<br>$0.0780\pm 0.0067$<br>$0.0734\pm 0.0022$<br>$\mathrm{TabR}$<br>$0.0803\pm 0.0066$<br>$0.0759\pm 0.0046$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.0807\pm 0.0088$<br>–<br>$\mathrm{MNCA}$<br>$0.0809\pm 0.0072$<br>$0.0784\pm 0.0062$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.0825\pm 0.0090$<br>$0.0793\pm 0.0072$<br>$\mathrm{TabM}$<br>$0.0777\pm 0.0099$<br>$0.0769\pm 0.0105$<br>$\mathrm{TabM[G]}$<br>$0.0783\pm 0.0103$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.0769\pm 0.0091$<br>$0.0758\pm 0.0097$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.0790\pm 0.0079$<br>$0.0770\pm 0.0086$ | Mercedes_Benz_Greener_Manufacturing ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$8.3045\pm 0.8708$<br>$8.2682\pm 0.8992$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$8.4434\pm 0.7982$<br>$8.3178\pm 0.8482$<br>$\mathrm{DCN2}$<br>$8.3540\pm 0.8314$<br>$8.3021\pm 0.8579$<br>$\mathrm{SNN}$<br>$8.2718\pm 0.8152$<br>$8.2236\pm 0.8479$<br>$\mathrm{Trompt}$<br>$8.3409\pm 0.9840$<br>–<br>$\mathrm{AutoInt}$<br>$8.4001\pm 0.9256$<br>$8.3237\pm 0.9658$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$8.2860\pm 0.8656$<br>$8.2398\pm 0.9023$<br>$\mathrm{Excel}$<br>$8.2244\pm 0.8514$<br>$8.1918\pm 0.9387$<br>$\mathrm{SAINT}$<br>$8.3556\pm 0.9566$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$8.2252\pm 0.8617$<br>$8.1616\pm 0.8834$<br>$\mathrm{T2G}$<br>$8.2120\pm 0.8485$<br>$8.1654\pm 0.9339$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$8.3045\pm 0.8708$<br>$8.2682\pm 0.8992$<br>$\mathrm{MLP^{\ddagger}}$<br>$8.3045\pm 0.8708$<br>$8.2682\pm 0.8992$<br>$\mathrm{MLP^{\dagger}}$<br>$8.3045\pm 0.8708$<br>$8.2682\pm 0.8992$<br>$\mathrm{XGBoost}$<br>$8.2177\pm 0.8175$<br>$8.2092\pm 0.8458$<br>$\mathrm{LightGBM}$<br>$8.2078\pm 0.8231$<br>$8.1618\pm 0.8566$<br>$\mathrm{CatBoost}$<br>$8.1629\pm 0.8193$<br>$8.1554\pm 0.8439$<br>$\mathrm{TabR}$<br>$8.3506\pm 0.8149$<br>$8.2694\pm 0.8399$<br>$\mathrm{TabR^{\ddagger}}$<br>$8.3187\pm 0.8186$<br>–<br>$\mathrm{MNCA}$<br>$8.2557\pm 0.8602$<br>$8.1771\pm 0.8710$<br>$\mathrm{MNCA^{\ddagger}}$<br>$8.2557\pm 0.8602$<br>$8.1771\pm 0.8710$<br>$\mathrm{TabM}$<br>$8.2215\pm 0.8940$<br>$8.1995\pm 0.9130$<br>$\mathrm{TabM[G]}$<br>$8.2206\pm 0.8827$<br>–<br>$\mathrm{TabM_{mini}}$<br>$8.2375\pm 0.8953$<br>$8.2161\pm 0.9253$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$8.2375\pm 0.8953$<br>$8.2161\pm 0.9253$ |
| KDDCup09_upselling ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.7759\pm 0.0137$<br>$0.7806\pm 0.0125$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.7811\pm 0.0124$<br>$0.7861\pm 0.0109$<br>$\mathrm{DCN2}$<br>$0.7850\pm 0.0161$<br>$0.7884\pm 0.0135$<br>$\mathrm{SNN}$<br>$0.7884\pm 0.0122$<br>$0.7940\pm 0.0116$<br>$\mathrm{Trompt}$<br>$0.7994\pm 0.0055$<br>–<br>$\mathrm{AutoInt}$<br>$0.8004\pm 0.0075$<br>$0.8037\pm 0.0063$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.7979\pm 0.0105$<br>$0.8010\pm 0.0094$<br>$\mathrm{Excel}$<br>$0.7903\pm 0.0074$<br>$0.7939\pm 0.0099$<br>$\mathrm{SAINT}$<br>$0.7942\pm 0.0112$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.7957\pm 0.0127$<br>$0.7960\pm 0.0139$<br>$\mathrm{T2G}$<br>$0.8037\pm 0.0100$<br>$0.7988\pm 0.0084$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.7962\pm 0.0093$<br>$0.7995\pm 0.0105$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.8005\pm 0.0097$<br>$0.8032\pm 0.0117$<br>$\mathrm{MLP^{\dagger}}$<br>$0.7925\pm 0.0123$<br>$0.7963\pm 0.0089$<br>$\mathrm{XGBoost}$<br>$0.7930\pm 0.0108$<br>$0.7950\pm 0.0102$<br>$\mathrm{LightGBM}$<br>$0.7932\pm 0.0119$<br>$0.7969\pm 0.0115$<br>$\mathrm{CatBoost}$<br>$0.7992\pm 0.0117$<br>$0.8010\pm 0.0121$<br>$\mathrm{TabR}$<br>$0.7838\pm 0.0136$<br>$0.7859\pm 0.0167$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.7908\pm 0.0123$<br>–<br>$\mathrm{MNCA}$<br>$0.7939\pm 0.0097$<br>$0.7989\pm 0.0115$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.7960\pm 0.0131$<br>$0.8008\pm 0.0110$<br>$\mathrm{TabM}$<br>$0.8002\pm 0.0103$<br>$0.8021\pm 0.0074$<br>$\mathrm{TabM[G]}$<br>$0.7974\pm 0.0124$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.7963\pm 0.0123$<br>$0.8018\pm 0.0076$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.8031\pm 0.0133$<br>$0.8039\pm 0.0114$ | kdd_ipums_la_97-small ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.8828\pm 0.0061$<br>$0.8845\pm 0.0055$<br>$\mathrm{TabPFN}$<br>–<br>$0.8578\pm 0.0046$<br>$\mathrm{ResNet}$<br>$0.8823\pm 0.0070$<br>$0.8824\pm 0.0060$<br>$\mathrm{DCN2}$<br>$0.8770\pm 0.0072$<br>$0.8824\pm 0.0068$<br>$\mathrm{SNN}$<br>$0.8722\pm 0.0093$<br>$0.8733\pm 0.0083$<br>$\mathrm{Trompt}$<br>$0.8847\pm 0.0070$<br>–<br>$\mathrm{AutoInt}$<br>$0.8808\pm 0.0083$<br>$0.8830\pm 0.0081$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.8762\pm 0.0100$<br>$0.8770\pm 0.0088$<br>$\mathrm{Excel}$<br>$0.8803\pm 0.0054$<br>$0.8823\pm 0.0071$<br>$\mathrm{SAINT}$<br>$0.8837\pm 0.0055$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.8795\pm 0.0077$<br>$0.8792\pm 0.0062$<br>$\mathrm{T2G}$<br>$0.8833\pm 0.0054$<br>$0.8841\pm 0.0062$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.8765\pm 0.0108$<br>$0.8765\pm 0.0108$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.8816\pm 0.0057$<br>$0.8818\pm 0.0048$<br>$\mathrm{MLP^{\dagger}}$<br>$0.8757\pm 0.0101$<br>$0.8756\pm 0.0104$<br>$\mathrm{XGBoost}$<br>$0.8825\pm 0.0089$<br>$0.8835\pm 0.0085$<br>$\mathrm{LightGBM}$<br>$0.8792\pm 0.0075$<br>$0.8802\pm 0.0067$<br>$\mathrm{CatBoost}$<br>$0.8793\pm 0.0088$<br>$0.8803\pm 0.0100$<br>$\mathrm{TabR}$<br>$0.8798\pm 0.0081$<br>$0.8819\pm 0.0078$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.8831\pm 0.0050$<br>–<br>$\mathrm{MNCA}$<br>$0.8819\pm 0.0054$<br>$0.8832\pm 0.0048$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.8837\pm 0.0062$<br>$0.8860\pm 0.0059$<br>$\mathrm{TabM}$<br>$0.8845\pm 0.0063$<br>$0.8848\pm 0.0070$<br>$\mathrm{TabM[G]}$<br>$0.8846\pm 0.0059$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.8827\pm 0.0054$<br>$0.8810\pm 0.0050$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.8775\pm 0.0094$<br>$0.8780\pm 0.0099$ |
| wine_quality ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.6707\pm 0.0178$<br>$0.6530\pm 0.0152$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.6687\pm 0.0166$<br>$0.6543\pm 0.0170$<br>$\mathrm{DCN2}$<br>$0.7010\pm 0.0171$<br>$0.6699\pm 0.0139$<br>$\mathrm{SNN}$<br>$0.6604\pm 0.0174$<br>$0.6245\pm 0.0140$<br>$\mathrm{Trompt}$<br>$0.6605\pm 0.0153$<br>–<br>$\mathrm{AutoInt}$<br>$0.6840\pm 0.0126$<br>$0.6478\pm 0.0146$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.6672\pm 0.0263$<br>$0.6294\pm 0.0200$<br>$\mathrm{Excel}$<br>$0.6881\pm 0.0182$<br>$0.6664\pm 0.0179$<br>$\mathrm{SAINT}$<br>$0.6797\pm 0.0161$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.6787\pm 0.0149$<br>$0.6564\pm 0.0250$<br>$\mathrm{T2G}$<br>$0.6783\pm 0.0170$<br>$0.6570\pm 0.0273$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.6569\pm 0.0167$<br>$0.6328\pm 0.0155$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.6532\pm 0.0133$<br>$0.6336\pm 0.0140$<br>$\mathrm{MLP^{\dagger}}$<br>$0.6721\pm 0.0180$<br>$0.6463\pm 0.0262$<br>$\mathrm{XGBoost}$<br>$0.6039\pm 0.0134$<br>$0.6025\pm 0.0139$<br>$\mathrm{LightGBM}$<br>$0.6135\pm 0.0138$<br>$0.6122\pm 0.0144$<br>$\mathrm{CatBoost}$<br>$0.6088\pm 0.0132$<br>$0.6060\pm 0.0137$<br>$\mathrm{TabR}$<br>$0.6315\pm 0.0097$<br>$0.6197\pm 0.0096$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.6412\pm 0.0105$<br>–<br>$\mathrm{MNCA}$<br>$0.6154\pm 0.0083$<br>$0.6058\pm 0.0149$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.6099\pm 0.0144$<br>$0.6028\pm 0.0157$<br>$\mathrm{TabM}$<br>$0.6169\pm 0.0123$<br>$0.6131\pm 0.0126$<br>$\mathrm{TabM[G]}$<br>$0.6225\pm 0.0114$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.6193\pm 0.0130$<br>$0.6138\pm 0.0140$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.6255\pm 0.0146$<br>$0.6194\pm 0.0150$ | isolet ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$2.2744\pm 0.2203$<br>$2.0018\pm 0.1111$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$2.2077\pm 0.2248$<br>$1.9206\pm 0.1478$<br>$\mathrm{DCN2}$<br>$2.2449\pm 0.1579$<br>$2.0176\pm 0.0770$<br>$\mathrm{SNN}$<br>$2.4269\pm 0.2382$<br>$2.1142\pm 0.1262$<br>$\mathrm{Trompt}$<br>$2.6219\pm 0.0315$<br>–<br>$\mathrm{AutoInt}$<br>$2.6130\pm 0.1658$<br>$2.3308\pm 0.1088$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$2.3344\pm 0.2073$<br>$2.0915\pm 0.1159$<br>$\mathrm{Excel}$<br>$2.8691\pm 0.0882$<br>$2.5989\pm 0.0664$<br>$\mathrm{SAINT}$<br>$2.7696\pm 0.0200$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$2.4879\pm 0.2524$<br>$2.1501\pm 0.1506$<br>$\mathrm{T2G}$<br>$2.2867\pm 0.2489$<br>$1.9179\pm 0.1530$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$2.2719\pm 0.1006$<br>$2.1026\pm 0.1088$<br>$\mathrm{MLP^{\ddagger}}$<br>$2.1832\pm 0.1124$<br>$2.0775\pm 0.0805$<br>$\mathrm{MLP^{\dagger}}$<br>$2.0979\pm 0.1779$<br>$1.9283\pm 0.1334$<br>$\mathrm{XGBoost}$<br>$2.7567\pm 0.0470$<br>$2.7294\pm 0.0366$<br>$\mathrm{LightGBM}$<br>$2.7005\pm 0.0296$<br>$2.6903\pm 0.0290$<br>$\mathrm{CatBoost}$<br>$2.8847\pm 0.0227$<br>$2.8574\pm 0.0148$<br>$\mathrm{TabR}$<br>$1.9760\pm 0.1738$<br>$1.7627\pm 0.1520$<br>$\mathrm{TabR^{\ddagger}}$<br>$1.9919\pm 0.1813$<br>–<br>$\mathrm{MNCA}$<br>$1.7905\pm 0.1594$<br>$1.6205\pm 0.1676$<br>$\mathrm{MNCA^{\ddagger}}$<br>$1.8912\pm 0.1851$<br>$1.7147\pm 0.1348$<br>$\mathrm{TabM}$<br>$1.8831\pm 0.1194$<br>$1.8578\pm 0.1088$<br>$\mathrm{TabM[G]}$<br>$1.9549\pm 0.1319$<br>–<br>$\mathrm{TabM_{mini}}$<br>$1.9966\pm 0.0923$<br>$1.9311\pm 0.0862$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$1.8378\pm 0.0803$<br>$1.8126\pm 0.0692$ |
| cpu_act ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$2.6814\pm 0.2291$<br>$2.4953\pm 0.1150$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$2.3933\pm 0.0641$<br>$2.3005\pm 0.0397$<br>$\mathrm{DCN2}$<br>$2.7868\pm 0.1999$<br>$2.4884\pm 0.0327$<br>$\mathrm{SNN}$<br>$2.5811\pm 0.1480$<br>$2.3863\pm 0.0324$<br>$\mathrm{Trompt}$<br>$2.2133\pm 0.0221$<br>–<br>$\mathrm{AutoInt}$<br>$2.2537\pm 0.0536$<br>$2.1708\pm 0.0349$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$2.3079\pm 0.0829$<br>$2.1831\pm 0.0470$<br>$\mathrm{Excel}$<br>$2.3094\pm 0.2401$<br>$2.1411\pm 0.0767$<br>$\mathrm{SAINT}$<br>$2.2781\pm 0.0630$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$2.2394\pm 0.0508$<br>$2.1494\pm 0.0268$<br>$\mathrm{T2G}$<br>$2.2111\pm 0.0413$<br>$2.1330\pm 0.0316$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$2.2730\pm 0.0457$<br>$2.1899\pm 0.0419$<br>$\mathrm{MLP^{\ddagger}}$<br>$2.2671\pm 0.0383$<br>$2.1940\pm 0.0433$<br>$\mathrm{MLP^{\dagger}}$<br>$2.3309\pm 0.0719$<br>$2.2516\pm 0.0574$<br>$\mathrm{XGBoost}$<br>$2.5237\pm 0.3530$<br>$2.4723\pm 0.3789$<br>$\mathrm{LightGBM}$<br>$2.2223\pm 0.0894$<br>$2.2067\pm 0.0916$<br>$\mathrm{CatBoost}$<br>$2.1239\pm 0.0489$<br>$2.1092\pm 0.0499$<br>$\mathrm{TabR}$<br>$2.2980\pm 0.0529$<br>$2.2228\pm 0.0501$<br>$\mathrm{TabR^{\ddagger}}$<br>$2.1278\pm 0.0783$<br>–<br>$\mathrm{MNCA}$<br>$2.2603\pm 0.0479$<br>$2.2339\pm 0.0508$<br>$\mathrm{MNCA^{\ddagger}}$<br>$2.2105\pm 0.0483$<br>$2.1396\pm 0.0474$<br>$\mathrm{TabM}$<br>$2.1940\pm 0.0523$<br>$2.1677\pm 0.0487$<br>$\mathrm{TabM[G]}$<br>$2.2033\pm 0.0552$<br>–<br>$\mathrm{TabM_{mini}}$<br>$2.2254\pm 0.0734$<br>$2.1877\pm 0.0541$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$2.1572\pm 0.0376$<br>$2.1222\pm 0.0358$ | bank-marketing ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.7860\pm 0.0057$<br>$0.7887\pm 0.0052$<br>$\mathrm{TabPFN}$<br>–<br>$0.7894\pm 0.0091$<br>$\mathrm{ResNet}$<br>$0.7921\pm 0.0076$<br>$0.7932\pm 0.0066$<br>$\mathrm{DCN2}$<br>$0.7859\pm 0.0068$<br>$0.7917\pm 0.0078$<br>$\mathrm{SNN}$<br>$0.7836\pm 0.0074$<br>$0.7882\pm 0.0054$<br>$\mathrm{Trompt}$<br>$0.7975\pm 0.0080$<br>–<br>$\mathrm{AutoInt}$<br>$0.7917\pm 0.0071$<br>$0.7956\pm 0.0058$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.7954\pm 0.0059$<br>$0.8001\pm 0.0048$<br>$\mathrm{Excel}$<br>$0.7957\pm 0.0090$<br>$0.7985\pm 0.0106$<br>$\mathrm{SAINT}$<br>$0.7953\pm 0.0058$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.7918\pm 0.0076$<br>$0.7951\pm 0.0071$<br>$\mathrm{T2G}$<br>$0.7918\pm 0.0058$<br>$0.7955\pm 0.0047$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.7947\pm 0.0101$<br>$0.7977\pm 0.0117$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.7988\pm 0.0092$<br>$0.8024\pm 0.0093$<br>$\mathrm{MLP^{\dagger}}$<br>$0.7981\pm 0.0065$<br>$0.8008\pm 0.0057$<br>$\mathrm{XGBoost}$<br>$0.8013\pm 0.0081$<br>$0.8030\pm 0.0076$<br>$\mathrm{LightGBM}$<br>$0.8006\pm 0.0078$<br>$0.8013\pm 0.0072$<br>$\mathrm{CatBoost}$<br>$0.8026\pm 0.0068$<br>$0.8056\pm 0.0082$<br>$\mathrm{TabR}$<br>$0.7995\pm 0.0054$<br>$0.8015\pm 0.0037$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.8023\pm 0.0088$<br>–<br>$\mathrm{MNCA}$<br>$0.7961\pm 0.0065$<br>$0.8003\pm 0.0077$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.7977\pm 0.0081$<br>$0.8010\pm 0.0084$<br>$\mathrm{TabM}$<br>$0.7908\pm 0.0068$<br>$0.7915\pm 0.0068$<br>$\mathrm{TabM[G]}$<br>$0.7902\pm 0.0067$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.7938\pm 0.0064$<br>$0.7959\pm 0.0071$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.8003\pm 0.0087$<br>$0.8017\pm 0.0087$ |
| Brazilian_houses ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.0473\pm 0.0179$<br>$0.0440\pm 0.0207$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.0505\pm 0.0181$<br>$0.0458\pm 0.0207$<br>$\mathrm{DCN2}$<br>$0.0477\pm 0.0172$<br>$0.0427\pm 0.0207$<br>$\mathrm{SNN}$<br>$0.0630\pm 0.0162$<br>$0.0556\pm 0.0175$<br>$\mathrm{Trompt}$<br>$0.0404\pm 0.0266$<br>–<br>$\mathrm{AutoInt}$<br>$0.0470\pm 0.0192$<br>$0.0437\pm 0.0217$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.0513\pm 0.0234$<br>$0.0484\pm 0.0262$<br>$\mathrm{Excel}$<br>$0.0450\pm 0.0156$<br>$0.0418\pm 0.0190$<br>$\mathrm{SAINT}$<br>$0.0479\pm 0.0205$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.0438\pm 0.0181$<br>$0.0412\pm 0.0204$<br>$\mathrm{T2G}$<br>$0.0468\pm 0.0165$<br>$0.0436\pm 0.0211$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.0426\pm 0.0180$<br>$0.0397\pm 0.0206$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.0437\pm 0.0203$<br>$0.0407\pm 0.0230$<br>$\mathrm{MLP^{\dagger}}$<br>$0.0421\pm 0.0209$<br>$0.0409\pm 0.0226$<br>$\mathrm{XGBoost}$<br>$0.0541\pm 0.0270$<br>$0.0535\pm 0.0287$<br>$\mathrm{LightGBM}$<br>$0.0603\pm 0.0249$<br>$0.0589\pm 0.0271$<br>$\mathrm{CatBoost}$<br>$0.0468\pm 0.0312$<br>$0.0456\pm 0.0332$<br>$\mathrm{TabR}$<br>$0.0490\pm 0.0152$<br>$0.0454\pm 0.0170$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.0451\pm 0.0163$<br>–<br>$\mathrm{MNCA}$<br>$0.0527\pm 0.0157$<br>$0.0509\pm 0.0180$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.0553\pm 0.0192$<br>$0.0511\pm 0.0191$<br>$\mathrm{TabM}$<br>$0.0443\pm 0.0213$<br>$0.0431\pm 0.0233$<br>$\mathrm{TabM[G]}$<br>$0.0450\pm 0.0202$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.0480\pm 0.0194$<br>$0.0452\pm 0.0221$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.0460\pm 0.0206$<br>$0.0439\pm 0.0228$ | MagicTelescope ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.8539\pm 0.0060$<br>$0.8566\pm 0.0061$<br>$\mathrm{TabPFN}$<br>–<br>$0.8579\pm 0.0064$<br>$\mathrm{ResNet}$<br>$0.8589\pm 0.0068$<br>$0.8651\pm 0.0049$<br>$\mathrm{DCN2}$<br>$0.8432\pm 0.0074$<br>$0.8490\pm 0.0046$<br>$\mathrm{SNN}$<br>$0.8536\pm 0.0052$<br>$0.8567\pm 0.0047$<br>$\mathrm{Trompt}$<br>$0.8605\pm 0.0102$<br>–<br>$\mathrm{AutoInt}$<br>$0.8522\pm 0.0056$<br>$0.8560\pm 0.0034$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.8571\pm 0.0080$<br>$0.8624\pm 0.0044$<br>$\mathrm{Excel}$<br>$0.8480\pm 0.0090$<br>$0.8543\pm 0.0075$<br>$\mathrm{SAINT}$<br>$0.8595\pm 0.0060$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.8588\pm 0.0046$<br>$0.8643\pm 0.0037$<br>$\mathrm{T2G}$<br>$0.8553\pm 0.0055$<br>$0.8595\pm 0.0051$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.8591\pm 0.0061$<br>$0.8626\pm 0.0044$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.8575\pm 0.0056$<br>$0.8605\pm 0.0051$<br>$\mathrm{MLP^{\dagger}}$<br>$0.8593\pm 0.0054$<br>$0.8621\pm 0.0037$<br>$\mathrm{XGBoost}$<br>$0.8550\pm 0.0094$<br>$0.8589\pm 0.0110$<br>$\mathrm{LightGBM}$<br>$0.8547\pm 0.0085$<br>$0.8556\pm 0.0086$<br>$\mathrm{CatBoost}$<br>$0.8586\pm 0.0070$<br>$0.8588\pm 0.0077$<br>$\mathrm{TabR}$<br>$0.8682\pm 0.0058$<br>$0.8729\pm 0.0038$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.8641\pm 0.0052$<br>–<br>$\mathrm{MNCA}$<br>$0.8602\pm 0.0061$<br>$0.8628\pm 0.0041$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.8622\pm 0.0085$<br>$0.8681\pm 0.0064$<br>$\mathrm{TabM}$<br>$0.8607\pm 0.0058$<br>$0.8622\pm 0.0050$<br>$\mathrm{TabM[G]}$<br>$0.8585\pm 0.0057$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.8581\pm 0.0053$<br>$0.8597\pm 0.0055$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.8616\pm 0.0080$<br>$0.8646\pm 0.0075$ |
| Ailerons ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{DCN2}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{SNN}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{Trompt}$<br>$0.0002\pm 0.0000$<br>–<br>$\mathrm{AutoInt}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{Excel}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{SAINT}$<br>$0.0002\pm 0.0000$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{T2G}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{MLP^{\dagger}}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{XGBoost}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{LightGBM}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{CatBoost}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{TabR}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.0002\pm 0.0000$<br>–<br>$\mathrm{MNCA}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{TabM}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{TabM[G]}$<br>$0.0002\pm 0.0000$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.0002\pm 0.0000$<br>$0.0002\pm 0.0000$ | MiamiHousing2016 ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.1614\pm 0.0033$<br>$0.1574\pm 0.0043$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.1548\pm 0.0030$<br>$0.1511\pm 0.0027$<br>$\mathrm{DCN2}$<br>$0.1683\pm 0.0099$<br>$0.1575\pm 0.0047$<br>$\mathrm{SNN}$<br>$0.1618\pm 0.0029$<br>$0.1557\pm 0.0021$<br>$\mathrm{Trompt}$<br>$0.1478\pm 0.0028$<br>–<br>$\mathrm{AutoInt}$<br>$0.1537\pm 0.0035$<br>$0.1478\pm 0.0027$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.1527\pm 0.0037$<br>$0.1479\pm 0.0033$<br>$\mathrm{Excel}$<br>$0.1519\pm 0.0038$<br>$0.1442\pm 0.0022$<br>$\mathrm{SAINT}$<br>$0.1507\pm 0.0022$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.1514\pm 0.0029$<br>$0.1462\pm 0.0031$<br>$\mathrm{T2G}$<br>$0.1523\pm 0.0023$<br>$0.1478\pm 0.0024$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.1514\pm 0.0025$<br>$0.1479\pm 0.0017$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.1512\pm 0.0019$<br>$0.1470\pm 0.0024$<br>$\mathrm{MLP^{\dagger}}$<br>$0.1461\pm 0.0015$<br>$0.1433\pm 0.0022$<br>$\mathrm{XGBoost}$<br>$0.1440\pm 0.0029$<br>$0.1434\pm 0.0029$<br>$\mathrm{LightGBM}$<br>$0.1461\pm 0.0025$<br>$0.1455\pm 0.0030$<br>$\mathrm{CatBoost}$<br>$0.1417\pm 0.0021$<br>$0.1408\pm 0.0026$<br>$\mathrm{TabR}$<br>$0.1417\pm 0.0025$<br>$0.1390\pm 0.0020$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.1392\pm 0.0023$<br>–<br>$\mathrm{MNCA}$<br>$0.1503\pm 0.0040$<br>$0.1477\pm 0.0032$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.1475\pm 0.0031$<br>$0.1438\pm 0.0024$<br>$\mathrm{TabM}$<br>$0.1483\pm 0.0030$<br>$0.1465\pm 0.0029$<br>$\mathrm{TabM[G]}$<br>$0.1487\pm 0.0029$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.1508\pm 0.0035$<br>$0.1484\pm 0.0036$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.1407\pm 0.0016$<br>$0.1387\pm 0.0008$ |
| OnlineNewsPopularity ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.8643\pm 0.0007$<br>$0.8632\pm 0.0005$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.8665\pm 0.0011$<br>$0.8639\pm 0.0000$<br>$\mathrm{DCN2}$<br>$0.8714\pm 0.0013$<br>$0.8648\pm 0.0004$<br>$\mathrm{SNN}$<br>$0.8692\pm 0.0015$<br>$0.8665\pm 0.0005$<br>$\mathrm{Trompt}$<br>$0.8623\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.8636\pm 0.0022$<br>$0.8596\pm 0.0008$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.8615\pm 0.0008$<br>$0.8598\pm 0.0004$<br>$\mathrm{Excel}$<br>$0.8605\pm 0.0024$<br>$0.8556\pm nan$<br>$\mathrm{SAINT}$<br>$0.8600\pm 0.0007$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.8629\pm 0.0019$<br>$0.8603\pm 0.0000$<br>$\mathrm{T2G}$<br>$0.8632\pm 0.0009$<br>$0.8572\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.8604\pm 0.0009$<br>$0.8591\pm 0.0004$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.8594\pm 0.0004$<br>$0.8585\pm 0.0001$<br>$\mathrm{MLP^{\dagger}}$<br>$0.8585\pm 0.0003$<br>$0.8581\pm 0.0001$<br>$\mathrm{XGBoost}$<br>$0.8545\pm 0.0002$<br>$0.8543\pm 0.0000$<br>$\mathrm{LightGBM}$<br>$0.8546\pm 0.0002$<br>$0.8544\pm 0.0000$<br>$\mathrm{CatBoost}$<br>$0.8532\pm 0.0003$<br>$0.8527\pm 0.0001$<br>$\mathrm{TabR}$<br>$0.8677\pm 0.0013$<br>$0.8633\pm 0.0009$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.8624\pm 0.0011$<br>–<br>$\mathrm{MNCA}$<br>$0.8651\pm 0.0003$<br>$0.8650\pm 0.0002$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.8647\pm 0.0010$<br>$0.8624\pm 0.0006$<br>$\mathrm{TabM}$<br>$0.8584\pm 0.0003$<br>$0.8581\pm 0.0001$<br>$\mathrm{TabM[G]}$<br>$0.8586\pm 0.0005$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.8592\pm 0.0004$<br>$0.8588\pm 0.0001$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.8560\pm 0.0015$<br>$0.8532\pm 0.0008$ | credit ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.7735\pm 0.0042$<br>$0.7729\pm 0.0047$<br>$\mathrm{TabPFN}$<br>–<br>$0.7636\pm 0.0045$<br>$\mathrm{ResNet}$<br>$0.7721\pm 0.0033$<br>$0.7738\pm 0.0027$<br>$\mathrm{DCN2}$<br>$0.7703\pm 0.0034$<br>$0.7746\pm 0.0026$<br>$\mathrm{SNN}$<br>$0.7712\pm 0.0045$<br>$0.7716\pm 0.0059$<br>$\mathrm{Trompt}$<br>$0.7740\pm 0.0006$<br>–<br>$\mathrm{AutoInt}$<br>$0.7737\pm 0.0050$<br>$0.7765\pm 0.0058$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.7748\pm 0.0038$<br>$0.7768\pm 0.0059$<br>$\mathrm{Excel}$<br>$0.7724\pm 0.0038$<br>$0.7740\pm 0.0069$<br>$\mathrm{SAINT}$<br>$0.7739\pm 0.0052$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.7745\pm 0.0041$<br>$0.7767\pm 0.0040$<br>$\mathrm{T2G}$<br>$0.7744\pm 0.0046$<br>$0.7762\pm 0.0057$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.7749\pm 0.0055$<br>$0.7767\pm 0.0075$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.7734\pm 0.0034$<br>$0.7747\pm 0.0043$<br>$\mathrm{MLP^{\dagger}}$<br>$0.7758\pm 0.0040$<br>$0.7772\pm 0.0055$<br>$\mathrm{XGBoost}$<br>$0.7698\pm 0.0027$<br>$0.7706\pm 0.0029$<br>$\mathrm{LightGBM}$<br>$0.7686\pm 0.0028$<br>$0.7726\pm 0.0034$<br>$\mathrm{CatBoost}$<br>$0.7734\pm 0.0035$<br>$0.7752\pm 0.0038$<br>$\mathrm{TabR}$<br>$0.7730\pm 0.0043$<br>$0.7740\pm 0.0040$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.7723\pm 0.0037$<br>–<br>$\mathrm{MNCA}$<br>$0.7739\pm 0.0032$<br>$0.7757\pm 0.0026$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.7734\pm 0.0045$<br>$0.7754\pm 0.0040$<br>$\mathrm{TabM}$<br>$0.7751\pm 0.0042$<br>$0.7755\pm 0.0049$<br>$\mathrm{TabM[G]}$<br>$0.7744\pm 0.0036$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.7747\pm 0.0039$<br>$0.7758\pm 0.0042$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.7748\pm 0.0026$<br>$0.7757\pm 0.0036$ |
| elevators ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.0020\pm 0.0001$<br>$0.0019\pm 0.0000$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.0019\pm 0.0000$<br>$0.0019\pm 0.0000$<br>$\mathrm{DCN2}$<br>$0.0019\pm 0.0000$<br>$0.0019\pm 0.0000$<br>$\mathrm{SNN}$<br>$0.0020\pm 0.0001$<br>$0.0019\pm 0.0000$<br>$\mathrm{Trompt}$<br>$0.0018\pm 0.0000$<br>–<br>$\mathrm{AutoInt}$<br>$0.0019\pm 0.0000$<br>$0.0018\pm 0.0000$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.0019\pm 0.0000$<br>$0.0018\pm 0.0000$<br>$\mathrm{Excel}$<br>$0.0019\pm 0.0000$<br>$0.0018\pm 0.0000$<br>$\mathrm{SAINT}$<br>$0.0018\pm 0.0000$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.0019\pm 0.0000$<br>$0.0018\pm 0.0000$<br>$\mathrm{T2G}$<br>$0.0019\pm 0.0000$<br>$0.0018\pm 0.0000$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.0019\pm 0.0000$<br>$0.0018\pm 0.0000$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.0018\pm 0.0000$<br>$0.0018\pm 0.0000$<br>$\mathrm{MLP^{\dagger}}$<br>$0.0018\pm 0.0000$<br>$0.0018\pm 0.0000$<br>$\mathrm{XGBoost}$<br>$0.0020\pm 0.0000$<br>$0.0020\pm 0.0000$<br>$\mathrm{LightGBM}$<br>$0.0020\pm 0.0000$<br>$0.0020\pm 0.0000$<br>$\mathrm{CatBoost}$<br>$0.0020\pm 0.0000$<br>$0.0019\pm 0.0000$<br>$\mathrm{TabR}$<br>$0.0049\pm 0.0000$<br>$0.0049\pm 0.0000$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.0019\pm 0.0001$<br>–<br>$\mathrm{MNCA}$<br>$0.0019\pm 0.0000$<br>$0.0019\pm 0.0000$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.0018\pm 0.0000$<br>$0.0018\pm 0.0000$<br>$\mathrm{TabM}$<br>$0.0019\pm 0.0000$<br>$0.0018\pm 0.0000$<br>$\mathrm{TabM[G]}$<br>$0.0019\pm 0.0000$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.0019\pm 0.0000$<br>$0.0019\pm 0.0000$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.0018\pm 0.0000$<br>$0.0018\pm 0.0000$ | fifa ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.8038\pm 0.0124$<br>$0.8011\pm 0.0143$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.8025\pm 0.0140$<br>$0.7985\pm 0.0149$<br>$\mathrm{DCN2}$<br>$0.8046\pm 0.0135$<br>$0.7993\pm 0.0129$<br>$\mathrm{SNN}$<br>$0.8074\pm 0.0140$<br>$0.8031\pm 0.0147$<br>$\mathrm{Trompt}$<br>$0.7880\pm 0.0180$<br>–<br>$\mathrm{AutoInt}$<br>$0.7923\pm 0.0128$<br>$0.7886\pm 0.0127$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.7936\pm 0.0119$<br>$0.7903\pm 0.0133$<br>$\mathrm{Excel}$<br>$0.7909\pm 0.0111$<br>$0.7862\pm 0.0161$<br>$\mathrm{SAINT}$<br>$0.7901\pm 0.0118$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.7928\pm 0.0132$<br>$0.7888\pm 0.0130$<br>$\mathrm{T2G}$<br>$0.7928\pm 0.0139$<br>$0.7904\pm 0.0183$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.7940\pm 0.0118$<br>$0.7898\pm 0.0141$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.7907\pm 0.0092$<br>$0.7870\pm 0.0096$<br>$\mathrm{MLP^{\dagger}}$<br>$0.7806\pm 0.0104$<br>$0.7800\pm 0.0114$<br>$\mathrm{XGBoost}$<br>$0.7800\pm 0.0108$<br>$0.7795\pm 0.0114$<br>$\mathrm{LightGBM}$<br>$0.7806\pm 0.0120$<br>$0.7787\pm 0.0122$<br>$\mathrm{CatBoost}$<br>$0.7835\pm 0.0116$<br>$0.7817\pm 0.0114$<br>$\mathrm{TabR}$<br>$0.7902\pm 0.0119$<br>$0.7863\pm 0.0120$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.7914\pm 0.0136$<br>–<br>$\mathrm{MNCA}$<br>$0.7967\pm 0.0138$<br>$0.7933\pm 0.0145$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.7909\pm 0.0107$<br>$0.7866\pm 0.0106$<br>$\mathrm{TabM}$<br>$0.7974\pm 0.0144$<br>$0.7954\pm 0.0160$<br>$\mathrm{TabM[G]}$<br>$0.7970\pm 0.0146$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.7981\pm 0.0136$<br>$0.7947\pm 0.0154$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.7783\pm 0.0114$<br>$0.7768\pm 0.0123$ |
| house_sales ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.1790\pm 0.0009$<br>$0.1763\pm 0.0003$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.1755\pm 0.0014$<br>$0.1738\pm 0.0006$<br>$\mathrm{DCN2}$<br>$0.1862\pm 0.0032$<br>$0.1778\pm 0.0015$<br>$\mathrm{SNN}$<br>$0.1800\pm 0.0008$<br>$0.1770\pm 0.0004$<br>$\mathrm{Trompt}$<br>$0.1667\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.1700\pm 0.0014$<br>$0.1670\pm 0.0008$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.1704\pm 0.0007$<br>$0.1690\pm 0.0005$<br>$\mathrm{Excel}$<br>$0.1713\pm 0.0010$<br>$0.1668\pm nan$<br>$\mathrm{SAINT}$<br>$0.1713\pm 0.0015$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.1690\pm 0.0010$<br>$0.1659\pm 0.0004$<br>$\mathrm{T2G}$<br>$0.1689\pm 0.0010$<br>$0.1664\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.1699\pm 0.0008$<br>$0.1687\pm 0.0007$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.1690\pm 0.0005$<br>$0.1676\pm 0.0003$<br>$\mathrm{MLP^{\dagger}}$<br>$0.1687\pm 0.0004$<br>$0.1681\pm 0.0001$<br>$\mathrm{XGBoost}$<br>$0.1694\pm 0.0003$<br>$0.1689\pm 0.0001$<br>$\mathrm{LightGBM}$<br>$0.1692\pm 0.0004$<br>$0.1686\pm 0.0001$<br>$\mathrm{CatBoost}$<br>$0.1669\pm 0.0001$<br>$0.1667\pm 0.0000$<br>$\mathrm{TabR}$<br>$0.1689\pm 0.0009$<br>$0.1657\pm 0.0003$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.1636\pm 0.0009$<br>–<br>$\mathrm{MNCA}$<br>$0.1737\pm 0.0013$<br>$0.1714\pm 0.0005$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.1694\pm 0.0007$<br>$0.1670\pm 0.0003$<br>$\mathrm{TabM}$<br>$0.1692\pm 0.0011$<br>$0.1680\pm 0.0005$<br>$\mathrm{TabM[G]}$<br>$0.1692\pm 0.0009$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.1687\pm 0.0009$<br>$0.1676\pm 0.0002$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.1656\pm 0.0005$<br>$0.1647\pm 0.0002$ | medical_charges ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.0816\pm 0.0001$<br>$0.0814\pm 0.0000$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.0824\pm 0.0003$<br>$0.0817\pm 0.0001$<br>$\mathrm{DCN2}$<br>$0.0818\pm 0.0003$<br>$0.0815\pm 0.0001$<br>$\mathrm{SNN}$<br>$0.0827\pm 0.0006$<br>$0.0817\pm 0.0001$<br>$\mathrm{Trompt}$<br>$0.0812\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.0822\pm 0.0007$<br>$0.0814\pm 0.0001$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.0814\pm 0.0002$<br>$0.0811\pm 0.0000$<br>$\mathrm{Excel}$<br>$0.0817\pm 0.0004$<br>$0.0813\pm nan$<br>$\mathrm{SAINT}$<br>$0.0814\pm 0.0002$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.0814\pm 0.0002$<br>$0.0812\pm 0.0000$<br>$\mathrm{T2G}$<br>$0.0813\pm 0.0002$<br>$0.0811\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.0812\pm 0.0002$<br>$0.0810\pm 0.0000$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.0812\pm 0.0001$<br>$0.0809\pm 0.0001$<br>$\mathrm{MLP^{\dagger}}$<br>$0.0812\pm 0.0000$<br>$0.0811\pm 0.0000$<br>$\mathrm{XGBoost}$<br>$0.0825\pm 0.0001$<br>$0.0825\pm 0.0000$<br>$\mathrm{LightGBM}$<br>$0.0820\pm 0.0000$<br>$0.0820\pm 0.0000$<br>$\mathrm{CatBoost}$<br>$0.0816\pm 0.0000$<br>$0.0815\pm 0.0000$<br>$\mathrm{TabR}$<br>$0.0815\pm 0.0002$<br>$0.0812\pm 0.0000$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.0811\pm 0.0001$<br>–<br>$\mathrm{MNCA}$<br>$0.0811\pm 0.0001$<br>$0.0810\pm 0.0000$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.0809\pm 0.0000$<br>$0.0808\pm 0.0000$<br>$\mathrm{TabM}$<br>$0.0813\pm 0.0001$<br>$0.0812\pm 0.0000$<br>$\mathrm{TabM[G]}$<br>$0.0812\pm 0.0000$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.0814\pm 0.0001$<br>$0.0813\pm 0.0000$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.0812\pm 0.0001$<br>$0.0812\pm 0.0000$ |
| pol ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$5.5244\pm 0.5768$<br>$4.9945\pm 0.5923$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$6.3739\pm 0.6286$<br>$5.8181\pm 0.6054$<br>$\mathrm{DCN2}$<br>$6.5374\pm 0.9479$<br>$5.1814\pm 0.7775$<br>$\mathrm{SNN}$<br>$6.1816\pm 0.7366$<br>$5.5959\pm 0.8243$<br>$\mathrm{Trompt}$<br>$3.2337\pm 0.0605$<br>–<br>$\mathrm{AutoInt}$<br>$3.3295\pm 0.3379$<br>$2.7999\pm 0.1776$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$3.2011\pm 0.2921$<br>$2.8698\pm 0.2577$<br>$\mathrm{Excel}$<br>$3.0682\pm 0.2389$<br>$2.5816\pm 0.0368$<br>$\mathrm{SAINT}$<br>$2.7203\pm 0.1858$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$2.6974\pm 0.1666$<br>$2.3718\pm 0.0724$<br>$\mathrm{T2G}$<br>$2.9539\pm 0.1994$<br>$2.6282\pm 0.0730$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$2.8239\pm 0.2173$<br>$2.5266\pm 0.0605$<br>$\mathrm{MLP^{\ddagger}}$<br>$2.5452\pm 0.1221$<br>$2.3700\pm 0.0867$<br>$\mathrm{MLP^{\dagger}}$<br>$2.4958\pm 0.1292$<br>$2.3651\pm 0.1223$<br>$\mathrm{XGBoost}$<br>$4.2963\pm 0.0644$<br>$4.2548\pm 0.0488$<br>$\mathrm{LightGBM}$<br>$4.2320\pm 0.3369$<br>$4.1880\pm 0.3110$<br>$\mathrm{CatBoost}$<br>$3.6320\pm 0.1006$<br>$3.5505\pm 0.0896$<br>$\mathrm{TabR}$<br>$6.0708\pm 0.5368$<br>$5.5578\pm 0.4036$<br>$\mathrm{TabR^{\ddagger}}$<br>$2.5770\pm 0.1689$<br>–<br>$\mathrm{MNCA}$<br>$5.7878\pm 0.4884$<br>$5.3773\pm 0.5463$<br>$\mathrm{MNCA^{\ddagger}}$<br>$2.9083\pm 0.1364$<br>$2.6717\pm 0.0530$<br>$\mathrm{TabM}$<br>$3.3595\pm 0.4017$<br>$3.2130\pm 0.3979$<br>$\mathrm{TabM[G]}$<br>$3.3465\pm 0.4226$<br>–<br>$\mathrm{TabM_{mini}}$<br>$3.6925\pm 0.4469$<br>$3.4727\pm 0.3074$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$2.4893\pm 0.1620$<br>$2.4175\pm 0.1124$ | superconduct ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$10.8740\pm 0.0868$<br>$10.4118\pm 0.0429$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$10.7711\pm 0.1454$<br>$10.3495\pm 0.0168$<br>$\mathrm{DCN2}$<br>$10.8108\pm 0.0957$<br>$10.4342\pm 0.0179$<br>$\mathrm{SNN}$<br>$10.8562\pm 0.1300$<br>$10.3342\pm 0.0509$<br>$\mathrm{Trompt}$<br>$10.4442\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$11.0019\pm 0.1391$<br>$10.4469\pm 0.0521$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$10.7502\pm 0.0800$<br>$10.3281\pm 0.0450$<br>$\mathrm{Excel}$<br>$11.0879\pm 0.1571$<br>$10.4094\pm nan$<br>$\mathrm{SAINT}$<br>$10.7807\pm 0.1074$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$10.8256\pm 0.1692$<br>$10.3391\pm 0.0794$<br>$\mathrm{T2G}$<br>$10.8310\pm 0.1406$<br>$10.3017\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$10.5058\pm 0.0758$<br>$10.2322\pm 0.0463$<br>$\mathrm{MLP^{\ddagger}}$<br>$10.5061\pm 0.0330$<br>$10.2440\pm 0.0127$<br>$\mathrm{MLP^{\dagger}}$<br>$10.7220\pm 0.0757$<br>$10.3758\pm 0.0606$<br>$\mathrm{XGBoost}$<br>$10.1610\pm 0.0201$<br>$10.1413\pm 0.0025$<br>$\mathrm{LightGBM}$<br>$10.1634\pm 0.0118$<br>$10.1552\pm 0.0050$<br>$\mathrm{CatBoost}$<br>$10.2422\pm 0.0222$<br>$10.2116\pm 0.0058$<br>$\mathrm{TabR}$<br>$10.8842\pm 0.1073$<br>$10.4800\pm 0.0280$<br>$\mathrm{TabR^{\ddagger}}$<br>$10.3835\pm 0.0562$<br>–<br>$\mathrm{MNCA}$<br>$10.4419\pm 0.0640$<br>$10.2926\pm 0.0261$<br>$\mathrm{MNCA^{\ddagger}}$<br>$10.5651\pm 0.0616$<br>$10.3155\pm 0.0253$<br>$\mathrm{TabM}$<br>$10.3379\pm 0.0338$<br>$10.1943\pm 0.0291$<br>$\mathrm{TabM[G]}$<br>$10.3395\pm 0.0529$<br>–<br>$\mathrm{TabM_{mini}}$<br>$10.3392\pm 0.0649$<br>$10.1866\pm 0.0400$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$10.2083\pm 0.0591$<br>$10.0737\pm 0.0222$ |
| jannis ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.7840\pm 0.0018$<br>$0.7872\pm 0.0007$<br>$\mathrm{TabPFN}$<br>–<br>$0.7419\pm 0.0018$<br>$\mathrm{ResNet}$<br>$0.7923\pm 0.0024$<br>$0.7958\pm 0.0010$<br>$\mathrm{DCN2}$<br>$0.7712\pm 0.0029$<br>$0.7825\pm 0.0009$<br>$\mathrm{SNN}$<br>$0.7818\pm 0.0025$<br>$0.7859\pm 0.0011$<br>$\mathrm{Trompt}$<br>$0.8027\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.7933\pm 0.0018$<br>$0.7983\pm 0.0013$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.7927\pm 0.0025$<br>$0.8019\pm 0.0012$<br>$\mathrm{Excel}$<br>$0.7954\pm 0.0015$<br>$0.8021\pm nan$<br>$\mathrm{SAINT}$<br>$0.7971\pm 0.0028$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.7940\pm 0.0028$<br>$0.7998\pm 0.0006$<br>$\mathrm{T2G}$<br>$0.7998\pm 0.0024$<br>$0.8052\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.7923\pm 0.0018$<br>$0.7945\pm 0.0010$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.7947\pm 0.0017$<br>$0.7967\pm 0.0011$<br>$\mathrm{MLP^{\dagger}}$<br>$0.7891\pm 0.0013$<br>$0.7900\pm 0.0006$<br>$\mathrm{XGBoost}$<br>$0.7967\pm 0.0019$<br>$0.7998\pm 0.0007$<br>$\mathrm{LightGBM}$<br>$0.7956\pm 0.0017$<br>$0.7968\pm 0.0005$<br>$\mathrm{CatBoost}$<br>$0.7985\pm 0.0018$<br>$0.8009\pm 0.0012$<br>$\mathrm{TabR}$<br>$0.7983\pm 0.0022$<br>$0.8023\pm 0.0018$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.8051\pm 0.0023$<br>–<br>$\mathrm{MNCA}$<br>$0.7993\pm 0.0019$<br>$0.8042\pm 0.0013$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.8068\pm 0.0021$<br>$0.8128\pm 0.0007$<br>$\mathrm{TabM}$<br>$0.8066\pm 0.0015$<br>$0.8075\pm 0.0004$<br>$\mathrm{TabM[G]}$<br>$0.8055\pm 0.0022$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.8046\pm 0.0026$<br>$0.8062\pm 0.0011$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.8059\pm 0.0018$<br>$0.8085\pm 0.0006$ | MiniBooNE ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.9480\pm 0.0007$<br>$0.9498\pm 0.0001$<br>$\mathrm{TabPFN}$<br>–<br>$0.9266\pm 0.0012$<br>$\mathrm{ResNet}$<br>$0.9488\pm 0.0011$<br>$0.9504\pm 0.0005$<br>$\mathrm{DCN2}$<br>$0.9433\pm 0.0011$<br>$0.9470\pm 0.0010$<br>$\mathrm{SNN}$<br>$0.9476\pm 0.0013$<br>$0.9491\pm 0.0010$<br>$\mathrm{Trompt}$<br>$0.9473\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.9447\pm 0.0014$<br>$0.9473\pm 0.0010$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.9446\pm 0.0014$<br>$0.9483\pm 0.0002$<br>$\mathrm{Excel}$<br>$0.9430\pm 0.0015$<br>$0.9451\pm nan$<br>$\mathrm{SAINT}$<br>$0.9471\pm 0.0009$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.9467\pm 0.0014$<br>$0.9486\pm 0.0010$<br>$\mathrm{T2G}$<br>$0.9475\pm 0.0014$<br>$0.9508\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.9466\pm 0.0009$<br>$0.9478\pm 0.0004$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.9473\pm 0.0010$<br>$0.9493\pm 0.0004$<br>$\mathrm{MLP^{\dagger}}$<br>$0.9482\pm 0.0008$<br>$0.9492\pm 0.0001$<br>$\mathrm{XGBoost}$<br>$0.9436\pm 0.0006$<br>$0.9452\pm 0.0003$<br>$\mathrm{LightGBM}$<br>$0.9422\pm 0.0009$<br>$0.9427\pm 0.0003$<br>$\mathrm{CatBoost}$<br>$0.9453\pm 0.0008$<br>$0.9459\pm 0.0005$<br>$\mathrm{TabR}$<br>$0.9487\pm 0.0008$<br>$0.9500\pm 0.0002$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.9475\pm 0.0007$<br>–<br>$\mathrm{MNCA}$<br>$0.9488\pm 0.0010$<br>$0.9505\pm 0.0001$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.9493\pm 0.0012$<br>$0.9501\pm 0.0008$<br>$\mathrm{TabM}$<br>$0.9500\pm 0.0005$<br>$0.9505\pm 0.0002$<br>$\mathrm{TabM[G]}$<br>$0.9489\pm 0.0010$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.9487\pm 0.0006$<br>$0.9494\pm 0.0002$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.9497\pm 0.0006$<br>$0.9508\pm 0.0003$ |
| nyc-taxi-green-dec-2016 ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.3951\pm 0.0009$<br>$0.3921\pm 0.0003$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.3899\pm 0.0016$<br>$0.3873\pm 0.0009$<br>$\mathrm{DCN2}$<br>$0.3919\pm 0.0009$<br>$0.3889\pm 0.0003$<br>$\mathrm{SNN}$<br>$0.3933\pm 0.0013$<br>$0.3899\pm 0.0004$<br>$\mathrm{Trompt}$<br>$0.3979\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.4084\pm 0.0256$<br>$0.3967\pm 0.0059$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.3914\pm 0.0026$<br>$0.3861\pm 0.0013$<br>$\mathrm{Excel}$<br>$0.3969\pm 0.0036$<br>$0.3897\pm nan$<br>$\mathrm{SAINT}$<br>$0.3905\pm 0.0013$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.3937\pm 0.0064$<br>$0.3889\pm 0.0018$<br>$\mathrm{T2G}$<br>$0.3908\pm 0.0045$<br>$0.3858\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.3812\pm 0.0018$<br>$0.3761\pm 0.0016$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.3795\pm 0.0016$<br>$0.3733\pm 0.0013$<br>$\mathrm{MLP^{\dagger}}$<br>$0.3680\pm 0.0006$<br>$0.3653\pm 0.0005$<br>$\mathrm{XGBoost}$<br>$0.3792\pm 0.0002$<br>$0.3787\pm 0.0000$<br>$\mathrm{LightGBM}$<br>$0.3688\pm 0.0002$<br>$0.3684\pm 0.0000$<br>$\mathrm{CatBoost}$<br>$0.3647\pm 0.0005$<br>$0.3632\pm 0.0003$<br>$\mathrm{TabR}$<br>$0.3577\pm 0.0222$<br>$0.3380\pm 0.0027$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.3725\pm 0.0091$<br>–<br>$\mathrm{MNCA}$<br>$0.3728\pm 0.0012$<br>$0.3720\pm 0.0010$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.3536\pm 0.0052$<br>$0.3407\pm 0.0009$<br>$\mathrm{TabM}$<br>$0.3866\pm 0.0006$<br>$0.3855\pm 0.0003$<br>$\mathrm{TabM[G]}$<br>$0.3862\pm 0.0005$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.3877\pm 0.0009$<br>$0.3857\pm 0.0004$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.3527\pm 0.0112$<br>$0.3478\pm 0.0009$ | particulate-matter-ukair-2017 ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.3759\pm 0.0004$<br>$0.3729\pm 0.0003$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$0.3743\pm 0.0007$<br>$0.3718\pm 0.0005$<br>$\mathrm{DCN2}$<br>$0.3759\pm 0.0012$<br>$0.3738\pm 0.0004$<br>$\mathrm{SNN}$<br>$0.3790\pm 0.0007$<br>$0.3744\pm 0.0002$<br>$\mathrm{Trompt}$<br>$0.3700\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.3723\pm 0.0011$<br>$0.3692\pm 0.0010$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.3741\pm 0.0010$<br>$0.3698\pm 0.0004$<br>$\mathrm{Excel}$<br>$0.3699\pm 0.0014$<br>$0.3652\pm nan$<br>$\mathrm{SAINT}$<br>$0.3704\pm 0.0014$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.3735\pm 0.0012$<br>$0.3686\pm 0.0004$<br>$\mathrm{T2G}$<br>$0.3676\pm 0.0024$<br>$0.3631\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.3665\pm 0.0008$<br>$0.3642\pm 0.0003$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.3657\pm 0.0007$<br>$0.3629\pm 0.0002$<br>$\mathrm{MLP^{\dagger}}$<br>$0.3649\pm 0.0011$<br>$0.3637\pm 0.0008$<br>$\mathrm{XGBoost}$<br>$0.3641\pm 0.0001$<br>$0.3640\pm 0.0000$<br>$\mathrm{LightGBM}$<br>$0.3637\pm 0.0001$<br>$0.3635\pm 0.0000$<br>$\mathrm{CatBoost}$<br>$0.3647\pm 0.0004$<br>$0.3637\pm 0.0002$<br>$\mathrm{TabR}$<br>$0.3613\pm 0.0005$<br>$0.3590\pm 0.0002$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.3596\pm 0.0004$<br>–<br>$\mathrm{MNCA}$<br>$0.3670\pm 0.0004$<br>$0.3649\pm 0.0002$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.3646\pm 0.0001$<br>$0.3643\pm 0.0000$<br>$\mathrm{TabM}$<br>$0.3686\pm 0.0006$<br>$0.3679\pm 0.0003$<br>$\mathrm{TabM[G]}$<br>$0.3683\pm 0.0007$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.3690\pm 0.0009$<br>$0.3675\pm 0.0004$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.3603\pm 0.0005$<br>$0.3589\pm 0.0003$ |
| road-safety ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.7857\pm 0.0019$<br>$0.7873\pm 0.0004$<br>$\mathrm{TabPFN}$<br>–<br>$0.7338\pm 0.0032$<br>$\mathrm{ResNet}$<br>$0.7875\pm 0.0007$<br>$0.7898\pm 0.0008$<br>$\mathrm{DCN2}$<br>$0.7781\pm 0.0014$<br>$0.7823\pm 0.0012$<br>$\mathrm{SNN}$<br>$0.7847\pm 0.0010$<br>$0.7865\pm 0.0002$<br>$\mathrm{Trompt}$<br>$0.7804\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$0.7826\pm 0.0030$<br>$0.7883\pm 0.0013$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$0.7878\pm 0.0032$<br>$0.7919\pm 0.0015$<br>$\mathrm{Excel}$<br>$0.7864\pm 0.0053$<br>$0.7907\pm nan$<br>$\mathrm{SAINT}$<br>$0.7584\pm 0.0584$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.7907\pm 0.0012$<br>$0.7943\pm 0.0007$<br>$\mathrm{T2G}$<br>$0.7912\pm 0.0026$<br>$0.7961\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.7867\pm 0.0018$<br>$0.7903\pm 0.0002$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.7853\pm 0.0014$<br>$0.7881\pm 0.0007$<br>$\mathrm{MLP^{\dagger}}$<br>$0.7899\pm 0.0009$<br>$0.7935\pm 0.0003$<br>$\mathrm{XGBoost}$<br>$0.8101\pm 0.0017$<br>$0.8129\pm 0.0004$<br>$\mathrm{LightGBM}$<br>$0.7982\pm 0.0012$<br>$0.7996\pm 0.0005$<br>$\mathrm{CatBoost}$<br>$0.8012\pm 0.0009$<br>$0.8022\pm 0.0002$<br>$\mathrm{TabR}$<br>$0.8403\pm 0.0014$<br>$0.8441\pm 0.0005$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.8374\pm 0.0013$<br>–<br>$\mathrm{MNCA}$<br>$0.8080\pm 0.0013$<br>$0.8121\pm 0.0006$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.8232\pm 0.0017$<br>$0.8287\pm 0.0008$<br>$\mathrm{TabM}$<br>$0.7946\pm 0.0013$<br>$0.7961\pm 0.0005$<br>$\mathrm{TabM[G]}$<br>$0.7945\pm 0.0009$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.7931\pm 0.0011$<br>$0.7946\pm 0.0010$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.8015\pm 0.0034$<br>$0.8060\pm 0.0015$ | year ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$8.9628\pm 0.0232$<br>$8.8931\pm 0.0066$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>$8.9658\pm 0.0239$<br>$8.8755\pm 0.0066$<br>$\mathrm{DCN2}$<br>$9.2761\pm 0.0401$<br>$9.0640\pm 0.0156$<br>$\mathrm{SNN}$<br>$9.0054\pm 0.0256$<br>$8.9351\pm 0.0073$<br>$\mathrm{Trompt}$<br>$8.9707\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>$9.0430\pm 0.0280$<br>$8.9619\pm 0.0092$<br>$\mathrm{MLP\texttt{-}Mixer}$<br>$8.9589\pm 0.0182$<br>$8.9086\pm 0.0177$<br>$\mathrm{Excel}$<br>$9.0395\pm 0.0266$<br>$8.9551\pm nan$<br>$\mathrm{SAINT}$<br>$9.0248\pm 0.0225$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$9.0005\pm 0.0215$<br>$8.9360\pm 0.0013$<br>$\mathrm{T2G}$<br>$8.9775\pm 0.0138$<br>$8.8979\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$8.9355\pm 0.0103$<br>$8.9063\pm 0.0030$<br>$\mathrm{MLP^{\ddagger}}$<br>$8.9455\pm 0.0173$<br>$8.9083\pm 0.0046$<br>$\mathrm{MLP^{\dagger}}$<br>$8.9379\pm 0.0206$<br>$8.8753\pm 0.0038$<br>$\mathrm{XGBoost}$<br>$9.0307\pm 0.0028$<br>$9.0245\pm 0.0015$<br>$\mathrm{LightGBM}$<br>$9.0200\pm 0.0025$<br>$9.0128\pm 0.0015$<br>$\mathrm{CatBoost}$<br>$9.0370\pm 0.0073$<br>$9.0054\pm 0.0028$<br>$\mathrm{TabR}$<br>$9.0069\pm 0.0152$<br>$8.9132\pm 0.0088$<br>$\mathrm{TabR^{\ddagger}}$<br>$8.9721\pm 0.0105$<br>–<br>$\mathrm{MNCA}$<br>$8.9476\pm 0.0152$<br>$8.8977\pm 0.0037$<br>$\mathrm{MNCA^{\ddagger}}$<br>$8.8973\pm 0.0082$<br>$8.8550\pm 0.0031$<br>$\mathrm{TabM}$<br>$8.8701\pm 0.0110$<br>$8.8517\pm 0.0022$<br>$\mathrm{TabM[G]}$<br>$8.8715\pm 0.0116$<br>–<br>$\mathrm{TabM_{mini}}$<br>$8.8958\pm 0.0087$<br>$8.8810\pm 0.0020$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$8.8825\pm 0.0087$<br>$8.8560\pm 0.0015$ |

**Table 20: Extended results for the main benchmark. Results are grouped by datasets.**
| sberbank-housing ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.2529\pm 0.0078$<br>$0.2474\pm 0.0052$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>–<br>–<br>$\mathrm{DCN2}$<br>$0.2616\pm 0.0049$<br>$0.2506\pm 0.0015$<br>$\mathrm{SNN}$<br>$0.2671\pm 0.0140$<br>$0.2555\pm 0.0033$<br>$\mathrm{Trompt}$<br>$0.2509\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>–<br>–<br>$\mathrm{MLP\texttt{-}Mixer}$<br>–<br>–<br>$\mathrm{Excel}$<br>$0.2533\pm 0.0046$<br>$0.2485\pm nan$<br>$\mathrm{SAINT}$<br>$0.2467\pm 0.0019$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.2440\pm 0.0038$<br>$0.2367\pm 0.0010$<br>$\mathrm{T2G}$<br>$0.2416\pm 0.0025$<br>$0.2343\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.2528\pm 0.0055$<br>$0.2503\pm 0.0029$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.2412\pm 0.0031$<br>$0.2355\pm 0.0006$<br>$\mathrm{MLP^{\dagger}}$<br>$0.2383\pm 0.0032$<br>$0.2327\pm 0.0009$<br>$\mathrm{XGBoost}$<br>$0.2419\pm 0.0012$<br>$0.2416\pm 0.0007$<br>$\mathrm{LightGBM}$<br>$0.2468\pm 0.0009$<br>$0.2467\pm 0.0002$<br>$\mathrm{CatBoost}$<br>$0.2482\pm 0.0034$<br>$0.2473\pm 0.0016$<br>$\mathrm{TabR}$<br>$0.2820\pm 0.0323$<br>$0.2603\pm 0.0048$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.2542\pm 0.0101$<br>–<br>$\mathrm{MNCA}$<br>$0.2593\pm 0.0053$<br>$0.2520\pm 0.0032$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.2448\pm 0.0039$<br>$0.2404\pm 0.0025$<br>$\mathrm{TabM}$<br>$0.2469\pm 0.0035$<br>$0.2440\pm 0.0026$<br>$\mathrm{TabM[G]}$<br>$0.2480\pm 0.0049$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.2440\pm 0.0025$<br>$0.2425\pm 0.0008$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.2357\pm 0.0025$<br>$0.2333\pm 0.0007$ | ecom-offers ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.5989\pm 0.0017$<br>$0.5995\pm 0.0011$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>–<br>–<br>$\mathrm{DCN2}$<br>$0.5996\pm 0.0043$<br>$0.6039\pm 0.0028$<br>$\mathrm{SNN}$<br>$0.5912\pm 0.0056$<br>$0.5961\pm 0.0033$<br>$\mathrm{Trompt}$<br>$0.5803\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>–<br>–<br>$\mathrm{MLP\texttt{-}Mixer}$<br>–<br>–<br>$\mathrm{Excel}$<br>$0.5759\pm 0.0066$<br>$0.5759\pm nan$<br>$\mathrm{SAINT}$<br>$0.5812\pm 0.0098$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.5775\pm 0.0063$<br>$0.5817\pm 0.0021$<br>$\mathrm{T2G}$<br>$0.5791\pm 0.0056$<br>$0.5824\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.5800\pm 0.0029$<br>$0.5819\pm 0.0011$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.5846\pm 0.0048$<br>$0.5872\pm 0.0018$<br>$\mathrm{MLP^{\dagger}}$<br>$0.5949\pm 0.0013$<br>$0.5953\pm 0.0006$<br>$\mathrm{XGBoost}$<br>$0.5763\pm 0.0072$<br>$0.5917\pm 0.0035$<br>$\mathrm{LightGBM}$<br>$0.5758\pm 0.0006$<br>$0.5758\pm 0.0003$<br>$\mathrm{CatBoost}$<br>$0.5596\pm 0.0068$<br>$0.5067\pm 0.0011$<br>$\mathrm{TabR}$<br>$0.5943\pm 0.0019$<br>$0.5977\pm 0.0009$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.5762\pm 0.0052$<br>–<br>$\mathrm{MNCA}$<br>$0.5765\pm 0.0087$<br>$0.5820\pm 0.0047$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.5758\pm 0.0050$<br>$0.5796\pm 0.0009$<br>$\mathrm{TabM}$<br>$0.5948\pm 0.0006$<br>$0.5952\pm 0.0004$<br>$\mathrm{TabM[G]}$<br>$0.5959\pm 0.0010$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.5948\pm 0.0009$<br>$0.5954\pm 0.0004$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.5919\pm 0.0016$<br>$0.5926\pm 0.0006$ |
| --- | --- |
| maps-routing ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.1625\pm 0.0001$<br>$0.1621\pm 0.0000$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>–<br>–<br>$\mathrm{DCN2}$<br>$0.1656\pm 0.0004$<br>$0.1636\pm 0.0001$<br>$\mathrm{SNN}$<br>$0.1634\pm 0.0002$<br>$0.1625\pm 0.0000$<br>$\mathrm{Trompt}$<br>$0.1624\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>–<br>–<br>$\mathrm{MLP\texttt{-}Mixer}$<br>–<br>–<br>$\mathrm{Excel}$<br>$0.1628\pm 0.0001$<br>$0.1621\pm nan$<br>$\mathrm{SAINT}$<br>$0.1634\pm nan$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.1625\pm 0.0003$<br>$0.1619\pm 0.0001$<br>$\mathrm{T2G}$<br>$0.1616\pm 0.0001$<br>$0.1608\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.1618\pm 0.0002$<br>$0.1613\pm 0.0000$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.1618\pm 0.0002$<br>$0.1613\pm 0.0001$<br>$\mathrm{MLP^{\dagger}}$<br>$0.1620\pm 0.0002$<br>$0.1614\pm 0.0000$<br>$\mathrm{XGBoost}$<br>$0.1616\pm 0.0001$<br>$0.1614\pm 0.0000$<br>$\mathrm{LightGBM}$<br>$0.1618\pm 0.0000$<br>$0.1616\pm 0.0000$<br>$\mathrm{CatBoost}$<br>$0.1619\pm 0.0001$<br>$0.1615\pm 0.0000$<br>$\mathrm{TabR}$<br>$0.1639\pm 0.0003$<br>$0.1622\pm 0.0002$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.1622\pm 0.0002$<br>–<br>$\mathrm{MNCA}$<br>$0.1625\pm 0.0001$<br>$0.1621\pm 0.0001$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.1627\pm 0.0002$<br>$0.1623\pm 0.0001$<br>$\mathrm{TabM}$<br>$0.1612\pm 0.0001$<br>$0.1609\pm 0.0000$<br>$\mathrm{TabM[G]}$<br>$0.1612\pm 0.0001$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.1613\pm 0.0002$<br>$0.1609\pm 0.0000$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.1610\pm 0.0001$<br>$0.1607\pm 0.0001$ | homesite-insurance ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.9506\pm 0.0005$<br>$0.9514\pm 0.0001$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>–<br>–<br>$\mathrm{DCN2}$<br>$0.9398\pm 0.0053$<br>$0.9432\pm 0.0018$<br>$\mathrm{SNN}$<br>$0.9473\pm 0.0013$<br>$0.9484\pm 0.0007$<br>$\mathrm{Trompt}$<br>$0.9588\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>–<br>–<br>$\mathrm{MLP\texttt{-}Mixer}$<br>–<br>–<br>$\mathrm{Excel}$<br>$0.9622\pm 0.0004$<br>$0.9635\pm nan$<br>$\mathrm{SAINT}$<br>$0.9613\pm nan$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.9622\pm 0.0006$<br>$0.9633\pm 0.0001$<br>$\mathrm{T2G}$<br>$0.9624\pm 0.0006$<br>$0.9637\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.9609\pm 0.0009$<br>$0.9626\pm 0.0003$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.9617\pm 0.0004$<br>$0.9630\pm 0.0002$<br>$\mathrm{MLP^{\dagger}}$<br>$0.9582\pm 0.0014$<br>$0.9599\pm 0.0002$<br>$\mathrm{XGBoost}$<br>$0.9601\pm 0.0002$<br>$0.9602\pm 0.0000$<br>$\mathrm{LightGBM}$<br>$0.9603\pm 0.0002$<br>$0.9604\pm 0.0001$<br>$\mathrm{CatBoost}$<br>$0.9606\pm 0.0003$<br>$0.9609\pm 0.0001$<br>$\mathrm{TabR}$<br>$0.9487\pm 0.0014$<br>$0.9505\pm 0.0001$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.9556\pm 0.0021$<br>–<br>$\mathrm{MNCA}$<br>$0.9514\pm 0.0038$<br>$0.9522\pm 0.0027$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.9620\pm 0.0006$<br>$0.9635\pm 0.0002$<br>$\mathrm{TabM}$<br>$0.9641\pm 0.0004$<br>$0.9644\pm 0.0003$<br>$\mathrm{TabM[G]}$<br>$0.9640\pm 0.0004$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.9642\pm 0.0003$<br>$0.9644\pm 0.0001$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.9627\pm 0.0002$<br>$0.9630\pm 0.0001$ |
| cooking-time ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.4828\pm 0.0002$<br>$0.4822\pm 0.0000$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>–<br>–<br>$\mathrm{DCN2}$<br>$0.4834\pm 0.0003$<br>$0.4822\pm 0.0001$<br>$\mathrm{SNN}$<br>$0.4835\pm 0.0006$<br>$0.4818\pm 0.0002$<br>$\mathrm{Trompt}$<br>$0.4809\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>–<br>–<br>$\mathrm{MLP\texttt{-}Mixer}$<br>–<br>–<br>$\mathrm{Excel}$<br>$0.4821\pm 0.0005$<br>$0.4808\pm nan$<br>$\mathrm{SAINT}$<br>$0.4840\pm nan$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.4820\pm 0.0008$<br>$0.4813\pm 0.0005$<br>$\mathrm{T2G}$<br>$0.4809\pm 0.0008$<br>$0.4797\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.4811\pm 0.0004$<br>$0.4805\pm 0.0001$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.4809\pm 0.0006$<br>$0.4804\pm 0.0003$<br>$\mathrm{MLP^{\dagger}}$<br>$0.4812\pm 0.0004$<br>$0.4807\pm 0.0002$<br>$\mathrm{XGBoost}$<br>$0.4823\pm 0.0001$<br>$0.4821\pm 0.0000$<br>$\mathrm{LightGBM}$<br>$0.4826\pm 0.0001$<br>$0.4825\pm 0.0001$<br>$\mathrm{CatBoost}$<br>$0.4823\pm 0.0001$<br>$0.4820\pm 0.0001$<br>$\mathrm{TabR}$<br>$0.4828\pm 0.0008$<br>$0.4814\pm 0.0004$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.4818\pm 0.0006$<br>–<br>$\mathrm{MNCA}$<br>$0.4825\pm 0.0004$<br>$0.4819\pm 0.0003$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.4818\pm 0.0005$<br>$0.4809\pm 0.0003$<br>$\mathrm{TabM}$<br>$0.4803\pm 0.0006$<br>$0.4797\pm 0.0003$<br>$\mathrm{TabM[G]}$<br>$0.4803\pm 0.0005$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.4804\pm 0.0006$<br>$0.4796\pm 0.0000$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.4805\pm 0.0007$<br>$0.4795\pm 0.0003$ | homecredit-default ↑<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.8538\pm 0.0014$<br>$0.8566\pm 0.0005$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>–<br>–<br>$\mathrm{DCN2}$<br>$0.8471\pm 0.0019$<br>$0.8549\pm 0.0002$<br>$\mathrm{SNN}$<br>$0.8541\pm 0.0016$<br>$0.8569\pm 0.0010$<br>$\mathrm{Trompt}$<br>$0.8355\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>–<br>–<br>$\mathrm{MLP\texttt{-}Mixer}$<br>–<br>–<br>$\mathrm{Excel}$<br>$0.8513\pm 0.0024$<br>$0.8564\pm nan$<br>$\mathrm{SAINT}$<br>$0.8377\pm nan$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.8571\pm 0.0023$<br>$0.8611\pm 0.0013$<br>$\mathrm{T2G}$<br>$0.8597\pm 0.0007$<br>$0.8629\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.8598\pm 0.0009$<br>$0.8607\pm 0.0003$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.8572\pm 0.0011$<br>$0.8590\pm 0.0003$<br>$\mathrm{MLP^{\dagger}}$<br>$0.8568\pm 0.0039$<br>$0.8614\pm 0.0014$<br>$\mathrm{XGBoost}$<br>$0.8670\pm 0.0005$<br>$0.8674\pm 0.0001$<br>$\mathrm{LightGBM}$<br>$0.8664\pm 0.0004$<br>$0.8667\pm 0.0000$<br>$\mathrm{CatBoost}$<br>$0.8627\pm nan$<br>–<br>$\mathrm{TabR}$<br>$0.8501\pm 0.0027$<br>$0.8548\pm 0.0003$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.8547\pm 0.0021$<br>–<br>$\mathrm{MNCA}$<br>$0.8531\pm 0.0018$<br>$0.8569\pm 0.0004$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.8544\pm 0.0033$<br>$0.8606\pm 0.0024$<br>$\mathrm{TabM}$<br>$0.8583\pm 0.0010$<br>$0.8599\pm 0.0006$<br>$\mathrm{TabM[G]}$<br>$0.8583\pm 0.0011$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.8577\pm 0.0017$<br>$0.8598\pm 0.0004$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.8632\pm 0.0017$<br>$0.8656\pm 0.0003$ |
| delivery-eta ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$0.5493\pm 0.0007$<br>$0.5478\pm 0.0006$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>–<br>–<br>$\mathrm{DCN2}$<br>$0.5516\pm 0.0014$<br>$0.5495\pm 0.0004$<br>$\mathrm{SNN}$<br>$0.5495\pm 0.0008$<br>$0.5479\pm 0.0001$<br>$\mathrm{Trompt}$<br>$0.5519\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>–<br>–<br>$\mathrm{MLP\texttt{-}Mixer}$<br>–<br>–<br>$\mathrm{Excel}$<br>$0.5552\pm 0.0030$<br>$0.5524\pm nan$<br>$\mathrm{SAINT}$<br>$0.5528\pm nan$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$0.5542\pm 0.0026$<br>$0.5523\pm 0.0018$<br>$\mathrm{T2G}$<br>$0.5527\pm 0.0016$<br>$0.5512\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$0.5521\pm 0.0014$<br>$0.5512\pm 0.0005$<br>$\mathrm{MLP^{\ddagger}}$<br>$0.5535\pm 0.0019$<br>$0.5526\pm 0.0009$<br>$\mathrm{MLP^{\dagger}}$<br>$0.5521\pm 0.0019$<br>$0.5511\pm 0.0007$<br>$\mathrm{XGBoost}$<br>$0.5468\pm 0.0002$<br>$0.5463\pm 0.0001$<br>$\mathrm{LightGBM}$<br>$0.5468\pm 0.0001$<br>$0.5465\pm 0.0000$<br>$\mathrm{CatBoost}$<br>$0.5465\pm 0.0001$<br>$0.5461\pm 0.0000$<br>$\mathrm{TabR}$<br>$0.5514\pm 0.0024$<br>$0.5480\pm 0.0005$<br>$\mathrm{TabR^{\ddagger}}$<br>$0.5520\pm 0.0015$<br>–<br>$\mathrm{MNCA}$<br>$0.5498\pm 0.0007$<br>$0.5488\pm 0.0002$<br>$\mathrm{MNCA^{\ddagger}}$<br>$0.5507\pm 0.0013$<br>$0.5494\pm 0.0006$<br>$\mathrm{TabM}$<br>$0.5510\pm 0.0015$<br>$0.5504\pm 0.0004$<br>$\mathrm{TabM[G]}$<br>$0.5517\pm 0.0016$<br>–<br>$\mathrm{TabM_{mini}}$<br>$0.5519\pm 0.0015$<br>$0.5511\pm 0.0006$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$0.5508\pm 0.0013$<br>$0.5497\pm 0.0003$ | weather ↓<br>Method<br>Single model<br>Ensemble<br>$\mathrm{MLP}$<br>$1.5378\pm 0.0054$<br>$1.5111\pm 0.0029$<br>$\mathrm{TabPFN}$<br>–<br>–<br>$\mathrm{ResNet}$<br>–<br>–<br>$\mathrm{DCN2}$<br>$1.5606\pm 0.0057$<br>$1.5292\pm 0.0028$<br>$\mathrm{SNN}$<br>$1.5280\pm 0.0085$<br>$1.5013\pm 0.0034$<br>$\mathrm{Trompt}$<br>$1.5187\pm nan$<br>–<br>$\mathrm{AutoInt}$<br>–<br>–<br>$\mathrm{MLP\texttt{-}Mixer}$<br>–<br>–<br>$\mathrm{Excel}$<br>$1.5131\pm 0.0022$<br>$1.4707\pm nan$<br>$\mathrm{SAINT}$<br>$1.5097\pm 0.0045$<br>–<br>$\mathrm{FT\texttt{-}T}$<br>$1.5104\pm 0.0097$<br>$1.4719\pm 0.0040$<br>$\mathrm{T2G}$<br>$1.4849\pm 0.0087$<br>$1.4513\pm nan$<br>$\mathrm{MLP^{\ddagger-lite}}$<br>$1.5170\pm 0.0040$<br>$1.4953\pm 0.0023$<br>$\mathrm{MLP^{\ddagger}}$<br>$1.5139\pm 0.0031$<br>$1.4978\pm 0.0020$<br>$\mathrm{MLP^{\dagger}}$<br>$1.5162\pm 0.0020$<br>$1.5066\pm 0.0008$<br>$\mathrm{XGBoost}$<br>$1.4671\pm 0.0006$<br>$1.4629\pm 0.0002$<br>$\mathrm{LightGBM}$<br>$1.4625\pm 0.0008$<br>$1.4581\pm 0.0003$<br>$\mathrm{CatBoost}$<br>$1.4688\pm 0.0019$<br>–<br>$\mathrm{TabR}$<br>$1.4666\pm 0.0039$<br>$1.4547\pm 0.0008$<br>$\mathrm{TabR^{\ddagger}}$<br>$1.4458\pm 0.0018$<br>–<br>$\mathrm{MNCA}$<br>$1.5062\pm 0.0054$<br>$1.4822\pm 0.0013$<br>$\mathrm{MNCA^{\ddagger}}$<br>$1.5008\pm 0.0034$<br>$1.4782\pm 0.0011$<br>$\mathrm{TabM}$<br>$1.4786\pm 0.0039$<br>$1.4715\pm 0.0020$<br>$\mathrm{TabM[G]}$<br>$1.4796\pm 0.0037$<br>–<br>$\mathrm{TabM_{mini}}$<br>$1.4809\pm 0.0027$<br>$1.4717\pm 0.0012$<br>$\mathrm{TabM_{mini}^{\dagger}}$<br>$1.4709\pm 0.0047$<br>$1.4611\pm 0.0023$ |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: - Table 2, for the MLP (Maps Routing dataset) the time unit is missing. I am guessing it is 10 minutes.
soundness: 3
strengths: - The proposed method achieves better performance compared to the other methods with tuned hyperparameters. 
- The set of considered baselines is extensive.
- Extensive results on classification/regression datasets. Additionally, results are provided on well-known benchmarks in the domain.
- The authors provide extensive ablations of the different components of the method. The authors additionally provide a time comparison of the different methods.
summary: The authors propose TabM, a variation of an MLP with $k$ heads that uses a shared backbone, and only certain aspects of the architecture in the beginning  and in the end are specialized. The heads are trained independently. The work is compared with prior DL methods (plain and attention architectures), retrieval-based methods, and tree-based methods on a diverse collection of classification and regression tasks. Based on the results, the proposed method manages to outperform all baselines with tuned hyperparameters.
weaknesses: - The work is difficult to read, the captions of Figure 2 and Figure 3 are long and blend with the core manuscript text, making it hard to keep track of the manuscript flow.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See weakness.
soundness: 3
strengths: - [Major] This method is simple yet effective.
- [Major] The paper is easy to read.
summary: This paper come up with an efficient ensembling method for tabular deep learning — which integrate BatchEnsemble with multilayer perceptron. This method exhibits great improvements in their benchmark.
weaknesses: - [Medium] The proposed method is not much novel. Note that this concern is not the main reason that I chose to reject this paper. If other concerns listed can be addressed (e.g., more technical contributions, comprehensive empirical study for depper understandings), I think this paper is still very useful.
- [Major] More experiments are needed to fully understand the behavior of TabM. In the abstract, the authors mentioned “we show that parameter-efficient ensembling is not an arbitrary trick, but rather a highly effective way to reduce overfitting and improve optimization dynamics of tabular MLPs.” However, in the paper, there is no discussion why TabM helps with reducing overfitting and optimization dynamics. Of course the experiments proves this, but it seems like a general benefits of ensemble models. At least, some explanation supported by experiments is needed to fully understand this method — especially this method is already very simple and not much novel, understanding this better is important to have sufficient technical contribution.
- [Major] Baselines missing. This method is an ensemble method. At least we could compare it to other naive ensemble method. For instance, how much speed up it provides compared to normal ensemble; how robust it compared to normal ensemble; etc.
- [Major] Datasets missing. Yes this paper test on many datasets — but the more important thing is the diversity. For instance, how about high-dimensional datasets and large datasets (I know it is on Table 2, but only evaluated on two datasets and three baselines, moreover, only TabMmini has been evaluated). I suggest the authors to use TabZilla hard benchmark, and then evaluate the method’s capability under different circumstances. These kind of experiments can make the findings more interesting.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. In Table 2 (p.10), the last column of the MLP row lists "10." Could the authors confirm if it is measured in minutes, as there’s no unit mentioned?
2. In line 1027, the authors state, *For numerical features, by default, we used a slightly modified version of the quantile normalization from the Scikit-learn package, with rare exceptions when it turned out to be detrimental.* Could the authors clarify how they define “detrimental” in this context and provide further explanation?
3. Could the authors provide more detailed diversity metrics for the predictions of the 32 base models?
4. Line 431 notes, *Interestingly, the best prediction head of TabM performs no better than the plain MLP.* Would an ensemble of 32 independently and undertrained MLPs yield similarly strong performance?
5. The ranking method used is unconventional. Is this ranking method novel to this paper? Could the authors also provide results for a more traditional average rank metric to facilitate comparison, as it would be interesting to see how these two measures differ?
6. One advantage of using deep learning for tabular data is the ability to generate embeddings that are useful for downstream tasks and multimodal learning. How does TabM determine the representation of each sample? Is it based on concatenation, averaging across base models, or some other method? Additionally, could the authors provide a comparison of TabM’s embeddings with those from other deep learning approaches on specific tasks to showcase their effectiveness?

---
Lastly, if the authors can address the concerns noted in the weaknesses and questions, I will consider raising my score accordingly.
soundness: 2
strengths: 1. The paper is well-structured and clearly written, making the methods, results, and analysis easy to understand.
2. TabM demonstrates competitive performance, positioning it as an efficient alternative for tabular data modeling.
3. By introducing BatchEnsemble techniques into tabular data modeling, the authors have adapted and enhanced a previously underutilized approach in this domain.
4. The paper provides a thorough analysis of each component's impact on model performance, contributing valuable insights.
summary: This paper proposes TabM, a deep learning model for tabular data based on MLP enhanced with BatchEnsemble, along with custom modifications designed to improve the model's efficiency and performance. The authors position TabM as an efficient alternative to more complex architectures like Transformer-based models, claiming it achieves superior performance without their added computational cost. The paper also provides a detailed analysis, suggesting that parameter-efficient ensembling can effectively address overfitting and improve optimization dynamics in tabular MLPs.
weaknesses: 1. Although the authors mention similar methods, a more detailed comparison with other related approaches for tabular data, such as Trompt, would be helpful. Despite structural differences, both methods share commonalities in prediction, such as averaging multiple head (cycle) outputs and summing losses across heads (cycles). 
2. Comparisons with standard ensemble methods are insufficient. Incorporating results from methods like model soup could help position TabM's performance.
3. Dataset scope is somewhat limited, particularly for classification tasks. For a more comprehensive evaluation, the authors might consider incorporating the datasets from Tabzilla, which could provide richer classification benchmarks and enable a more detailed analysis, such as comparing performance across binary, multiclass, and regression tasks or evaluating model performance on datasets of varying sizes.

[1] Kuan-Yu Chen, Ping-Han Chiang, Hsin-Rung Chou, Ting-Wei Chen, Tien-Hao Chang: Trompt: Towards a Better Deep Neural Network for Tabular Data. In ICML

[2] Duncan C. McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C., Ganesh Ramakrishnan, Micah Goldblum, Colin White: When Do Neural Nets Outperform Boosted Trees on Tabular Data? In NeurIPS

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Are number of layers for MLP and TabM tuned as hyper-parameters? What are typical values for each?

- Why is k not included in the hyper-parameter search for TabM?

- How do you ensure diversity for the MLPs in the ensemble in Figure 6?

- How do you initialize the last layers in TabM to ensure diversity?

- Is it possible that TabM_mini outperforms the real ensemble because of more diversity in the initialization?

- Is it correct that when doing the full ensemble, adding an adapter into each model would lead to an equivalent ensemble? I was wondering why this ablation was not performed, but I assume random initialization of weights for each member of the ensemble makes the adapters irrelevant.
soundness: 3
strengths: The paper discusses the adoption of the BatchNorm architecture for the tabular setting, and describes several interesting ablations.
The empirical evaluation is broad and in-depth and well presented.
The empirical results are quite strong against a broad variety of baselines.
The paper is well written.
summary: The paper applies the BatchEnsemble technique to MLPs for tabular data, and investigates several modifications. The results show an improvement over MLPs and several deep baseline models and GBRT on a broach benchmark of 50 datasets from the literature. The experiments show that in the first adapter layer in particular is extremely critical, and results in the majority of gains.
weaknesses: - The technical novelty of the paper is somewhat small; however, this is made up for by the in-depth analysis with somewhat surprising results and the good performance of the proposed model.
- The paper doesn't describe the relationship to dropout, even though dropout was original describes as an efficient ensemble approach. Given the simple nature of TabM_mini, there seem to be some obvious parallels between drop-out and TabM_mini that I think are worth discussing. In particular, TabM_mini actually contains a dropout layer, and it would be interesting to evaluate how the two forms of ensembling complement each other.
- The comparison doesn't include TabPFN, an extremely strong baseline. McElfresh showed that even subsampling data to at most 3000 datapoints, TabPFN outperforms most other models.
- Given the focus on improving MLP performance, a comparison with regularization cocktails and "Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data" might be relevant but not essential.
- Given the somewhat surprising result about the importance of the initial adapter, it would be great to have more ablations on the different components. In particular, it's unclear in how far the initial adapter and last layer are coupled. How does the performance change if the best performing last layer is used for all the initial adapters? I.e. are the initial layer and last layer co-adapted or do they work independently? 

## Minor comments
Figure 2 cuts off points at 8%. It seems there's a lot of points on exactly that line, which seems a bit suspicious. Are these all the outliers that are clipped to the limits of the figure? It would be great to indicate outliers in the plot.

Table 2 is far below the mention and maybe should be moved up.

Table 2, MLP on the Maps dataset is missing units in duration.

The title of section 5.3 should read "How does the performance of TabM depend on k?"

Line 257: "requires a special care" should be "requires special care"

Line 160: multiplication with 100% is not mathematically meaningful.

Meta-review: The authors propose TabM an efficient ensembling architecture that uses MLPs as its backbone, where the main weight matrix of a linear layer is shared between the ensemble members, and only member-specific adapters are learned. The proposed method outperforms traditional ensembles of neural networks and it offers a faster runtime compared to the plain ensemble counterpart. 

The experimental protocol features a comparison with an extensive number of baselines/datasets where the proposed method achieves the best overall performance. However, in terms of novelty, the proposed approach is somehow limited.

Initially, the reviewers had concerns regarding the efficiency of the proposed method and the complexity of the datasets included in the experimental protocol. However, the authors provided a thorough rebuttal. After the rebuttal the reviewers agree that the proposed method has a strong performance and it includes an extensive experimental protocol, yielding valuable results that would be helpful to the community.

I agree with the reviewers that the novelty of the proposed method is limited, however, the proposed work provides valuable insights and a strong baseline in the realm of tabular data. Based on the previous points, I recommend acceptance.

Decision: Accept (Poster)