Title: Better by default: Strong pre-tuned MLPs and boosted trees on tabular data
Venue: NeurIPS 2024
Accepted: yes
Keywords: tabular data

benchmark

default parameters

neural networks

deep learning

multilayer perceptron

gradient-boosted decision trees
Abstract: For classification and regression on tabular data, the dominance of gradient-boosted decision trees (GBDTs) has recently been challenged by often much slower deep learning methods with extensive hyperparameter tuning. We address this discrepancy by introducing (a) RealMLP, an improved multilayer perceptron (MLP), and (b) strong meta-tuned default parameters for GBDTs and RealMLP. We tune RealMLP and the default parameters on a meta-train benchmark with 118 datasets and compare them to hyperparameter-optimized versions on a disjoint meta-test benchmark with 90 datasets, as well as the GBDT-friendly benchmark by Grinsztajn et al. (2022). Our benchmark results on medium-to-large tabular datasets (1K--500K samples) show that RealMLP offers a favorable time-accuracy tradeoff compared to other neural baselines and is competitive with GBDTs in terms of benchmark scores. Moreover, a combination of RealMLP and GBDTs with improved default parameters can achieve excellent results without hyperparameter tuning. Finally, we demonstrate that some of RealMLP's improvements can also considerably improve the performance of TabR with default parameters.

Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data

Abstract

For classification and regression on tabular data, the dominance of gradient-boosted decision trees (GBDTs) has recently been challenged by often much slower deep learning methods with extensive hyperparameter tuning. We address this discrepancy by introducing (a) RealMLP, an improved multilayer perceptron (MLP), and (b) strong meta-tuned default parameters for GBDTs and RealMLP. We tune RealMLP and the default parameters on a meta-train benchmark with 118 datasets and compare them to hyperparameter-optimized versions on a disjoint meta-test benchmark with 90 datasets, as well as the GBDT-friendly benchmark by Grinsztajn et al. (2022). Our benchmark results on medium-to-large tabular datasets (1K-500K samples) show that RealMLP offers a favorable time-accuracy tradeoff compared to other neural baselines and is competitive with GBDTs in terms of benchmark scores. Moreover, a combination of RealMLP and GBDTs with improved default parameters can achieve excellent results without hyperparameter tuning. Finally, we demonstrate that some of RealMLP's improvements can also considerably improve the performance of TabR with default parameters.

1 Introduction

Perhaps the most common type of data in practical machine learning (ML) is tabular data, characterized by a fixed number of features (columns) that can take different types such as numerical or categorical, as well as a lack of the spatiotemporal structure found in image or text data. The moderate dimension and lack of symmetries make tabular data accessible to a wide variety of machine learning methods. Although tabular data is very diverse and no method is dominant on all datasets, gradient-boosted decision trees (GBDTs) exhibit excellent results on benchmarks [18, 43, 58, 69], although their superiority has been challenged by a variety of deep learning methods [3].

While many architectures for neural networks (NNs) have been proposed [3], variants of the simple multilayer perceptron (MLP) have repeatedly been shown to be good baselines for tabular NNs [15, 16, 30, 54]. Moreover, in terms of training time, MLPs are often slower than GBDTs but still considerably faster than many other architectures [18, 43]. Therefore, we study how MLPs can be improved in terms of architecture, training, preprocessing, hyperparameters, and initialization. We also demonstrate that at least some of these improvements can successfully improve TabR [17].

Even with fast and accurate NNs, the cost of extensive hyperparameter optimization can be problematic and hinder the adoption of new methods. To address this issue, we investigate the potential of better dataset-independent default parameters for MLPs and GBDTs. Specifically, we compare the library defaults (D) to our tuned defaults (TD) and (dataset-dependent) hyperparameter optimization (HPO). Unlike McElfresh et al. [43], who argue in favor HPO on GBDTs over trying NNs, our results show a better time-accuracy trade-off for trying different (tuned) default models, as is done by modern AutoML systems [10, 11].

1.1 Contribution

The problem of finding better default parameters can be seen as a meta-learning problem [64]. We employ a meta-train benchmark consisting of 118 datasets on which the default hyperparameters are optimized, and a disjoint meta-test benchmark consisting of 90 datasets on which they are evaluated. We consider separate default parameters for classification, optimized for classification error, and for regression, optimized for RMSE. Our benchmarks do not contain missing numerical values, and we restrict ourselves to sizes between 1K and 500K samples, cf. Section 2.

In Section 3, we introduce RealMLP , which improves on standard MLPs through a bag of tricks and better default parameters , tuned entirely on the meta-train benchmark. We introduce many novel or nonstandard components , such as preprocessing using robust scaling and smooth clipping, a new numerical embedding variant, a diagonal weight layer, new schedules, different initialization methods, etc. Our benchmark results demonstrate that it often outperforms other comparably fast NNs from the literature and can be competitive with GBDTs. To demonstrate that our bag of tricks is useful for other models, we introduce RealTabR-D , a version of TabR [17] including some of our tricks that, despite less extensive tuning, achieves excellent benchmark results.

In Section 4, we provide new default parameters , tuned on the meta-train benchmark, for XGBoost [9], LightGBM [31], and CatBoost [51]. While they cannot match HPO on average, they outperform the library defaults on the meta-test benchmark.

In Section 5, we evaluate these and other models on the meta-test benchmark and the benchmark by Grinsztajn et al. [18]. We also investigate several possibilities for algorithm selection and ensembling, demonstrating that algorithm selection over default methods provides a better time-performance tradeoff than HPO, thanks to our new improved default parameters and MLP.

The code for our benchmarks, including scikit-learn interfaces for the models, is available at https://github.com/dholzmueller/pytabkit

Our code and data are archived at https://doi.org/10.18419/darus-4555 .

1.2 Related Work

Neural networks Borisov et al. [3] review deep learning on tabular data and identify three main classes of methods: Data transformation methods, specialized architectures, and regularization models. In particular, recent research has mainly focused on specialized architectures based on attention [1, 7, 15, 27], including attention between datapoints [17, 37, 53, 56, 60]. However, these methods are usually significantly slower than MLPs or even GBDTs [17, 18, 43]. Our research instead expands on improvements to MLPs for tabular data such as the SELU activation function [35], bias initialization methods [61], regularization methods [30], categorical embedding layers [19], and numerical embedding layers [16].

Benchmarks Shwartz-Ziv and Armon [58] benchmarked three deep learning methods and noticed that they performed better on the datasets from their own papers than on other datasets. We address this issue by using more datasets and evaluating our methods on datasets that they were not tuned on. Grinsztajn et al. [18], McElfresh et al. [43], and Ye et al. [69] propose larger benchmarks and find that GBDTs still outperform deep learning methods on average, analyzing why and when this is the case. Kohli et al. [36] also emphasize the need for large benchmarks. We evaluate our methods on the benchmark by Grinsztajn et al. [18] as well as datasets from the AutoML benchmark [13] and the OpenML-CTR23 regression benchmark [12].

Better defaults Probst et al. [50] study the tunability of ML methods, i.e., the difference in benchmark scores between the best fixed hyperparameters and tuned hyperparameters. While their

approach involves finding better defaults, they do not evaluate them on a separate meta-test benchmark, only consider classification, and do not provide defaults for LightGBM, CatBoost, and NNs.

Meta-learning The problem of finding the best fixed hyperparameters is a meta-learning problem [4, 64]. Although we do not introduce or employ a fully automated method to find good defaults, we use a meta-learning benchmark setup to properly evaluate them. Wistuba et al. [66] and Pfisterer et al. [49] learn portfolios of configurations and van Rijn et al. [63] learn symbolic defaults, but neither of these papers considers GBDTs or NNs. Salinas and Erickson [55] learn large portfolios of configurations on an extensive benchmark, without studying the best defaults for individual model families. Such portfolios are successfully applied in modern AutoML methods [10, 11]. At the other end of the meta-learning spectrum, TabPFN [23] meta-learns a (tuning-free) learning method on small synthetic datasets. Unlike TabPFN, we only meta-learn hyperparameters and can therefore use fewer but larger and more realistic meta-train datasets, resulting in methods that scale to larger datasets.

2 Methodology

To evaluate a fixed hyperparameter configuration H , we need a collection B train of benchmark datasets and a scoring function that computes a benchmark score S ( B train , H ) by aggregating the errors attained by the method with hyperparameters H on each dataset. However, when optimizing H on B train , we might overfit to the benchmark and therefore ideally need a second benchmark B test to get an unbiased score for H . We refer to B train , B test as meta-train and meta-test benchmarks and subdivide them into classification and regression benchmarks B train class , B train reg , B test class , and B test reg . We also use the Grinsztajn et al. [18] benchmark B Grinsztajn , which allows us to run more expensive baselines, since it limits training set sizes to 10K samples and contains fewer datasets due to more strict dataset inclusion criteria. Since B train contains groups of datasets that are variants of the same dataset, for example by using different columns as targets, we use weighting factors inversely proportional to the group size.

Table 1 shows some characteristics of the considered benchmarks. The meta-test benchmark includes datasets that are more extreme in several dimensions, allowing us to test whether our default parameters generalize 'out of distribution'. For all datasets, we remove rows with missing numerical values and encode missing categorical values as a separate category.

2.1 Benchmark Data Selection

The meta-train set consists of medium-sized datasets from the UCI Repository [32], adapted from Steinwart [61]. The meta-test set consists of the datasets from the AutoML Benchmark [13] as well as the OpenML-CTR23 regression benchmark [12] with a few modifications: we subsample some large datasets and remove datasets that are already contained in the meta-train set, are too small, or have categories with too large cardinality. More details on the datasets and preprocessing can be found in Appendix C.3.

2.2 Aggregate Benchmark Score

To optimize the default parameters, we need to define a single benchmark score. To this end, we evaluate a method on N splits = 10 random training-validation-test splits (60%-20%-20%) on each dataset. As metrics on individual dataset splits, we use classification error ( 100% -accuracy) or 1-AUROC(one-vs-rest) for classification and

for regression. There are various options to aggregate these errors into a single score. Some, such as average rank or mean normalized error, depend on which other methods are included in the evaluation, hindering an independent optimization. We would like to use the geometric mean error because arguably, an error reduction from 0 . 02 to 0 . 01 is more valuable than an error reduction from 0 . 42 to 0 . 41 . However, since the geometric mean error is too sensitive to cases with zero error (especially for classification error), we instead use a shifted geometric mean error , where a small value ε := 0 . 01 is added to the errors err ij before taking the geometric mean:

Here, we use weights w i = 1 /N datasets on the meta-test set and Grinsztajn et al. [18] benchmark. On the meta-train set, we make the w i dependent on the number of related datasets, cf. Appendix C.3. In Appendix B.10, we present results for other aggregation strategies.

3 Improving Neural Networks

The following section presents RealMLP-TD, our improved MLP with tuned defaults, which was designed based on experiments on the meta-train benchmark. A simplified version called RealMLPTD-S is also described. To demonstrate that our improvements can be useful for other architectures, we introduce RealTabR-D, a version of TabR that includes some of our improvements but has not been tuned as extensively as RealMLP-TD.

Data preprocessing In the first step of RealMLP, we apply one-hot encoding to categorical columns with at most eight distinct values (not counting missing values). Binary categories are encoded to a single feature with values {-1 , 1 } . Missing values in categorical columns are encoded to zero. After that, all numerical columns, including the one-hot encoded ones, are preprocessed independently as follows: Let x 1 , . . . , x n ∈ R be the values in column i , and let q p be the p -quantile of ( x 1 , . . . , x n ) for p ∈ [0 , 1] . Then,

̸

̸

In scikit-learn [48], this corresponds to applying a RobustScaler (first case) or MinMaxScaler (second case), and then the function f , which smoothly clips its input to the range ( -3 , 3) . Smooth clipping functions like f have been used by, e.g., Holzmüller et al. [24] and Hafner et al. [20]. Intuitively, when features have large outliers, smooth clipping prevents the outliers from affecting the result too strongly, while robust scaling prevents the outliers from affecting the inlier scaling.

NNarchitecture Our architecture, visualized in Figure 1 (a), is a multilayer perceptron (MLP) with three hidden layers containing 256 neurons each, except for the following additions and modifications:

RealMLP-TD employs categorical embedding layers [19] to embed the remaining categorical features with cardinality > 8 .

For numerical features, excluding the one-hot encoded ones, we introduce PBLD (periodic bias linear DenseNet) embeddings, which concatenate the original value to the PL embeddings proposed by Gorishniy et al. [16] and use a different periodic embedding with biases,

One-hot encoding

Robust scale

Smooth-clip

Num./cat. embeddings

Learnable scaling

Linear

Parametric activation

Dropout

Linear

3

×

(a) Preprocessing and NN architecture

for RealMLP-TD.

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

f

(

t

)

coslog

4

flat cos

0

10

20

Vanilla MLP

Robust scale + smooth clip

One-hot for small cat.

No early stopping

Last best epoch

coslog

4

Adam

β

2

= 0

.

95

Label smoothing (class.)

Output clipping (reg.)

NT parametrization

Act. fn. SELU / Mish

Parametric act. fn.

Scaling layer

Num. embeddings: PL

PL emb.

→

PBLD emb.

Dropout

p

= 0

.

15

Dropout sched:

flat cos

Weight decay wd

= 0

.

02

wd sched:

flat cos

Bias init: he+5

Weight init: data-driven

= RealMLP

Preprocessing

Hyperparameters

Architecture

Regularization

Initialization

meta-train-class

0

10

20

meta-train-reg

Benchmark score improvement (%) vs. vanilla

New

Unusual

(b) The coslog

4

and flat

_

cos schedules.

(c) From a vanilla MLP to RealMLP-TD.

inspired by Huang et al. [26] and Rahimi and Recht [52], respectively. PBLD embeddings apply separate small two-layer MLPs to each feature x i as

For efficiency reasons, we use 4-dimensional embeddings with w (1 ,i ) emb , b (1 ,i ) emb ∈ R 16 , b (2 ,i ) emb ∈ R 3 , W (2 ,i ) emb ∈ R 3 × 16 .

Our linear layers use the neural tangent parametrization (NTP) as proposed by Jacot et al. [28], i.e., they compute z ( l +1) = d -1 / 2 l W ( l ) x ( l ) + b ( l ) , where d l is the dimension of the layer input x ( l ) . The motivation behind the use of the NTP here is that it effectively modifies the learning rate for the weight matrices depending on the input dimension d l , hopefully preventing too large steps whenever the number of columns is large. We did not observe improvements when using the Adam version of the maximal update parametrization [68].

To encourage (soft) feature selection, we introduce a scaling layer before the first linear layer, which is simply a matrix-vector product with a diagonal weight matrix. In other words, it computes x i, out = s i · x i, in , with a learnable scaling factor s i for each feature i . We found it beneficial to use a larger learning rate for this layer.

RealMLP-TD uses parametric activation functions inspired by PReLU [21]. In general, for an activation function σ , we define a parametric version with separate learnable α i for each neuron i :

We use dropout after each activation function. We do not use the Alpha-dropout variant originally proposed for SELU [35], as we were not able to obtain good results with it.

When α i = 1 , this recovers σ , and when α i = 0 , the activation function is linear. As activation functions, we use SELU [35] for classification and Mish [45] for regression.

For regression, at test time, the MLP outputs are clipped to the observed range during training. (We observed that this is mainly helpful for suboptimal hyperparameters.)

Initialization The parameters s i of the scaling layer are initialized to 1 , making it an identity function at initialization. Similarly, the parameters α i of the parametric activation functions are initialized to 1 , recovering the standard activation functions at initialization. We initialize weights and biases in a data-dependent fashion during a forward pass on the (possibly subsampled) training set. We rescale rows of standard-normal-initialized weight matrices to scale the variance of the output pre-activations over the dataset to one. For the biases, we use the data-dependent he+5 initialization method [called hull+5 in 61].

Training Like Gorishniy et al. [15], we use the AdamW optimizer [34, 40]. We set its momentum hyperparameters to β 1 = 0 . 9 and β 2 = 0 . 95 instead of the default β 2 = 0 . 999 . The idea to use a smaller value for β 2 is adopted from the fastai tabular MLP [25]. RealMLP is optimized for 256 epochs with a batch size of 256. As a loss function for classification, we use softmax + cross-entropy with label smoothing [62] with parameter ε = 0 . 1 . For regression, we use the MSE loss and affinely transform the targets to have zero mean and unit variance on the training and validation set.

Hyperparameters We allow parameter-specific scheduled hyperparameters computed in each iteration using a base value, optional parameter-specific factors, and a schedule, as

allowing us, for example, to use a high learning rate factor for scaling layer parameters. Because we do not tune the number of epochs separately on each dataset, we use a multi-cycle learning rate schedule, providing multiple valleys that are usually preferable for stopping the training, while allowing high learning rates in between. Our schedule is similar to Loshchilov and Hutter [39] and Smith [59], but with a simpler analytical expression:

We set k = 4 to obtain four cycles as shown in Figure 1 (b). To allow stopping at different levels of regularization, we schedule dropout and weight decay using the following schedule, cf. Figure 1 (b): 2

The detailed hyperparameters can be found in Table A.1.

Best-epoch selection Due to the multi-cycle learning rate schedule, we do not perform classical early stopping. Instead, we always train for the full 256 epochs and then revert the model to the epoch with the lowest validation error, which in this paper is based on classification error, or RMSE for regression. In case of a tie, we found it beneficial to use the last of the tied best epochs.

RealMLP-TD-S Since certain aspects of RealMLP-TD are somewhat complex to implement, we introduce a simplified (and faster) variant called RealMLP-TD-S in Appendix A. Among the simplifications are: omitting embedding layers, using non-parametric activations, using a simpler initialization method, and omitting dropout and weight decay.

RealTabR-D For RealTabR-D, we adapt TabR-S-D by using our numerical preprocessing, setting Adam's β 2 to 0 . 95 , using our scaling layer with a modification to amplify the effective learning rate by a factor of 96, adding PBLD embeddings for numerical features, and adding label smoothing for classification. More details can be found in Appendix A.3.

4 Gradient-Boosted Decision Trees

To find better default hyperparameters for GBDTs, we employ a semi-automatic approach: We use hyperparameter optimization libraries like hyperopt [2] and SMAC3 [38] to explore a reasonably large hyperparameter space, evaluating the benchmark score of each configuration on the meta-train benchmarks, and then perform some small manual adjustments like rounding the best obtained hyperparameters. To balance efficiency and accuracy, we fix the number of estimators to 1000 and use the hist method for XGBoost. We only consider the libraries' default tree-building strategies since it is one of their main differences. The tuned defaults (TD) for LightGBM (LGBM), XGBoost (XGB), and CatBoost can be found in Table C.1, C.2, and C.3, respectively.

While some of the obtained hyperparameter values might be sensitive to the tuning and benchmark setup, we observe some general trends. First, row subsampling is used in all tuned defaults, while column subsampling is rarely applied. Second, trees are generally allowed to be deeper for regression than for classification. Third, the Bernoulli bootstrap in CatBoost is competitive with the Bayesian bootstrap while also being faster.

5 Experiments

In the following, we evaluate different methods with library defaults (D), tuned defaults (TD), and hyperparameter optimization (HPO). Recall that TD uses fixed parameters optimized on the meta-train benchmarks, while HPO tunes hyperparameters on each dataset split independently. All methods except random forests select the best iteration/epoch on the validation set of the respective dataset split based on accuracy / RMSE. All NN-based regression methods standardize the labels for training.

5.1 Methods

We provide methods in the following variants:

D : Default parameters, taken from the original library if possible (Appendix C.1).

HPO : Hyperparameters optimized separately for every train-test split on every dataset, using 50 steps of random search. Search spaces are specified in Appendix C.2 and are usually adapted from original or popular papers.

TD : Tuned default parameters from Section 3 and Section 4.

As tree-based methods, we use XGBoost ( XGB ), LightGBM ( LGBM ), and CatBoost from the respective libraries, as well as random forest ( RF ) from scikit-learn. The variant XGB-PBB-D uses meta-learned default parameters from Probst et al. [50]. For neural methods, we compare to MLP , ResNet , and FT-Transformer ( FTT ) from Gorishniy et al. [15], MLP-PLR from Gorishniy et al. [16], as well as TabR and TabR-S (without numerical embeddings) from Gorishniy et al. [17]. We compare these methods to RealMLP and RealTabR from Section 3. In addition, we investigate Best , which on each dataset split selects the method with the best validation score out of XGB, LGBM, CatBoost, and MLP-PLR (for Best-D) or RealMLP (for Best-TD and Best-HPO). Ensemble builds a weighted ensemble out of the same methods as Best, using the method of Caruana et al. [5] with 40 greedy selection steps as in Salinas and Erickson [55].

We do not run FTT, RF-HPO, and TabR-HPO on all benchmarks since some benchmarks (especially meta-test) are more expensive to run and these methods may run into out-of-memory errors.

5.2 Results

Figure 2 shows the results of the aforementioned methods on all benchmarks, along with their runtimes on a CPU. Note that XGB results on some (mainly meta-test) datasets are affected by a bug in handling rare categories, see Appendix B.

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

0475

0

.

0500

0

.

0525

0

.

0550

0

.

0575

0

.

0600

0

.

0625

0

.

0650

Shifted geometric mean of

classification errors

Meta-train classification benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

20

0

.

22

0

.

24

0

.

26

0

.

28

0

.

30

0

.

32

Shifted geometric mean of

nRMSEs

Meta-train regression benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

105

0

.

110

0

.

115

0

.

120

0

.

125

0

.

130

0

.

135

0

.

140

Shifted geometric mean of

classification errors

Meta-test classification benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

34

0

.

36

0

.

38

0

.

40

0

.

42

0

.

44

0

.

46

0

.

48

Shifted geometric mean of

nRMSEs

Meta-test regression benchmark

better

10

-

1

10

0

10

1

10

2

10

3

10

4

Average training

time (CPU)

per 1K samples [s]

0

.

175

0

.

180

0

.

185

0

.

190

0

.

195

0

.

200

Shifted geometric mean of

classification errors

better

10

-

1

10

0

10

1

10

2

10

3

10

4

Average training

time (CPU)

per 1K samples [s]

0

.

29

0

.

30

0

.

31

0

.

32

0

.

33

0

.

34

0

.

35

0

.

36

Shifted geometric mean of

nRMSEs

better

D = defaults

TD = tuned defaults

HPO = hyperparameter optimization

Best/Ensemble: out of XGB, LGBM, CatBoost, (Real)MLP

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

055

0

.

060

0

.

065

0

.

070

0

.

075

0

.

080

0

.

085

Shifted geometric mean of

1-AUC(one-vs-rest)

Meta-test classification benchmark

better

10

-

1

10

0

10

1

10

2

10

3

Average training

time (CPU)

per 1K samples [s]

0

.

100

0

.

105

0

.

110

0

.

115

0

.

120

0

.

125

Shifted geometric mean of

1-AUC(one-vs-rest)

better

How good are tuned defaults on new datasets? To answer this question, we compare the relative gaps between TD and HPO benchmark scores on the meta-test benchmarks to those on the meta-train benchmarks. The gap between RealMLP-HPO and RealMLP-TD is not much larger on the meta-test benchmarks, indicating that the tuned defaults transfer very well to the meta-test benchmark. For GBDTs, tuned defaults are competitive with HPO on the meta-train set, but not as good on the meta-test set. Still, they are considerably better than the untuned defaults on the meta-test set. Note that we did not limit the TD parameters to the literature search spaces for the HPO models (cf. Appendix C.2); for example, XGB-TD uses a smaller value of min_child_weight for classification and CatBoost-TD uses deeper trees and Bernoulli boosting. The XGBoost defaults XGB-PBB-D from Probst et al. [50] outperform XGB-TD on B test class , perhaps because their benchmark is more similar to B test class or because XGB-PBB-D uses more estimators (4168) and deeper trees.

RealMLP and RealTabR perform strongly among NNs. On most benchmarks, RealMLP-TD and RealTabR-D bring considerable improvements over MLP-PLR-D and TabR-S-D, at slightly larger runtimes, respectively. Similarly, RealMLP-HPO improves the results of MLP-PLR-HPO. TabR and FTT are notably slower than MLP-based methods on CPUs, while the difference is less pronounced on GPUs (Figure C.2). While RealMLP-TD beats TabR-S-D on many benchmarks, RealTabR-D performs even better on four out of six benchmarks, especially all regression benchmarks. On the Grinsztajn et al. [18] benchmark where we can afford to run more baselines, TabR-HPO performs best according to many aggregation metrics. It performs especially well on the electricity dataset, where MLPs struggle to learn high-frequency patterns [18].

RealMLP and RealTabR are competitive with tree-based models. On the meta-train and metatest benchmarks, RealMLP and RealTabR perform better than GBDTs in terms of shifted geometric mean error, while also being comparable or slightly better in terms of other aggregations like mean normalized error (Appendix B.10) or win-rates (Appendix B.12). On the Grinsztajn et al. [18] benchmark, RealMLP performs worse than CatBoost for classification and comparably for regression, while RealTabR-D performs comparably to CatBoost-TD for classification and better for regression.

Among GBDTs, CatBoost defaults are better and slower. Several papers have found CatBoost to perform favorably among GBDTs while being more computationally expensive to train [8, 33, 43, 51, 69]. We observe the same for our tuned defaults on most benchmarks.

Simply trying all default algorithms is faster and very often better than (naive) single-algorithm HPO. When comparing Best-TD to 50-step HPO on RealMLP or GBDTs, we notice that Best-TD is faster on average, while also being competitive with the best of the HPO models. In comparison, BestD is often outperformed by RealMLP-HPO. We also note that ensemble selection [5] usually gives 0-3% improvement on the benchmark score compared to selecting the best model, and can potentially be further improved [6]. Unlike McElfresh et al. [43], who argue in favor of CatBoost-HPO over trying NNs, our results favor model portfolios as used in modern AutoML systems [10].

Analyzing NN improvements Figure 1 (c) shows how adding the proposed RealMLP components to a simple MLP improves the meta-train benchmark performance. However, these results depend on the order in which components are added, which is addressed by a separate ablation study in Appendix B. For example, the large weight decay value makes RealMLP-TD sensitive to changes in some other hyperparameters like β 2 . We also show in Appendix B.8 that our architectural improvements alone are beneficial when applied to MLP-D directly, although non-architectural aspects are at least as important. In particular, our numerical preprocessing is easy to adopt and often beneficial for other NNs as well (Appendix B.7). The scaling layer and PBLD embeddings are easy to use and turned out to be effective within RealTabR-D as well. If affordable, larger stopping patiences and the use of (cyclic) learning rate schedules can be useful, while label smoothing is influential but can be detrimental for metrics like AUROC (Figure 3, Appendix B.5).

Dependence on benchmark choices We observe that choices in benchmark design can affect the interpretation of the results. The use of different aggregation metrics than the shifted geometric mean reduces the advantage of TD methods (Appendix B.10). For classification, using AUROC instead of classification error (Figure 3, Appendix B.5) favors GBDTs. Different dataset selection and preprocessing criteria on different benchmarks lead to large differences between benchmarks in the average errors, as indicated by the y -axis scaling in Figure 2.

Further insights In Appendix B, we present additional experimental results. We compare bagging and refitting for RealMLP-TD and LGBM-TD, finding that refitting multiple models is often better on average. We demonstrate that GBDTs benefit from high early stopping patiences for classification, especially when using accuracy as the stopping metric. When considering AUROC as a stopping metric, we show that stopping on cross-entropy is preferable to accuracy (Appendix B.5).

Limitations While our benchmarks cover medium-to-large tabular datasets in standard settings, it is unclear to which extent the obtained defaults can generalize to very small datasets, distribution shifts, datasets with missing numerical values, and other metrics such as log-loss. Additionally, runtimes and the resulting tradeoffs may change with different parallelization, hardware, or (time-aware) HPO algorithms. For computational reasons, we only use a single training-validation split per train-test split. This means that HPO can overfit the validation set more easily than in a cross-validation setup. While we extensively benchmark different NN models from the literature, we do not attempt to equalize non-architectural aspects, and our work should therefore not be seen as a comparison of architectures. We compared to TabR-S-D as a recent promising method with good default parameters [17, 69]. However, due to a surge of recently published deep tabular models [e.g., 7, 8, 29, 33, 41, 57, 67], it is unclear what the current 'best' deep tabular model is. In particular, ExcelFormer [7] also promises strong-performing default parameters. For GBDTs, due to the cost of running the benchmarks, our limits on the depth and number of trees are on the lower side of the literature.

6 Conclusion

In this paper, we studied the potential of improved default parameters for GBDTs and an improved MLP, evaluated on a large separate meta-test benchmark as well as the benchmark by Grinsztajn et al. [18], and investigated the time-accuracy tradeoffs of various algorithm selection and ensembling scenarios. Our improved MLP mostly outperforms other NNs from the literature with moderate runtime and is competitive with GBDTs in terms of benchmark scores. Since many of the proposed improvements to NNs are orthogonal to the improvements in other papers, they offer exciting opportunities for combinations, as we demonstrated with our RealTabR variant. While the 'NNs vs GBDTs' debate remains interesting, our results demonstrate that with good default parameters, it is worth trying both algorithm families even with a moderate training time budget.

Acknowledgments and Disclosure of Funding

We thank Gaël Varoquaux, Frank Sehnke, Katharina Strecker, Ravid Shwartz-Ziv, Lennart Purucker, and Francis Bach for helpful discussions. We thank Katharina Strecker for help with code refactoring.

Funded by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC 2075 - 390740016. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting David Holzmüller. LG acknowledges support in part by the French Agence Nationale de la Recherche under Grant ANR-20CHIA-0026 (LearnI). Part of this work was performed on the computational resource bwUniCluster funded by the Ministry of Science, Research and the Arts Baden-Württemberg and the Universities of the State of Baden-Württemberg, Germany, within the framework program bwHPC. Part of this work was performed using HPC resources from GENCI-IDRIS (Grant 2023-AD011012804R1 and 2024-AD011012804R2).

Contribution statement DH and IS conceived the project. DH implemented and experimentally validated the newly proposed methods and wrote the initial paper draft. DH and LG contributed to benchmarking, plotting, and implementing baseline methods. LG and IS helped revise the draft. IS supervised the project and contributed dataset downloading code.

References

Sercan O. Arik and Tomas Pfister. TabNet: Attentive interpretable tabular learning. In AAAI Conference on Artificial Intelligence , 2021.

James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International Conference on Machine Learning , 2013.

Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Pavel Brazdil, Christophe Giraud Carrier, Carlos Soares, and Ricardo Vilalta. Metalearning: Applications to Data Mining . Springer Science & Business Media, 2008.

Rich Caruana, Alexandru Niculescu-Mizil, Geoff Crew, and Alex Ksikes. Ensemble selection from libraries of models. In International Conference on Machine Learning , 2004.

Rich Caruana, Art Munson, and Alexandru Niculescu-Mizil. Getting the most out of ensemble selection. In International Conference on Data Mining , pages 828-833. IEEE, 2006.

Jintai Chen, Jiahuan Yan, Qiyuan Chen, Danny Z. Chen, Jian Wu, and Jimeng Sun. Can a Deep Learning Model be a Sure Bet for Tabular Prediction? In Conference on Knowledge Discovery and Data Mining . ACM, August 2024.

Kuan-Yu Chen, Ping-Han Chiang, Hsin-Rung Chou, Ting-Wei Chen, and Tien-Hao Chang. Trompt: Towards a better deep neural network for tabular data. In International Conference on Machine Learning , 2023.

Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In International Conference on Knowledge Discovery and Data Mining , 2016.

Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. AutoGluon-Tabular: Robust and accurate AutoML for structured data. In 7th ICML Workshop on Automated Machine Learning , 2020.

Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto-sklearn 2.0: Hands-free automl via meta-learning. The Journal of Machine Learning Research , 23(261), 2022.

Sebastian Felix Fischer, Matthias Feurer, and Bernd Bischl. OpenML-CTR23-A curated tabular regression benchmarking suite. In AutoML Conference 2023 (Workshop) , 2023.

Pieter Gijsbers, Marcos LP Bueno, Stefan Coors, Erin LeDell, Sébastien Poirier, Janek Thomas, Bernd Bischl, and Joaquin Vanschoren. AMLB: an AutoML benchmark. Journal of Machine Learning Research , 25(101):1-65, 2024. URL https://www.jmlr.org/papers/v25/ 22-0493.html .

Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association , 102(477):359-378, 2007.

Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Neural Information Processing Systems , 2021.

Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. Neural Information Processing Systems , 2022.

Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. TabR: Tabular deep learning meets nearest neighbors. In International Conference on Learning Representations , 2024.

Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Neural Information Processing Systems , 2022.

Cheng Guo and Felix Berkhahn. Entity embeddings of categorical variables. arXiv:1604.06737 , 2016.

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv:2301.04104 , 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In IEEE International Conference on Computer Vision , pages 1026-1034, 2015.

Steffen Herbold. Autorank: A Python package for automated ranking of classifiers. Journal of Open Source Software , 5(48):2173, 2020. doi: 10.21105/joss.02173. URL https://doi. org/10.21105/joss.02173 . Publisher: The Open Journal.

Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. In International Conference on Learning Representations , 2022.

David Holzmüller, Viktor Zaverkin, Johannes Kästner, and Ingo Steinwart. A framework and benchmark for deep batch active learning for regression. Journal of Machine Learning Research , 24(164), 2023.

Jeremy Howard and Sylvain Gugger. Fastai: A layered API for deep learning. Information , 11 (2):108, 2020.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In Computer Vision and Pattern Recognition , pages 47004708, 2017.

Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. TabTransformer: Tabular data modeling using contextual embeddings. arXiv:2012.06678 , 2020.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Neural Information Processing Systems , 2018.

Manu Joseph and Harsh Raj. GANDALF: Gated Adaptive Network for Deep Automated Learning of Features. arXiv:2207.08548 , 2024.

Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. In Neural Information Processing Systems , 2021.

Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. LightGBM: A highly efficient gradient boosting decision tree. In Neural Information Processing Systems , 2017.

Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. The UCI Machine Learning Repository. URL https://archive.ics.uci.edu .

Myung Jun Kim, Léo Grinsztajn, and Gaël Varoquaux. CARTE: pretraining and transfer for tabular learning. In International Conference on Machine Learning , 2024.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.

Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Neural Information Processing Systems , 2017.

Ravin Kohli, Matthias Feurer, Katharina Eggensperger, Bernd Bischl, and Frank Hutter. Towards Quantifying the Effect of Datasets for Benchmarking: A Look at Tabular Machine Learning. In ICLR 2024 Data-centric Machine Learning Research Workshop , 2024.

Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Thomas Rainforth, and Yarin Gal. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. In Neural Information Processing Systems , 2021.

Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter. SMAC3: A versatile Bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research , 23(54), 2022.

Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations , 2017.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.

Sascha Marton, Stefan Lüdtke, Christian Bartelt, and Heiner Stuckenschmidt. GRANDE: Gradient-based decision tree ensembles for tabular data. In International Conference on Learning Representations , 2024.

Calvin McCarter. The kernel density integral transformation. Transactions on Machine Learning Research , 2023.

Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? In Neural Information Processing Systems , 2023.

Dmytro Mishkin and Jiri Matas. All you need is a good init. In International Conference on Learning Representations , 2016.

Diganta Misra. Mish: A self regularized non-monotonic activation function. In British Machine Vision Conference , 2020.

Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, and Michael I. Jordan. Ray: A distributed framework for emerging AI applications. In USENIX Symposium on Operating Systems Design and Implementation , 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, and Luca Antiga. PyTorch: An imperative style, high-performance deep learning library. Neural Information Processing Systems , 32, 2019.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, and Vincent Dubourg. Scikitlearn: Machine learning in Python. Journal of Machine Learning Research , 12(85), 2011.

Florian Pfisterer, Jan N. Van Rijn, Philipp Probst, Andreas C. Müller, and Bernd Bischl. Learning multiple defaults for machine learning algorithms. In Genetic and Evolutionary Computation Conference , July 2021.

Philipp Probst, Anne-Laure Boulesteix, and Bernd Bischl. Tunability: Importance of hyperparameters of machine learning algorithms. Journal of Machine Learning Research , 20(53), 2019.

Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: Unbiased boosting with categorical features. In Neural Information Processing Systems , 2018.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Neural Information Processing Systems , 2007.

Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, and Michael K. Kopp. Hopfield networks is all you need. In International Conference on Learning Representations , 2020.

Ivan Rubachev, Nikolay Kartashev, Yury Gorishniy, and Artem Babenko. TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks. arXiv:2406.19380 , 2024.

David Salinas and Nick Erickson. TabRepo: A large scale repository of tabular model evaluations and its AutoML applications. In AutoML Conference , 2024.

Bernhard Schäfl, Lukas Gruber, Angela Bitto-Nemling, and Sepp Hochreiter. Modern Hopfield networks as memory for iterative learning on tabular data. In NeurIPS Workshop on Associative Memory & Hopfield Networks in 2023 , 2023.

Junhong Shen, Liam Li, Lucio M. Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. Cross-modal fine-tuning: Align then refine. In International Conference on Machine Learning , 2023.

Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion , 81:84-90, 2022.

Leslie N. Smith. Cyclical learning rates for training neural networks. In Winter Conference on Applications of Computer Vision , 2017.

Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training. In NeurIPS 2022 Table Representation Learning Workshop , 2022.

Ingo Steinwart. A sober look at neural network initializations. arXiv:1903.11482 , 2019.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Computer Vision and Pattern Recognition , 2016.

Jan N. van Rijn, Florian Pfisterer, Janek Thomas, Andreas Muller, Bernd Bischl, and Joaquin Vanschoren. Meta learning for defaults: Symbolic defaults. In NeurIPS 2018 Workshop on Meta-Learning , 2018.

Joaquin Vanschoren. Meta-learning: A survey. arXiv:1810.03548 , 2018.

Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. OpenML: Networked science in machine learning. ACM SIGKDD Explorations Newsletter , 15(2):49-60, 2014. Publisher: ACM New York, NY, USA.

Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Learning hyperparameter optimization initializations. In International Conference on Data Science and Advanced Analytics , pages 1-10, 2015.

Chenwei Xu, Yu-Chao Huang, Jerry Yao-Chieh Hu, Weijian Li, Ammar Gilani, Hsi-Sheng Goan, and Han Liu. BiSHop: Bi-directional cellular learning for tabular data with generalized sparse modern Hopfield model. In International Conference on Machine Learning , 2024.

Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In Neural Information Processing Systems , 2021.

Han-Jia Ye, Si-Yang Liu, Hao-Run Cai, Qi-Le Zhou, and De-Chuan Zhan. A closer look at deep learning on tabular data. arXiv:2407.00956 , 2024.

Appendices

Appendix Contents.

A Further Details on Neural Networks

The detailed hyperparameter settings for RealMLP-TD and RealMLP-TD-S are listed in Table A.1.

A.1 RealMLP-TD Details

Architecture To make the binary and multi-class cases more similar, we use two output neurons in the binary case, using the same loss function as in the multi-class case.

Initialization We initialize categorical embedding parameters from N (0 , 1) . We initialize the components of w (1 ,i ) emb from N (0 , 0 . 1 2 ) and of b (1 ,i ) emb from U [ -π, π ] . The other numerical embedding parameters are initialized according to PyTorch's default initialization, that is, from the uniform distribution U [ -1 / √ 16 , 1 / √ 16] . For weights and biases of the linear layers, we use a data-dependent initialization. The initialization is performed on the fly during a first forward pass of the network on the training set (which can be subsampled adaptively not to use more than 1 GB of RAM). We realize this by providing fit_transform() methods similar to a pipeline in scikit-learn. For the weight matrices, we use a custom two-step procedure: First, we initialize all entries from N (0 , 1) . Then, we rescale each row of the weight matrix such that the outputs 1 √ d l W ( l ) x ( l ) j have variance 1 over the dataset (i.e. when considering the sample index j ∈ { 1 , . . . , n } as a uniformly distributed random variable). This is somewhat similar to the LSUV initialization method [44]. For the biases, we use the data-dependent he+5 initialization method [called hull+5 in 61].

Training We implement weight decay as in PyTorch using θ ← θ -lr · wd · θ , which includes the learning rate unlike the original version [40].

A.2 RealMLP-TD-S Details

For RealMLP-TD-S, we make the following changes compared to RealMLP-TD:

We apply one-hot encoding to all categorical variables and do not apply categorical embeddings.

We use the standard non-parametric versions of the SELU and Mish activation functions.

We do not apply numerical embeddings.

We do not use dropout and weight decay.

We do not clip the outputs, even in the regression case.

We use simpler weight and bias initializations: We initialize weights and biases from N (0 , 1) , except in the last layer, where we initialize them to zero.

We apply a different base learning rate in the regression case.

A.3 RealTabR-D Details

To obtain RealTabR-D, we modify TabR-S-D in the following ways:

We replace the standard numerical preprocessing (a modified quantile transform) with our robust scaling and smooth clipping.

We use our scaling layer, but modify it to obtain a higher effective learning rate. We do this by modifying the forward pass to

We set Adam's β 2 to 0 . 95 instead of 0 . 999 .

while initializing s i to 1 /γ . This will multiply the gradients of s i by γ , which will be ignored by Adam's normalization (when neglecting Adam's ε parameter). It will also multiply the optimizer updates by γ , leading to approximately the same effect as multiplying the learning rate by γ . However, a difference is that multiplying the learning rate by γ will also lead to stronger weight decay updates in PyTorch's AdamW implementation, while the introduction of γ does not increase the relative magnitude of weight decay updates. We chose the version with γ for simplicity of implementation. While RealMLP-TD uses a learning rate factor of 6 for the scaling layer, it uses a higher base learning rate due to the use of the neural tangent parametrization. For all layers except the first one, which have width 256, the neural tangent

parametrization in RealMLP-TD uses a factor similar to γ , which is set to 1 / 16 = 1 / √ 256 . Hence, RealMLP-TD without NTP should use a base learning rate for these layers that is smaller by a factor of 1 / 16 , and therefore use a learning rate factor of 6 · 16 = 96 for the scaling layer. Consequently, we set γ := 96 for RealTabR-D without further tuning, noting that it performed significantly better than γ = 6 on the meta-train benchmarks.

For classification, we use label smoothing with parameter ε = 0 . 1 .

We use our PBLD embeddings for numerical features before the scaling layer, instead of no numerical embeddings in TabR-S-D. In order to make every experiment run on a GPU with 24GB RAM, we decrease the dimension of the hidden embedding layer from 16 to 8, although using 16 would have performed slightly better in our experiments on the meta-train benchmarks.

Since we adapted hyperparameters like learning rate and weight decay from TabR-S-D without meta-learning them, we refer to the resulting method as RealTabR-D and not RealTabR-TD. We did not include other tricks from RealMLP-TD for various reasons:

Brief experiments with NTP and the Mish activation deteriorated the performance.

Other tricks were not tried due to limited time of experimentation, expected increases in the already somewhat large runtime, and/or implementation complexity.

Parametric activations and increased stopping patience showed small improvements but were excluded due to a larger runtime.

A.4 Details on Cumulative Ablation

Here, we provide more details on the vanilla MLP and the ablation steps from Figure 1 (c). For each step, we choose the best default learning rate out of a learning rate grid, using { 0 . 0004 , 0 . 0007 , 0 . 001 , 0 . 0015 , 0 . 0025 , 0 . 004 , 0 . 007 , 0 . 01 , 0 . 015 } for NNs using standard parametrization and { 0 . 01 , 0 . 02 , 0 . 03 , 0 . 04 , 0 . 07 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } for NNs using neural tangent parametrization.

Vanilla MLP: We use three hidden layers with 256 hidden neurons in each layer, just like RealMLP-TD, and the ReLU activation function. Each linear layer uses standard parametrization and the PyTorch default initialization, which is uniform from [ -1 / √ fan_in , 1 / √ fan_in ] for both weights and biases, where fan_in is the input dimension. Categorical features are embedded using embedding layers, using eight-dimensional embeddings for each feature. Numerical features are transformed using a scikit-learn QuantileTransformer to approximately normal-distributed features. Optimization is performed using Adam with constant learning rate and default parameters β 1 = 0 . 9 , β 2 = 0 . 999 , ε = 10 -8 for at most 256 epochs with batch size 256, with constant learning rate. If the best validation error (classification error or RMSE) does not improve for 40 epochs, training is stopped. In each case, the model is reverted to the parameters of the epoch with the best validation score, using the first best epoch in case of a tie.

One-hot for small cat.: As in RealMLP-TD, we use one-hot encoding for categories with at most eight values, not counting missing values.

Robust scale + smooth clip: We replace the QuantileTransformer with robust scaling and smooth clipping.

No early stopping: We always train the full 256 epochs.

coslog 4 lr sched: We use the coslog 4 learning rate schedule instead of a constant one.

Last best epoch: In case of a tie, we use the last of the best epochs.

Adam β 2 = 0 . 95 : We set β 2 = 0 . 95 .

Output clipping (reg.): For regression, outputs are clipped to the min-max range observed during training.

Label smoothing (class.): We enable label smoothing with ε = 0 . 1 in the classification case.

NT parametrization: We use the neural tangent parametrization for linear layers, setting the bias learning rate factor to 0 . 1 .

Parametric act. fn.: We use parametric versions of the activation functions, with a learning rate factor of 0 . 1 for the parameters.

Act. fn. SELU / Mish: We change the activation function from ReLU to SELU (classification) or Mish (regression).

Scaling layer: We use a scaling layer with a learning rate factor of 6 before the first linear layer.

Num. embeddings: PBLD: We apply our PBLD embeddings instead.

Num. embeddings: PL: We apply the PL embeddings [16] to numerical features.

Dropout p = 0 . 15 : We apply dropout with probability 0 . 15 .

Weight decay wd = 0 . 02 : We apply weight decay (as in AdamW, PyTorch version) with value 0 . 02 .

Dropout sched: flat _ cos : We apply the flat _ cos schedule to the dropout probability.

wd sched: flat _ cos : We apply the flat _ cos schedule to weight decay.

Weight init: data-driven: We apply our data-driven weight initialization method.

Bias init: he+5: We apply the he+5 bias initialization method from Steinwart [61] (originally called hull+5).

A.5 Discussion

Here, we discuss some of the design decisions behind RealMLP-TD and possible trade-offs. First, our implementation allows us to train RealMLP-TD in a vectorized fashion on multiple train-validationtest splits at the same time. On the one hand, this can lead to speedups on GPUs when training multiple models in parallel, including on the benchmarks. On the other hand, it can hinder the implementation of certain methods like patience-based early stopping or loss-based learning rate schedules. While our ablations in Appendix B.1 show the advantage of our multi-cycle schedule over decreasing learning rate schedules, the latter ones could potentially enable a faster average training time through low-patience early stopping. An interesting follow-up question could be whether the multi-cycle schedule still works well with larger-patience early stopping.

Regarding categorical embeddings, our meta-train benchmark does not contain many high-cardinality categorical variables, and we were not able to conclude whether categorical embeddings are helpful or harmful compared to one-hot encoding (see Appendix B.1). Our motivation to include categorical embeddings stems from Guo and Berkhahn [19] as well as their potential to be more efficient for high-cardinality categorical variables. However, in practice, we find pure one-hot encoding to be faster on most datasets. Regarding the embedding size, we found that 4 already gave good results for numerical embeddings and decided to use 8 for categorical variables.

Additionally, other speed-accuracy tradeoffs are possible. Especially for regression, we observed that more epochs and larger hidden layers can be helpful. When faster networks are desired, the omission of numerical and categorical embedding layers as well as parametric activations from RealMLP-TD can be helpful, while the other omissions in RealMLP-TD-S do not considerably affect the training time. Of course, using larger batch sizes can also be helpful for larger datasets.

One caveat for classification is that cross-entropy with label smoothing is not a proper scoring rule, that is, in the infinite-sample limit, it is not minimized by the true probabilities P ( y | x ) [14]. Hence, label smoothing might not be suitable when other classification error metrics are used, as demonstrated in Appendix B.5 for AUROC.

B More Experiments

In this section, we present more experimental results. Note that XGBoost results are affected by a bug where, if a categorical value is not present in the training or validation set, it could cause adjacent categorical values to be encoded differently during training, validation, and evaluation. This affects the results mainly on the meta-test benchmarks, where the SGM scores for XGB-TD and XGB-D are around 2% lower after fixing the bug. These differences are not large enough to affect our qualitative conclusions. Due to the large computational cost, we did not rerun XGB-HPO and XGB-PBB-D after fixing the bug, and we provide the old XGB-TD and XGB-D results for a fair comparison to XGB-HPO and XGB-PBB-D.

B.1 MLP Ablations

To assess the importance of different improvements in RealMLP-TD, we perform an ablation study. We perform the ablation study only on the meta-train benchmarks, first because they are considerably faster to run, and second because we tune the default parameters only on the meta-train benchmarks. Since the hyperparameters of RealMLP-TD have been tuned on the meta-train benchmarks, the ablation scores are not unbiased but represent some of the considerations that have been made when tuning the defaults. For each ablation, we multiply the default learning rate by learning rate factors from the grid { 0 . 1 , 0 . 15 , 0 . 25 , 0 . 35 , 0 . 5 , 0 . 7 , 1 . 0 , 1 . 4 , 2 . 0 , 3 . 0 , 4 . 0 } and pick the best one. Table B.1 shows the results of the ablation study in terms of the relative increase of the benchmark score for each ablation.

In general, we observe that ablations often lead to much larger changes for regression than for classification. Perhaps this is because nRMSE is more sensitive to outliers compared to classification error. Another factor could be that the classification benchmark contains more datasets than the regression benchmark. For the specific ablations, we observe a few things:

For the numerical embeddings , we see that PBLD outperforms PL, PLR, and no numerical embeddings. Contrary to Gorishniy et al. [16], PL embeddings perform better than PLR

embeddings in our setting. While the configurations with PLR and no numerical embeddings appear extremely bad for regression, we observed that they can perform more benignly with lower weight decay values.

Using a cosine decay learning rate schedule instead of our multi-cycle schedule leads to small deteriorations. A constant learning rate schedule performs even worse, especially for regression.

Using the Adam default value of β 2 = 0 . 999 instead of our default β 2 = 0 . 95 leads to considerably worse performance, especially for regression. As for numerical embeddings, we observed that the difference is less pronounced at lower weight decay values.

Not employing label smoothing for classification is detrimental by around 1.8%.

The use of parametric activations results in a considerable 4.8% improvement for regression but is insignificant for classification. We observed that parametric activations can sometimes alleviate optimization difficulties with weight decay.

The learnable scaling layer yields improvements around 1.2% on both benchmarks.

The differences between activation functions are rather small. For classification, Mish is competitive with SELU in this ablation but we found it to be worse in some other hyperparameter settings, so we keep SELU as the default. For regression, Mish performs best.

When comparing the standard parametrization (SP) to the neural tangent parametrization (NTP), we disable weight decay for a fair comparison. Moreover, for SP, we set the learning rate factors for weight and bias layers to 1 / 16 = 1 / √ 256 . This is because, for the weights in NTP, the effective updates by Adam are damped by this factor in all hidden layers except the first one. Compared to NTP without weight decay, SP without weight decay performs insignificantly worse on both benchmarks. It is unclear to us why the parametrization, which has a considerable influence on how the effective learning speed of the first linear layer scales with the number of features, is apparently of little importance.

For dropout and weight decay , we observe that they yield comparable but not always significant benefits for classification and regression. Scheduling dropout and weight decay parameters with the flat _ cos schedule is helpful for regression, but not for classification in this setting.

When comparing the data-dependent initialization of RealMLP-TD to a vanilla initialization with standard normal weights and zero biases, we see that the data-dependent initialization gains around 1% on both benchmarks.

We do not observe a significant difference when using one-hot encoding for all categorical variables, since our benchmarks contain only very few datasets with large-cardinality categorical variables.

For selecting the best epoch, we consider selecting the first best epoch instead of the last best epoch in case of a tie. This is only relevant for classification metrics like classification error, where ties are somewhat likely to occur, especially on small and 'easy' datasets. We observe a non-significant 0.4% deterioration in the benchmark score.

B.2 MLP Preprocessing

In Table B.2, we compare different preprocessing methods for numerical features. Since we want to compare these methods in a relatively conventional setting, we apply them to RealMLP-TD-S (without numerical embeddings) and before one-hot encoding. We compare the following methods:

Robust scaling and smooth clipping, our method used in RealMLP-TD and RealMLP-TD-S and described in Section 3.

Standardization, i.e. subtracting the mean and dividing by the standard deviation. If the standard deviation of a feature is zero, we set the feature to zero.

Robust scaling without smooth clipping.

Standardization followed by smooth clipping.

A variant of the quantile transform, which we call the RTDL version, used by Gorishniy et al. [15] and Gorishniy et al. [17]. This version uses a dataset-size-dependent number of quantiles and adds some noise before fitting the transformation. It also uses a normal output distribution.

The quantile transformation from scikit-learn [48] with normal output distribution, which is popular in recent works [16-18, 43].

The recent kernel density integral transform [42] with normal output distribution, which interpolates between the quantile transformation and min-max scaling, with default parameter α = 1 .

Table B.2 shows that on the meta-train benchmark, robust scaling and smooth clipping performs best for both classification and regression.

B.3 Bagging, Refitting, and Ensembling

In our benchmark, for each training-test split, we only train one model on one training-validation split for efficiency reasons. However, ensembling and cross-validation techniques usually allow additional improvements to models. Here, we study multiple variants for RealMLP-TD and LGBM-TD. Let D be the available data for training and validation, split into five equal-size subsets D 1 , . . . , D 5 . (When |D| is not divisible by five, D 1 ∪ . . . ∪ D 5 ⊊ D since we need equal-size validation sets for vectorized NNs.) Let f D ,t ( X ) be the predictions on inputs X of the model trained on training set D after t ∈ { 1 , . . . , T } epochs (for NNs) or iterations (for LGBM). For classification, we consider the class probabilities as predictions. Let L D ′ ( f D ,t ) be the loss of f D ,t on dataset D ′ . Then, we compare the test errors of an ensemble of M = 1 or M = 5 models, trained using bagging or refitting, with individual or joint stopping (best-epoch selection), which is formally given as follows:

Here, each model is trained with a different random seed. For LGBM, since we use an early stopping patience of 300 for each of the individual models, the argmin in the definition of t ∗ i can only go up to the minimum stopping iteration T across the considered models.

The results of our experiments can be found in Table B.3 for LGBM-TD and in Table B.4 for RealMLP-TD. As expected, five models are considerably better than one. We find that refitting is mostly better than bagging, although a disadvantage of refitted models is that no validation scores are available, and it is unclear how HPO would affect this comparison. Comparing individual stopping to joint stopping, we find that individual stopping has a slight advantage in five-model bagging, while joint stopping performs better for single-model refitting. In the other two scenarios, joint stopping appears slightly better for RealMLP-TD and slightly worse for LGBM-TD. We also observe that the benefit of using five models instead of one appears to be larger for RealMLP-TD than for LGBM-TD.

B.4 Early stopping for GBDTs

In Figure B.1 and Figure B.2, we study the influence of different early stopping patiences and metrics on the resulting benchmark performance of XGB-TD, LGBM-TD, and CatBoost-TD. While the regression results only deteriorate slightly for low patiences of 10 or 20 iterations, classification results are much more hurt by low patiences. In the classification setting, we evaluate the use of

10

20

50

100

300

1000

Stopping patience

-

2

0

2

4

6

8

10

12

14

Error increase in %

XGB-TD

10

20

50

100

300

1000

Stopping patience

LGBM-TD

10

20

50

100

300

1000

Stopping patience

CatBoost-TD

stopped on classification error

stopped on Brier loss

stopped on cross-entropy loss

different losses for early stopping and for best-epoch selection: classification error, Brier score, and cross-entropy loss. In each case, cross-entropy loss is used as the training loss, and classification error is used for evaluating the models on the test sets in the computation of the benchmark score. We observe that models stopped on classification error strongly deteriorate at low patiences ( ≲ 100 ), while our default patience of 300 achieves close-to-optimal results. Models stopped on cross-entropy loss deteriorate much less at low patiences, but achieve roughly 2% worse benchmark score at high patiences. Stopping on Brier loss achieves very good high-patience performance and is still only slightly more sensitive to the patience than stopping on cross-entropy loss. An interesting follow-up question would be if HPO can attenuate the differences between different settings.

B.5 Results for AUROC

For classification, there are many different metrics to capture model performance. In the main paper, we use classification error to evaluate models. All TD configurations were tuned for classification error, early stopping and best-epoch selection were performed for classification error, and HPO was performed for classification error. Here, we evaluate models on the area under the ROC curve, also known as AUROC, AUC ROC, or AUC. For the multi-class case, we use the one-vs-rest formulation of AUC, which is faster to evaluate than one-vs-one. Higher AUC values are better and the optimal value is 1 . Since we are interested in the shifted geometric mean error, we use 1 -AUC instead.

10

20

50

100

300

1000

Stopping patience

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

Error increase in %

XGB-TD

10

20

50

100

300

1000

Stopping patience

LGBM-TD

10

20

50

100

300

1000

Stopping patience

CatBoost-TD

stopped on RMSE

We compare two settings:

A variant of the original setting where early stopping and the selection of the best epoch/iteration is based on accuracy but HPO is performed on 1 -AUC . (Thanks to using random search, we do not have to re-run the HPO for this.)

A setting where we use the cross-entropy loss for stopping and selecting the best epoch/iteration. While it would be possible to stop on AUC directly, this can be significantly slower since AUC is slower to evaluate. We do not perform HPO in this setting since it is expensive to run.

In both settings, we also evaluate RealMLP without label smoothing (no ls). Figure 3 shows the results optimized for accuracy and Figure B.3 shows the results optimized for cross-entropy. We make a few observations:

Stopping for cross-entropy generally performs better than stopping for classification error.

Tuned defaults are mostly still better than the library defaults, except for XGBoost on B test class .

Label smoothing harms RealMLP for AUC, perhaps because the stopping metric does not use label smoothing, or because it encourages near-constant logits in areas where the model is relatively certain.

RealMLP without label smoothing is still competitive with GBDTs on the meta-test benchmark but does not perform better than GBDTs unlike what we observed for classification error.

B.6 Results Without Missing-Value Datasets

To assess whether the results are influenced by our choices in missing value handling and exclusion, Figure B.4 presents results on all meta-test datasets that originally did not contain missing values. Only six meta-test datasets originally contain missing values: Three from B test class (kick, okcupid-stem, and porto-seguro) and three from B test reg (fps_benchmark, house_prices_nominal, SAT11-HAND-runtimeregression). While RealMLP deteriorates slightly, especially due to the exclusion of fps_benchmark, qualitative takeaways remain similar.

B.7 Comparing Preprocessing Methods for NNs

In the other sections of this paper, we run each NN using the preprocessing from the respective paper that introduced it. Specifically, we use robust scaling and smooth clipping for RealMLP and the RTDL version of the quantile transform for the other papers (see also Appendix B.2). Here,

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

029

0

.

030

0

.

031

0

.

032

0

.

033

0

.

034

0

.

035

0

.

036

Shifted geometric mean of

1-AUC(one-vs-rest)

Meta-train classification benchmark

better

10

-

1

10

0

10

1

Average training

time (CPU)

per 1K samples [s]

0

.

055

0

.

060

0

.

065

0

.

070

0

.

075

0

.

080

0

.

085

Shifted geometric mean of

1-AUC(one-vs-rest)

Meta-test classification benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

1075

0

.

1100

0

.

1125

0

.

1150

0

.

1175

0

.

1200

0

.

1225

0

.

1250

Shifted geometric mean of

1-AUC(one-vs-rest)

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

10

0

.

11

0

.

12

0

.

13

0

.

14

Shifted geometric mean of

classification errors

B

test

class

without missing value datasets

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

36

0

.

38

0

.

40

0

.

42

0

.

44

0

.

46

0

.

48

0

.

50

Shifted geometric mean of

nRMSEs

B

test

reg

without missing value datasets

better

we evaluate if robust scaling and smooth clipping can improve MLP, ResNet, MLP-PLR, FTT, and TabR-S as well. This also yields a more direct comparison of the architectures, although the nets still differ in other aspects such as initialization and regularization.

Figure B.5 includes results with robust scaling and smooth clipping (RS+SC) for MLP, ResNet, MLP-PLR, FTT, and TabR-S. While the results look promising for some methods (MLP, TabR) and not so promising for others (MLP-PLR), at least without re-tuning their default parameters, our results also show that trying both preprocessing methods can already give considerable improvements on most benchmarks.

B.8 Results for Varying Architecture

Table B.5 shows the effects of including the preprocessing and architecture of RealMLP within other models. In particular, we study the benefits of our architectural changes, cf. Figure 1 (c),

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

050

0

.

052

0

.

054

0

.

056

0

.

058

0

.

060

0

.

062

Shifted geometric mean of

classification errors

Meta-train classification benchmark

better

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

22

0

.

24

0

.

26

0

.

28

0

.

30

Shifted geometric mean of

nRMSEs

Meta-train regression benchmark

better

10

-

1

10

0

10

1

Average training

time (CPU)

per 1K samples [s]

0

.

110

0

.

115

0

.

120

0

.

125

0

.

130

0

.

135

Shifted geometric mean of

classification errors

Meta-test classification benchmark

better

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

38

0

.

40

0

.

42

0

.

44

Shifted geometric mean of

nRMSEs

Meta-test regression benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

180

0

.

185

0

.

190

0

.

195

0

.

200

Shifted geometric mean of

classification errors

better

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

30

0

.

31

0

.

32

0

.

33

0

.

34

0

.

35

0

.

36

Shifted geometric mean of

nRMSEs

better

D = defaults

TD = tuned defaults

HPO = hyperparameter optimization

Best/Ensemble: out of XGB, LGBM, CatBoost, (Real)MLP

when applied directly to the setting of MLP-D. To this end, we approximately reproduce MLP-D in our codebase without weight decay (since the optimal value changes when including the NTP) and with marginally different early stopping thresholding logic. We also determine the best default learning rate on the meta-train benchmark, similar to Appendix A.4. Our reproduction achieves benchmark scores within 1% of the benchmark scores of the MLP-D (RS+SC) version. Adding the PL embeddings from Gorishniy et al. [16] with our default settings sometimes gives good results but is significantly worse on B test class , indicating that they need more tuning. In contrast, incorporating the RealMLP architectural changes (including their associated learning rate factors) improves scores on all benchmarks by around 5% or more, although they alone do not match the results of TabR-S-D. However, the non-architectural changes in RealMLP-TD make an even larger difference.

B.9 Comparing HPO Methods

In Figure B.6, we compare two different HPO methods for GBDTs:

Random search (HPO), as used in the main paper, with 50 steps.

Tree parzen estimator (HPO-TPE) as implemented in hyperopt [2], with 50 steps. The first 20 of these steps use random search.

While TPE often performs slightly better, the differences in benchmark scores are relatively small.

B.10 More Time-Error Plots

Here, we provide more time-vs-error plots. Figure B.7 shows results for the arithmetic mean error, Figure B.8 shows results for the arithmetic mean rank, and Figure B.9 shows results for the arithmetic mean normalized error. For the normalized error, the scores are affinely rescaled on each dataset split such that the worst score is 1 and the best score is 0 .

B.11 Critical Difference Diagrams

Figure B.10 analyzes the external validity of differences in average ranks between methods, i.e., whether they will generalize to new datasets from a distribution. While establishing external validity requires a large number of datasets, our meta-test benchmarks show at least the improvements of RealMLP-TD over MLP-D to be externally valid.

B.12 Win-rate Plots

For pairs of methods, we analyze the percentage of (dataset, split) combinations on which the first method has a lower error than the second method. We plot these win-rates in marix plots: Figure B.11 shows the results on B train class , Figure B.12 shows the results on B test class , Figure B.13 shows the results on B Grinsztajn class , Figure B.14 shows the results on B train reg , Figure B.15 shows the results on B test reg , and Figure B.16 shows the results on B Grinsztajn reg .

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

050

0

.

052

0

.

054

0

.

056

Shifted geometric mean of

classification errors

Meta-train classification benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

22

0

.

24

0

.

26

0

.

28

0

.

30

Shifted geometric mean of

nRMSEs

Meta-train regression benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

105

0

.

110

0

.

115

0

.

120

0

.

125

0

.

130

0

.

135

Shifted geometric mean of

classification errors

Meta-test classification benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

36

0

.

38

0

.

40

0

.

42

0

.

44

0

.

46

Shifted geometric mean of

nRMSEs

Meta-test regression benchmark

better

D = defaults

TD = tuned defaults

HPO = hyperparameter optimization

Best/Ensemble: out of XGB, LGBM, CatBoost, (Real)MLP

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

075

0

.

080

0

.

085

0

.

090

0

.

095

Arithmetic mean of

classification errors

Meta-train classification benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

32

0

.

34

0

.

36

0

.

38

0

.

40

Arithmetic mean of

nRMSEs

Meta-train regression benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

160

0

.

165

0

.

170

0

.

175

0

.

180

0

.

185

0

.

190

0

.

195

Arithmetic mean of

classification errors

Meta-test classification benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

48

0

.

50

0

.

52

0

.

54

0

.

56

0

.

58

Arithmetic mean of

nRMSEs

Meta-test regression benchmark

better

10

-

1

10

0

10

1

10

2

10

3

10

4

Average training

time (CPU)

per 1K samples [s]

0

.

200

0

.

205

0

.

210

0

.

215

0

.

220

Arithmetic mean of

classification errors

better

10

-

1

10

0

10

1

10

2

10

3

10

4

Average training

time (CPU)

per 1K samples [s]

0

.

44

0

.

45

0

.

46

0

.

47

0

.

48

0

.

49

Arithmetic mean of

nRMSEs

better

D = defaults

TD = tuned defaults

HPO = hyperparameter optimization

Best/Ensemble: out of XGB, LGBM, CatBoost, (Real)MLP

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

8

10

12

14

16

18

20

Arithmetic mean of

ranks

Meta-train classification benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

5

10

15

20

25

Arithmetic mean of

ranks

Meta-train regression benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

6

8

10

12

14

16

18

20

Arithmetic mean of

ranks

Meta-test classification benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

15

.

0

17

.

5

20

.

0

22

.

5

Arithmetic mean of

ranks

Meta-test regression benchmark

better

10

-

1

10

0

10

1

10

2

10

3

10

4

Average training

time (CPU)

per 1K samples [s]

5

10

15

20

25

Arithmetic mean of

ranks

better

10

-

1

10

0

10

1

10

2

10

3

10

4

Average training

time (CPU)

per 1K samples [s]

5

10

15

20

25

Arithmetic mean of

ranks

better

D = defaults

TD = tuned defaults

HPO = hyperparameter optimization

Best/Ensemble: out of XGB, LGBM, CatBoost, (Real)MLP

normalized classification errors

Arithmetic mean of

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Arithmetic mean of

normalized classification errors

Meta-train classification benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

Arithmetic mean of

normalized nRMSEs

Meta-train regression benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

0

.

45

0

.

50

0

.

55

Meta-test classification benchmark

better

10

-

1

10

0

10

1

10

2

Average training

time (CPU)

per 1K samples [s]

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

Arithmetic mean of

normalized nRMSEs

Meta-test regression benchmark

better

10

-

1

10

0

10

1

10

2

10

3

10

4

Average training

time (CPU)

per 1K samples [s]

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

Arithmetic mean of

normalized classification errors

better

10

-

1

10

0

10

1

10

2

10

3

10

4

Average training

time (CPU)

per 1K samples [s]

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Arithmetic mean of

normalized nRMSEs

better

D = defaults

TD = tuned defaults

HPO = hyperparameter optimization

Best/Ensemble: out of XGB, LGBM, CatBoost, (Real)MLP

1

2

3

4

5

6

7

8

23 22 21 20 19 18 17 16 15 14 13 12 11 10 9

MLP-D

ResNet-D

MLP-HPO

RF-D

FTT-D

ResNet-HPO

MLP-PLR-D

XGB-D

MLP-PLR-HPO

TabR-S-D

XGB-HPO

LGBM-D

RealMLP-TD-S

XGB-PBB-D

CatBoost-D

LGBM-TD

LGBM-HPO

RealTabR-D

XGB-TD

CatBoost-HPO

RealMLP-TD

CatBoost-TD

RealMLP-HPO

CD

meta-train-class

1

2

3

4

5

6

7

8

9

22 21 20 19 18 17 16 15 14 13 12 11 10

ResNet-D

MLP-D

XGB-D

LGBM-D

CatBoost-D

MLP-HPO

MLP-PLR-D

ResNet-HPO

RF-D

RealMLP-TD-S

FTT-D

XGB-TD

CatBoost-HPO

XGB-HPO

LGBM-TD

CatBoost-TD

MLP-PLR-HPO

TabR-S-D

RealMLP-TD

LGBM-HPO

RealMLP-HPO

RealTabR-D

CD

meta-train-reg

1

2

3

4

5

6

7

8

9

22 21 20 19 18 17 16 15 14 13 12 11 10

MLP-D

RF-D

ResNet-D

XGB-D

MLP-HPO

LGBM-D

MLP-PLR-D

RealMLP-TD-S

XGB-TD

TabR-S-D

MLP-PLR-HPO

ResNet-HPO

LGBM-TD

XGB-HPO

CatBoost-D

XGB-PBB-D

RealTabR-D

CatBoost-TD

RealMLP-TD

CatBoost-HPO

LGBM-HPO

RealMLP-HPO

CD

meta-test-class

1

2

3

4

5

6

7

8

9

21 20 19 18 17 16 15 14 13 12 11 10

XGB-D

RF-D

ResNet-D

MLP-D

XGB-TD

MLP-PLR-D

LGBM-D

RealMLP-TD-S

ResNet-HPO

LGBM-TD

MLP-HPO

CatBoost-D

TabR-S-D

CatBoost-TD

RealMLP-TD

XGB-HPO

RealTabR-D

MLP-PLR-HPO

CatBoost-HPO

LGBM-HPO

RealMLP-HPO

CD

meta-test-reg

1

2

3

4

5

6

7

8

2625242322212019181716151413121110 9

RF-D

XGB-D

MLP-D

RealMLP-TD-S

ResNet-D

MLP-HPO

XGB-TD

LGBM-D

RF-HPO

ResNet-HPO

TabR-S-D

RealMLP-TD

FTT-D

RealTabR-D

FTT-HPO

XGB-HPO

LGBM-TD

MLP-PLR-D

MLP-PLR-HPO

XGB-PBB-D

RealMLP-HPO

LGBM-HPO

CatBoost-TD

CatBoost-HPO

CatBoost-D

TabR-HPO

CD

grinsztajn-class

1

2

3

4

5

6

7

8

25242322212019181716151413121110 9

ResNet-D

XGB-D

MLP-D

RF-D

ResNet-HPO

XGB-TD

MLP-HPO

MLP-PLR-D

RealMLP-TD-S

FTT-D

LGBM-D

RF-HPO

LGBM-TD

TabR-S-D

FTT-HPO

CatBoost-TD

CatBoost-D

MLP-PLR-HPO

XGB-HPO

CatBoost-HPO

RealTabR-D

RealMLP-HPO

TabR-HPO

LGBM-HPO

CD

grinsztajn-reg

RealMLP-TD

RF-D

MLP-D

ResNet-D

MLP-HPO

XGB-D

FTT-D

MLP-PLR-D

LGBM-D

XGB-HPO

ResNet-HPO

XGB-PBB-D

MLP-PLR-HPO

CatBoost-D

RealMLP-TD-S

TabR-S-D

Best-D

LGBM-HPO

LGBM-TD

XGB-TD

CatBoost-TD

CatBoost-HPO

RealTabR-D

Ensemble-D

RealMLP-TD

RealMLP-HPO

Best-HPO

Best-TD

Ensemble-TD

Ensemble-HPO

RF-D

MLP-D

ResNet-D

MLP-HPO

XGB-D

FTT-D

MLP-PLR-D

LGBM-D

XGB-HPO

ResNet-HPO

XGB-PBB-D

MLP-PLR-HPO

CatBoost-D

RealMLP-TD-S

TabR-S-D

Best-D

LGBM-HPO

LGBM-TD

XGB-TD

CatBoost-TD

CatBoost-HPO

RealTabR-D

Ensemble-D

RealMLP-TD

RealMLP-HPO

Best-HPO

Best-TD

Ensemble-TD

Ensemble-HPO

45.3 45.6 44.4 35.7 39.1 40.0 34.2 29.3 39.7 24.9 35.6 29.6 33.0 37.4 25.0 24.6 21.6 19.2 20.7 20.4 28.6 21.6 27.4 24.1 19.6 17.1 15.4 16.8

54.7

53.0 45.7 43.6 43.3 39.9 42.2 42.1 35.8 38.1 32.9 36.6 31.2 32.1 26.9 29.5 28.5 28.9 29.3 26.0 26.6 24.5 22.7 21.4 19.1 18.5 16.8 17.1

54.4 47.0

46.9 46.0 43.9 38.2 44.1 45.0 30.7 41.3 33.3 39.4 30.5 29.6 31.2 34.3 35.2 32.1 33.6 29.5 23.3 28.4 21.8 22.4 22.0 20.7 19.0 18.8

55.6 54.3 53.1

45.6 42.1 40.9 43.1 44.2 36.9 39.9 32.9 39.3 32.1 31.8 28.8 31.7 30.5 31.3 29.6 29.0 26.8 26.8 22.1 23.3 21.1 18.9 18.2 19.5

64.3 56.4 54.0 54.4

50.1 49.1 43.4 39.4 50.8 36.5 47.4 42.5 43.7 42.8 32.0 28.3 29.8 27.2 25.6 25.3 35.8 27.2 34.2 31.1 24.4 21.1 20.3 21.8

60.9 56.7 56.1 57.9 49.9

50.4 48.4 46.4 46.7 44.4 39.9 46.3 41.7 42.5 36.1 38.1 35.6 35.8 36.2 33.9 31.7 31.1 30.5 25.8 23.5 24.1 22.4 22.2

60.0 60.1 61.8 59.1 50.9 49.6

47.3 48.7 50.9 44.1 39.4 48.4 42.1 41.2 35.9 37.0 35.2 34.9 36.2 36.7 32.3 28.2 29.1 26.7 25.1 23.6 23.0 22.6

65.8 57.8 55.9 56.9 56.6 51.6 52.7

46.1 49.6 46.4 48.7 42.4 46.2 46.7 34.2 30.9 30.5 32.0 27.2 28.6 38.6 29.3 34.5 34.7 25.9 21.8 21.5 22.4

70.7 57.9 55.0 55.8 60.6 53.6 51.3 53.9

52.8 49.5 49.9 46.5 44.4 43.9 35.9 32.2 33.8 33.3 30.4 30.4 34.5 29.6 34.8 33.3 26.3 21.2 18.3 20.5

60.3 64.2 69.3 63.1 49.2 53.3 49.1 50.4 47.2

44.9 43.2 44.9 41.0 40.5 38.0 39.6 38.7 38.4 38.2 36.2 30.5 34.7 27.9 27.7 26.6 24.7 23.1 24.0

75.1 61.9 58.7 60.1 63.5 55.6 55.9 53.6 50.5 55.1

51.0 49.9 47.0 48.5 40.9 35.3 36.7 38.0 34.3 32.9 39.0 34.3 37.3 37.4 30.7 25.9 25.1 26.2

64.4 67.1 66.7 67.1 52.6 60.1 60.6 51.3 50.1 56.8 49.0

49.2 48.8 48.3 42.1 39.5 35.6 38.4 38.3 36.4 39.4 32.7 35.1 33.0 27.6 26.2 24.4 24.6

70.4 63.4 60.6 60.7 57.5 53.7 51.6 57.6 53.5 55.1 50.1 50.8

45.6 46.5 39.1 43.8 44.4 44.2 39.0 40.6 39.7 36.4 37.0 38.9 34.1 28.1 28.0 29.7

67.0 68.8 69.5 67.9 56.3 58.3 57.9 53.8 55.6 59.0 53.0 51.2 54.4

49.3 44.8 46.7 45.7 46.3 44.2 45.6 37.0 42.9 32.7 31.5 29.2 28.7 27.9 26.3

62.6 67.9 70.4 68.2 57.2 57.5 58.8 53.3 56.1 59.5 51.5 51.7 53.5 50.7

44.8 46.1 42.3 43.4 44.0 42.3 35.7 42.5 39.5 36.4 31.8 30.7 28.5 28.6

75.0 73.1 68.8 71.2 68.0 63.9 64.1 65.8 64.1 62.0 59.1 57.9 60.9 55.2 55.2

49.3 48.1 48.2 43.8 47.4 46.0 40.4 44.2 42.4 36.0 31.6 31.0 30.8

75.4 70.5 65.7 68.3 71.7 61.9 63.0 69.1 67.8 60.4 64.7 60.5 56.2 53.3 53.9 50.7

51.1 51.7 49.5 48.3 46.1 44.5 43.5 43.3 34.7 33.3 28.7 26.7

78.4 71.5 64.8 69.5 70.2 64.4 64.8 69.5 66.2 61.3 63.3 64.4 55.6 54.3 57.7 51.9 48.9

50.3 46.8 49.3 47.9 46.2 45.0 41.8 34.1 33.3 29.6 28.9

80.8 71.1 67.9 68.7 72.8 64.2 65.1 68.0 66.7 61.6 62.0 61.6 55.8 53.7 56.6 51.8 48.3 49.7

48.9 47.7 46.8 46.2 45.2 43.9 36.4 32.8 32.1 30.2

79.3 70.7 66.4 70.4 74.4 63.8 63.8 72.8 69.6 61.8 65.7 61.7 61.0 55.8 56.0 56.2 50.5 53.2 51.1

48.7 46.7 50.6 42.9 44.6 37.9 33.2 32.6 33.1

79.6 74.0 70.5 71.0 74.7 66.1 63.3 71.4 69.6 63.8 67.1 63.6 59.4 54.4 57.7 52.6 51.7 50.7 52.3 51.3

45.1 48.1 43.3 42.4 36.1 33.1 30.7 30.9

71.4 73.4 76.7 73.2 64.2 68.3 67.7 61.4 65.5 69.5 61.0 60.6 60.3 63.0 64.3 54.0 53.9 52.1 53.2 53.3 54.9

51.5 48.3 47.7 41.0 39.7 37.8 37.8

78.4 75.5 71.6 73.2 72.8 68.9 71.8 70.7 70.4 65.3 65.7 67.3 63.6 57.1 57.5 59.6 55.5 53.8 53.8 49.4 51.9 48.5

45.0 43.9 38.4 33.1 32.9 33.8

72.6 77.3 78.2 77.9 65.8 69.5 70.9 65.5 65.2 72.1 62.7 64.9 63.0 67.3 60.5 55.8 56.5 55.0 54.8 57.1 56.7 51.7 55.0

43.2 38.3 40.6 37.7 34.8

75.9 78.6 77.6 76.7 68.9 74.2 73.3 65.3 66.7 72.3 62.6 67.0 61.1 68.5 63.6 57.6 56.7 58.2 56.1 55.4 57.6 52.3 56.1 56.8

44.1 44.1 43.6 36.7

80.4 80.9 78.0 78.9 75.6 76.5 74.9 74.1 73.7 73.4 69.3 72.4 65.9 70.8 68.2 64.0 65.3 65.9 63.6 62.1 63.9 59.0 61.6 61.7 55.9

51.6 48.3 40.8

82.9 81.5 79.3 81.1 78.9 75.9 76.4 78.2 78.8 75.3 74.1 73.8 71.9 71.3 69.3 68.4 66.7 66.7 67.2 66.8 66.9 60.3 66.9 59.4 55.9 48.4

46.1 44.9

84.6 83.2 81.0 81.8 79.7 77.6 77.0 78.5 81.7 76.9 74.9 75.6 72.0 72.1 71.5 69.0 71.3 70.4 67.9 67.4 69.3 62.2 67.1 62.3 56.4 51.7 53.9

47.5

83.2 82.9 81.2 80.5 78.2 77.8 77.4 77.6 79.5 76.0 73.8 75.4 70.3 73.7 71.4 69.2 73.3 71.1 69.8 66.9 69.1 62.2 66.2 65.2 63.3 59.2 55.1 52.5

Meta-train classification benchmark , percentage of row wins

0

20

40

60

80

100

RF-D

MLP-D

XGB-D

ResNet-D

MLP-HPO

MLP-PLR-D

LGBM-D

RealMLP-TD-S

XGB-TD

MLP-PLR-HPO

TabR-S-D

ResNet-HPO

LGBM-TD

CatBoost-D

XGB-HPO

XGB-PBB-D

Best-D

CatBoost-TD

RealMLP-TD

RealTabR-D

CatBoost-HPO

Ensemble-D

LGBM-HPO

RealMLP-HPO

Best-TD

Best-HPO

Ensemble-TD

Ensemble-HPO

RF-D

MLP-D

XGB-D

ResNet-D

MLP-HPO

MLP-PLR-D

LGBM-D

RealMLP-TD-S

XGB-TD

MLP-PLR-HPO

TabR-S-D

ResNet-HPO

LGBM-TD

CatBoost-D

XGB-HPO

XGB-PBB-D

Best-D

CatBoost-TD

RealMLP-TD

RealTabR-D

CatBoost-HPO

Ensemble-D

LGBM-HPO

RealMLP-HPO

Best-TD

Best-HPO

Ensemble-TD

Ensemble-HPO

47.0

41.5

46.4

44.9

42.6

34.5

37.7

35.1

36.4

45.2

39.6

29.4

27.3

22.8

26.3

25.7

22.8

30.7

33.2

20.6

23.5

22.2

28.5

20.2

20.9

18.1

17.9

53.0

47.5

45.6

38.1

40.7

45.9

37.6

41.0

33.6

36.0

30.1

35.6

35.6

33.9

34.5

26.7

33.1

26.0

21.4

24.9

24.7

28.2

24.9

17.6

18.4

18.9

16.0

58.5

52.5

52.1

49.0

46.2

52.5

41.5

39.8

40.3

44.5

44.6

30.9

34.5

27.0

26.7

22.9

23.5

29.1

32.9

23.1

23.3

21.0

28.0

19.0

20.8

17.7

18.1

53.6

54.4

47.9

44.9

45.1

46.4

40.4

41.6

39.8

38.1

27.7

35.2

35.7

36.6

36.9

28.6

31.9

30.4

23.3

26.7

28.6

27.9

25.4

20.2

20.4

21.5

16.0

55.1

61.9

51.0

55.1

50.0

46.4

42.3

45.3

37.9

40.5

34.5

38.3

36.4

37.6

36.3

32.4

34.4

28.8

22.0

28.9

30.0

30.4

25.4

20.0

19.3

19.4

17.0

57.4

59.3

53.8

54.9

50.0

50.5

43.4

44.7

33.9

45.6

42.2

38.3

36.6

30.4

32.8

30.3

32.1

29.7

31.8

29.8

23.5

26.3

27.6

18.1

20.1

21.6

17.5

65.5

54.1

47.5

53.6

53.6

49.5

49.4

42.4

44.6

49.8

46.7

34.3

34.5

30.4

33.0

26.5

24.6

34.7

37.1

25.5

24.2

24.5

29.7

21.1

20.7

21.5

18.0

62.3

62.4

58.5

59.6

57.7

56.6

50.6

52.7

46.0

46.1

46.6

45.8

43.4

42.7

37.5

38.4

39.2

27.3

32.9

38.4

34.7

32.7

25.1

21.3

20.2

23.2

18.0

64.9

59.0

60.2

58.4

54.7

55.3

57.6

47.3

49.0

48.3

50.0

34.9

43.3

41.8

37.8

36.7

36.7

35.8

39.7

34.7

33.4

29.2

31.9

22.3

24.2

21.0

19.4

63.6

66.4

59.7

60.2

62.1

66.1

55.4

54.0

51.0

53.8

52.5

45.6

43.5

41.6

40.7

40.8

39.3

33.5

37.4

34.5

33.6

32.3

30.1

21.0

20.1

21.6

16.5

54.8

64.0

55.5

61.9

59.5

54.4

50.2

53.9

51.7

46.3

48.4

45.1

44.9

44.6

45.1

38.6

40.6

36.1

25.2

40.0

36.8

38.1

33.4

29.0

27.4

29.6

23.9

60.4

69.9

55.4

72.3

65.5

57.8

53.3

53.4

50.0

47.5

51.6

46.0

42.3

43.8

45.0

38.0

36.8

38.2

34.6

37.0

35.7

34.8

31.5

26.0

24.0

25.9

20.4

70.6

64.4

69.1

64.8

61.7

61.7

65.7

54.2

65.1

54.4

54.9

54.0

48.3

46.7

45.4

43.5

46.9

42.6

43.6

43.6

37.9

34.5

39.3

27.5

27.1

27.0

21.8

72.7

64.4

65.5

64.3

63.6

63.4

65.5

56.6

56.7

56.5

55.1

57.7

51.7

46.4

43.5

42.1

37.0

45.3

46.5

36.1

40.4

40.1

42.5

33.6

31.5

30.9

29.2

77.2

66.1

73.0

63.4

62.4

69.6

69.6

57.3

58.2

58.4

55.4

56.3

53.3

53.6

50.5

50.8

47.4

44.8

46.4

42.4

40.8

32.8

41.1

28.1

26.8

27.6

21.5

73.7

65.5

73.3

63.1

63.8

67.2

67.0

62.5

62.2

59.3

54.9

55.0

54.6

56.5

49.5

53.9

50.3

39.7

45.4

47.9

46.6

38.2

36.4

29.9

27.3

30.6

24.3

74.3

73.3

77.1

71.4

67.6

69.7

73.5

61.6

63.3

59.2

61.4

62.0

56.5

57.9

49.2

46.1

45.6

44.8

49.6

44.4

40.7

42.4

43.0

31.8

31.0

33.1

27.0

77.2

66.9

76.5

68.1

65.6

67.9

75.4

60.8

63.3

60.7

59.4

63.2

53.1

63.0

52.6

49.7

54.4

48.8

49.5

48.3

48.8

45.6

46.5

35.9

35.5

32.9

29.1

69.3

74.0

70.9

69.6

71.3

70.3

65.3

72.7

64.2

66.5

63.9

61.8

57.4

54.7

55.2

60.3

55.2

51.2

50.8

51.3

47.8

51.1

34.7

36.3

29.6

29.6

22.6

66.8

78.6

67.1

76.7

78.0

68.2

62.9

67.1

60.3

62.6

74.8

65.4

56.4

53.5

53.6

54.6

50.4

50.5

49.2

47.9

45.8

47.4

42.5

37.5

33.5

35.6

29.1

79.4

75.1

76.9

73.3

71.1

70.2

74.5

61.6

65.3

65.5

60.0

63.0

56.4

63.9

57.6

52.1

55.6

51.7

48.8

52.1

51.7

44.9

45.0

33.1

32.1

31.8

25.4

76.5

75.3

76.7

71.4

70.0

76.5

75.8

65.3

66.6

66.4

63.2

64.3

62.1

59.6

59.2

53.4

59.3

51.3

52.2

54.2

48.3

48.3

49.8

33.9

33.6

29.1

26.7

77.8

71.8

79.0

72.1

69.6

73.8

75.5

67.3

70.8

67.7

61.9

65.2

65.5

59.9

67.2

61.8

57.6

54.4

48.9

52.6

55.1

51.7

47.4

32.2

33.1

31.4

23.2

71.5

75.1

72.0

74.6

74.6

72.4

70.3

74.9

68.1

69.9

66.6

68.5

60.7

57.5

58.9

63.6

57.0

53.5

65.3

57.5

55.0

50.2

52.6

43.8

38.0

35.8

22.8

79.8

82.4

81.0

79.8

80.0

81.9

78.9

78.7

77.7

79.0

71.0

74.0

72.5

66.4

71.9

70.1

68.2

64.1

63.8

62.5

66.9

66.1

67.8

56.2

43.9

40.9

31.3

79.1

81.6

79.2

79.6

80.7

79.9

79.3

79.8

75.8

79.9

72.6

76.0

72.9

68.5

73.2

72.7

69.0

64.5

70.4

66.5

67.9

66.4

66.9

62.0

56.1

48.3

30.3

81.9

81.1

82.3

78.5

80.6

78.4

78.5

76.8

79.0

78.4

70.4

74.1

73.0

69.1

72.4

69.4

66.9

67.1

70.4

64.4

68.2

70.9

68.6

64.2

59.1

51.7

37.1

82.1

84.0

81.9

84.0

83.0

82.5

82.0

82.0

80.6

83.5

76.1

79.6

78.2

70.8

78.5

75.7

73.0

70.9

77.4

70.9

74.6

73.3

76.8

77.2

68.8

69.7

62.9

Meta-test classification benchmark , percentage of row wins

0

20

40

60

80

100

RF-D

XGB-D

MLP-D

ResNet-D

MLP-HPO

RealMLP-TD-S

LGBM-D

XGB-TD

RF-HPO

ResNet-HPO

FTT-D

MLP-PLR-D

MLP-PLR-HPO

TabR-S-D

LGBM-TD

FTT-HPO

RealMLP-TD

XGB-HPO

RealTabR-D

RealMLP-HPO

XGB-PBB-D

LGBM-HPO

CatBoost-TD

Best-TD

CatBoost-D

Best-D

CatBoost-HPO

TabR-HPO

Best-HPO

Ensemble-D

Ensemble-TD

Ensemble-HPO

RF-D

XGB-D

MLP-D

ResNet-D

MLP-HPO

RealMLP-TD-S

LGBM-D

XGB-TD

RF-HPO

ResNet-HPO

FTT-D

MLP-PLR-D

MLP-PLR-HPO

TabR-S-D

LGBM-TD

FTT-HPO

RealMLP-TD

XGB-HPO

RealTabR-D

RealMLP-HPO

XGB-PBB-D

LGBM-HPO

CatBoost-TD

Best-TD

CatBoost-D

Best-D

CatBoost-HPO

TabR-HPO

Best-HPO

Ensemble-D

Ensemble-TD

Ensemble-HPO

41.7 44.2 39.4 41.4 30.3 19.7 16.9 16.9 28.9 24.4 25.6 24.2 26.1 10.0 19.4 21.1 12.2 13.1 21.7 8.3 10.0 8.6

7.5 16.9 6.9

7.5 12.2 9.4

3.1

5.0

5.6

58.3

47.2 42.5 44.7 35.8 30.3 17.8 33.9 37.5 29.4 29.2 29.4 32.5 18.3 29.7 28.6 13.9 23.6 27.2 13.1 13.6

8.6 11.1 17.2 14.2 13.1 16.4 10.6

8.3

8.1

5.8

55.8 52.8

42.2 37.8 38.3 36.1 42.5 35.6 32.8 28.9 28.3 30.6 38.9 31.4 27.5 31.9 30.6 32.2 31.7 25.8 20.8 26.9 22.2 25.6 19.2 18.9 23.3 22.2 11.7 11.4 15.0

60.6 57.5 57.8

49.4 45.3 39.2 48.1 36.7 31.9 28.3 34.2 31.4 41.1 35.0 28.1 35.0 30.8 32.8 30.6 27.2 25.0 26.7 31.7 27.2 26.1 23.6 23.1 20.8 16.7 16.9 12.8

58.6 55.3 62.2 50.6

49.2 43.9 47.2 36.4 35.6 31.1 36.4 31.4 46.1 38.3 30.6 37.5 40.0 37.8 32.5 35.3 28.6 29.2 30.0 27.8 28.6 24.2 27.8 23.6 18.3 14.7 18.1

69.7 64.2 61.7 54.7 50.8

38.6 48.3 44.2 41.9 36.9 36.7 38.1 38.3 38.1 31.7 31.7 39.7 28.6 26.9 36.9 33.1 33.1 25.8 26.7 27.2 28.3 24.4 21.4 18.6 14.2 13.3

80.3 69.7 63.9 60.8 56.1 61.4

50.6 54.7 58.9 42.8 41.4 39.4 44.4 33.9 44.4 40.8 31.4 36.1 43.6 30.6 27.2 21.1 25.3 21.9 20.6 22.5 30.0 23.9 12.8 16.1 16.1

83.1 82.2 57.5 51.9 52.8 51.7 49.4

50.8 50.0 47.5 43.3 39.2 43.3 35.8 42.2 36.7 36.7 34.2 35.8 28.9 32.8 30.0 28.6 26.7 29.7 25.8 22.8 23.9 20.0 13.9 17.2

83.1 66.1 64.4 63.3 63.6 55.8 45.3 49.2

60.6 43.1 41.9 43.9 47.8 32.8 40.8 44.2 31.7 39.7 43.9 29.2 30.0 24.2 26.9 27.8 22.5 24.4 29.7 26.7 18.3 17.8 19.7

71.1 62.5 67.2 68.1 64.4 58.1 41.1 50.0 39.4

34.7 39.4 41.4 51.1 42.2 36.7 46.1 40.0 41.9 38.9 39.4 35.8 30.6 34.7 32.5 30.3 32.5 26.9 28.9 19.7 22.2 16.4

75.6 70.6 71.1 71.7 68.9 63.1 57.2 52.5 56.9 65.3

48.1 48.3 53.9 46.1 37.8 49.2 44.2 46.9 39.4 44.7 43.3 35.8 35.0 31.7 31.7 35.6 29.4 28.3 23.3 21.7 20.8

74.4 70.8 71.7 65.8 63.6 63.3 58.6 56.7 58.1 60.6 51.9

50.6 51.1 46.9 50.3 47.8 45.8 51.1 40.3 40.8 37.2 38.3 36.7 30.6 31.1 38.9 34.2 31.7 15.3 20.8 21.4

75.8 70.6 69.4 68.6 68.6 61.9 60.6 60.8 56.1 58.6 51.7 49.4

54.7 47.5 46.1 50.0 46.7 52.5 41.7 45.0 38.1 38.1 38.1 35.6 35.6 36.1 35.8 27.5 20.6 19.7 19.4

73.9 67.5 61.1 58.9 53.9 61.7 55.6 56.7 52.2 48.9 46.1 48.9 45.3

48.9 45.6 51.9 48.6 43.1 53.6 51.7 44.2 46.9 45.6 39.2 41.1 39.4 28.3 40.3 33.9 30.6 29.7

90.0 81.7 68.6 65.0 61.7 61.9 66.1 64.2 67.2 57.8 53.9 53.1 52.5 51.1

53.9 44.2 49.2 48.6 45.0 40.8 37.8 39.7 35.6 36.1 36.1 35.0 31.7 27.8 23.6 17.5 16.1

80.6 70.3 72.5 71.9 69.4 68.3 55.6 57.8 59.2 63.3 62.2 49.7 53.9 54.4 46.1

50.6 45.3 47.8 47.2 45.0 43.1 39.7 38.9 31.9 35.3 37.2 30.0 32.5 26.7 19.4 17.8

78.9 71.4 68.1 65.0 62.5 68.3 59.2 63.3 55.8 53.9 50.8 52.2 50.0 48.1 55.8 49.4

49.2 48.9 39.7 48.6 47.2 46.9 42.5 35.6 37.5 43.1 37.2 34.2 26.7 15.6 21.7

87.8 86.1 69.4 69.2 60.0 60.3 68.6 63.3 68.3 60.0 55.8 54.2 53.3 51.4 50.8 54.7 50.8

49.7 47.5 45.6 38.3 39.4 35.8 37.8 37.2 33.1 37.2 30.8 21.9 20.3 20.3

86.9 76.4 67.8 67.2 62.2 71.4 63.9 65.8 60.3 58.1 53.1 48.9 47.5 56.9 51.4 52.2 51.1 50.3

50.8 45.0 48.1 41.7 42.8 38.6 37.2 39.4 32.8 40.8 28.9 25.6 27.2

78.3 72.8 68.3 69.4 67.5 73.1 56.4 64.2 56.1 61.1 60.6 59.7 58.3 46.4 55.0 52.8 60.3 52.5 49.2

52.8 50.6 48.9 46.4 43.1 41.4 41.9 39.4 36.7 28.9 21.7 13.6

91.7 86.9 74.2 72.8 64.7 63.1 69.4 71.1 70.8 60.6 55.3 59.2 55.0 48.3 59.2 55.0 51.4 54.4 55.0 47.2

46.4 42.2 39.4 43.9 44.4 44.4 33.1 30.8 28.3 19.7 20.0

90.0 86.4 79.2 75.0 71.4 66.9 72.8 67.2 70.0 64.2 56.7 62.8 61.9 55.8 62.2 56.9 52.8 61.7 51.9 49.4 53.6

49.2 43.6 47.5 51.4 50.6 35.8 35.3 26.9 24.4 24.7

91.4 91.4 73.1 73.3 70.8 66.9 78.9 70.0 75.8 69.4 64.2 61.7 61.9 53.1 60.3 60.3 53.1 60.6 58.3 51.1 57.8 50.8

45.0 46.7 50.6 43.9 36.7 36.7 30.3 25.0 24.7

92.5 88.9 77.8 68.3 70.0 74.2 74.7 71.4 73.1 65.3 65.0 63.3 61.9 54.4 64.4 61.1 57.5 64.2 57.2 53.6 60.6 56.4 55.0

46.4 49.2 49.2 40.8 43.3 36.7 18.6 27.8

83.1 82.8 74.4 72.8 72.2 73.3 78.1 73.3 72.2 67.5 68.3 69.4 64.4 60.8 63.9 68.1 64.4 62.2 61.4 56.9 56.1 52.5 53.3 53.6

49.4 47.8 45.0 42.5 26.1 32.8 28.1

93.1 85.8 80.8 73.9 71.4 72.8 79.4 70.3 77.5 69.7 68.3 68.9 64.4 58.9 63.9 64.7 62.5 62.8 62.8 58.6 55.6 48.6 49.4 50.8 50.6

50.3 44.2 42.5 22.5 30.3 27.8

92.5 86.9 81.1 76.4 75.8 71.7 77.5 74.2 75.6 67.5 64.4 61.1 63.9 60.6 65.0 62.8 56.9 66.9 60.6 58.1 55.6 49.4 56.1 50.8 52.2 49.7

40.8 44.2 31.7 31.4 27.2

87.8 83.6 76.7 76.9 72.2 75.6 70.0 77.2 70.3 73.1 70.6 65.8 64.2 71.7 68.3 70.0 62.8 62.8 67.2 60.6 66.9 64.2 63.3 59.2 55.0 55.8 59.2

53.3 41.4 42.5 37.8

90.6 89.4 77.8 79.2 76.4 78.6 76.1 76.1 73.3 71.1 71.7 68.3 72.5 59.7 72.2 67.5 65.8 69.2 59.2 63.3 69.2 64.7 63.3 56.7 57.5 57.5 55.8 46.7

42.5 34.7 23.3

96.9 91.7 88.3 83.3 81.7 81.4 87.2 80.0 81.7 80.3 76.7 84.7 79.4 66.1 76.4 73.3 73.3 78.1 71.1 71.1 71.7 73.1 69.7 63.3 73.9 77.5 68.3 58.6 57.5

43.3 39.2

95.0 91.9 88.6 83.1 85.3 85.8 83.9 86.1 82.2 77.8 78.3 79.2 80.3 69.4 82.5 80.6 84.4 79.7 74.4 78.3 80.3 75.6 75.0 81.4 67.2 69.7 68.6 57.5 65.3 56.7

38.1

94.4 94.2 85.0 87.2 81.9 86.7 83.9 82.8 80.3 83.6 79.2 78.6 80.6 70.3 83.9 82.2 78.3 79.7 72.8 86.4 80.0 75.3 75.3 72.2 71.9 72.2 72.8 62.2 76.7 60.8 61.9

Grinsztajn et al. (2022) classification benchmark , percentage of row wins

0

20

40

60

80

100

ResNet-D

XGB-D

LGBM-D

MLP-D

CatBoost-D

RF-D

MLP-PLR-D

MLP-HPO

ResNet-HPO

FTT-D

RealMLP-TD-S

XGB-TD

CatBoost-HPO

Best-D

CatBoost-TD

LGBM-TD

XGB-HPO

Ensemble-D

RealMLP-TD

MLP-PLR-HPO

TabR-S-D

LGBM-HPO

RealMLP-HPO

RealTabR-D

Best-TD

Ensemble-TD

Best-HPO

Ensemble-HPO

ResNet-D

XGB-D

LGBM-D

MLP-D

CatBoost-D

RF-D

MLP-PLR-D

MLP-HPO

ResNet-HPO

FTT-D

RealMLP-TD-S

XGB-TD

CatBoost-HPO

Best-D

CatBoost-TD

LGBM-TD

XGB-HPO

Ensemble-D

RealMLP-TD

MLP-PLR-HPO

TabR-S-D

LGBM-HPO

RealMLP-HPO

RealTabR-D

Best-TD

Ensemble-TD

Best-HPO

Ensemble-HPO

39.9

44.2

30.1

43.8

35.9

20.6

17.8

9.8

21.2

14.5

26.7

25.1

13.9

23.4

23.8

23.2

9.7

12.0

10.7

12.2

19.6

6.3

7.1

4.2

5.6

3.8

3.1

60.1

52.6

48.8

57.0

41.7

44.1

44.9

46.4

41.1

38.5

15.6

8.5

19.1

8.3

11.1

10.8

10.5

31.3

24.1

25.3

6.0

24.3

18.7

5.9

3.7

5.2

3.5

55.8

47.4

44.1

58.5

43.6

39.1

43.3

46.3

38.6

37.4

28.7

16.2

12.5

17.6

18.9

12.2

9.6

30.1

23.5

23.7

7.5

23.8

17.0

11.1

10.2

6.8

4.3

69.9

51.2

55.9

52.4

45.1

34.8

27.3

29.9

28.7

28.4

32.8

30.3

24.9

28.6

29.6

30.5

22.3

19.3

16.5

22.1

21.7

8.0

14.3

8.6

6.5

5.2

4.0

56.2

43.0

41.5

47.6

41.0

41.6

46.1

47.8

40.2

40.2

34.3

11.9

17.3

24.5

29.0

22.3

16.2

30.3

25.8

25.1

17.0

26.8

18.6

14.1

14.2

9.4

7.5

64.1

58.3

56.4

54.9

59.0

48.5

49.1

47.3

42.9

40.0

19.4

34.9

32.3

17.9

12.3

20.5

22.6

34.9

28.9

28.2

9.5

27.4

18.7

9.5

6.9

4.9

3.8

79.4

55.9

60.9

65.2

58.4

51.5

51.3

53.9

32.8

44.9

39.3

40.5

30.1

35.2

35.9

36.2

17.4

24.6

14.7

32.8

27.9

10.1

19.6

12.2

12.0

8.0

6.3

82.2

55.1

56.7

72.7

53.9

50.9

48.7

49.2

43.1

35.5

38.9

38.9

30.9

38.6

35.0

35.5

29.3

24.7

24.9

33.1

27.4

10.5

17.6

12.1

11.4

6.3

5.1

90.2

53.6

53.7

70.1

52.2

52.7

46.1

50.8

37.5

38.8

40.7

40.5

30.9

38.6

37.1

35.4

30.3

23.8

19.4

29.7

27.5

13.6

15.3

16.1

13.0

9.0

6.2

78.8

58.9

61.4

71.3

59.8

57.1

67.2

56.9

62.5

51.9

49.9

46.4

43.5

45.8

46.4

46.6

39.1

26.0

26.3

38.3

39.3

11.6

20.4

11.5

14.4

7.4

4.7

85.5

61.5

62.6

71.6

59.8

60.0

55.1

64.5

61.2

48.1

43.8

43.7

41.5

41.3

37.4

40.6

38.7

27.8

34.1

44.4

29.8

13.8

29.8

16.5

17.2

9.8

7.9

73.3

84.4

71.3

67.2

65.7

80.6

60.7

61.1

59.3

50.1

56.2

48.2

47.9

45.0

32.2

40.1

43.6

37.9

41.2

35.1

27.0

32.1

24.7

14.0

3.6

10.3

4.5

74.9

91.5

83.8

69.7

88.1

65.1

59.5

61.1

59.5

53.6

56.3

51.8

54.0

34.9

42.6

37.8

47.8

46.8

42.4

38.0

29.0

39.8

32.9

20.9

17.8

11.7

5.8

86.1

80.9

87.5

75.1

82.7

67.7

69.9

69.1

69.1

56.5

58.5

52.1

46.0

43.6

42.3

42.7

27.3

44.1

33.0

45.7

34.3

32.0

27.9

19.1

16.1

10.7

8.4

76.6

91.7

82.4

71.4

75.5

82.1

64.8

61.4

61.4

54.2

58.7

55.0

65.1

56.4

46.5

50.0

50.5

47.7

46.8

38.8

40.4

38.2

29.0

22.5

9.6

14.8

6.7

76.2

88.9

81.1

70.4

71.0

87.7

64.1

65.0

62.9

53.6

62.6

67.8

57.4

57.7

53.5

53.0

54.9

45.0

45.0

37.2

35.6

38.1

28.9

20.6

10.8

14.0

5.7

76.8

89.2

87.8

69.5

77.7

79.5

63.8

64.5

64.6

53.4

59.4

59.9

62.2

57.3

50.0

47.0

54.2

46.2

43.1

40.1

33.3

40.6

29.9

29.2

25.0

20.1

8.9

90.3

89.5

90.4

77.7

83.8

77.4

82.6

70.7

69.7

60.9

61.3

56.4

52.2

72.7

49.5

45.1

45.8

45.9

40.7

43.4

38.5

37.6

28.5

16.7

15.4

9.6

7.1

88.0

68.7

69.9

80.7

69.7

65.1

75.4

75.3

76.2

74.0

72.2

62.1

53.2

55.9

52.3

55.0

53.8

54.1

56.0

53.7

48.5

28.3

41.3

29.3

26.3

21.2

17.6

89.3

75.9

76.5

83.5

74.2

71.1

85.3

75.1

80.6

73.7

65.9

58.8

57.6

67.0

53.2

55.0

56.9

59.3

44.0

45.9

49.4

28.8

37.7

18.6

21.4

14.0

10.2

87.8

74.7

76.2

77.9

74.9

71.8

67.2

66.9

70.3

61.7

55.6

64.9

62.0

54.3

61.2

62.8

59.9

56.6

46.3

54.1

56.3

42.8

19.0

37.1

37.3

34.8

31.5

80.4

94.0

92.5

78.3

83.0

90.5

72.1

72.6

72.5

60.7

70.2

73.0

71.0

65.7

59.6

64.4

66.7

61.5

51.5

50.6

43.7

45.7

37.0

32.2

25.1

22.0

8.0

93.8

75.7

76.2

92.0

73.2

72.6

89.9

89.5

86.4

88.4

86.2

67.9

60.2

68.0

61.8

61.9

59.4

62.4

71.7

71.2

57.2

54.3

50.8

44.3

37.9

32.5

15.7

92.9

81.3

83.0

85.7

81.4

81.3

80.4

82.4

84.7

79.6

70.2

75.3

67.1

72.1

71.0

71.1

70.1

71.5

58.7

62.3

81.0

63.0

49.2

51.0

52.8

42.3

39.0

95.8

94.1

88.9

91.4

85.9

90.5

87.8

87.9

83.9

88.5

83.5

86.0

79.1

80.9

77.5

79.4

70.8

83.3

70.7

81.4

62.9

67.8

55.7

49.0

32.4

35.2

23.9

94.4

96.3

89.8

93.5

85.8

93.1

88.0

88.6

87.0

85.6

82.8

96.4

82.2

83.9

90.4

89.2

75.0

84.6

73.7

78.6

62.7

74.9

62.1

47.2

67.6

43.2

22.5

96.2

94.8

93.2

94.8

90.6

95.1

92.0

93.7

91.0

92.6

90.2

89.7

88.3

89.3

85.2

86.0

79.9

90.4

78.8

86.0

65.2

78.0

67.5

57.7

64.8

56.8

18.0

96.9

96.5

95.7

96.0

92.5

96.2

93.7

94.9

93.8

95.3

92.1

95.5

94.2

91.6

93.3

94.3

91.1

92.9

82.4

89.8

68.5

92.0

84.3

61.0

76.1

77.5

82.0

Meta-train regression benchmark , percentage of row wins

0

20

40

60

80

100

100

XGB-D

RF-D

MLP-D

ResNet-D

XGB-TD

LGBM-D

MLP-PLR-D

LGBM-TD

RealMLP-TD-S

ResNet-HPO

MLP-HPO

CatBoost-D

CatBoost-TD

TabR-S-D

Best-D

RealMLP-TD

XGB-HPO

CatBoost-HPO

MLP-PLR-HPO

RealTabR-D

Ensemble-D

LGBM-HPO

Best-TD

RealMLP-HPO

Ensemble-TD

Best-HPO

Ensemble-HPO

XGB-D

RF-D

MLP-D

ResNet-D

XGB-TD

LGBM-D

MLP-PLR-D

LGBM-TD

RealMLP-TD-S

ResNet-HPO

MLP-HPO

CatBoost-D

CatBoost-TD

TabR-S-D

Best-D

RealMLP-TD

XGB-HPO

CatBoost-HPO

MLP-PLR-HPO

RealTabR-D

Ensemble-D

LGBM-HPO

Best-TD

RealMLP-HPO

Ensemble-TD

Best-HPO

Ensemble-HPO

60.0

35.2

36.7

20.0

28.6

33.1

19.8

27.6

29.5

29.8

21.7

13.6

22.4

12.7

16.4

10.0

6.9

17.6

13.8

4.2

7.4

8.6

13.8

4.3

8.1

4.3

40.0

35.5

35.7

23.8

24.8

26.7

18.1

27.4

31.0

31.9

22.4

16.0

23.8

14.8

16.2

9.8

12.1

17.6

16.0

12.4

10.7

11.0

12.1

6.2

7.9

4.3

64.8

64.5

50.7

48.6

43.8

44.8

40.7

42.6

31.4

29.3

33.1

31.2

25.7

27.6

22.1

22.9

25.5

19.5

21.0

21.0

18.8

16.0

16.4

12.9

14.3

8.3

63.3

64.3

49.3

47.4

42.1

39.5

43.3

39.3

22.6

33.6

36.0

36.7

28.8

25.7

22.9

25.5

24.0

19.5

21.0

22.4

23.6

19.3

13.8

17.1

11.7

5.7

80.0

76.2

51.4

52.6

52.4

47.4

30.0

44.0

44.0

42.1

39.0

27.1

37.6

30.7

29.0

24.8

25.5

25.7

26.7

24.8

16.0

18.9

26.2

6.0

14.0

6.9

71.4

75.2

56.2

57.9

47.6

49.3

42.4

47.1

45.5

45.0

36.2

30.5

38.1

25.0

31.2

19.8

19.3

28.1

31.2

16.0

14.8

20.2

20.2

13.3

14.3

5.0

66.9

73.3

55.2

60.5

52.6

50.7

48.6

48.8

46.2

44.0

37.6

41.2

36.7

28.7

29.3

33.3

29.5

22.9

28.1

21.2

29.8

23.6

20.0

18.1

16.9

9.0

80.2

81.9

59.3

56.7

70.0

57.6

51.4

50.2

47.6

47.6

43.6

38.1

41.7

35.0

36.7

32.4

34.8

31.2

32.6

35.2

21.2

26.8

24.5

11.1

15.7

6.0

72.4

72.6

57.4

60.7

56.0

52.9

51.2

49.8

47.1

48.8

43.6

44.0

41.2

41.2

27.9

37.6

37.6

37.4

32.6

36.4

35.5

26.9

18.3

24.8

17.6

8.8

70.5

69.0

68.6

77.4

56.0

54.5

53.8

52.4

52.9

49.5

43.8

41.9

38.3

38.8

33.6

33.3

33.8

30.7

29.0

34.0

31.7

26.2

20.0

22.9

16.9

7.1

70.2

68.1

70.7

66.4

57.9

55.0

56.0

52.4

51.2

50.5

43.6

43.6

43.8

41.4

34.8

35.5

38.8

29.5

33.6

37.1

35.7

28.6

20.5

20.7

18.3

10.2

78.3

77.6

66.9

64.0

61.0

63.8

62.4

56.4

56.4

56.2

56.4

49.0

48.1

36.4

43.3

36.0

24.5

37.1

38.8

26.9

36.9

33.3

31.0

24.0

22.4

12.1

86.4

84.0

68.8

63.3

72.9

69.5

58.8

61.9

56.0

58.1

56.4

51.0

48.6

41.9

38.1

47.1

42.9

36.2

42.1

38.8

36.9

28.3

29.3

14.5

19.8

8.1

77.6

76.2

74.3

71.2

62.4

61.9

63.3

58.3

58.8

61.7

56.2

51.9

51.4

46.7

46.7

37.6

40.7

39.3

35.5

42.9

39.0

36.9

34.0

27.1

25.7

15.7

87.3

85.2

72.4

74.3

69.3

75.0

71.3

65.0

58.8

61.2

58.6

63.6

58.1

53.3

45.2

47.4

40.5

42.9

42.4

30.8

42.1

36.4

35.5

29.0

24.3

13.8

83.6

83.8

77.9

77.1

71.0

68.8

70.7

63.3

72.1

66.4

65.2

56.7

61.9

53.3

54.8

52.6

49.0

48.1

48.3

49.0

48.3

41.2

27.4

25.6

22.1

10.5

90.0

90.2

77.1

74.5

75.2

80.2

66.7

67.6

62.4

66.7

64.5

64.0

52.9

62.4

52.6

47.4

44.0

46.7

46.0

48.6

39.0

41.2

40.2

26.4

27.4

11.9

93.1

87.9

74.5

76.0

74.5

80.7

70.5

65.2

62.4

66.2

61.2

75.5

57.1

59.3

59.5

51.0

56.0

48.3

46.4

48.6

48.3

40.7

39.8

27.4

28.2

12.3

82.4

82.4

80.5

80.5

74.3

71.9

77.1

68.8

62.6

69.3

70.5

62.9

63.8

60.7

57.1

51.9

53.3

51.7

49.5

50.7

55.7

44.0

34.5

35.0

23.3

13.1

86.2

84.0

79.0

79.0

73.3

68.8

71.9

67.4

67.4

71.0

66.4

61.2

57.9

64.5

57.6

51.7

54.0

53.6

50.5

52.9

50.2

43.1

38.8

33.8

31.2

18.1

95.8

87.6

79.0

77.6

75.2

84.0

78.8

64.8

63.6

66.0

62.9

73.1

61.2

57.1

69.2

51.0

51.4

51.4

49.3

47.1

46.9

42.4

40.0

31.0

28.1

14.8

92.6

89.3

81.2

76.4

84.0

85.2

70.2

78.8

64.5

68.3

64.3

63.1

63.1

61.0

57.9

51.7

61.0

51.7

44.3

49.8

53.1

46.2

38.8

29.3

29.3

11.2

91.4

89.0

84.0

80.7

81.1

79.8

76.4

73.2

73.1

73.8

71.4

66.7

71.7

63.1

63.6

58.8

58.8

59.3

56.0

56.9

57.6

53.8

37.6

27.4

28.6

12.1

86.2

87.9

83.6

86.2

73.8

79.8

80.0

75.5

81.7

80.0

79.5

69.0

70.7

66.0

64.5

72.6

59.8

60.2

65.5

61.2

60.0

61.2

62.4

49.0

38.9

11.5

95.7

93.8

87.1

82.9

94.0

86.7

81.9

88.9

75.2

77.1

79.3

76.0

85.5

72.9

71.0

74.4

73.6

72.6

65.0

66.2

69.0

70.7

72.6

51.0

43.1

19.5

91.9

92.1

85.7

88.3

86.0

85.7

83.1

84.3

82.4

83.1

81.7

77.6

80.2

74.3

75.7

77.9

72.6

71.8

76.7

68.8

71.9

70.7

71.4

61.1

56.9

16.2

95.7

95.7

91.7

94.3

93.1

95.0

91.0

94.0

91.2

92.9

89.8

87.9

91.9

84.3

86.2

89.5

88.1

87.7

86.9

81.9

85.2

88.8

87.9

88.5

80.5

83.8

Meta-test regression benchmark , percentage of row wins

0

20

40

60

80

100

ResNet-D

XGB-D

MLP-D

RF-D

ResNet-HPO

MLP-HPO

XGB-TD

MLP-PLR-D

RealMLP-TD-S

FTT-D

LGBM-D

RF-HPO

LGBM-TD

RealMLP-TD

TabR-S-D

FTT-HPO

MLP-PLR-HPO

CatBoost-D

CatBoost-TD

XGB-HPO

Best-D

CatBoost-HPO

Best-TD

RealMLP-HPO

RealTabR-D

LGBM-HPO

TabR-HPO

Ensemble-D

Best-HPO

Ensemble-TD

Ensemble-HPO

ResNet-D

XGB-D

MLP-D

RF-D

ResNet-HPO

MLP-HPO

XGB-TD

MLP-PLR-D

RealMLP-TD-S

FTT-D

LGBM-D

RF-HPO

LGBM-TD

RealMLP-TD

TabR-S-D

FTT-HPO

MLP-PLR-HPO

CatBoost-D

CatBoost-TD

XGB-HPO

Best-D

CatBoost-HPO

Best-TD

RealMLP-HPO

RealTabR-D

LGBM-HPO

TabR-HPO

Ensemble-D

Best-HPO

Ensemble-TD

Ensemble-HPO

43.9 37.9 55.4 20.0 33.6 40.0 22.5 28.9 26.1 18.6 27.5 22.9 20.7 13.6 17.9 11.1 14.6 15.4 18.6

9.6

10.0 12.1 10.4

6.8

9.3

7.5

6.4

8.6

6.4

3.9

56.1

51.1 51.4 46.8 43.2 33.6 35.7 36.8 35.0 20.4 30.7 23.6 29.6 26.1 26.4 22.9 22.1 11.8 10.0 11.6

8.6

7.1

13.9 11.1

6.8

9.6

2.1

6.1

2.5

1.4

62.1 48.9

57.9 42.5 30.4 40.4 30.4 38.9 28.6 23.2 30.4 23.9 23.6 21.4 16.8 15.0 17.5 19.6 18.6 12.9 13.9 14.6 10.7 11.1 10.0

9.6

5.0

8.6

7.5

4.3

44.6 48.6 42.1

37.1 37.9 35.0 31.8 33.9 31.1 32.1 20.7 22.1 26.1 33.2 27.1 26.4 28.9 23.6 24.3 25.7 21.8 16.8 21.4 18.2 12.9 20.7 18.6 11.4

7.1

6.8

80.0 53.2 57.5 62.9

44.6 55.0 34.3 41.8 31.8 27.9 37.5 31.8 32.1 25.7 22.9 20.7 23.2 23.2 26.8 16.4 13.9 16.8 13.2 13.6 13.2 12.1 11.4

8.9

8.6

5.0

66.4 56.8 69.6 62.1 55.4

46.4 42.9 41.4 43.6 31.8 37.1 27.5 30.4 37.1 29.3 22.9 28.2 28.6 29.3 21.1 21.1 17.9 15.4 18.6 16.1 13.6 11.4 11.8 11.4

5.0

60.0 66.4 59.6 65.0 45.0 53.6

47.1 43.9 45.0 43.6 40.0 22.1 34.6 33.6 33.6 31.1 36.1 18.2 20.4 28.9 23.2 14.8 25.0 19.3 11.4 17.9 13.6

9.6

3.2

4.3

77.5 64.3 69.6 68.2 65.7 57.1 52.9

48.9 35.7 36.4 48.6 39.3 30.4 34.3 18.6 20.7 25.7 33.9 28.6 18.9 21.8 21.8 17.1 16.4 20.0 15.4

4.3

10.7

7.5

4.6

71.1 63.2 61.1 66.1 58.2 58.6 56.1 51.1

47.5 44.6 47.1 37.1 37.1 36.1 37.5 35.7 41.1 33.6 39.3 31.8 29.6 19.3 13.2 22.5 25.4 22.9 18.9

8.9

11.8

5.0

73.9 65.0 71.4 68.9 68.2 56.4 55.0 64.3 52.5

43.9 48.6 43.2 39.3 41.1 25.0 31.4 27.1 38.9 32.5 22.9 24.6 27.9 22.9 21.1 21.8 20.0

9.6

12.9 10.4

4.3

81.4 79.6 76.8 67.9 72.1 68.2 56.4 63.6 55.4 56.1

52.5 43.9 48.2 49.6 42.9 43.2 38.2 35.4 28.6 23.2 21.8 27.9 27.9 28.6

9.6

20.7

2.5

13.9

8.9

3.9

72.5 69.3 69.6 79.3 62.5 62.9 60.0 51.4 52.9 51.4 47.5

48.2 45.7 51.8 42.9 40.4 36.8 41.1 37.5 34.3 36.4 33.9 33.9 37.5 27.1 34.3 27.5 21.8 17.1 10.7

77.1 76.4 76.1 77.9 68.2 72.5 77.9 60.7 62.9 56.8 56.1 51.8

47.1 48.6 43.9 45.0 41.4 33.2 33.9 37.1 34.3 24.3 33.9 31.4 20.0 23.6 18.9 13.9

3.4

5.4

79.3 70.4 76.4 73.9 67.9 69.6 65.4 69.6 62.9 60.7 51.8 54.3 52.9

44.3 46.4 43.9 40.4 43.6 42.9 37.5 33.9 31.6 21.1 29.3 33.2 25.4 21.8 13.6

7.9

3.6

86.4 73.9 78.6 66.8 74.3 62.9 66.4 65.7 63.9 58.9 50.4 48.2 51.4 55.7

48.6 47.1 45.4 42.1 42.9 39.6 30.0 35.4 34.3 21.4 28.2 17.5 21.4 21.1 16.1 11.1

82.1 73.6 83.2 72.9 77.1 70.7 66.4 81.4 62.5 75.0 57.1 57.1 56.1 53.6 51.4

45.7 41.4 45.4 40.4 38.9 32.9 35.4 31.8 33.9 31.1 27.5 17.1 17.1 16.8

6.1

88.9 77.1 85.0 73.6 79.3 77.1 68.9 79.3 64.3 68.6 56.8 59.6 55.0 56.1 52.9 54.3

47.1 45.0 44.3 40.7 32.5 32.1 25.7 32.5 29.3 29.6 20.0 13.9 17.1

6.4

85.4 77.9 82.5 71.1 76.8 71.8 63.9 74.3 58.9 72.9 61.8 63.2 58.6 59.6 54.6 58.6 52.9

51.1 41.8 36.3 31.1 39.6 42.1 32.1 30.7 32.5 11.3 24.3 21.4

7.9

84.6 88.2 80.4 76.4 76.8 71.4 81.8 66.1 66.4 61.1 64.6 58.9 66.8 56.4 57.9 54.6 55.0 48.9

45.4 41.4 34.3 35.7 37.9 37.1 28.9 29.6 24.6 21.1

6.1

6.1

81.4 90.0 81.4 75.7 73.2 70.7 79.6 71.4 60.7 67.5 71.4 62.5 66.1 57.1 57.1 59.6 55.7 58.2 54.6

50.7 48.6 48.2 43.6 38.6 39.6 35.7 27.1 29.6 25.4 12.1

90.4 88.4 87.1 74.3 83.6 78.9 71.1 81.1 68.2 77.1 76.8 65.7 62.9 62.5 60.4 61.1 59.3 63.7 58.6 49.3

42.1 40.4 43.2 35.7 37.1 33.2

8.0

25.0 22.5

6.8

90.0 91.4 86.1 78.2 86.1 78.9 76.8 78.2 70.4 75.4 78.2 63.6 65.7 66.1 70.0 67.1 67.5 68.9 65.7 51.4 57.9

47.1 48.9 41.1 42.1 36.8 30.0 27.7 21.4

6.1

87.9 92.9 85.4 83.2 83.2 82.1 85.2 78.2 80.7 72.1 72.1 66.1 75.7 68.4 64.6 64.6 67.9 60.4 64.3 51.8 59.6 52.9

45.4 48.6 43.2 42.5 39.6 24.6

8.8

7.1

89.6 86.1 89.3 78.6 86.8 84.6 75.0 82.9 86.8 77.1 72.1 66.1 66.1 78.9 65.7 68.2 74.3 57.9 62.1 56.4 56.8 51.1 54.6

50.7 44.3 45.4 38.6 31.4 29.6

7.3

93.2 88.9 88.9 81.8 86.4 81.4 80.7 83.6 77.5 78.9 71.4 62.5 68.6 70.7 78.6 66.1 67.5 67.9 62.9 61.4 64.3 58.9 51.4 49.3

52.5 40.7 46.4 37.5 34.3 21.4

90.7 93.2 90.0 87.1 86.8 83.9 88.6 80.0 74.6 78.2 90.4 72.9 80.0 66.8 71.8 68.9 70.7 69.3 71.1 60.4 62.9 57.9 56.8 55.7 47.5

42.9 36.4 35.5 28.2 14.3

92.5 90.4 90.4 79.3 87.9 86.4 82.1 84.6 77.1 80.0 79.3 65.7 76.4 74.6 82.5 72.5 70.4 67.5 70.4 64.3 66.8 63.2 57.5 54.6 59.3 57.1

49.3 38.9 33.9 22.1

93.6 97.9 95.0 81.4 88.6 88.6 86.4 95.7 81.1 90.4 97.5 72.5 81.1 78.2 78.6 82.9 80.0 88.7 75.4 72.9 92.0 70.0 60.4 61.4 53.6 63.6 50.7

44.6 36.8 16.1

91.4 93.9 91.4 88.6 91.1 88.2 90.4 89.3 91.1 87.1 86.1 78.2 86.1 86.4 78.9 82.9 86.1 75.7 78.9 70.4 75.0 72.3 75.4 68.6 62.5 64.5 61.1 55.4

48.9 11.3

93.6 97.5 92.5 92.9 91.4 88.6 96.8 92.5 88.2 89.6 91.1 82.9 96.6 92.1 83.9 83.2 82.9 78.6 93.9 74.6 77.5 78.6 91.2 70.4 65.7 71.8 66.1 63.2 51.1

19.3

96.1 98.6 95.7 93.2 95.0 95.0 95.7 95.4 95.0 95.7 96.1 89.3 94.6 96.4 88.9 93.9 93.6 92.1 93.9 87.9 93.2 93.9 92.9 92.7 78.6 85.7 77.9 83.9 88.7 80.7

Grinsztajn et al. (2022) regression benchmark , percentage of row wins

0

20

40

60

80

C Benchmark Details

C.1 Default Configurations

The parameters for RealMLP-TD and RealMLP-TD-S have already been given in Table A.1. Table C.1 shows the hyperparameters of LGBM-TD and LGBM-D. Table C.2 shows the hyperparameters of XGB-TD and XGB-D. Table C.3 shows the hyperparameters of CatBoost-TD and CatBoost-D. The parameters for LGBM-D, XGB-D, and CatBoost-D have been taken from the respective libraries at the time of writing and are given here for completeness. We also provide tables for MLP-D (Table C.4), ResNet-D (Table C.6), MLP-PLR-D (Table C.5), FTT-D (Table C.7), TabR-S-D (Table C.8), and RealTabR-D (Table C.9). By 'RTDL quantile transform', we refer to the version adding noise before fitting the quantile transform.

For XGB-PBB-D, we use the default parameters from Probst et al. [50], with the following modifications: We use hist gradient boosting since it is the new default in XGBoost 2.0. Moreover, since we have high-cardinality categories, we limit one-hot encoding to categories with less than 20 distinct values (not counting missing values) and use XGBoost's native categorical feature handling for the remaining categorical features. For RF-D, we use the default parameters from scikit-learn, do not give RF-D access to the validation set (to make it more similar to other methods that do not use nested cross-validation), and encode categorical columns using ordinal encoding with a random shuffling of categories.

C.2 Hyperparameter Optimization

For all methods, we run 50 steps of random search with the search spaces presented in the following. The search spaces for LGBM-HPO (Table C.10), XGB-HPO (Table C.11), and CatBoost-HPO (Table C.12) are adapted from the 'tree-friendly' literature, using n_estimators=1000 in each case. The search space for RF-HPO (Table C.13) is taken from Grinsztajn et al. [18].

For RealMLP-HPO, we provide a custom search space specified in Table C.14. The search spaces for MLP-HPO (Table C.15), MLP-PLR-HPO (Table C.16), ResNet-HPO (Table C.17), FTT-HPO (Table C.18), and TabR-HPO (Table C.19) are adapted from the literature, with minor modifications to decrease RAM usage.

{

}

C.3 Dataset Selection and Preprocessing

C.3.1 Meta-train Benchmarks

For the meta-train benchmarks, we adapt code from Steinwart [61] to collect all datasets from the UCI repository that follow certain criteria:

Between 2,500 and 50,000 samples.

Labeled as classification or regression task.

Number of features at most 1,000.

Description made it straightforward to convert the original dataset into a numeric .csv format.

Uploaded before 2019-05-08.

We remove rows with missing values and keep only those datasets that still have at least 2,500 samples. 3 Some datasets are labeled both as regression and classification datasets, in which case we use them for both. Some datasets contain different versions (e.g., different target columns), in which case we use all of them. To avoid biasing the results towards one dataset, we compute benchmark scores using weights proportional to 1 / # versions. In total, we obtain 71 classification datasets (including versions) out of 46 original datasets, and 47 regression datasets (including versions) out of 26 original datasets. Tables C.20 and C.21 summarize key characteristics of these datasets. We count datasets with the same prefix (before the first underscore) as being versions of the same dataset for weighting, except for the two 'facebook' datasets in B train reg , which we count as distinct because they are taken from different sources. For regression, we standardize the targets to have mean zero and variance 1 on the whole dataset. This does not introduce leakage since all neural networks standardize regression targets based on the training set, and tree-based methods are invariant to affine rescaling.

During earlier development of the MLP, the meta-train benchmark used to include an epileptic seizure recognition dataset, which has since been removed from the UCI repository, hence we do not report results on it.

C.3.2 Meta-test Benchmarks

The meta-test benchmarks consist of datasets from the AutoML Benchmark [13] and additional regression datasets from the OpenML-CTR23 benchmark [12], obtained from OpenML [65].

We make the following modifications:

We use brazilian_houses from OpenML-CTR23 and exclude Brazilian_houses from the AutoML regression benchmark, since the latter contains three additional features that should not be used for predicting the target.

We excluded the following datasets because versions of them were already contained in the meta-training set:

We use another version of the sarcos dataset where the original test set is not included, since the original test set consists of duplicates of training samples.

-For classification: kr-vs-kp, wilt, ozone-level-8hr, first-order-theorem-proving, GesturePhaseSegmentationProcessed, PhishingWebsites, wine-quality-white, nomao, bankmarketing, adult

-For regression: wine_quality, abalone, OnlineNewsPopularity, Brazilian_houses, physicochemical_protein, naval_propulsion_plant, superconductivity, white_wine, red_wine, grid_stability

We preprocess the datasets as follows:

We remove rows with missing continuous values

We encode missing categorical values as a separate category.

We subsample large datasets to contain at most 500,000 samples. Since the dionis dataset was particularly slow to train with GBDT models due to its 355 classes, we subsampled it to 100,000 samples.

For regression, we standardize the targets to have mean zero and variance 1. This does not introduce leakage since all neural networks standardize regression targets based on the training set, and tree-based methods are invariant to affine rescaling.

After preprocessing, we

exclude datasets with less than 1,000 samples, these were

-for regression: boston, cars, colleges, energy_efficiency, forest_fires, Moneyball, QSAR_fish_toxicity, sensory, student_performance_por, tecator, us_crime

-for classification: albert, APSFailure, arcene, Australian, blood-transfusion-servicecenter, eucalyptus, KDDCup09_appetency, KDDCup09-Upselling, micro-mass, vehicle

exclude datasets that have more than 10,000 features after one-hot encoding. These were Amazon_employee_access, Click_prediction_small, and sf-police-incidents (all classification).

C.3.3 Grinsztajn et al. [18] Benchmarks

We select the datasets as follows:

We use the newer version of the benchmark on OpenML.

We exclude the eye_movements dataset since a leak in the dataset was reported by Gorishniy et al. [17].

When a dataset is used both in benchmarks with and without categorical features, we use the version with categorical features.

C.4 Comparison with Standard Grinsztajn et al. [18] Benchmark

Here, we compare two versions of the Grinsztajn et al. [18] benchmark:

(a) The 'new' version, using our benchmarking setup with the datasets of the Grinsztajn et al. [18] benchmark. This version is used in all plots except Figure C.2 and Figure C.3.

(b) The 'old' version, which is a slightly modified version of the original code, described in Appendix C.5.

The corresponding results for the most comparable metrics are shown in Figure C.1 for the new paper version, and Figure C.2 for the old version. We decided to use the new version for multiple reasons, including a more realistic validation setting, having the exact same baselines, and having more options for evaluation and plotting. Here is a list of differences in our adapted version:

In the new version, we removed the eye_movements dataset due to a leak reported in Gorishniy et al. [17].

We always standardize targets, to make our results independent of the scaling of the datasets. (In contrast, HPO methods on the original benchmark have standardization as an option in their tuning space.)

We subsample datasets after downloading them to 500K samples (all train-test splits are performed on the same 500K samples).

We limit training+validation set sizes to 13333, such that at most 10K samples are used for training. Of these samples, we always use 25% for validation, unlike the original benchmark, which limits the training and validation set sizes separately to 10K and 50K samples.

With the new version, we mostly report results using different aggregation strategies and using nRMSE instead of R 2 for regression, but try to provide comparable aggregated metrics in Figure C.1.

The new version does not use separate validation sets for early stopping and for HPO, which avoids unfairly disadvantaging D and TD methods compared to HPO methods.

The new version uses different random hyperparameter configurations on different train-test splits, which should provide more accurate results and allows computing confidence intervals as in Appendix C.6.

The new version measures all runtimes on the CPU, while the old version measures NN runtimes on the GPU.

The new version uses ten train-test splits on all datasets, instead of a smaller dataset-sizedependent number.

The old version uses slightly different baseline configurations:

-The old version sometimes replaces search spaces like Choice([0, LogUniform[1e-6, 1e-3]]) with more simple spaces.

-The old version uses (in the code) a simplified version of the MLP without dropout and without weight decay.

-The new version doesn't use as large categorical embedding sizes for ResNet and MLP models (up to 64 instead of [64, 512]).

-In the old version, ResNet-HPO tunes the normalization, unlike the original paper [15].

-The old version uses larger stopping patiences for default models than in the original literature [15].

-In the old version, the batch size is tuned for some models.

-The old version uses different versions of quantile preprocessing for NN methods, while we use the RTDL quantile transform for all methods except RealMLP.

-In the old version, XGBoost uses the exact tree method with one-hot encoding, while in the new version, we use the hist method that supports native categorical feature handling. This makes XGBoost slower but also more accurate in the older version.

Both the new and the old version use early stopping and best-epoch selection on accuracy (for classification) / RMSE (for regression).

C.5 Closer-to-original Version of the Grinsztajn et al. [18] Benchmark

In the following, we document the benchmark settings for obtaining the results in Figure C.2 and Figure C.3. The results were obtained using a modification of the original code.

10

-

1

10

0

10

1

10

2

10

3

10

4

Average training

time (CPU)

per 1K samples [s]

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

Arithmetic mean of

custom-normalized classification errors

better

10

-

1

10

0

10

1

10

2

10

3

10

4

Average training

time (CPU)

per 1K samples [s]

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

Arithmetic mean of

custom-normalized

1

-

R

2

better

10

-

1

10

0

10

1

10

2

10

3

Average training

time

per 1K samples (s)

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

Normalized

accuracy

better

10

0

10

1

10

2

10

3

Average training

time

per 1K samples (s)

0

.

2

0

.

4

0

.

6

0

.

8

Normalized

R2 score

better

The datasets are taken from the benchmarks described in Grinsztajn et al. [18]. When a dataset is used both in benchmarks with and without categorical features, we use the version with categorical features. We preprocess the datasets following the same steps as in Grinsztajn et al. [18]:

For neural networks, we quantile-transform the features to have a Gaussian distribution. For TabR [17], we use the modified quantile transform from the TabR paper. For RealMLP, we use the preprocessing described in Section 3, namely robust scaling and smooth clipping.

For neural networks, we add as a hyperparameter the possibility to normalize the target variable for the model fit and transform it back for evaluation (via scikit-learn's Transformed-

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

GBT

GBT

GBT

GBT

GBT

GBT

GBT

GBT

GBT

MLP

MLP

MLP

MLP

MLP

MLP

MLP

MLP

MLP

RF

RF

RF

RF

RF

RF

RF

RF

RF

RealMLP

RealMLP

RealMLP

RealMLP

RealMLP

RealMLP

RealMLP

RealMLP

RealMLP

ResNet

ResNet

ResNet

ResNet

ResNet

ResNet

ResNet

ResNet

ResNet

TabR-S

TabR-S

TabR-S

TabR-S

TabR-S

TabR-S

TabR-S

TabR-S

TabR-S

XGB

XGB

XGB

XGB

XGB

XGB

XGB

XGB

XGB

0.6

0.7

0.8

0.9

1

3

10

30

Number of random search iterations

Normalized test accuracy of best

model (on valid set) up to this iteration

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

GBT

GBT

GBT

GBT

GBT

GBT

GBT

GBT

GBT

MLP

MLP

MLP

MLP

MLP

MLP

MLP

MLP

MLP

RF

RF

RF

RF

RF

RF

RF

RF

RF

RealMLP

RealMLP

RealMLP

RealMLP

RealMLP

RealMLP

RealMLP

RealMLP

RealMLP

TabR-S

TabR-S

TabR-S

TabR-S

TabR-S

TabR-S

TabR-S

TabR-S

TabR-S

XGB

XGB

XGB

XGB

XGB

XGB

XGB

XGB

XGB

0.4

0.5

0.6

0.7

0.8

0.9

1

3

10

30

Number of random search iterations

Normalized test R2 score of best

model (on valid set) up to this iteration

TargetRegressor and StandardScaler, which differs from the QuantileTransformer from the original paper, as we found it to work better). The same standardization is also applied to all default-parameter versions of neural networks.

Train size is restricted to 10,000 samples and test and validation size to 50,000 samples.

For models that do not handle categorical variables natively, we encode categorical features using OneHotEncoder from scikit-learn.

Note that the datasets from the original benchmark are already slightly preprocessed, e.g., heavy-tailed targets are standardized and missing values are removed. More details can be found in the original paper.

Results normalization For Figure C.2, as in the original paper, we normalize the R2 or accuracy score for each dataset before averaging them. We use an affine normalization between 0 and 1, 1 corresponding to the score of the best model for each dataset, and 0 corresponding to the score of the worst model (for classification) and the 10th percentile of the scores (for regression). We use slightly different percentiles compared to the original paper as we normalize across the scores of the tuned and default models, and not all steps of the random search, which reduces the number of outliers. Other aggregation metrics are shown in Appendix B.10.

Time measurement We follow the original paper and run neural networks on a GPU and the other models on 1 core of an AMD EPYC 7742 64-Core processor, and we average the time across all random steps (for each random step, the time is averaged across splits). To compute the runtime of neural networks, we restrict ourselves to steps ran on the same GPU model (NVIDIA A100-40GB), which means that we exclude datasets for which we have less than 15 steps of each model on this GPU (leaving us with 11 datasets for classification and 15 for regression). We then compute the average runtime per 1000 samples on each dataset and average them.

Other details We rerun classification results for neural networks compared to the original results to early stop on accuracy rather than on cross-entropy, to make results more comparable with the rest of this paper.

At https://github.com/LeoGrin/tabular-benchmark/tree/better_by_default , we provide code for the adapted original Grinsztajn et al. [18] benchmark.

C.6 Confidence Intervals

Here, we specify how our confidence intervals are computed. Let X ij denote the score (error/rank) of a method on dataset i and split j , with i ∈ { 1 , . . . , n } and j ∈ { 1 , . . . , m } . Then, the benchmark score S can be written as

where f = g = id for the arithmetic mean. For the shifted geometric mean, we instead have g = exp and f ( x ) = log( x + ε ) , ε = 0 . 01 . We interpret the benchmark datasets as fixed, but the splits as random. For each dataset i , X i 1 , . . . , X im are i.i.d. random variables. We first take the dataset averages

The random variables X 1 j , . . . , X nj are independent but not identically distributed. Still, for lack of a better option, we assume that the Z j are normally distributed with unknown mean and variance. We know that the Z j are i.i.d., hence we use the confidence intervals from the Student's t -distribution for normally distributed random variables with unknown mean and variance. This gives us a confidence interval [ a, b ] for 1 m ∑ m j =1 Z j . Since g is increasing, we hence obtain a confidence interval [ g ( a ) , g ( b )] for S = g ( 1 m ∑ m j =1 Z j ) .

Comparison of two methods We often compute the error increase in % in the benchmark score of method A compared to method B with the shifted geometric mean, given by

Here, we leverage that the shifted geometric mean uses g = exp to write

which is of the same form as Eq. (1). Hence, we obtain confidence intervals for this quantity using the same method.

C.7 Time Measurements

For our meta-train and meta-test benchmarks, we report training times measured as follows: We run all methods on a single compute node with a 32-core AMD Ryzen Threadripper Pro 3975 WX CPU, using 32 threads for GBDTs and the PyTorch default settings for NNs. No method is run on GPUs. We run methods sequentially on one split on each dataset of the meta-train-class and meta-train-reg benchmarks. For random-search-based HPO methods, we only run one (TabR-HPO, FTT-HPO) or two (other methods) random search steps and extrapolate the runtime to 50 steps. Runtimes for combinations of models (Best and Ensemble) are computed as the sum of the individual runtimes. We compute the runtime per 1000 samples on each dataset and then average them. For simplicity, we do not use the dataset-dependent weighting employed otherwise on the meta-train benchmark.

C.8 Compute Resources

While we did not measure compute resources precisely, our experiments required at least around 3000 hours on RTX 3090 GPUs and other GPUs, as well as roughly 10,000 hours on HPC CPU nodes (32-64 cores).

C.9 Used Libraries

Our implementation uses various libraries, out of which we would like to particularly acknowledge PyTorch [47], Scikit-learn [48], Ray [46], XGBoost [9], LightGBM [31], and CatBoost [51]. For using XGBoost, LightGBM, and CatBoost, we adapted wrapping code from the CatBoost quality benchmarks [51].

D Results for Individual Datasets

Here, we provide and compare the results of central methods per dataset. Figures D.1 - D.7 show scatterplot comparisons for different models.

Table D.1 and Table D.2 show results on B train class . Table D.3 and Table D.4 show results on B train reg . Table D.5 and Table D.6 show results on B test class . Table D.7 and Table D.8 show results on B test reg . Table D.9 and Table D.10 show results on Grinsztajn . Table D.11 and Table D.12 show results on

B class B Grinsztajn reg .

Best-TD better

CatBoost-HPO better

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for Best-TD (

↓

)

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for CatBoost-HPO (

↓

)

Meta-train classification benchmark

Best-TD better

CatBoost-HPO better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for Best-TD (

↓

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for CatBoost-HPO (

↓

)

Meta-train regression benchmark

Best-TD better

CatBoost-HPO better

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for Best-TD (

↓

)

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for CatBoost-HPO (

↓

)

Meta-test classification benchmark

Best-TD better

CatBoost-HPO better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for Best-TD (

↓

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for CatBoost-HPO (

↓

)

Meta-test regression benchmark

Best-TD better

CatBoost-HPO better

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Classification error for Best-TD (

)

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Classification error for CatBoost-HPO (

↓

)

Best-TD better

CatBoost-HPO better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for Best-TD (

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for CatBoost-HPO (

↓

)

↓

↓

Best-TD better

Best-HPO better

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for Best-TD (

↓

)

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for Best-HPO (

↓

)

Meta-train classification benchmark

Best-TD better

Best-HPO better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for Best-TD (

↓

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for Best-HPO (

↓

)

Meta-train regression benchmark

Best-TD better

Best-HPO better

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for Best-TD (

↓

)

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for Best-HPO (

↓

)

Meta-test classification benchmark

Best-TD better

Best-HPO better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for Best-TD (

↓

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for Best-HPO (

↓

)

Meta-test regression benchmark

Best-TD better

Best-HPO better

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Classification error for Best-TD (

)

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Classification error for Best-HPO (

↓

)

Best-TD better

Best-HPO better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

nRMSE for Best-TD (

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

nRMSE for Best-HPO (

↓

)

↓

↓

RealMLP-TD better

CatBoost-TD better

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for RealMLP-TD (

↓

)

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for CatBoost-TD (

↓

)

Meta-train classification benchmark

RealMLP-TD better

CatBoost-TD better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for RealMLP-TD (

↓

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for CatBoost-TD (

↓

)

Meta-train regression benchmark

RealMLP-TD better

CatBoost-TD better

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for RealMLP-TD (

↓

)

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for CatBoost-TD (

↓

)

Meta-test classification benchmark

RealMLP-TD better

CatBoost-TD better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for RealMLP-TD (

↓

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for CatBoost-TD (

↓

)

Meta-test regression benchmark

RealMLP-TD better

CatBoost-TD better

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Classification error for RealMLP-TD (

)

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Classification error for CatBoost-TD (

↓

)

RealMLP-TD better

CatBoost-TD better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

nRMSE for RealMLP-TD (

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

nRMSE for CatBoost-TD (

↓

)

↓

↓

RealMLP-HPO better

CatBoost-HPO better

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Classification error for RealMLP-HPO (

↓

)

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Classification error for CatBoost-HPO (

↓

)

Meta-train classification benchmark

RealMLP-HPO better

CatBoost-HPO better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for RealMLP-HPO (

↓

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for CatBoost-HPO (

↓

)

Meta-train regression benchmark

RealMLP-HPO better

CatBoost-HPO better

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for RealMLP-HPO (

↓

)

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for CatBoost-HPO (

↓

)

Meta-test classification benchmark

RealMLP-HPO better

CatBoost-HPO better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for RealMLP-HPO (

↓

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for CatBoost-HPO (

↓

)

Meta-test regression benchmark

RealMLP-HPO better

CatBoost-HPO better

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Classification error for RealMLP-HPO (

)

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Classification error for CatBoost-HPO (

↓

)

RealMLP-HPO better

CatBoost-HPO better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for RealMLP-HPO (

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for CatBoost-HPO (

↓

)

↓

↓

Ensemble-TD better

Best-TD better

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for Ensemble-TD (

↓

)

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for Best-TD (

↓

)

Meta-train classification benchmark

Ensemble-TD better

Best-TD better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for Ensemble-TD (

↓

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for Best-TD (

↓

)

Meta-train regression benchmark

Ensemble-TD better

Best-TD better

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for Ensemble-TD (

↓

)

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for Best-TD (

↓

)

Meta-test classification benchmark

Ensemble-TD better

Best-TD better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for Ensemble-TD (

↓

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for Best-TD (

↓

)

Meta-test regression benchmark

Ensemble-TD better

Best-TD better

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Classification error for Ensemble-TD (

)

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Classification error for Best-TD (

↓

)

Ensemble-TD better

Best-TD better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

nRMSE for Ensemble-TD (

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

nRMSE for Best-TD (

↓

)

↓

↓

RealMLP-TD better

RealMLP-HPO better

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for RealMLP-TD (

↓

)

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for RealMLP-HPO (

↓

)

Meta-train classification benchmark

RealMLP-TD better

RealMLP-HPO better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for RealMLP-TD (

↓

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for RealMLP-HPO (

↓

)

Meta-train regression benchmark

RealMLP-TD better

RealMLP-HPO better

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for RealMLP-TD (

↓

)

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for RealMLP-HPO (

↓

)

Meta-test classification benchmark

RealMLP-TD better

RealMLP-HPO better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for RealMLP-TD (

↓

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for RealMLP-HPO (

↓

)

Meta-test regression benchmark

RealMLP-TD better

RealMLP-HPO better

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Classification error for RealMLP-TD (

)

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Classification error for RealMLP-HPO (

↓

)

RealMLP-TD better

RealMLP-HPO better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

nRMSE for RealMLP-TD (

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

nRMSE for RealMLP-HPO (

↓

)

↓

↓

CatBoost-TD better

CatBoost-HPO better

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for CatBoost-TD (

↓

)

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for CatBoost-HPO (

↓

)

Meta-train classification benchmark

CatBoost-TD better

CatBoost-HPO better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for CatBoost-TD (

↓

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for CatBoost-HPO (

↓

)

Meta-train regression benchmark

CatBoost-TD better

CatBoost-HPO better

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for CatBoost-TD (

↓

)

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Classification error for CatBoost-HPO (

↓

)

Meta-test classification benchmark

CatBoost-TD better

CatBoost-HPO better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for CatBoost-TD (

↓

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for CatBoost-HPO (

↓

)

Meta-test regression benchmark

CatBoost-TD better

CatBoost-HPO better

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Classification error for CatBoost-TD (

)

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Classification error for CatBoost-HPO (

↓

)

CatBoost-TD better

CatBoost-HPO better

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for CatBoost-TD (

)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

nRMSE for CatBoost-HPO (

↓

)

↓

↓

Dataset

RealMLP-TD

RealTabR-D

TabR-S-D

MLP-PLR-D

MLP-D

CatBoost-TD

LGBM-TD

XGB-TD

RF-D

abalone

0.447

±

0.014

0.445

±

0.006

0.440

±

0.010

0.453

±

0.010

0.448

±

0.014

0.458

±

0.009

0.455

±

0.012

0.451

±

0.013

0.457

±

0.010

adult

0.140

±

0.004

0.134

±

0.004

0.142

±

0.004

0.144

±

0.003

0.144

±

0.003

0.130

±

0.003

0.131

±

0.003

0.131

±

0.003

0.146

±

0.003

anuran_calls_families

0.006

±

0.001

0.006

±

0.001

0.008

±

0.002

0.009

±

0.002

0.009

±

0.002

0.007

±

0.002

0.009

±

0.002

0.008

±

0.003

0.012

±

0.003

anuran_calls_genus

0.007

±

0.002

0.008

±

0.002

0.008

±

0.002

0.011

±

0.002

0.010

±

0.002

0.008

±

0.002

0.009

±

0.002

0.009

±

0.002

0.012

±

0.003

anuran_calls_species

0.006

±

0.001

0.007

±

0.001

0.010

±

0.002

0.008

±

0.002

0.009

±

0.002

0.007

±

0.001

0.008

±

0.002

0.008

±

0.002

0.010

±

0.001

avila

0.000

±

0.000

0.000

±

0.000

0.001

±

0.000

0.003

±

0.001

0.016

±

0.002

0.001

±

0.000

0.001

±

0.001

0.001

±

0.000

0.011

±

0.001

bank_marketing

0.089

±

0.002

0.087

±

0.003

0.088

±

0.002

0.089

±

0.001

0.091

±

0.001

0.088

±

0.002

0.090

±

0.002

0.090

±

0.002

0.091

±

0.002

bank_marketing_additional

0.085

±

0.003

0.084

±

0.002

0.086

±

0.003

0.085

±

0.002

0.086

±

0.002

0.084

±

0.003

0.084

±

0.002

0.085

±

0.002

0.086

±

0.002

chess

0.005

±

0.002

0.013

±

0.003

0.015

±

0.003

0.010

±

0.004

0.008

±

0.003

0.008

±

0.003

0.011

±

0.002

0.005

±

0.001

0.015

±

0.003

chess_krvk

0.081

±

0.004

0.120

±

0.005

0.128

±

0.004

0.121

±

0.006

0.141

±

0.009

0.153

±

0.002

0.147

±

0.003

0.146

±

0.003

0.293

±

0.017

crowd_sourced_mapping

0.032

±

0.004

0.031

±

0.003

0.028

±

0.001

0.037

±

0.003

0.034

±

0.004

0.035

±

0.003

0.031

±

0.003

0.033

±

0.003

0.058

±

0.004

default_credit_card

0.179

±

0.004

0.181

±

0.004

0.182

±

0.003

0.179

±

0.003

0.181

±

0.004

0.179

±

0.004

0.178

±

0.004

0.180

±

0.004

0.183

±

0.003

eeg_eye_state

0.016

±

0.002

0.011

±

0.001

0.107

±

0.024

0.176

±

0.014

0.120

±

0.017

0.051

±

0.002

0.052

±

0.001

0.051

±

0.002

0.083

±

0.003

electrical_grid_stability_simulated

0.032

±

0.004

0.036

±

0.003

0.048

±

0.004

0.039

±

0.004

0.057

±

0.003

0.051

±

0.004

0.053

±

0.004

0.057

±

0.003

0.086

±

0.005

facebook_live_sellers_thailand_status

0.134

±

0.007

0.137

±

0.005

0.139

±

0.007

0.137

±

0.008

0.139

±

0.007

0.131

±

0.005

0.135

±

0.006

0.133

±

0.008

0.141

±

0.004

firm_teacher_clave

0.129

±

0.006

0.128

±

0.005

0.130

±

0.005

0.133

±

0.007

0.134

±

0.005

0.150

±

0.006

0.149

±

0.003

0.149

±

0.005

0.191

±

0.006

first_order_theorem_proving

0.179

±

0.006

0.180

±

0.007

0.182

±

0.007

0.188

±

0.008

0.181

±

0.006

0.158

±

0.008

0.160

±

0.008

0.160

±

0.006

0.162

±

0.007

gas_sensor_drift_class

0.005

±

0.001

0.006

±

0.001

0.004

±

0.001

0.006

±

0.001

0.005

±

0.001

0.006

±

0.001

0.006

±

0.001

0.006

±

0.001

0.008

±

0.001

gesture_phase_segmentation_raw

0.087

±

0.006

0.079

±

0.009

0.079

±

0.006

0.108

±

0.006

0.106

±

0.007

0.071

±

0.005

0.067

±

0.005

0.066

±

0.004

0.074

±

0.005

gesture_phase_segmentation_va3

0.323

±

0.007

0.289

±

0.008

0.293

±

0.006

0.347

±

0.011

0.368

±

0.008

0.321

±

0.007

0.312

±

0.006

0.316

±

0.009

0.355

±

0.006

htru2

0.020

±

0.002

0.019

±

0.002

0.020

±

0.001

0.020

±

0.002

0.020

±

0.002

0.021

±

0.002

0.019

±

0.002

0.020

±

0.002

0.020

±

0.002

human_activity_smartphone

0.008

±

0.001

0.009

±

0.001

0.010

±

0.001

0.011

±

0.002

0.015

±

0.002

0.008

±

0.002

0.008

±

0.002

0.008

±

0.001

0.024

±

0.002

indoor_loc_building

0.002

±

0.000

0.002

±

0.000

0.002

±

0.000

0.002

±

0.000

0.002

±

0.000

0.002

±

0.000

0.003

±

0.001

0.002

±

0.000

0.002

±

0.000

indoor_loc_relative

0.070

±

0.002

0.081

±

0.005

0.090

±

0.002

0.084

±

0.003

0.099

±

0.004

0.077

±

0.003

0.059

±

0.003

0.066

±

0.007

0.077

±

0.003

insurance_benchmark

0.061

±

0.005

0.060

±

0.004

0.060

±

0.004

0.060

±

0.004

0.060

±

0.004

0.061

±

0.005

0.059

±

0.004

0.061

±

0.004

0.072

±

0.004

landsat_satimage

0.077

±

0.006

0.083

±

0.006

0.094

±

0.006

0.085

±

0.007

0.090

±

0.005

0.087

±

0.005

0.080

±

0.004

0.080

±

0.004

0.090

±

0.004

letter_recognition

0.019

±

0.001

0.019

±

0.001

0.020

±

0.001

0.039

±

0.002

0.030

±

0.002

0.034

±

0.003

0.032

±

0.002

0.033

±

0.002

0.045

±

0.002

madelon

0.340

±

0.016

0.401

±

0.019

0.443

±

0.011

0.349

±

0.027

0.435

±

0.019

0.139

±

0.010

0.211

±

0.010

0.210

±

0.014

0.320

±

0.011

magic_gamma_telescope

0.115

±

0.005

0.108

±

0.004

0.109

±

0.004

0.119

±

0.005

0.122

±

0.003

0.116

±

0.004

0.118

±

0.004

0.118

±

0.003

0.122

±

0.003

mushroom

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

musk

0.003

±

0.002

0.004

±

0.002

0.005

±

0.002

0.007

±

0.001

0.011

±

0.003

0.011

±

0.002

0.012

±

0.002

0.012

±

0.002

0.028

±

0.002

nomao

0.022

±

0.002

0.020

±

0.001

0.022

±

0.001

0.024

±

0.001

0.026

±

0.002

0.018

±

0.001

0.017

±

0.001

0.017

±

0.002

0.020

±

0.002

nursery

0.020

±

0.001

0.012

±

0.003

0.011

±

0.002

0.022

±

0.002

0.024

±

0.002

0.021

±

0.001

0.026

±

0.002

0.024

±

0.002

0.034

±

0.001

occupancy_detection

0.006

±

0.001

0.007

±

0.001

0.008

±

0.001

0.009

±

0.001

0.008

±

0.001

0.007

±

0.001

0.007

±

0.001

0.007

±

0.000

0.006

±

0.001

online_shoppers_attention

0.098

±

0.004

0.101

±

0.004

0.098

±

0.004

0.095

±

0.003

0.099

±

0.004

0.097

±

0.003

0.096

±

0.005

0.097

±

0.004

0.096

±

0.003

optical_recognition_handwritten_digits

0.011

±

0.003

0.012

±

0.002

0.015

±

0.003

0.020

±

0.003

0.018

±

0.003

0.017

±

0.003

0.015

±

0.003

0.016

±

0.003

0.020

±

0.004

ozone_level_1hr

0.036

±

0.007

0.037

±

0.008

0.035

±

0.008

0.035

±

0.008

0.035

±

0.008

0.036

±

0.009

0.035

±

0.007

0.035

±

0.007

0.035

±

0.007

ozone_level_8hr

0.071

±

0.010

0.070

±

0.010

0.072

±

0.012

0.067

±

0.011

0.072

±

0.010

0.071

±

0.011

0.070

±

0.012

0.067

±

0.011

0.070

±

0.009

page_blocks

0.028

±

0.004

0.026

±

0.005

0.027

±

0.004

0.026

±

0.004

0.028

±

0.004

0.025

±

0.003

0.025

±

0.003

0.025

±

0.004

0.026

±

0.003

pen_recognition_handwritten_characters

0.004

±

0.001

0.004

±

0.001

0.007

±

0.001

0.007

±

0.002

0.008

±

0.002

0.007

±

0.001

0.007

±

0.001

0.006

±

0.001

0.011

±

0.001

phishing

0.031

±

0.002

0.029

±

0.002

0.029

±

0.003

0.032

±

0.003

0.034

±

0.002

0.031

±

0.002

0.032

±

0.002

0.031

±

0.002

0.032

±

0.003

polish_companies_bankruptcy_1year

0.018

±

0.003

0.019

±

0.001

0.026

±

0.003

0.026

±

0.003

0.028

±

0.002

0.021

±

0.002

0.022

±

0.003

0.021

±

0.002

0.027

±

0.003

polish_companies_bankruptcy_2year

0.017

±

0.002

0.017

±

0.001

0.041

±

0.002

0.041

±

0.002

0.041

±

0.002

0.025

±

0.003

0.025

±

0.003

0.026

±

0.003

0.035

±

0.002

polish_companies_bankruptcy_3year

0.023

±

0.002

0.025

±

0.002

0.041

±

0.004

0.038

±

0.002

0.046

±

0.003

0.031

±

0.003

0.033

±

0.003

0.033

±

0.003

0.040

±

0.003

polish_companies_bankruptcy_4year

0.029

±

0.003

0.030

±

0.002

0.051

±

0.002

0.048

±

0.004

0.053

±

0.002

0.031

±

0.002

0.036

±

0.002

0.035

±

0.002

0.048

±

0.002

polish_companies_bankruptcy_5year

0.037

±

0.002

0.040

±

0.002

0.064

±

0.004

0.056

±

0.004

0.066

±

0.002

0.031

±

0.004

0.033

±

0.003

0.036

±

0.004

0.050

±

0.005

seismic_bumps

0.068

±

0.009

0.065

±

0.009

0.066

±

0.009

0.065

±

0.009

0.065

±

0.008

0.070

±

0.007

0.066

±

0.009

0.067

±

0.007

0.068

±

0.008

skill_craft

0.589

±

0.010

0.582

±

0.014

0.601

±

0.012

0.574

±

0.019

0.597

±

0.015

0.600

±

0.014

0.607

±

0.013

0.610

±

0.014

0.593

±

0.014

smartphone_human_activity

0.045

±

0.004

0.035

±

0.007

0.042

±

0.004

0.064

±

0.006

0.071

±

0.004

0.035

±

0.003

0.035

±

0.005

0.036

±

0.004

0.079

±

0.004

smartphone_human_activity_postural

0.009

±

0.002

0.008

±

0.002

0.010

±

0.001

0.011

±

0.002

0.013

±

0.002

0.006

±

0.001

0.007

±

0.001

0.007

±

0.001

0.025

±

0.002

spambase

0.052

±

0.008

0.059

±

0.005

0.052

±

0.007

0.051

±

0.006

0.054

±

0.006

0.048

±

0.008

0.049

±

0.008

0.048

±

0.008

0.052

±

0.008

superconductivity_class

0.058

±

0.003

0.063

±

0.002

0.059

±

0.002

0.067

±

0.002

0.063

±

0.003

0.057

±

0.003

0.058

±

0.003

0.058

±

0.003

0.059

±

0.002

thyroid_all_bp

0.026

±

0.004

0.027

±

0.003

0.028

±

0.003

0.027

±

0.006

0.029

±

0.005

0.024

±

0.004

0.025

±

0.003

0.022

±

0.003

0.027

±

0.003

thyroid_all_hyper

0.015

±

0.002

0.016

±

0.003

0.017

±

0.003

0.014

±

0.002

0.018

±

0.003

0.014

±

0.003

0.014

±

0.003

0.014

±

0.003

0.015

±

0.003

thyroid_all_hypo

0.008

±

0.002

0.012

±

0.002

0.018

±

0.004

0.013

±

0.002

0.020

±

0.002

0.003

±

0.002

0.005

±

0.002

0.005

±

0.002

0.007

±

0.002

thyroid_all_rep

0.009

±

0.002

0.009

±

0.003

0.009

±

0.002

0.013

±

0.002

0.016

±

0.005

0.005

±

0.002

0.006

±

0.002

0.005

±

0.002

0.009

±

0.002

thyroid_ann

0.008

±

0.002

0.009

±

0.002

0.012

±

0.004

0.006

±

0.002

0.010

±

0.002

0.003

±

0.001

0.003

±

0.001

0.005

±

0.002

0.003

±

0.001

thyroid_dis

0.013

±

0.002

0.013

±

0.002

0.014

±

0.002

0.015

±

0.003

0.016

±

0.003

0.010

±

0.002

0.011

±

0.002

0.010

±

0.003

0.013

±

0.002

thyroid_hypo

0.014

±

0.003

0.014

±

0.003

0.014

±

0.003

0.014

±

0.005

0.016

±

0.003

0.009

±

0.004

0.010

±

0.002

0.012

±

0.003

0.012

±

0.003

0.016

±

0.003

0.020

±

0.003

0.022

±

0.004

0.019

±

0.004

0.029

±

0.003

0.011

±

0.003

0.018

±

0.004

0.011

±

0.003

0.017

±

0.005

thyroid_sick_eu

0.000

±

0.000

0.000

±

0.001

0.001

±

0.001

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

turkiye_student_evaluation

0.016

±

0.002

0.016

±

0.002

0.017

±

0.003

0.016

±

0.002

0.033

±

0.005

0.016

±

0.002

0.016

±

0.002

0.017

±

0.003

0.113

±

0.005

wall_follow_robot_2

0.002

±

0.001

0.008

±

0.003

0.009

±

0.002

0.004

±

0.001

0.008

±

0.002

0.002

±

0.001

0.004

±

0.002

0.002

±

0.001

0.001

±

0.001

wall_follow_robot_24

0.029

±

0.004

0.042

±

0.003

0.040

±

0.007

0.016

±

0.004

0.042

±

0.005

0.002

±

0.001

0.005

±

0.002

0.004

±

0.001

0.007

±

0.001

wall_follow_robot_4

0.002

±

0.001

0.009

±

0.003

0.012

±

0.003

0.007

±

0.002

0.011

±

0.003

0.002

±

0.001

0.004

±

0.002

0.003

±

0.001

0.001

±

0.001

waveform

0.141

±

0.004

0.140

±

0.004

0.147

±

0.008

0.145

±

0.004

0.145

±

0.006

0.146

±

0.005

0.149

±

0.006

0.147

±

0.005

0.148

±

0.005

waveform_noise

0.139

±

0.006

0.145

±

0.010

0.161

±

0.012

0.140

±

0.009

0.143

±

0.007

0.145

±

0.007

0.146

±

0.009

0.145

±

0.007

0.150

±

0.008

wilt

0.012

±

0.003

0.011

±

0.002

0.011

±

0.002

0.014

±

0.002

0.011

±

0.002

0.014

±

0.002

0.014

±

0.003

0.014

±

0.003

0.016

±

0.002

wine_quality_all

0.377

±

0.008

0.353

±

0.007

0.352

±

0.010

0.422

±

0.016

0.412

±

0.015

0.346

±

0.008

0.351

±

0.007

0.349

±

0.007

0.338

±

0.009

wine_quality_type

0.004

±

0.001

0.004

±

0.002

0.004

±

0.001

0.005

±

0.002

0.004

±

0.001

0.004

±

0.001

0.005

±

0.001

0.003

±

0.001

0.006

±

0.001

wine_quality_white

0.371

±

0.009

0.342

±

0.007

0.345

±

0.008

0.403

±

0.021

0.405

±

0.018

0.334

±

0.008

0.343

±

0.009

0.338

±

0.008

0.330

±

0.008

Dataset

RealMLP-TD

RealTabR-D

TabR-S-D

MLP-PLR-D

MLP-D

CatBoost-TD

LGBM-TD

XGB-TD

RF-D

ada

0.148

±

0.012

0.146

±

0.013

0.151

±

0.009

0.146

±

0.013

0.149

±

0.011

0.139

±

0.008

0.141

±

0.011

0.140

±

0.011

0.144

±

0.007

airlines

0.335

±

0.001

0.331

±

0.001

0.332

±

0.001

0.337

±

0.001

0.337

±

0.001

0.332

±

0.001

0.333

±

0.001

0.337

±

0.001

0.382

±

0.001

amazon-commerce-reviews

0.209

±

0.021

0.242

±

0.016

0.402

±

0.028

0.576

±

0.102

0.372

±

0.019

0.197

±

0.017

0.285

±

0.020

0.290

±

0.020

0.407

±

0.022

Bioresponse

0.238

±

0.010

0.228

±

0.013

0.228

±

0.013

0.236

±

0.009

0.232

±

0.006

0.205

±

0.010

0.204

±

0.008

0.211

±

0.007

0.205

±

0.009

car

0.008

±

0.006

0.013

±

0.009

0.011

±

0.006

0.013

±

0.006

0.011

±

0.006

0.019

±

0.009

0.021

±

0.006

0.019

±

0.004

0.071

±

0.017

christine

0.293

±

0.012

0.293

±

0.009

0.289

±

0.007

0.271

±

0.011

0.284

±

0.015

0.269

±

0.012

0.266

±

0.012

0.273

±

0.015

0.281

±

0.013

churn

0.044

±

0.003

0.050

±

0.005

0.055

±

0.006

0.046

±

0.004

0.065

±

0.007

0.050

±

0.004

0.048

±

0.005

0.048

±

0.004

0.064

±

0.005

cmc

0.465

±

0.019

0.457

±

0.019

0.449

±

0.025

0.452

±

0.014

0.441

±

0.015

0.460

±

0.017

0.457

±

0.017

0.467

±

0.018

0.472

±

0.014

cnae-9

0.068

±

0.010

0.055

±

0.008

0.066

±

0.011

0.065

±

0.008

0.053

±

0.011

0.076

±

0.010

0.308

±

0.017

0.091

±

0.012

0.087

±

0.012

connect-4

0.130

±

0.003

0.135

±

0.003

0.135

±

0.003

0.149

±

0.002

0.149

±

0.002

0.143

±

0.003

0.136

±

0.003

0.142

±

0.003

0.181

±

0.003

covertype

0.029

±

0.001

0.026

±

0.000

0.029

±

0.001

0.056

±

0.002

0.069

±

0.002

0.105

±

0.000

0.058

±

0.001

0.072

±

0.000

0.055

±

0.001

credit-g

0.257

±

0.017

0.250

±

0.015

0.252

±

0.025

0.256

±

0.023

0.269

±

0.017

0.250

±

0.018

0.252

±

0.019

0.255

±

0.013

0.256

±

0.025

Diabetes130US

0.402

±

0.003

0.399

±

0.003

0.401

±

0.003

0.396

±

0.002

0.400

±

0.003

0.383

±

0.002

0.398

±

0.002

0.455

±

0.005

0.398

±

0.002

dilbert

0.010

±

0.002

0.014

±

0.002

0.020

±

0.002

0.019

±

0.003

0.024

±

0.003

0.013

±

0.002

0.013

±

0.002

0.012

±

0.002

0.039

±

0.004

dionis

0.089

±

0.002

0.093

±

0.001

0.099

±

0.002

0.129

±

0.002

0.114

±

0.001

0.199

±

0.008

0.128

±

0.023

0.435

±

0.003

0.123

±

0.002

dna

0.044

±

0.004

0.050

±

0.005

0.063

±

0.007

0.056

±

0.005

0.056

±

0.006

0.046

±

0.003

0.040

±

0.004

0.041

±

0.003

0.050

±

0.005

fabert

0.312

±

0.009

0.314

±

0.008

0.354

±

0.009

0.367

±

0.010

0.370

±

0.009

0.285

±

0.006

0.386

±

0.008

0.299

±

0.006

0.317

±

0.008

Fashion-MNIST

0.097

±

0.001

0.101

±

0.002

0.106

±

0.002

0.115

±

0.002

0.109

±

0.002

0.099

±

0.001

0.091

±

0.001

0.092

±

0.002

0.122

±

0.002

gina

0.053

±

0.005

0.060

±

0.005

0.080

±

0.006

0.079

±

0.008

0.090

±

0.005

0.047

±

0.005

0.053

±

0.005

0.061

±

0.005

0.077

±

0.008

guillermo

0.175

±

0.004

0.219

±

0.006

0.271

±

0.010

0.210

±

0.007

0.243

±

0.006

0.165

±

0.004

0.171

±

0.003

0.179

±

0.004

0.197

±

0.005

helena

0.617

±

0.002

0.599

±

0.003

0.602

±

0.002

0.634

±

0.002

0.623

±

0.002

0.631

±

0.003

0.638

±

0.003

0.718

±

0.003

0.647

±

0.002

Higgs

0.250

±

0.001

0.248

±

0.001

0.255

±

0.001

0.261

±

0.002

0.253

±

0.001

0.257

±

0.001

0.259

±

0.001

0.260

±

0.001

0.271

±

0.001

Internet-Advertisements

0.024

±

0.003

0.026

±

0.004

0.026

±

0.004

0.026

±

0.005

0.026

±

0.005

0.024

±

0.005

0.025

±

0.003

0.025

±

0.005

0.020

±

0.003

0.273

±

0.002

0.262

±

0.002

0.271

±

0.002

0.276

±

0.002

0.291

±

0.002

0.282

±

0.002

0.282

±

0.002

0.285

±

0.002

0.302

±

0.002

jasmine

0.207

±

0.014

0.201

±

0.011

0.206

±

0.012

0.197

±

0.011

0.207

±

0.012

0.187

±

0.011

0.190

±

0.010

0.195

±

0.012

0.189

±

0.008

0.004

±

0.001

0.014

±

0.003

0.098

±

0.010

0.009

±

0.001

0.107

±

0.003

0.133

±

0.002

0.134

±

0.003

0.136

±

0.002

0.204

±

0.002

0.140

±

0.007

0.139

±

0.007

0.143

±

0.009

0.142

±

0.007

0.145

±

0.011

0.147

±

0.010

0.143

±

0.007

0.144

±

0.010

0.141

±

0.007

KDDCup99

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.002

±

0.000

0.000

±

0.000

0.000

±

0.000

kick

0.099

±

0.001

0.099

±

0.001

0.100

±

0.001

0.098

±

0.001

0.098

±

0.001

0.096

±

0.001

0.097

±

0.001

0.138

±

0.008

0.098

±

0.001

madeline

0.258

±

0.013

0.269

±

0.021

0.425

±

0.022

0.261

±

0.015

0.413

±

0.012

0.136

±

0.008

0.198

±

0.011

0.195

±

0.018

0.262

±

0.008

mfeat-factors

0.016

±

0.004

0.023

±

0.004

0.024

±

0.005

0.026

±

0.004

0.029

±

0.005

0.021

±

0.004

0.028

±

0.005

0.030

±

0.005

0.031

±

0.006

MiniBooNE

0.050

±

0.001

0.051

±

0.001

0.050

±

0.001

0.053

±

0.001

0.052

±

0.001

0.053

±

0.001

0.053

±

0.001

0.055

±

0.002

0.065

±

0.001

numerai28.6

0.479

±

0.004

0.414

±

0.003

0.421

±

0.002

0.481

±

0.002

0.480

±

0.002

0.480

±

0.003

0.481

±

0.003

0.483

±

0.004

0.489

±

0.003

okcupid-stem

0.253

±

0.004

0.246

±

0.003

0.247

±

0.003

0.248

±

0.004

0.249

±

0.004

0.243

±

0.003

0.246

±

0.003

0.410

±

0.016

0.262

±

0.003

pc4

0.095

±

0.012

0.105

±

0.014

0.101

±

0.019

0.098

±

0.009

0.094

±

0.008

0.099

±

0.012

0.099

±

0.013

0.100

±

0.013

0.104

±

0.012

philippine

0.284

±

0.011

0.268

±

0.009

0.305

±

0.012

0.271

±

0.008

0.301

±

0.008

0.249

±

0.012

0.251

±

0.011

0.253

±

0.009

0.254

±

0.010

phoneme

0.097

±

0.007

0.100

±

0.007

0.101

±

0.007

0.112

±

0.008

0.120

±

0.013

0.097

±

0.007

0.100

±

0.005

0.102

±

0.007

0.098

±

0.006

porto-seguro

0.038

±

0.000

0.038

±

0.000

0.038

±

0.000

0.038

±

0.000

0.038

±

0.000

0.038

±

0.000

0.038

±

0.000

0.038

±

0.000

0.038

±

0.000

qsar-biodeg

0.126

±

0.015

0.125

±

0.021

0.133

±

0.013

0.139

±

0.017

0.121

±

0.016

0.139

±

0.014

0.137

±

0.016

0.131

±

0.012

0.136

±

0.016

riccardo

0.002

±

0.000

0.002

±

0.001

0.004

±

0.001

0.011

±

0.001

0.006

±

0.001

0.003

±

0.001

0.003

±

0.001

0.003

±

0.001

0.048

±

0.003

0.488

±

0.008

0.522

±

0.006

0.574

±

0.004

0.544

±

0.023

0.579

±

0.007

0.487

±

0.006

0.464

±

0.006

0.471

±

0.008

0.570

±

0.008

Satellite

0.006

±

0.002

0.006

±

0.002

0.007

±

0.002

0.006

±

0.002

0.006

±

0.001

0.006

±

0.002

0.005

±

0.002

0.005

±

0.002

0.006

±

0.002

segment

0.077

±

0.009

0.074

±

0.010

0.071

±

0.007

0.080

±

0.009

0.082

±

0.008

0.069

±

0.008

0.071

±

0.006

0.070

±

0.006

0.072

±

0.006

shuttle

0.000

±

0.000

0.001

±

0.000

0.001

±

0.000

0.000

±

0.000

0.001

±

0.000

0.000

±

0.000

0.002

±

0.001

0.000

±

0.000

0.000

±

0.000

steel-plates-fault

0.241

±

0.019

0.225

±

0.015

0.232

±

0.011

0.227

±

0.017

0.250

±

0.014

0.223

±

0.014

0.223

±

0.011

0.220

±

0.011

0.241

±

0.013

sylvine

0.054

±

0.005

0.035

±

0.006

0.060

±

0.007

0.058

±

0.006

0.075

±

0.006

0.052

±

0.005

0.057

±

0.006

0.058

±

0.005

0.067

±

0.005

volkert

0.282

±

0.003

0.228

±

0.004

0.223

±

0.003

0.300

±

0.004

0.271

±

0.003

0.299

±

0.002

0.291

±

0.002

0.296

±

0.002

0.341

±

0.002

yeast

0.403

±

0.021

0.396

±

0.015

0.404

±

0.019

0.404

±

0.020

0.411

±

0.019

0.411

±

0.019

0.401

±

0.017

0.409

±

0.024

0.391

±

0.017

Dataset

RealMLP-HPO

MLP-PLR-HPO

ResNet-HPO

MLP-HPO

CatBoost-HPO

LGBM-HPO

XGB-HPO

ada

0.147

±

0.008

0.140

±

0.008

0.148

±

0.012

0.150

±

0.010

0.138

±

0.012

0.141

±

0.013

0.140

±

0.012

airlines

0.334

±

0.001

0.334

±

0.001

0.334

±

0.001

0.334

±

0.001

0.331

±

0.001

0.329

±

0.001

0.329

±

0.001

amazon-commerce-reviews

0.207

±

0.022

0.437

±

0.068

0.280

±

0.018

0.336

±

0.032

0.216

±

0.015

0.264

±

0.022

0.300

±

0.021

Bioresponse

0.219

±

0.010

0.233

±

0.010

0.225

±

0.008

0.229

±

0.011

0.209

±

0.010

0.206

±

0.012

0.204

±

0.011

car

0.004

±

0.003

0.013

±

0.007

0.016

±

0.013

0.012

±

0.007

0.017

±

0.008

0.017

±

0.010

0.022

±

0.007

christine

0.280

±

0.014

0.274

±

0.011

0.284

±

0.010

0.277

±

0.012

0.270

±

0.009

0.268

±

0.012

0.270

±

0.013

churn

0.042

±

0.003

0.045

±

0.003

0.059

±

0.007

0.054

±

0.006

0.048

±

0.004

0.048

±

0.005

0.048

±

0.005

cmc

0.472

±

0.022

0.456

±

0.034

0.450

±

0.027

0.447

±

0.020

0.471

±

0.021

0.470

±

0.016

0.454

±

0.018

cnae-9

0.079

±

0.020

0.066

±

0.010

0.064

±

0.012

0.053

±

0.011

0.066

±

0.009

0.079

±

0.013

0.095

±

0.015

connect-4

0.132

±

0.002

0.143

±

0.003

0.136

±

0.002

0.141

±

0.003

0.139

±

0.003

0.136

±

0.002

0.145

±

0.001

covertype

0.028

±

0.001

0.036

±

0.001

0.038

±

0.001

0.040

±

0.001

0.062

±

0.001

0.033

±

0.001

0.040

±

0.003

credit-g

0.262

±

0.023

0.276

±

0.022

0.272

±

0.028

0.271

±

0.018

0.234

±

0.024

0.268

±

0.027

0.248

±

0.017

Diabetes130US

0.395

±

0.002

0.392

±

0.002

0.398

±

0.003

0.401

±

0.003

0.384

±

0.003

0.390

±

0.002

0.387

±

0.002

dilbert

0.007

±

0.001

0.019

±

0.003

0.016

±

0.002

0.026

±

0.002

0.014

±

0.002

0.014

±

0.002

0.022

±

0.004

dionis

0.088

±

0.001

0.126

±

0.009

0.090

±

0.002

0.108

±

0.005

0.104

±

0.002

0.109

±

0.003

0.122

±

0.003

dna

0.043

±

0.005

0.056

±

0.008

0.046

±

0.003

0.054

±

0.006

0.043

±

0.004

0.040

±

0.003

0.040

±

0.003

fabert

0.309

±

0.006

0.343

±

0.014

0.363

±

0.011

0.367

±

0.006

0.286

±

0.006

0.298

±

0.007

0.303

±

0.007

Fashion-MNIST

0.093

±

0.003

0.107

±

0.002

0.103

±

0.002

0.105

±

0.002

0.097

±

0.002

0.091

±

0.001

0.094

±

0.002

gina

0.046

±

0.006

0.077

±

0.006

0.073

±

0.006

0.086

±

0.006

0.053

±

0.005

0.050

±

0.005

0.058

±

0.005

guillermo

0.165

±

0.002

0.202

±

0.006

0.228

±

0.006

0.242

±

0.005

0.170

±

0.002

0.167

±

0.002

0.169

±

0.003

helena

0.614

±

0.003

0.627

±

0.005

0.603

±

0.003

0.620

±

0.006

0.622

±

0.002

0.624

±

0.003

0.626

±

0.002

Higgs

0.247

±

0.002

0.252

±

0.001

0.244

±

0.001

0.252

±

0.001

0.258

±

0.001

0.255

±

0.001

0.257

±

0.001

Internet-Advertisements

0.024

±

0.002

0.021

±

0.004

0.024

±

0.004

0.025

±

0.004

0.025

±

0.005

0.025

±

0.004

0.026

±

0.004

0.269

±

0.002

0.278

±

0.004

0.279

±

0.003

0.287

±

0.002

0.281

±

0.002

0.278

±

0.002

0.279

±

0.002

jasmine

0.213

±

0.012

0.205

±

0.014

0.208

±

0.011

0.218

±

0.011

0.202

±

0.011

0.196

±

0.016

0.188

±

0.006

0.003

±

0.001

0.008

±

0.001

0.115

±

0.005

0.032

±

0.005

0.133

±

0.002

0.133

±

0.003

0.134

±

0.002

0.143

±

0.010

0.153

±

0.010

0.139

±

0.006

0.142

±

0.005

0.142

±

0.008

0.143

±

0.010

0.144

±

0.005

KDDCup99

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

kick

0.098

±

0.001

0.098

±

0.001

0.097

±

0.001

0.098

±

0.001

0.096

±

0.001

0.096

±

0.001

0.097

±

0.001

madeline

0.166

±

0.012

0.215

±

0.018

0.411

±

0.011

0.406

±

0.016

0.150

±

0.013

0.153

±

0.010

0.162

±

0.014

mfeat-factors

0.015

±

0.003

0.029

±

0.005

0.019

±

0.003

0.030

±

0.006

0.022

±

0.003

0.029

±

0.005

0.034

±

0.006

MiniBooNE

0.049

±

0.001

0.051

±

0.001

0.049

±

0.001

0.051

±

0.001

0.053

±

0.001

0.052

±

0.001

0.053

±

0.001

numerai28.6

0.479

±

0.004

0.479

±

0.003

0.481

±

0.003

0.480

±

0.003

0.480

±

0.003

0.479

±

0.004

0.481

±

0.002

okcupid-stem

0.250

±

0.004

0.247

±

0.003

0.248

±

0.003

0.247

±

0.003

0.242

±

0.003

0.245

±

0.003

0.254

±

0.004

pc4

0.103

±

0.009

0.111

±

0.017

0.093

±

0.011

0.103

±

0.009

0.096

±

0.012

0.103

±

0.015

0.104

±

0.014

philippine

0.273

±

0.015

0.272

±

0.010

0.301

±

0.010

0.296

±

0.010

0.250

±

0.008

0.241

±

0.010

0.245

±

0.007

phoneme

0.098

±

0.007

0.099

±

0.006

0.116

±

0.009

0.107

±

0.007

0.102

±

0.004

0.102

±

0.008

0.112

±

0.006

porto-seguro

0.038

±

0.000

0.038

±

0.000

0.038

±

0.000

0.038

±

0.000

0.038

±

0.000

0.038

±

0.000

0.038

±

0.000

qsar-biodeg

0.129

±

0.014

0.134

±

0.016

0.126

±

0.017

0.122

±

0.016

0.135

±

0.012

0.136

±

0.012

0.134

±

0.020

riccardo

0.002

±

0.001

0.002

±

0.000

0.005

±

0.001

0.005

±

0.001

0.003

±

0.001

0.003

±

0.001

0.003

±

0.001

0.478

±

0.007

0.496

±

0.007

0.544

±

0.006

0.555

±

0.008

0.486

±

0.008

0.467

±

0.005

0.475

±

0.008

Satellite

0.006

±

0.001

0.007

±

0.002

0.006

±

0.001

0.007

±

0.001

0.007

±

0.002

0.006

±

0.002

0.005

±

0.002

segment

0.079

±

0.007

0.081

±

0.008

0.077

±

0.007

0.082

±

0.009

0.070

±

0.007

0.072

±

0.008

0.073

±

0.006

shuttle

0.000

±

0.000

0.000

±

0.000

0.001

±

0.000

0.001

±

0.000

0.000

±

0.000

0.000

±

0.000

0.000

±

0.000

steel-plates-fault

0.239

±

0.014

0.244

±

0.010

0.247

±

0.012

0.248

±

0.014

0.222

±

0.012

0.223

±

0.008

0.232

±

0.017

sylvine

0.053

±

0.005

0.058

±

0.005

0.074

±

0.004

0.073

±

0.005

0.055

±

0.005

0.053

±

0.004

0.058

±

0.003

volkert

0.272

±

0.004

0.288

±

0.006

0.235

±

0.003

0.256

±

0.004

0.301

±

0.003

0.285

±

0.004

0.290

±

0.003

yeast

0.407

±

0.020

0.408

±

0.014

0.399

±

0.026

0.410

±

0.020

0.393

±

0.020

0.404

±

0.022

0.391

±

0.017

Dataset

RealMLP-HPO

MLP-PLR-HPO

ResNet-HPO

MLP-HPO

CatBoost-HPO

LGBM-HPO

XGB-HPO

airfoil_self_noise

0.174

±

0.011

0.210

±

0.011

0.329

±

0.015

0.233

±

0.017

0.227

±

0.011

0.241

±

0.015

0.248

±

0.012

Airlines_DepDelay_10M

0.979

±

0.000

0.980

±

0.001

0.982

±

0.000

0.983

±

0.000

0.980

±

0.001

0.980

±

0.001

0.982

±

0.001

Allstate_Claims_Severity

0.651

±

0.006

0.655

±

0.006

0.658

±

0.006

0.659

±

0.005

0.652

±

0.005

0.656

±

0.008

0.656

±

0.006

auction_verification

0.101

±

0.016

0.067

±

0.025

0.178

±

0.013

0.162

±

0.014

0.061

±

0.020

0.130

±

0.030

0.088

±

0.010

black_friday

0.686

±

0.003

0.687

±

0.001

0.690

±

0.002

0.693

±

0.002

0.679

±

0.002

0.679

±

0.001

0.681

±

0.001

brazilian_houses

0.788

±

0.345

0.742

±

0.109

1.623

±

1.223

0.606

±

0.147

0.711

±

0.369

0.878

±

0.340

0.565

±

0.079

Buzzinsocialmedia_Twitter

0.266

±

0.015

0.254

±

0.018

0.286

±

0.025

0.275

±

0.017

0.320

±

0.018

0.289

±

0.018

0.222

±

0.014

california_housing

0.413

±

0.008

0.427

±

0.008

0.426

±

0.010

0.435

±

0.008

0.402

±

0.007

0.398

±

0.008

0.400

±

0.008

concrete_compressive_strength

0.290

±

0.029

0.295

±

0.024

0.314

±

0.028

0.314

±

0.029

0.271

±

0.030

0.279

±

0.021

0.278

±

0.023

cps88wages

0.834

±

0.015

0.833

±

0.016

0.835

±

0.015

0.834

±

0.015

0.835

±

0.015

0.835

±

0.016

0.834

±

0.016

cpu_activity

0.125

±

0.005

0.124

±

0.004

0.127

±

0.003

0.137

±

0.006

0.122

±

0.003

0.122

±

0.007

0.119

±

0.006

diamonds

0.134

±

0.002

0.138

±

0.003

0.141

±

0.002

0.141

±

0.002

0.136

±

0.002

0.138

±

0.003

0.135

±

0.002

elevators

0.276

±

0.005

0.306

±

0.013

0.315

±

0.005

0.731

±

0.039

0.307

±

0.007

0.318

±

0.006

0.322

±

0.006

fifa

0.457

±

0.025

0.466

±

0.027

0.494

±

0.028

0.512

±

0.026

0.465

±

0.026

0.466

±

0.027

0.486

±

0.021

fps_benchmark

0.004

±

0.002

0.006

±

0.001

0.039

±

0.002

0.008

±

0.001

0.038

±

0.019

0.018

±

0.002

0.033

±

0.007

geographical_origin_of_music

0.899

±

0.038

0.934

±

0.033

0.919

±

0.039

0.906

±

0.030

0.871

±

0.017

0.869

±

0.024

0.861

±

0.020

health_insurance

0.775

±

0.004

0.775

±

0.005

0.777

±

0.006

0.775

±

0.005

0.775

±

0.005

0.775

±

0.004

0.774

±

0.005

house_16H

0.564

±

0.014

0.558

±

0.011

0.551

±

0.012

0.570

±

0.018

0.573

±

0.011

0.571

±

0.013

0.575

±

0.014

house_prices_nominal

0.399

±

0.051

0.378

±

0.031

0.445

±

0.063

0.384

±

0.046

0.361

±

0.022

0.383

±

0.030

0.374

±

0.039

house_sales

0.320

±

0.013

0.320

±

0.010

0.340

±

0.010

0.340

±

0.011

0.310

±

0.011

0.319

±

0.008

0.324

±

0.007

kin8nm

0.238

±

0.003

0.264

±

0.004

0.279

±

0.004

0.298

±

0.006

0.378

±

0.009

0.424

±

0.014

0.461

±

0.006

kings_county

0.327

±

0.010

0.317

±

0.008

0.341

±

0.013

0.339

±

0.009

0.309

±

0.008

0.323

±

0.008

0.325

±

0.007

Mercedes_Benz_Greener_Manufacturing

0.668

±

0.032

0.670

±

0.030

0.680

±

0.029

0.677

±

0.030

0.664

±

0.031

0.669

±

0.029

0.664

±

0.031

miami_housing

0.267

±

0.009

0.272

±

0.007

0.287

±

0.010

0.295

±

0.013

0.257

±

0.006

0.269

±

0.008

0.272

±

0.011

MIP-2016-regression

0.843

±

0.037

0.829

±

0.032

0.835

±

0.030

0.846

±

0.043

0.788

±

0.034

0.788

±

0.038

0.807

±

0.030

nyc-taxi-green-dec-2016

0.613

±

0.031

0.665

±

0.020

0.643

±

0.019

0.692

±

0.065

0.656

±

0.014

0.655

±

0.013

0.661

±

0.016

pol

0.059

±

0.004

0.065

±

0.005

0.169

±

0.006

0.130

±

0.006

0.119

±

0.004

0.106

±

0.005

0.108

±

0.006

pumadyn32nh

0.586

±

0.007

0.589

±

0.007

0.606

±

0.009

0.626

±

0.010

0.594

±

0.006

0.599

±

0.006

0.602

±

0.007

QSAR-TID-10980

0.598

±

0.014

0.636

±

0.010

0.614

±

0.008

0.612

±

0.011

0.596

±

0.011

0.582

±

0.010

0.590

±

0.010

QSAR-TID-11

0.514

±

0.016

0.552

±

0.015

0.524

±

0.017

0.527

±

0.011

0.526

±

0.014

0.509

±

0.015

0.517

±

0.015

quake

1.012

±

0.012

1.004

±

0.010

1.006

±

0.007

1.003

±

0.006

1.005

±

0.008

1.001

±

0.007

1.005

±

0.010

Santander_transaction_value

0.879

±

0.024

0.856

±

0.026

0.934

±

0.016

0.921

±

0.012

0.873

±

0.017

0.843

±

0.020

0.851

±

0.019

sarcos

0.102

±

0.002

0.110

±

0.002

0.109

±

0.002

0.113

±

0.003

0.136

±

0.002

0.128

±

0.002

0.132

±

0.003

SAT11-HAND-runtime-regression

0.444

±

0.053

0.489

±

0.035

0.477

±

0.031

0.464

±

0.038

0.515

±

0.032

0.501

±

0.029

0.531

±

0.041

socmob

0.383

±

0.054

0.299

±

0.041

0.459

±

0.055

0.412

±

0.080

0.364

±

0.054

0.404

±

0.052

0.417

±

0.054

solar_flare

1.017

±

0.089

0.981

±

0.067

0.975

±

0.069

0.975

±

0.066

0.984

±

0.046

0.972

±

0.054

1.009

±

0.156

space_ga

0.495

±

0.022

0.516

±

0.015

0.489

±

0.021

0.499

±

0.019

0.548

±

0.026

0.546

±

0.026

0.564

±

0.025

topo_2_1

0.968

±

0.005

0.972

±

0.005

0.970

±

0.005

0.968

±

0.004

0.970

±

0.004

0.964

±

0.004

0.968

±

0.007

video_transcoding

0.052

±

0.005

0.057

±

0.005

0.068

±

0.006

0.063

±

0.006

0.073

±

0.002

0.067

±

0.004

0.072

±

0.002

wave_energy

0.003

±

0.001

0.007

±

0.001

0.044

±

0.002

0.029

±

0.004

0.049

±

0.001

0.081

±

0.004

0.095

±

0.009

Yolanda

0.786

±

0.001

0.791

±

0.002

0.786

±

0.001

0.791

±

0.002

0.810

±

0.001

0.795

±

0.002

0.800

±

0.003

yprop_4_1

0.965

±

0.007

0.965

±

0.004

0.963

±

0.009

0.965

±

0.009

0.963

±

0.008

0.949

±

0.005

0.954

±

0.008

Dataset

RealMLP-TD

RealTabR-D

TabR-S-D

MLP-PLR-D

MLP-D

CatBoost-TD

LGBM-TD

XGB-TD

RF-D

0.348

±

0.003

0.350

±

0.001

0.349

±

0.001

0.346

±

0.001

0.348

±

0.002

0.347

±

0.002

0.347

±

0.002

0.363

±

0.005

0.353

±

0.001

bank-marketing

0.206

±

0.008

0.198

±

0.009

0.196

±

0.004

0.201

±

0.006

0.207

±

0.006

0.193

±

0.009

0.196

±

0.008

0.195

±

0.006

0.200

±

0.006

Bioresponse

0.240

±

0.011

0.233

±

0.007

0.236

±

0.011

0.249

±

0.013

0.240

±

0.008

0.228

±

0.013

0.227

±

0.010

0.229

±

0.006

0.232

±

0.008

california

0.114

±

0.003

0.090

±

0.004

0.092

±

0.003

0.113

±

0.003

0.122

±

0.003

0.095

±

0.002

0.097

±

0.002

0.097

±

0.003

0.111

±

0.002

compas-two-years

0.325

±

0.009

0.332

±

0.008

0.326

±

0.007

0.328

±

0.007

0.326

±

0.005

0.325

±

0.008

0.329

±

0.006

0.331

±

0.009

0.377

±

0.009

covertype

0.122

±

0.002

0.096

±

0.001

0.101

±

0.001

0.145

±

0.004

0.144

±

0.002

0.138

±

0.001

0.143

±

0.002

0.146

±

0.003

0.153

±

0.001

credit

0.227

±

0.005

0.224

±

0.005

0.226

±

0.007

0.224

±

0.007

0.225

±

0.006

0.223

±

0.007

0.226

±

0.007

0.227

±

0.008

0.235

±

0.005

default-of-credit-card-clients

0.281

±

0.005

0.284

±

0.005

0.286

±

0.005

0.284

±

0.004

0.285

±

0.006

0.286

±

0.005

0.284

±

0.005

0.287

±

0.005

0.292

±

0.003

Diabetes130US

0.397

±

0.002

0.396

±

0.001

0.397

±

0.001

0.396

±

0.001

0.396

±

0.001

0.395

±

0.001

0.399

±

0.002

0.398

±

0.001

0.438

±

0.001

electricity

0.170

±

0.007

0.154

±

0.015

0.110

±

0.007

0.161

±

0.002

0.170

±

0.004

0.117

±

0.002

0.115

±

0.001

0.111

±

0.001

0.137

±

0.002

heloc

0.284

±

0.007

0.284

±

0.009

0.284

±

0.009

0.276

±

0.008

0.282

±

0.009

0.281

±

0.008

0.282

±

0.007

0.284

±

0.006

0.283

±

0.006

Higgs

0.288

±

0.002

0.292

±

0.002

0.307

±

0.001

0.288

±

0.001

0.310

±

0.002

0.290

±

0.001

0.289

±

0.001

0.294

±

0.002

0.300

±

0.001

house_16H

0.116

±

0.004

0.119

±

0.002

0.115

±

0.003

0.113

±

0.003

0.116

±

0.005

0.116

±

0.004

0.115

±

0.004

0.118

±

0.004

0.121

±

0.003

0.222

±

0.002

0.226

±

0.001

0.257

±

0.002

0.221

±

0.001

0.249

±

0.002

0.222

±

0.001

0.224

±

0.001

0.230

±

0.001

0.235

±

0.001

MagicTelescope

0.137

±

0.004

0.131

±

0.004

0.130

±

0.003

0.136

±

0.004

0.139

±

0.004

0.136

±

0.004

0.139

±

0.005

0.138

±

0.005

0.144

±

0.004

MiniBooNE

0.063

±

0.001

0.065

±

0.001

0.067

±

0.001

0.066

±

0.001

0.064

±

0.001

0.064

±

0.001

0.063

±

0.001

0.064

±

0.001

0.078

±

0.001

pol

0.012

±

0.002

0.014

±

0.001

0.032

±

0.002

0.017

±

0.002

0.037

±

0.004

0.014

±

0.001

0.015

±

0.001

0.014

±

0.001

0.016

±

0.002

road-safety

0.233

±

0.002

0.232

±

0.002

0.230

±

0.001

0.238

±

0.004

0.241

±

0.002

0.233

±

0.001

0.237

±

0.001

0.242

±

0.001

0.243

±

0.001

E Broader Impact

We present NN models with an improved speed-accuracy tradeoff and hope that this can reduce the resource consumption of tabular models in applications and further benchmarks. While tabular ML has many potential applications, we feel that none must be particularly highlighted here.

NeurIPS Paper Checklist

1. Claims

Answer: [Yes]

Justification: See the paper.

Guidelines:

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See the paper.

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide a paragraph on limitations in Section 5.

Guidelines:

2. Limitations

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors are encouraged to create a separate "Limitations" section in their paper.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

3. Theory Assumptions and Proofs

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Justification: We do not have theoretical results.

Answer: [NA]

Guidelines:

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

The answer NA means that the paper does not include theoretical results.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Theorems and Lemmas that the proof relies upon should be properly referenced.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

4. Experimental Result Reproducibility

Justification: We provide implementation details in the paper and the appendix, especially Appendix C. We cannot provide all details on dataset preprocessing, but these are provided with the code.

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes]

Guidelines:

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

The answer NA means that the paper does not include experiments.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

5. Open access to data and code

Answer: [Yes]

Guidelines:

Justification: We provide code for the meta-train and meta-test benchmarks in the supplementary material. For the camera-ready version, we will provide more complete documentation, the code for the Grinsztajn et al. [18] benchmark, and the experimental data.

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Guidelines:

Justification: These details are provided in the paper and appendix.

The answer NA means that the paper does not include experiments.

The full details can be provided either with the code, in appendix, or as supplemental material.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Guidelines:

Justification: We provide error bars for fixed datasets, quantifying the uncertainty over the random splits, as described in the appendix. We also provide critical-difference diagrams in Appendix B.11.

The answer NA means that the paper does not include experiments.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

The assumptions made should be given (e.g., Normally distributed errors).

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Justification: We did not track these resources in detail, but a rough estimate for the total resources can be found in Appendix C.8.

Answer: [No]

Guidelines:

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The answer NA means that the paper does not include experiments.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

9. Code Of Ethics

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Justification: The research appears to conform to the code of ethics but we did not check all ≈ 200 datasets used in this paper.

Answer: [NA]

Guidelines:

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

10. Broader Impacts

Answer: [Yes]

Guidelines:

Justification: See Appendix E, but this work is foundational research and the impact is unclear.

The answer NA means that there is no societal impact of the work performed.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Justification: We only use publicly available tabular datasets without obvious safety risks and do not release pretrained models.

Answer: [NA]

Guidelines:

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

The answer NA means that the paper poses no such risks.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

12. Licenses for existing assets

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Justification: We use ≈ 200 datasets from online repositories, so we cite the repositories / benchmark curators but not the individual datasets.

Answer: [No]

Guidelines:

The authors should cite the original paper that produced the code package or dataset.

The answer NA means that the paper does not use existing assets.

The authors should state which version of the asset is used and, if possible, include a URL.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Justification: We only provide download code for existing datasets.

Answer: [NA]

Guidelines:

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The answer NA means that the paper does not release new assets.

The paper should discuss whether and how consent was obtained from people whose asset is used.

14. Crowdsourcing and Research with Human Subjects

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Guidelines:

Justification:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce RealMLP, an improved multilayer perceptron (MLP), alongside improved default parameters for GBDTs and RealMLP. The authors tune RealMLP on a meta-train benchmark with 71 classification and 47 regression datasets and compare them to hyperparameter-optimized versions on a disjoint meta-test benchmark with 48 classification and 42 regression datasets, as \nwell as the GBDT-friendly benchmark by Grinsztajn et al. (2022), claiming that RealMLP offers a better time-accuracy tradeoff than other neural nets and is competitive with GBDTs.\n\nSTRENGTHS:\n1. The inclusion of the standard recent benchmark of Grinsztajn et al. (2022) is welcome, and gives more credence to the authors' claims.\n2. The description of the authors' improvements in Sec. 3 is unusually good, containing sufficient detail that, if needed, someone could most likely recreate the method from scratch.\n3. The goal of improving the performance of MLPs on tabular data is both useful and interesting.\n\nWEAKNESSES:\nMAJOR\n\n1. The support for a key claim (\"RealMLP offers a better time-accuracy tradeoff than other neural nets and is competitive with GBDTs\") is, at best, inconsistent. In Fig. 2, utilizing the authors' own benchmark suite, RealMLP is consistently less performant than the best GBDT. The claim that it outperforms other neural nets is not strongly supported by Fig. 2, since only a handful of the baseline NN methods from [1] are included. Particularly notable is the exclusion of [2], despite the fact that the authors reference it in their prior work. [2] requires less than a second to train and there exist methods in the literature to extend it to datasets of arbitrary size. As for Fig. 3, it seemingly contains no confidence intervals. However, the authors' method appears to perform comparably to FT-Transformer and Tab-R for classification tasks; on regression tasks, it is faster, but less performant, than FT-Transformer.\n2. Sec 2.1 explains how the new benchmark was assembled, but not why it was necessary to do so. [1], [2] and Grinsztajn et al. have already introduced robust tabular benchmark suites into the literature. The authors also mention, but utilize only parts of, the AutoML benchmark and the OpenML-CTR23 regression benchmark.\n3. While the concerns raised in Sec. 2.2 about some previously used metrics are reasonable, the choice of shifted geometric mean error, which adds noise to extremely low error scores, is also problematic. A better approach would have been to report multiple metrics.\n\nMINOR\n\n1. The 95% confidence intervals in the figures are very difficult to see, and should be made clearer.\n\n[1] When do neural nets outperform boosted trees on tabular data?\n[2] TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second\n\nQUESTIONS:\nQUESTIONS\n\n* Why was it necessary to create a new benchmark suite? Why is this the fairest choice of benchmark for these methods?\n* Why report only one non-standard metric?\n* Why are so few deep tabular methods included in the core experiments?\n\nSUMMARY\n\nWhile the detailed description of the authors method is helpful and the ideas contained therein could be of interest to the NeurIPS audience, the experimental results are not yet robust enough to warrant publication at NeurIPS.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper \"Better by default: Strong pre-tuned MLPs and boosted trees on tabular data\", the authors make two major contributions: (i) they propose a multi-layer perceptron configurations that is tuned on a set of training datasets and (ii) investigate in a large scale empirical study how different default parameterizations (library vs tuned on a set of training datasets) compares to optimized hyperparameters on a set of test datasets. The study reveals interesting results regarding the performance of default parameterizations set in standard ML libraries, better default parameterizations exist (at least for the scope of the benchmark), and that the proposed MLP often performs competitive to gradient-boosted decision trees when it comes to a Pareto-optimal tradeoff between accuracy and training time.\n\nSTRENGTHS:\n- The paper is very well written and easy to follow. Sufficient details on the architecture of RealMLP and the implementation of tuning the default parameterization or optimizing the hyperparameters are given.\n- Generally speaking, the authors are very eager to provide all the details to make their work reproducible which is of high importance for such a paper.\n- In principle, I like the 2D-comparisons but a summary plot how the D vs TD vs HPO behaves for the different methods would be very much appreciated.\n\nWEAKNESSES:\n- Intuitively one would expect that the performance of TD lies in between D and HPO but I could also imagine that TD is oftentimes very close to HPO or might even exceed HPO's performance due to overtuning. But this comparison is not considered as an overview.\n- It is not clear why random search has been used as HPO technique in the paper, especially, since the authors even mention more sophisticated techniques such as SMAC that are based on Bayesian optimization.\n- The style of writing in Section 3 could be improved as there are various content-wise enumerations in the form of sentences starting with \"We\".\n\nQUESTIONS:\n- How would a CASH approach compare here if we would first select the learning algorithm according to the (tuned) default parameterization and then optimize its hyperparameters? As it is done in Mohr, F., Wever, M. Naive automated machine learning. Mach Learn 112, 1131–1170 (2023). https://doi.org/10.1007/s10994-022-06200-0\n- Why was random search used for hyperparameter optimization?\n- Is there any intuition why using the last of tied epochs as the final model as stated on line 161?\n- Why are hyperparameter values rounded as stated on line 170?\n\nLIMITATIONS:\nLimitations have properly been described in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work investigates/provides better default hyperparameter configurations for MLPs and gradient-boosted decision trees. Additionally, it proposes an augmented neural network with a series of enhancements. Experimental results are provided showing that the neural network is on par with gradient-boosted decision trees. Moreover, the authors argue that the neural network and the gradient-boosted decision trees with the new defaults are competitive with their HPO-tuned per-dataset counterparts and motivate practitioners to use the different methods with the new defaults and perform ensembling.\n\nSTRENGTHS:\n- The work is written well.\n- The work uses a wide collection of benchmarks that are well-known in the community.\n- The work includes results with default, tuned-per-benchmark, and tuned-per-dataset hyperparameter configurations for the main methods investigated.\n\nWEAKNESSES:\n- The novelty of the paper is limited in my opinion. The majority of the components that are proposed in Figure 1. C are not novel and are used in AutoML methods. Moreover, it is not clear what subset of the proposed components could actually benefit the other methods and not only the MLP. \n- The baselines used are not consistent, for example, FT-Transformer and SAINT are included for the Grinsztajn et al. benchmark, but not for the other benchmarks.\n- The protocol is not fully consistent, as for example, for the results of the meta-train and meta-test benchmark, HPO is not applied for the ResNet method or Tab-R.\n- Figure 2 shows results for accuracy, where RealMLP-TD strongly outperforms gradient-boosted decision tree methods for the meta-test classification and regression benchmark. However, AUROC is a metric that considers class imbalance and provides a better view of the method performances, Figure B.3 which reports results for AUROC shows a different picture, where, gradient-boosted decision trees outperform the RealMLP-TD. As such, I would propose to place Figure B.3 in the main paper and Figure 2 in the Appendix.\n\nQUESTIONS:\n- Could the authors provide results for the missing baselines and for the baselines that do not feature HPO results?\n- How does this work fair in comparison to using a portfolio of configurations using dataset meta-features and selecting the default configuration per dataset based on a similarity distance between the portfolio of configurations and the context dataset?\n\nLIMITATIONS:\n- I believe the authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces RealMLP, an enhanced Multilayer Perceptron (MLP) designed for classification and regression tasks on tabular data. It also proposes optimized default parameters for both RealMLP and Gradient Boosted Decision Trees (GBDTs). Through benchmarking on diverse datasets, the authors demonstrate that RealMLP achieves a superior balance between time efficiency and accuracy compared to other neural networks, while remaining competitive with GBDTs. The integration of RealMLP and GBDTs using optimized defaults shows promising performance on medium-sized tabular datasets, alleviating the need for extensive hyperparameter tuning.\n\nSTRENGTHS:\n1. The study includes comprehensive benchmarking on a meta-train dataset comprising 71 classification and 47 regression datasets, and a meta-test dataset comprising 48 classification and 42 regression datasets.\n\n2. Detailed experimental settings and hyperparameters are explicitly specified in the paper and appendices, ensuring reproducibility.\n\n3. The experiments establish statistical significance, presenting error bars and critical-difference diagrams.\n\n4. RealMLP demonstrates superior efficiency by offering a better time-accuracy tradeoff compared to other neural networks.\n\n5. The optimized combination of RealMLP and GBDTs with improved default parameters achieves outstanding performance without the need for hyperparameter tuning.\n\nWEAKNESSES:\n1. The finding presented in this work is not entirely novel. Similar conclusions have been reached by Tree-hybrid MLPs `[1]`, although the methods are not quite the same. Moreover, ensemble different methods, including combinations like MLP + GBDTs, have long been recognized as effective approaches in Kaggle competitions and do not necessitate rediscovery.\n\n2. While mentioning good performance by default, it's worth noting that Excelformer achieves one of the best performances under default settings. However, a direct comparison was not included, which would enhance the clarity of the findings. Please include the comparison in the rebuttal.\n\n3. Despite emphasizing the applicability to medium-sized datasets, practical applications often involve diverse tabular data with varying feature lengths and training data sizes. The small application scope could potentially limit the impact of this study. Besides, most of tabular data sizes are often small.\n\n4. The compared methods are evasive. Since the authors want to claim the speed-performance balance, the effective approaches Net-DNF `[2]`, TabNet `[3]`, and TabCaps `[4]` are not compared. The compared models FT-Transformer and SAINT, especially SAINT, are heavy.\n\n**REF**\n\n`[1]` Team up GBDTs and DNNs: Advancing Efficient and Effective Tabular Prediction with Tree-hybrid MLPs\n\n`[2]` Net-DNF: Effective Deep Modeling of Tabular Data\n\n`[3]` Tabnet: Attentive interpretable tabular learning\n\n`[4]` Tabcaps: A capsule neural network for tabular data classification with bow routing\n\nQUESTIONS:\nThe effective default performance achieved by MLP + GBDTs may not be suitable for all scenarios, such as those requiring tabular pre-training or zero-shot scenarios, where pure tabular models might be more appropriate. How does the proposed method address applications in tabular pre-training scenarios?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an enhanced version of the tabular MLP model -- RealMLP. By using multiple tricks over simple MLP, the proposed method becomes much more competitive with GBDTs than a simple MLP. Moreover, the authors provide strong \"tuned default\" configurations for GBDTs and RealMLP. These configs considerably reduce training time in exchange for arguably small test error increase compared to hyperparameter optimization procedures.\n\nSTRENGTHS:\n* The paper addresses two important problems in tabular DL:  1) lack of an efficient strong DL baseline  2) high cost of extensive hyperparameter tuning for a good performance\n* Extensive evaluation of the proposed method. Appendix A and B are also provide important ablations on RealMLP. \n* Great experimental setup in terms of meta-train and meta-test splits that allow to find tuned-defaults.\n* RealMLP is considerably better than MLP. \n* Tuned-defaults for GBDTs.\n* Everything necessary to reproduce the results is provided.\n\nWEAKNESSES:\n1. One of the main contributions of the paper is the different DL techniques from Figure 1 that have been added to MLP. Although there are enough useful experiments and ablations on these techniques for RealMLP (Figure 1 and Appendix B), these tricks can be applied to any DL model. From Figure 3 we observe that TabR-S-D and FT-Transformer-D are much better than MLP-D and slightly worse than RealMLP-TD. So, I think it is essential to ablate the mentioned techniques to improve these models and compare the results with RealMLP. I expect that the techniques improve stronger models much less compared to MLP, and it would be a beneficial result. It will show that RealMLP-TD is a good baseline while TabR-S-D-with-tricks is only slightly better than TabR-S-D. Otherwise, if improved TabR or FT-T considerably outperform RealMLP, the results will be even closer to GBDTs' which is also a positive outcome. I think that the results in Table B.5, which show an advantage of RC+SF preprocessing for TabR-S-D , also support my idea. \n2. Table B.1. from Appendix: changing Adam's $\\beta_2$ from 0.95 to 0.999 significantly increases the error rate while changing Adam's $\\beta_2$ from 0.999 to 0.95 on Figure 1 has almost no effect. I would make a conclusion that the resulting RealMLP-TD architecture with all tricks is very sensitive to hyperparameters. \n3. Also Table B.1: Strangely, PLR embedding is much worse than PL one. The only difference is ReLU if I am not mistaken.\n4. No non-parametric DL baselines in Figure 2. Only TabR-S-D is presented. Also, it would be interesting to compare tuned MLP-PLR [1]. Authors from [1] use extensive HPO and do not fix embedding size, so a comparison with RealMLP-TD would be interesting to see if a properly tuned MLP-PLR is better/worse than RealMLP with tricks and fixed embedding size.\n\nQUESTIONS:\n1. Please, see weaknesses. \n2. Probably I missed it in the text but how many HPO iterations have you used to obtain -TD configs?\n3. Have you measured inference efficiency? I think MLP is usually a bit faster than GBDTs but I wonder if parametric activations or proposed numerical embeddings make MLP slower.\n\nLIMITATIONS:\nLimitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}