Better by default: Strong pre-tuned MLPs and boosted trees on tabular data
Authors: David Holzmüller
Leo Grinsztajn
Ingo Steinwart
Venue: NeurIPS 2024 poster (2024)
Keywords: tabular data
benchmark
default parameters
neural networks
deep learning
multilayer perceptron
gradient-boosted decision trees
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We propose better default parameters for boosted decision trees and improved neural networks on tabular data, evaluate them on separate large benchmarks, and show that they can achieve excellent results with moderate runtime.

Abstract: For classification and regression on tabular data, the dominance of gradient-boosted decision trees (GBDTs) has recently been challenged by often much slower deep learning methods with extensive hyperparameter tuning. We address this discrepancy by introducing (a) RealMLP, an improved multilayer perceptron (MLP), and (b) strong meta-tuned default parameters for GBDTs and RealMLP. We tune RealMLP and the default parameters on a meta-train benchmark with 118 datasets and compare them to hyperparameter-optimized versions on a disjoint meta-test benchmark with 90 datasets, as well as the GBDT-friendly benchmark by Grinsztajn et al. (2022). Our benchmark results on medium-to-large tabular datasets (1K--500K samples) show that RealMLP offers a favorable time-accuracy tradeoff compared to other neural baselines and is competitive with GBDTs in terms of benchmark scores. Moreover, a combination of RealMLP and GBDTs with improved default parameters can achieve excellent results without hyperparameter tuning. Finally, we demonstrate that some of RealMLP's improvements can also considerably improve the performance of TabR with default parameters.

Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data
David Holzmüller* SIERRA Team, Inria Paris
Léo Grinsztajn SODA Team, Inria Saclay
Ecole Normale Superieure PSL University
Ingo Steinwart
University Stuttgart
Faculty of Mathematics and Physics Institute for Stochastics and Applications
Abstract
For classification and regression on tabular data, the dominance of maadient-booster decision trees (GBDTs) has recently been challenged by often much slower deep learning methods with extensive hyperparameter tuning. We address this dis- crepancy by introducing (a) RealMLP, improved multilayer perceptron (MLP), and (b) strong meta-tuned default parameters for GBDTs and ReaIMLP. We tune RealMLP and the default parameters meta-train benchmark with 118 datasets and compare them to hyperparameter-opiiiine benchmark with 90 datasets, as well as the GBDT-friendly benchmark by Grin- sztajn et al. (2022). Our benchmark results on medium-to-lee (1K-500K samples) show that RealMLP offers favorable acceracy tradeoff compared to other neural baselines and is competitive with GBDTs in terms of benchmark scores. Moreover, a combination of RealMLP and GBDTs with im- proved default parameters can achieve excellent results without hyperparameter tuning. Finally, we demonstrate that some of RealMLPPss improvements can also considerably improve the performance of TabR with default parameters.
versions on disjoint meta-test
tabular datasets
Introduction
Perhaps the most common type of data in practical machine learning (ML) is tabular data, char- acterized by fixed number of features (columns) that take different types such as numerical or categorical, as well as a lack of the spatiotemporal structure found in image or text data. The moderate dimension and lack of symmetries make tabular data accessible learning methods. Although tabular data is very diverse and no method is dominant on all datasets, gradient-boosted decision trees (GBDTs) exhibit excellent results on benchmarks [[8]
wide variety of machine
58
although their superiority has been challenged by variety of deep learning methods B
While many architectures for neural networks (NNs) have been proposed 33) variants of the simple multilayer perceptron (MLP) have repeatedly been shown to be good baselines for tabular NNs
15 16 30 54] Moreover, in terms of training time, MLPs are often slower than GBDTs but still considerably faster than many other architectures [88] 43 Therefore, we study how MLPs can be improved in terms of architecture, training, preprocessing, hyperparameters, and initialization. We also demonstrate that at least some of these improvements can successfully improve TabR [17
Work done partially while still at University of Stuttgart.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Even with fast and accurate NNs. the cost of extensive hyperparameter optimization can be problem- atic and hinder the adoption of new methods. To address this issue, we investigate the potential of better dataset-independenn default parameters for MIPs and GBDTs. Specifically, we compare the library defaults (D) to our tuned defaults (TD) and (dataset-dependent) hyperparameter optimization (HPO). Unlike McElfresh et al. [3]. who argue in favor HPO GBDTs over trying NNs, our results show a better aieeaccuuacyy trade-off for trying different (tuned) default models, as done by modern AutoML systems [0
1.1 Contribution
The problem of finding better default parameters can be seen meta-learning problem [64]. We employ a meta-train benchmark consisting of 118 datasets on which the default hyperparameters are optimized, and a disjoint meta-test benchmark consisting of 90 datasets on which they evaluated. We consider separate default parameters for classification, optimized for classification error, and for regression, optimized for RMSE. Our benchmarks do not contain missing numerical values, and restrict ourselves to sizes between IK and 500K samples, cf. [Section 2
In Section 3.
we introduce ReaLMLP, which improves standard MLPs through bag of tricks and better default parameters, tuned entirely on the meta-train benchmark. We introduce many novel or nonstandard components, such as preprocessing using robust scaling and smooth clipping, a new numerical embedding variant, a diagonal weight layer, new schedules, different initialization methods, etc. Our benchmark results demonstrate that it often outperforms other comparably fast NNs from the literature and can he competitive with GBDT's To demonstrate that our bag of tricks is useful for other models, introduce RealTabR-D, version of TabR [17 including some of our tricks that, despite less extensive tuning, achieves excellent benchmark results.
In Section 4 we provide new default parameters, tuned the meta-train benchmark, for XGBoost 9 LightGBM (31, and CatBoost [51) While they cannot match HPO on average, they outperform the library defaults on the meta-test benchmark.
In Section 5] we evaluate these and other models on the meta-test benchmark and the benchmark by Grinzttjjn et al. [18]. We also investigate several possibilities for algorithm selection and ensembling, demonstrating that algorithm selection over default methods provides better iine-pefformnnce tradeoff than HPO, thanks to our new improved default parameters and ML..
The code for our benchmarks, including scikit-learn interfaces for the models, is available at
Our code and data are archived at https::///// httt:::://doiorree 10. htt.:/o.oorrr1...188//aaaaassss
1.2
Related Work
Neural networks Borisov et al. 3 review deep learning on tabular data and identify three main classes of methods: Data transformation methods, specialized architectures, and regularization models. In particular, recent research has mainly focused on specialized architectures based on
attention 77 27], including attention between datapoints [17 37
53
56, 60). However, these Our research
methods are usually significantly slower than MLPs or even GBDTss 17 18
instead expands on improvements to MLPs for tabular data such as the SELU activation function (35, bias initialization methods [61], regularization methods (30], categorical embedding layers [19. and numerical embedding layers [16].
Benchmarks Shwartz-Zii Ziv and Armon 58 benchmarked three deep learning methods and noticed that they performed better the datasets from their own papers than on other datasets. We address this issue by using more datasets and evaluating our methods on datasets that they were not tuned on. Grinsztajn et al. [18), McElfresh et al. 43 and Ye et al. [69 propose larger benchmarks and find that GBDT's still outperform deep learning methods on average, analyzing why and when this is the case. Kohli et al. [36] also emphasize the need for large benchmarks. We evaluate our methods on the benchmark by Grinsztajn et al. [18] as well as datasets from the AutoML benchmark [3 and the
OpenML-CTR23 regression benchmark 12
Better defaults Probst et al. 50 study the tunability of ML. methods, i.e., the difference in benchmark scores between the best fixed hyperparameters and tuned hyperparameters. While their
2

---

Table I: Characteristics the meta-train and meta-test sets,
cluss
Blest classs
@Grinsztajn class
Btrain reg
B
Brrisstaaja reg
#datasets
71
48
18
47
42
28
#dataset groups
46
48
18
26
42
28
min #samples
1847
1000
3434
3338
1030
4052
max #samples
45222
500000
500000
48204
500000
500000
max #classes
26
355
2
0
0
max #features
561
10000
419
520
4991
359
max #categories
41
7019
14
38
359
20
approach involves finding better defaults, they do evaluate them on separate meta-test benchmark, only consider classification, and do not provide defaults for LightGBM, CatBoost, and NNs.
Meta-learning
The problem of finding the best fixed hyperparameters is meta-learning problem 4 [4]. Although we do not introduce or employ fully automated method to find good defaults, we use meta-learning benchmark setup to properly evaluate them. Wistuba al. [66] and Pfisterer et al. 49 learn portfolios of configurations and van Rijn et al. 63 learn symbolic defaults. but neither these papers considers GBDT's or NNs. Salinas and Erickson [55]
learn large portfolios of
extensive benchmark, without studying the best defaults for individual model families. Such portfolios are successfully applied in modern AutoML methods [IO. At the other end of the meta-learning spectrum, TabPFN [23] meta- small synthetic datasets. Unlike TabPFN, we only meta-learn hyperparameters and can therefore fewer but larger and more realistic eeta-train datasets, resulting in methods that scale to larger datasets.
configurations
(tuning-freee freee) learning method on
2 Methodology
To evaluate fixed hyperparameter configuration H. we need collection Btrain of benchmark datasets and a scoring function that computes a benchmark score S(Burain H) by aggregating the errors attained by the method with hyperparameters H. each dataset. However, when optimizing H on Btrain, we might overfit to the benchmark and therefore ideally need a second benchmark Brsst to get an unbiased score for H. We refer to Btrain Bustt meta-train and meta-test benchmarks and subdivide them into classification and regression benchmarks gtrain rog class and Blest. We
Blest
also use the Grinsztajn et al. [18] benchmark BGrinsztajn, which allows us to run more expensive baselines, since limits training set sizes to 10K samples contains fewer datasets due to more strict dataset inclusion criteria. Since gtrsin contains groups of datasets that are variants of the same dataset, for example by using different columns as targets, we use weighting factors inversely proportional to the group size.
Table I shows some characteristics of the considered benchmarks. The meta-test benchmark includes datasets that are more extreme in several dimensions, allowing us to test whether our default parame- ters generalize "out of distribution". For all datasets, we remove rows with missing numerical values and encode missing categorical values as separate category.
2.1 Benchmark Data Selection
The meta- veta-train set consists of medium-siee sized datasets from the UCI Repository 32). adapted from Steinwart [61] The meta-test set consists of the datasets from the AutoML Benchmark [13] as well as the OpenML-CTR23 regression benchmark [12] with few modifications: we subsample some large datasets and remove datasets that are already contained in the meta-train set, are too small, or have categories with too large cardinality. More details on the datasets and preprocessing can be found in Appendix C.3

---

2.2 Aggregate Benchmark Score
To optimize the default parameters, we need define a single benchmark score. To this end, we evaluate method on Naplits = 10 random valiiag-iiiiiiit-testt splits (60%-20%% on each dataset. As metries on individual dataset splits, we use classification error (100% accuracy) or 1 I-AUROC(one--s-res)) for classification and
RMSE
nRMSE standard deviation of targets
/1 R
for regression. There are various options to aggregate these errors into single score. Some, such as average rank mean normalized error, depend on which other methods are included in the evaluation, hindering an independent optimization. We would like to use the geometric mean error because arguably, error reduction from 0.02 to 0.01 is more valuable than an error reduction from 0.42 to 0.41. However, since the geometric mean error is too sensitive to cases with zero error (especially for classification error), instead use shifted geometric mean error, where small value 0.01 is added to the errors erty before taking the geometric mean:
Nextaents
Naplita
SGM, == exp
Wi
log(erri + +e)
Naplits
Here, we use weights w, 1/Naatasets on the meta-test set and Grinsztajn et al. [18 benchmark. On the meta-train set, we make the w; dependent on the number of related datasets, cf. Appendix C.3 In Appendixx B.10, we present results for other aggregation strategies.
3 Improving Neural Networks
The following section presents RealMLP-TD. our improved MLP with tuned defaults, which was designed based on experiments on the meta-train benchmark. A simplified version called RealMI.P. TD-S is also described. To demonstrate that our improvements can be useful for other architectures. we introduce RealTabb-DD, a version TabR that includes some of our improvements but has not been tuned extensively as RealMLP-TD.
Data preprocessing In the first step of RealMLP, we apply one-hot encoding to categorical columns with at most eight distinct values (not counting missing values). Binary categories are encoded to single feature with values {-1, 1,,1,, 1). Missing values in categorical columns are encoded zero. After that, all numerical columns, including the one-hot encoded ones, are preprocessed independently as follows: Let .e,., E be the values in column i, and let 9p the p-quantile of (.1, ..,.......0 T.) for p [0, 1]. Then,
ppproceeeed
f(s) (x, 91/2)), f(x)
VT+ (5)
43/4
+ 91/4
98/4-91/4 91/4
8j
43/4 41/4 and @1 + q0
otherwise.
In scikit-learn [48], this corresponds to applying a RobustScaler (first case) or MinMaxScaler (second case), and then the function f. which smoothly clips its input to the range (-3.3). Smooth clipping functions like if have been used by, e.g., Holzmüller et al. [24] and Hafner et al. [20]. Intuitively, when features have large outliers, smooth clipping prevents the outliers from affecting the result too strongly, while robust scaling prevents the outliers from affecting the inlier scaling.
NN architecture Our architecture, visualized in Figure (a), a multilayer perceptron (MLP) with three hidden layers containing 256 neurons each, except for the following additions and modifications:
RealMLP-TH employs categorical embedding layers [9 to embed the remaining categorical
features with cardinality
For numerical features, excluding the one-hot encoded ones, we introduce PBLD (periodic bias linear DenseNet) embeddings, which concatenate the original value to the PI. embed- dings proposed by Gorishniy et al. [16] and use different periodic embedding with biases,

---

meta-trin--llas
meta-ttrinn-ge
One-hot encoding
Robust scale
Hyperygannttr Pienoccooing
Vanilla MLP
Robust scale smooth clip
Smooth-clip
One hot for small cat.
No carly stopping
Last best epoch
Num./cat, embeddings
coslog, Ir sched
Adam 9a 0.95
Learnable scaling
Label smoothing (class.)
Outpat clipping (reg.)
Linear
NT parametrization
Act. fn. SELU Mish
3x
Parametric actívation
Archigeeture
Parmmetric act. fn.
Scaling layer
Dropout
Num, embedddings Pl.
PL cmb. PBLD cob..
Linear
Regular Regularization brattco
Dropout p 0.15
Dropost sched: flat.cos
(a) Preprocessing and NN architecture for ReaIMLP-TD.
Weight docay wd = 0.02
wd sched: flat.cos
Bias init: liee5
Weight init data driven
RealMLP
Unuezzl
10
20
10
20
Benchmark score improvement (%) vs. vanilla
(b) The coslog, and flat_cco cos schedules.
(c) From vanilla MLP to ReaIMLP-TD.
Figure Components of ReaLMLP-TD. Part (c) shows result adding one component in each step, where the best default learning rate is found separately for each step. The vanilla MLP uses categorical embeddings, a quantile transform to preprocess numerical features, default PyTorch initialization, ReLU activation, early stopping, and is optimized with Adam with default parameters. For more details, see Appendix A.4 The error bars are approximate 95% confidence intervals for the limit #splits - oc, Appendix C.6
inspired by Huang et (26) and Rahimi and Recht [52), respectively. PBLD embeddings apply separate small two-layer MLPs to each feature 2; as
Ty. W (2.4) emb cos cos(22r (1.1),
emb 24 b (1l1))
amb E R
For efficiency reasons, we use 4-dimensional embeddings with u 3 (2.6) 3x16
(1.1) (1.t) emb
E RRI6
(2.1) emb E
cmb
To encourage (soft) feature selection, we introduce a scaling layer before the first linear layer, which simply matrix-vector product with diagonal weight matrix. In other words, it computes oi,out
Zijiu, with a learnable scaling factor for each feature i. We found it beneficial use larger learning rate for this layer.
Our linear layers use the neural tangent parametrization (NTP) as proposed by Jacot et al.
288 i.c., they compute z(+1] di 1/2 w (1) (!), where di the dimension of the layer input (1), The motivation behind the use the NTP here that it effectively modifies
the learning rate for the weight matrices depending on the input dimension di, hopefully
preventing too large steps whenever the number of columns large. We did not observe improvements when using the Adam version of the maximal update parametrization [68].

---

RealMLP- uses parametric activation functions inspired by PReLU [21] In general, for an activation function σ, we define a parametric version with separate learnable a; for each neuron i:
oa, (x:) (1 04) a,a((2))
When 0; = 1, this recovers Ø, and when a; = O, the activation function linear. As activation functions, use SELU (35 for classification and Mish [45] for regression.
We use dropout after each activation function. We do not use the Alpha-dropout variant originally proposed for SELU [5), we
able obtaín good results with it.
For regression, at test time, the MLP outputs are clipped to the observed range during training. (We observed that this mainly helpful for suboptimal hyperparameters.)
The parameters 8; of the scaling layer are initialized to 1, making it an identity function initialization. Similarly, the parameters 0; of the parametric activation functions initialized to I, recovering the standard activation functions at initialization. We initialize weights and biases in data-dependent fashion during a forward pass the (possibly subsampled) training set. We rescale rows of saandardnoormal-iittaalll
Initialization
weight matrices to scale the variance of the output pre-activations over the dataset to one. For the biases, we use the deta-deeenddnt he+5 initialization method [called hull+5 61].
Training Like Gorishniy et al. [15],
the A optimizer [34 40 We set its momentum
hyperparameters to 31 = 0.9 and 32 0.95 instead of the default 32 0.999. The idea to use a smaller value for 32 is adopted from the fastai tabular MLP (25). RealMLP optimized for 256 epochs with a batch size of 256. As a loss function for classification,, we use softmax cross-entropy with label smoothing [62] with parameter = 0.1. For regression, we use the MSE loss and affinely transform the targets to have zero mean and unit variance on the training and validation set.
We allow parameter-specifie scheduled hyperparameters computed in each iteration using base value, optional parameter-specific factors, and schedule, as
Hyperparameters
iteration
base_value: peram_facton
schedule #iterations
allowing for example, to high learning rate factor for scaling layer parameters. Because we do not tune the number of epochs separately on each dataset, we use a multi-cycle learning rate schedule, providing multiple valleys that are usually preferable for stopping the training, while allowing high learning rates in between. Our schedule similar to Loshchilov and Hutter [39 and Smith [59], but with simpler analytical expression:
coslogı (t) _ cos log, + (2* 1)t)))
We set obtain four cycles shown in Figure (b). To allow stopping different levels of regularization, we schedule dropout and weight decay using the following schedule, cf. Figure I (b)
flat fat_cco((t) = / cos (m
1))).
The detailed hyperparameters can be found in Table A.1
Best-epoch selection Due the multi-cycle learning rate schedule, we do not perform classical early stopping. Instead, we always train for the full 256 epochs and then revert the model to the epoch with the lowest validation error, which this paper based on classification error, or RMSE for regression. In case of tie, we found it beneficial to use the last of the tied best epochs.
RealMLP-TD-S
Since certain aspects ReaIMLP-TD are somewhat complex implement, we introduce a simplified (and faster) variant called RealMLP-TD-S in Appendix A Among the simplifications are: omitting embedding layers, using non-parametric activations, using a simpler initialization method, and omitting dropout and weight decay.
'inspired by
similar
schedule
in
http:://gtthub..oo/lesss//
Ranger- Deep- Learning- Swwniiinnn-ppiiiieenn Optimizer

---

RealTabR-DD For RealTabR-D, D, we adapt TabR-S-D by using our numerical preprocessing, setting Adam's B2 0.95, using our scaling layer with a modification to amplify the effective learning rate by factor of 96, adding PBLD embeddings for numerical features, and adding label smoothing for classification. More details can be found in Appendix A.3
4 Gradient-Boosted Decision Trees
To find better default hyperparameters for GBDTs, we employ a semi-automatic approach: We use hyperparameter optimization libraries like hyperopt [2] and SMAC3 [38 to explore a reasonably large hyperparameter space, evaluating the benchmark score of each configuration on the meta-train benchmarks, and then perform some small manual adjustments like rounding the best obtained hyperparameters. To balance efficiency and accuracy, fix the number of estimators to 1000 and use the hist method for XGBoost. We only consider the librarics' default tree-building strategies since it is one of their main differences. The tuned defaults (TD) for LightGBM (LGBM), XGBoost (XGB), and CatBoost can be found in Table C.I]
C.2
and
C,3
respectively.
While some of the obtained hyperparameter values might be sensitive to the tuning and benchmark setup, we observe some general trends. First, row subsampling is used in all tuned defaults, while column subsampling is rarely applied. Second, trees are generally allowed to be deeper for regression than for classification. Third, the Bernoulli bootstrap in CatBoost is competitive with the Bayesian bootstrap while also being faster.
Experiments
In the following, we evaluate different methods with library defaults (D), tuned defaults (TD), and hyperparameter optimization (HPO). Recall that TD uses fixed parameters optimized on the meta-train benchmarks, while HPO tunes hyperparameters on each dataset split independently. All methods except random forests select the best iteration/epoch on the validation set of the respective dataset split based on accuracy RMSE. All NN-based regression methods standardize the labels for training.
5.1 Methods
We provide methods in the following variants:
D: Default parameters, taken from the original library if possible (Appendix C.1
TD: Tuned default parameters from CoomSeetionn BanddSeection
HPO: Hyperparameters optimized separately for every train-test split every dataset, using 50 steps of random search. Search spaces are specified in Appendii C.2 and are usually adapted from original or popular papers.
As tree-based methods, we use XGBoost (XGB), LightGBM (LGBM), and CatBoost from the respective libraries, as well as random forest (RF) from scikit-learn. The variant XGB-PBB-D uses meta-learned default parameters from Probst et al. [50] For neural methods, we compare MLP. ResNet, and FT- TT-Trassfrrmrr (FTT) from Gorishniy et al. [15), MI.P-LRR from Gorishniy et al. 16], as well as TabR and TabR-S (without numerical embeddings) from Gorishniy et [7.. We compare these methods to RealMLP and RealTabR from Section 3] In addition, we investigate Best, which on each dataset split selects the method with the best validation score out of XGB, LGBM, CatBoost, and MLP-PLR (for Best-D) or RealMLP (for Best-TD and Best-HPO). Ensemble builds weighted ensemble out of the same methods as Best, using the method of Caruana et al. with 40 greedy selection steps as in Salinas and Erickson (55).
We do not run FTT, RF-HPO, and TabR- on all benchmarks since some benchmarks (especially meta-test) are more expensive to run and these methods may run into out-of- errors.
5.2 Results
shows the results of the aforementioned methods on all benchmarks, along with their runtimes on a CPU. Note that XGB results on some (mainly meta-test) datasets are affected by a bug in handling rare categories, see Appendix B
Figure 2
7

---

Meta-train classification benchmark
Meta-train regression benchmark
0.32
0,0630
ResNeiDD
Shited geometii mean of diassfication errors
AF
CalBBost
BBB-D
Shitted geomerie moan nRMSSS
MLP-D
0,0000
MIP HPOL
00.29
RF
XGB-HPO
PPs
FTTC
CaiBones-HPP HPG MLP.HPO
1GBM
Red Set-POO
CalBooothDD
XGB-HPO
0.25
LGBM TO
HOBM-HOO
TTD
0,0030
abR-S-D
ReaMLP-TD'S d Bemmmblo
Best
0,0025
LGEM-TD
XGB-TD
P-TD-S
Bostr
Enfemble
CBM-APa
Cartooost--P
0.20
CatBoosi-TD
RealTabR-D
ReaIMLPP-D
0,0000
better
RoaIMLP.TD
RealML P-MPO
0.22
better
Re P-HPO
Best
Beat-tff
Best-TD
R
Hea--Hho
0UNNT
EmarbeeTT
Ensembee-HPP
Frsemsie
10-11
10
13 Average training (CPU) per samples [s]
10-1
Average training (CPU) per samples
classification
Meta-test regression benchmark
0.48
0:140
0.46
cisssfication
Shitted goomeric mean of ARIASEs
AGB-D
GBV-D
Restwsn-
MIP ResNeeDD
0:130
XGB
XGB
XOB-TTD
CatBrost--
ju-P |-PLIR-D R-D
Res
9:135
ULP
0.42
LGBM-TD
TabR-Sii
CatBooott-D
keB HPO
MI PLR R-RR THiBthee-PP
0.40
CatBoost-TDD
Best-F
HPC
Tabe
LGBM HPO
Farthooat-H HPG
MCP-PP
Shifte geametric
GGB HPO
Ensember
0:155
Beat-D
Erswmblee
RaaTTaRR-
ReaIMiP.TD
better
ReSIMLP-TD
Te
ReaIMLP.HPO
better
Ersemble
Best-T
Eraerbu-TD
Ensemble HPO
0.3L
Ensembles: iPo
10 1
IIf Average training (CPU) per samples [s]
Average training (CPU) per samples
Grinsztain et (2022) classification benchmark
Grinsztajn al. (2022) regression benchmark
0.200
MUP-D
0.36
D
Shifled geom mean ilcatinn
RasNesD
MLP-HPOL
Shitte geomeriic mean of RRMSES
0.35
MuP
HPs
Has
0.106
RealMLP TD
MUP-PER- TabR-siq
MLPHPG
XGB.8 c
0.150
LGBM
ReaiMLP-TDL
SR-SSD
HPO
PLP.PLB FTTHPQ
00:5
KF
XGB
0.185
LGBM-TD
XGB-TD
PoaMMLP-HPP
0,32
CatBoost LGBM-TD
Yoe-TD
Mes Parss
GaiBuast-
HBO
0.150
CalBooat-TD TD
EnsembleDD
HealTwbr
CalBooa HPO
HPP
pesto
LORk
MLP-PLR-PP
Ensembee11
e essmmlewpng
Ensemble
BearTabR-D
TabR-Hin
*
better
better
TealMLL-HPO
1.175
Best HPC
0.20
1C7
101
Id
1c!
Ic?
107
10
Average training (CPU) per samples
Average training time (CPU) per samples [s]
D defaults
TD tuned defaults
HPO hyperparameter optimization
Best/Ensemble: out of XGB, LGBM, CatBoost, (Real)MLP
Figure 2: Benchmark scores on all benchmarks vs. average training time. The y-axis shows the shifted geometric mean (SGM.) classification error (left) nRMSE (right) explained in Section 2.2 The z-axis shows average training times per 1000 samples (measured on grrain for efficiency reasons), see Appendix C.7 The error bars are approximate 95% confidence intervals for the limit #splits + oc, see Appendix C.6 Note that XGB results on some (mainly meta-test) datasets are affected by a bug handling rare categories, see Appendix

---

Meta-test classification benchmark
Grinsztajn et al. (2022) classification benchmark
0.085
0.125
PeWNMPPTOOS
Shifted geometric 1-AUUC(ono-v--res)
GBM-D
Shifted geometre mean of 1-AUG(one-e--eal)
MLP-DV
0.060
T. (no
MLP-HPOL
0.120
ResNot-D
XGB-4
MLP PLR BBP.//P--DD
DesNe:HHIO
0.075
RF-D
LGBM-TTI
TD-
DUIIULP-TO UID (nd be
MIP-PUR-HPO
ResNor
XG8-T0
RF-HPO
eaIMLP-HPO
0,060
XGB-6
XCa-TDT
RESPPUR-
..P.
HOaIMLAIDD ID-S (no LS)
T
HPO
CatBoost-TD
CatBooohp
LGBM-TD
Real T
KGG RBB D
XGB HPO
ReaaMER TD
ResNeshiPo
RealTabR-D (no LS)
HP
0.00G
HLC-
Best-TD
LGMMHPO
Best-HIO
Co
CaiB
Real Miaao We
XGBB-P8--o
CGB-HP
0.100
0,060
better
RealTabil-D (no
better
Bess.D
Beet-TD
CatBoost- HPO
Best-PP
0,100
HPO
0.055
10-1
10²
10
10F
20-1
Jif
10
10a 10P
Average training time (CPU) per 1K samples [s]
Average training time (CPU) per 1K samples [s]
Figure 3: Benchmark scores average training time for AUC. Methods labeled "no LS" deac- tivate label smoothing. Stopping and best-epoch selection are performed on accuracy, while HPO is performed AUC. See Figure B.3]for for stopping on cross-entropy. The y-axis shows the shifted geometric mean (SGM)) - AUC as explained in Section 2.2] The æ-axis shows average training times per 1000 samples (measured on gtrain for efficiency reasons), see Appeedix C.7 The error bars are approximate 95% confidence intervals for the limit #splits + oo, see Appendix C.6
How good are tuned defaults on new datasets? To answer this question, we compare the relative gaps between TD and HPO benchmark scores on the meta-test benchmarks those on the meta-train benchmarks. The gap between ReaIMLP-HPO and RealMLP benchmarks, indicating that the tuned defaults transfer very well to the meta-test benchmark. For GBDTs, tuned defaults are competitive with HPO on the meta-train set, but not as good on the meta-test set. Still, they are considerably better than the untuned defaults on the meta-test set. Note that did not limit the TD parameters the literature search spaces for the HPO models (cf. Appendix C.2); for example, XGB-TD uses a smaller value of min child weight for classification and CatBoost-TDD uses deeper trees and Bernoulli boosting. The XGBoost defaults XGB-PBB-D from Probst et al. (50) outperform XGB-TD on Blest perhaps because their benchmark is more similar to Blest or because XGB-PBB-D uses more estimators (4168) and deeper trees.
is not much larger the meta-test
RealMLP and RealTabR perform strongly among NNs. On most benchmarks. RealMI.P-TD and RealTabbRDD bring considerable improvements over MIP-PLR-DD and TabR-S-D, at slightly larger runtimes, respectively. Similarly, RealMLP--PP improves the results of MLP-PLRR-HPO. TabR and FTT are notably slower than MLP-based methods on CPUs, while the difference is less pronounced on GPUs (Figure C.2). While RealMLP-TD beats TabR-S-D on many benchmarks, RealTabR-D performs even better on four out of six benchmarks, especially all regression benchmarks. On the Grinsztajn et [18] benchmark where we can afford run more baselines, TabR-Han -PO performs best according to many aggregation metrics. It performs especially well on the electricity dataset, where MLPs struggle to learn figh-frequency patterns [88]
RealMLP and RealTabR are competitive with tree-based models. On the meta meta-train and meta- test benchmarks, RealMLP and RealTabR perform better than GBDTs in terms of shifted geometric mean error, while also being comparable or slightly better in terms of other aggregations like mean normalized error (Appendix T0 or win-rates (Appendix B.12). On the Grinsztajn et al. [8 benchmark, RealMi.P performs worse than CatBoost for classification and comparably for regression, while Real TabR-D performs comparably to CatBoost-TD for classification and better for regression.
Among GBDTs, CatBoost defaults are better and slower. Several papers have found CotBoost to perform favorably among GBDT's while being more computationally expensive to train 33 51 69]. We observe the same for our tuned defaults most benchmarks.

---

Simply trying all default algorithms is faster and very often better than (naive) single-algorithm HPO. When comparing Best-TD 50-step HPO on RealMLP or GBDTs, we notice that Best-TD is faster on average, while also being competitive with the best of the HPO models. In comparison, Best- is often outperformed by ReaIMI.P-HPO. We also note that ensemble selection [5] usually gives 0-3% improvement on the benchmark score compared to selecting the best model, and can potentially be further improved [6]. Unlike McElfresh et al. [43] who argue in favor of CatBoost-HPP over trying NNs, our results favor model portfolios as used in modern AutoML systems [10]
Analyzing NN improvements Figure 1 (c) shows how adding the proposed RealMLP components
to simple MLP improves the meta-train benchmark performance. However, these results depend on the order in which components are added, which is addressed by a separate ablation study in Appendix B For example, the large weight decay value makes ReaIMLP-TD sensitive to changes in some other hyperparameters like 82. We also show in [Appendix B.8 that our architectural improvements alone are beneficial when applied to MLP-D directly, although non-architectural aspects are at least as important. In particular, our numerical preprocessing is easy to adopt and often beneficial for other NNs as well (Appendix B.7). The scaling layer and PBLD embeddings are easy to use and turned out to be effective within RealTabR-DD as well. If affordable. larger stopping patiences and the use of (cyclic) learning rate schedules can be useful, while label smoothing is influential but can be detrimental for metrics like AUROC (Figure 3 Appendix B.5).
Dependence on benchmark choices We observe that choices in benchmark design can affect the interpretation of the results. The use of different aggregation metries than the shifted geometric mean reduces the advantage of TD methods (Appendix B.1O). For classification, using AUROC instead of classification error (Figure 3) Appendix B.5) favors GBDTs. Different dataset selection and preprocessing criteria on different benchmarks lead to large differences between benchmarks in the average erros,, indicated by the y-axis scaling in Figure 2
Further insights In Appendix B we present additional experimental results. We compare bagging and refitting for RealMLP-TD and LGBM-TD, finding that refitting multiple models is often better on average. We demonstrate that GBDTs benefit from high early stopping patiences for classification, especially when using accuracy as the stopping metric. When considering AUROC as stopping metric, we show that stopping on cross-entropy is preferable to accuracy (Appendix B.5)
Limitations While our benchmarks cover medium torge tabular datasets in standard settings, it is unclear to which extent the obtained defaults can generalize to very small datasets, distribution shifts, datasets with missing numerical values, and other metrics such as log-loss. Additionally, runtimes and the resulting tradeoff's may change with different parallelization, hardware, or (time-aware) HPO algorithms. For computational reasons, we only use single training-validation split per train-test split. This means that HPÔ overfit the validation set more easily than in a vols--valiationn setup. While we extensively benchmark different NN models from the literature, we do not attempt to equalize non-architectural aspects, and our work should therefore not be seen as comparison of architectures. We compared to TabR-S-D as a recent promising method with good default parameters 17 69)
However, due to surge of recently published deep tabular models [e.g.. 7 29 33
67 is
unclear what the current "best" deep tabular model is. In particular, ExcelFormer 7 also promises strong-perte pronng-eroormigg default parameters. For GBDTs, due to the cost of running the benchmarks. our limits on the depth and number of trees are on the lower side of the literature.
6 Conclusion
In this paper. we studied the potential of improved default parameters for GBDTs and an improved MLP, evaluated on large separate meta-test benchmark as well as the benchmark by Grinsztajn et al. [8] and investigated the time-accuracy tradeoffs of various algorithm selection and ensembling scenarios. Our improved MLP mostly outperforms other NNs from the literature with moderate runtime and is competitive with GBDT's in terms of benchmark scores, Since many of the proposed improvements to NNs are orthogonal to the improvements in other papers, they offer exciting opportunities for combinations, as we demonstrated with our RealTabR variant. While the "NNs vs GBDTs" debate remains interesting, our results demonstrate that with good default parameters, worth trying both algorithm families even with a moderate training time budget.
10

---

Acknowledgments and Disclosure of Funding
We thank Gaël Varoquaux, Frank Schnke, Katharina Strecker, Ravid Shwartz-Ziv, Lennart Purucker, and Francis Bach for helpful discussions. We thank Katharina Strecker for help with code refactoring.
Funded by Deutsche Forschungsgemeinnccatt (DFG, German Research Foundation) under Ger- many's Excellence Strategy EXC 2075 390740016. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting David Holzmüller. LG acknowledges support in part by the French Agence Nationale de la Recherche under Grant ANR-20- CHIA-0026 (Learnl). Part of this work performed the computational resource bwUnCCuster funded by the Ministry of Science, Research and the Arts Baden- Württemberg and the Universities of the State of Baden- Württemberg, Germany, within the framework program bwHPC. Part of this work was performed using HPC resources from GENCI-IDRIS (Grant 2023-AD01 1012804RI and 2024-AD01 1012804R2).
Contribution statement DH and IS conceived the project. DH implemented and experimentally validated the newly proposed methods and wrote the initial paper draft. DH and L.G contributed to benchmarking, plotting, and implementing baseline methods. I.G and IS helped revise the draft. IS supervised the project and contributed dataset downloading code.
References
[1] Sercan O. Arik and Tomas Pfister. TabNet: Attentive interpretable tabular learning. In AAAI Conference on Artificial Intelligence, 2021.
[2] James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyper- parameter optimization in hundreds of dimensions for vision architectures. In International Conference on Machine Learning, 2013.
[3] Vadim Borisov, Tobias Leemann, Kathrin SeBler, Johannes Haug. Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey, IEEE Transactions Neural Networks and Learning Systems, 2022.
[4] Pavel Brazdil, Christophe Giraud Carrier, Carlos Soares, and Ricardo Vilalta. Metalearning: Applications Data Mining. Springer Science & Business Media, 2008.
[5] Rich Caruana, Alexandru Niculescu-Mizil, Geoff Crew, and Alex Ksikes. Ensemble selection from libraries of models. In International Conference on Machine Learning. 2004.
[6] Rich Caruana, Art Munson, and Alexandru Niculescu-Mizii. Getting the most out of ensemble selection. In International Conference on Data Mining, pages 828-833 IEEE, 2006.
[7] Jintai Chen, Jiahuan Yan, Qiyuan Chen, Danny Z. Chen, Jian Wu, and Jimeng Sun. Can Deep Learning Model be Sure Bet for Tabular Prediction? In Conference on Knowledge Discovery and Data Mining. ACM, August 2024.
[8] Kuan-Yo Yu Chen, Ping-Han Chiang, Hsin-Rung Chou, Ting- Wei Chen, and Tien-Hao Chang. Trompt: Towards a better deep neural network for tabular data. In International Conference on Machine Learning, 2023.
[9] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In International Conference on Knowledge Discovery and Data Mining, 2016.
[10] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. CatoolluonTaabllla Robust and accurate AutoML for structured data. In 7th ICML Workshop on Automated Machine Learning, 2020.
[11] Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto-sklearn 2.0: Hands-free automl via meta-learning The Journal of Machine Learning Research, 23(261), 2022.
[12] Sebastian Felix Fischer, Matthias Feurer, and Bernd Bischl. OpenML-CCTR2-44 curated tabular regression benchmarking suite. In AutoML Conference 2023 (Workshop), 2023.
11

---

[13] Pieter Gijsbers, Marcos LP Bueno, Stefan Coors, Erin LeDell, Sébastien Poirier, Janek Thomas, Bernd Bischl, and Joaquin Vanschoren. AMLB: an AutoML benchmark. Journal of Ma- chine Learning Research,
2024. URL htl.hhtps:://eee jmlr. http://wwww...o/ppppp/pr////////
22-0493.1 .ttml
[14] Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-375 2007.
[15] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Neural Information Processing Systems, 2021.
[16] Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. Neural Information Processing Systems, 2022.
[17] Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. TabR: Tabular deep learning meets nearest neighbors. International Conference Learning Representations, 2024.
[18] Léo Grinsztajn, Edouard Oyallon, and Gaäl Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Neural Information Processing Systems, 2022.
[19] Cheng Guo and Felix Berkhahn. Entity embeddings of categorical variables. arXiv:1604.0677 2016.
[20] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv:2301.04100 2023.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Sur- passing human-level performance on imagenet classification. In IEEE International Conference on Computer Vision, pages 1026-10344 2015.
[22] Steffen Herbold. Autorank: A Python package for automated ranking of classifiers. Journal of Open Source Software, 5(48):2173, 2020. doi: 10.21 105/joss.022 02173. URL htt/ssts:///o org/10. 21.00.211//osss 02173 Publisher: The Open Journal.
[23] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: transformer that solves small tabular classification problems in second. In International Conference on Learning Representations, 2022.
[24] David Holzmüller, Viktor Zaverkin, Johannes Kästner, and Ingo Steinwart. A framework and benchmark for deep batch active learning for regression. Journal of Machine Learning Research, 24(164), 2023.
[25] Jeremy Howard and Sylvain Gugger. Fastai: layered API for deep learning. Information, 11 (2):108, 2020.
[26] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In Computer Vision and Patlern Recognition, pages 4700- 4708, 2017.
[27] Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. TabTransformer: Tabular data modeling using contextual embeddings. arXiv:2012.06678, 2020.
[28] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnel: Convergence and generalization in neural networks. Neural Information Processing Systems, 2018.
[29] Manu Joseph and Harsh Raj. GANDALF: Gated Adaptive Network for Deep Automated Learning of Features. 28845482070 2024.
[30] Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. In Neural Information Processing Systems, 2021.
12

---

[31] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. LightGBM: A highly efficient gradient boosting decision tree. In Neural Information Processing Systems, 2017.
[32] Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. The UCI Machine Learning Reposi- tory. URL https: pettps://archiveee ics. uci edu
[33] Myung Jun Kim, Léo Grinsztajn. and Gaël Varoquaux. CARTE: pretraining and transfer for tabular learning. In International Conference on Machine Learming, 2024.
[34] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- lional Conference on Learning Representations, 2015.
[35] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self seff-normalizing neural networks. In Neural Information Processing Systems, 2017.
[36] Ravin Kohli, Matthias Feurer, Katharina Eggensperger, Bernd Bischl, and Frank Hutter. Towards Quantifying the Effect of Datasets for Benchmarking: A Look at Tabular Machine Learning. In ICLR 2024 Data-centric Machine Learning Research Workshop, 2024.
[37] Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Thomas Rainforth, and Yarin Gal. Sclf-attention between datapoints: Going beyond individual input-output pairs in deep learning. In Neural Information Processing Systems, 2021.
[38] Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter. SMAC3: A versatile Bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research, 23(54), 2022.
[39] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
[40] Uya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2018
[41] Sascha Marton, Stefan Lüdtke, Christian Bartelt, and Heiner Stuckenschmidt. GRANDE: Gradient-based decision tree ensembles for tabular data. In International Conference on Learning Representations, 2024.
[42] Calvin McCarter. The kernel density integral transformation. Transactions on Machine Learning Research, 2023.
[43] Dunean McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakr- ishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? In Neural Information Processing Systems, 2023.
[44] Dmytro Mishkin and Jiri Matas. All you need is good init. In International Conference Learning Representations 2016.
[45] Diganta Misra. Mish: A self regularized non-monotonic activation function. In British Machine Vision Conference, 2020.
[46] Philipp Moritz, Robert Nishihara, Stephanic Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, and Michael I. Jordan. Ray: A distributed framework for emerging Al applications. In USENIX Symposium Operating Systems Design and Implementation, 2018.
[47] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, and Luca Antiga. PyTorch: An imperative style, high- performefer deep learning library. Neural Information Processing Systems, 32, 2019.
[48] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, and Vincent Dubourg. Scikit- learn: Machine learning in Python. Journal of Machine Learning Research, 12(85), 2011.
13

---

[49] Florian Pfisterer, Jan N. Van Rijn, Philipp Probst, Andreas C. Müller, and Bernd Bischl. Learning multiple defaults for machine learning algorithms. In Genetic and Evolutionary Computation Conference, July 2021.
[50] Philipp Probst, Anne-Laure Boulesteix, and Bernd Bischl. Tunability: Importance of hyper- parameters of machine learning algorithms. Journal of Machine Learning Research, 20(53), 2019.
[51] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: Unbiased boosting with categorical features. In Neural Information Processing Systems, 2018.
[52] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Neural Information Processing Systems, 2007.
[53] Hubert Ramsauer, Bernhard Schäft, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, and Michael K. Kopp. Hopfield networks is all you need. In International Conference on Learning Representations, 2020.
[54] Ivan Rubachev, Nikolay Kartashev, Yury Gorishniy, and Artem Babenko. TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks. arXiv:2406. 19380, 2024.
[55] David Salinas and Nick Erickson. TabRepo: A large scale repository of tabular model evalua- tions and its AutoML applications. In AutoML Conference, 2024.
[56] Bernhard Schäfl, Lukas Gruber, Angela Bitto-Nemling, and Sepp Hochreiter. Modern Hopfield networks as memory for iterative learning on tabular data. In NeurIPS Workshop on Associative Memory & Hopfield Networks in 2023, 2023.
[57] Junhong Shen, Liam Li, Lucio M. Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. Cross-modal fine-tuning: Align then refine. In International Conference on Machine Learning, 2023.
[58] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning not all you need. Information Fusion, 81:84-90, 2022.
[59] Leslie Smith. Cyclical learning rates for training neural networks. In Winter Conference on Applications of Computer Vision, 2017.
[60] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training. In NeurlPS 2022 Table Representation Learning Workshop, 2022.
[61] Ingo Steinwart. A sober look at neural network initializations. arXiv: 2019.
[62] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Computer Vision and Patlern Recognition, 2016.
[63] Jan N. van Rijn, Florian Pfisterer, Janek Thomas, Andreas Muller, Bernd Bischl, and Joaquin Vanschoren. Meta learning for defaults: Symbolic defaults. In NeurIPS 2018 Workshop on Meaningg 2018.
[64] Joaquin Vanschoren. Meta-learning A survey, arXiv:1810.03548, 2018
[65] Joaquin Vanschoren, Jan N. van Rijn, Bemnd Bischl, and Luis Torgo. OpenML: Networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):99 60, 2014. Publisher: ACM New York, NY, USA.
[66] Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme Learning hyperparameter optimization initializations. In International Conference on Data Science and Advanced Analytics, pages 1-10, 2015.
14

---

[67] Chenwei Xu, Yu-Char Huang, Jerry Yao-Chick Hu, Weijian Li, Ammar Gilani, Hsi-Sheng Goan, and Han Liu. BiSHop: Bi-directional cellular learning for tabular data with generalized sparse modern Hopfield model. In International Conference on Machine Learning, 2024.
[68] Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In Neural Information Processing Systems, 2021.
[69] Han-Jia Ye, Si-Yan Yang Liu, Hao-Run Cai, Qi-Le Zhou, and De-Chuan Zhan. A closer look at deep learning on tabular data, arXiv:2407.00956 2024.
15

---

Appendices
Appendix Contents.
Further Details on Neural Networks
17
A.1
DetM
17
A.2
ReaDMLP-TD-S Details
17
A.3
RealTabR-D Details
17
A.4
Details on Cumulative Ablation
19
A.5
Discussion
20
More Experiments
20
MLP Ablations
20
B.2
MLP Preprocessing
21
B.3
Bagging, Refitting, and Ensembling
22
B.4
Early stopping for GBDTs
B.5
Results for AUROC
24
B.6
Results Without Missing- Value Datasets
25
B.7
Comparing Preprocessing Methods for NNs
25
B.8
Results for Varying Architecture
26
B,9
Comparing HPO Methods
28
10
More Time- Error Plots
28
Critical Difference Diagrams
28
B. 12 Win-rate Plots
28
Benchmark Details
40
Default Configurations/
40
C.2
Hyperparameter Optimization
40
Dataset Selection and Preprocessing
48
Comparison with Standard Grinsttajn et al. [18 Benchmark
54
C.5
Closer-to-original Version of the Grinsztajn et al. 18 Benchmark
54
C.6
Confidence Intervals
57
Time Measurements
57
Compute Resources
57
9
Used Libraries
57
Results for Individual Datasets
58
Broader Impact
76
16

---

A Further Details on Neural Networks
The detailed hyperparameter settings for ReaIMLP-TD and ReaIMLP-TD-S are listed in Table A.1
A.1 RealMLP-TD Details
Architecture To make the binary and multi-class cases more similar, we use two output neurons in the binary case, using the same loss function as in the multi-class case.
Initialization We initialize categorical embedding parameters from (N 1). We initialize the ,, T.. The other numerical embedding parameters are initialized according to PyTorch's 's default initialization, that is, from the uniform (1.6) components of to emb from (0, 0.1 and of ba from 2[|=,,
distribution 11[1//116, 1/ I6). For weights and biases of the linear layers, we use data data-dependent initialization. The initialization is performed on the fly during first forward pass of the network on the training set (which can be subsampled adaptively not to use more than GB of RAM). We realize this by providing tett_rransfrm () methods similar to a pipeline in scikit-learn, For the weight matrices, we use a custom two-step procedure: First, we initialize all entries from N 1). Then, we x have variance 1 over the dataset (i.e. when considering the sample index j E {l...., n as a uniformly distributed random variable). This is somewhat similar to the LSUV initialization method [44]. For the biases, we use the data-dependent he+5 initialization method [called hull+5 in i6]
rescale each row of the weight matrix such that the outputs + W (1)
Vd.
Training We implement weight decay as in PyTorrch using 0 + 0 - Ir wd 0, which includes the learning rate unlike the original version [40].
A.2 RealMLP-TD-S Details
For RealMLP-TD-S, we make the following changes compared to ReaIMLP-TD:
We apply one-hot encoding to all categorical variables and do not apply categorical embed-
dings.
We do not apply numerical embeddings.
We use the standard non-parametric versions of the SELU and Mish activation functions.
We do not use dropout and weight decay.
We use simpler weight and bias initializations: We initialize weights and biases from N'O. 1). except in the last layer, where we initialize them to zero.
We do not clip the outputs, even in the regression case.
We apply different base learning rate in the regression case.
A.3 RealTabR-DD Details
To obtain RealTabR-D,, we modify TabR-S-D in the following ways:
We replace the standard numerical preprocessing (a modified quantile transform) with our robust scaling and smooth clipping.
We set Adam's 's f32 to 0.95 instead of 0.999.
We use our scaling layer, but modify to obtain a higher effective learning rate. We this by modifying the forward pass to
Piout 7 S; Tg,in
while initializing to 1/7. This will multiply the gradients of s; by y, which will be ignored by Adam's normalization (when neglecting Adam's parameter). It will also multiply the optimizer updates by leading to approximately the same effect as multiplying the learning rate by y. However, difference is that multiplying the learning rate by y will also lead to stronger weight decay updates in Py PyTorch's 's AdamW implementation, while the introduction of y does not increase relative magnitude of weight decay updates. We chose version with for simplicity of implementation. While ReaIMLP-TTD uses a learning rate factor of 6 for the scaling layer, it uses higher base learning rate due to the use of the neural tangent parametrization. For all layers except the first one, which have width 256, the neural tangent
17

---

Table A.I: Overview hyperparameters for RealMLP-TD and RealMLL-TD-S.
RealMLP-TD
RealMLP-TD-S
Hyperparameter
classification
regression
classification
regression
Num. embedding type
PBLD
PBLD
None
None
Num. embedding periodic init std.
0.1
0.1
Num, emhedding hidden dimension
Num, embedding dimension
4
Max one-hot size (without missing)
8
8
Num. preprocessing
robust scale smooth clip
Categorical embedding dimension
Categorical embedding initialization
A( 1)
((,, 1)
Use scaling layer
Scaling layer initialization
1.0 (constant)
Number of linear layers
Hidden layer sizes
[256, 256, 256]
Activation function
SELU yes
Mish
SELU
Mish
Use parametric activation function
no
Parametric activation functton initialization
1.0
1.0
Linear layer parametrization
NTP
Last linear layer weight initialization
data-driven data-driven hc+5
data-driven data-driven
zero
zero std normal
Other linear layer weight initialization
std normal
Last linear layer bias initialization
hc+5
Other linear layer bias initialization
he+5
he+5
std normal
std normal
Optimizer Batch size Number of epochs Adam $1 Adam 82
Adam
256
256
0.9
0.95
Adam E
le-8
Learning rate (base value)
0.04
0.2
0.04
0.07
Learning rate schedule
coslog.
Learning rate (num. emb. factor) Learning rate (scaling layer factor)
0.1
0.1
Learning rate (bias factor)
0.1
Learning rate (param. act. factor)
0.1
0.1
Dropout probability (base value)
0.15
0.15
0.0
0.0
Dropout schedule
flat_cos
flat_cos
Weight decay (base value)
0.02
0.02
0.0
Weight decay schedule
flat. cos
flat cos
Weight decay (bias factor)
0.0
0.0
Loss function
cross-entropy 0.1
MSE
cross-entropy 0.1
MSE
Label smoothing
Standardize targets during training
yes yes MSE
yes
Output min-mas max clipping
no
Best epoch selection metric
class. error
class. error
MSE
Best epach selection method
last best validation
parametrization in RealMLP-TD uses a factor similar to which is set to 1/16 = 1/6256.
Hence, ReaIMLP-TD without NTP should use base learning rate for these layers that is smaller by factor of 1/16, and therefore use a learning rate factor of 6 16 = 96 for the scaling layer. Consequently, we set 7 96 for RealTabR-D without further tuning, noting that it performed significantly better than = 6 on the meta-train benchmarks.
We use our PBLD embeddings for numerical features before the scaling layer, instead of no numerical embeddings in TabR-S-D. D. In order to make every experiment run on GPU with 24GB RAM, we decrease the dimension of the hidden embedding layer from 16 to although using 16 would have performed slightly better in our experiments the meta-train benchmarks.
For classification, label smoothing with parameter 0.1.
18

---

Since we adapted hyperparameters like learning rate and weight decay from TabR-S-D without meta-learning them, we refer the resulting method as RealTabR-D and not RealTabR-TD. We did not include other tricks from RealMLPP-TD for various reasons:
Brief experiments with NTP and the Mish activation deteriorated the performance. Parametric activations and increased stopping patience showed small improvements but were excluded due to larger runtime.
Other tricks were not tried due to limited time of experimentation, expected increases in the already somewhat large runtime, and/or implementation complexity.
A.4 Details on Cumulative Ablation
Here, we provide more details the vanilla MLP and the ablation steps from Figure (c). For each step, we choose the best default learning rate out of a learning rate grid, using (0.0004, 0.0007, 0.001, 0.0015, 0.0025, 0.004, 0.007, 0.01, 0.015] for NNs using standard parametrization and {0.01, 0.02, 0.03, 0.04, 0.07, 0.1, 0.2, 0.3, 0.4} for NNs using neural tangent parametrization.
Vanilla MLP: We use three hidden layers with 256 hidden neurons in each layer, just like ReaIMLP-TD, and the ReLU activation function. Each linear layer uses stan- dard parametrization and the PyTorch default initialization, which is uniform from [-1/vfan in, 1/ /////aanjiji in for both weights and biases, where fan in is the input dimen- sion. Categorical features are embedded using embedding layers, using eight-dimensional embeddings for each feature. Numerical features are transformed using scikit-learn QuantileTTransf
approximately normal-distributed features. Optimization is performed using Adam with constant learning rate and default parameters 31 = 0.9, B2 0.999, = 10-8 for at most 256 epochs with batch size 256, with constant learning rate. If the best validation error (classification error RMSE) does not improve for 40 epochs, training is stopped. In each case, the model is reverted to the parameters of the epoch with the best validation score, using the first best epoch in case of tie.
Robust scale smooth clip: We replace the QuantileTrane wwrrrrnnsformee former with robust scaling and smooth clipping.
One-hot for small cat.: As in ReaIMLP-TD, we use one-hot encoding for categories with at most eight values, not counting missing values.
No early stopping: We always train the full 256 epochs.
Last best epoch: In case of a tie, we use the last of the best epochs.
coslog, Ir sched: We use the coslog, learning rate schedule instead of constant one.
Adam 32- 0.95: We 83-- 0.95.
Label smoothing (class.): We enable label smoothing with 0.1 in the classification case.
Output clipping (reg.): For regression, outputs are clipped the min-max range observed during training.
NT parametrization: We use the neural tangent parametrization for linear layers, setting the bias learning rate factor to 0.1.
Act. SELU Mish: We change the activation function from ReLU to SELU (classification) or Mish (regression).
Parametric act. fn.: We use parametric versions of the activation functions, with a learning rate factor of 0.1 for the parameters.
Scaling layer: We use a scaling layer with a learning rate factor of before the first linear
layer.
Num. embeddings: PL: We apply the PL embeddings [6] to numerical features.
Num. embeddings: PBLD: We apply our PBLD embeddings instead.
Dropout p 0.15: We apply dropout with probability 0.15.
Dropout sched: flat_cos: We apply the flat_ schedule the dropout probability.
Weight decay wd = 0.02: We apply weight decay (as in Adam Py version) with
value 0.02.
wd sched: flat_cos: We apply the flat_cos schedule to weight decay.
Bias init: he+5: We apply the he+5 bias initialization method from Steinwart 61 (originally
called hull+5).
Weight init: data-driven: We apply our data-driven weight initialization method.
19

---

A.5 Discussion
Here, we discuss some of the design decisions behind RealMLP-TD and possible trade-offs. First, our implementation allows us to train RealMLP-TD in vectorized fashion on multiple train-validation- test splits at the same time. On the one hand, this can lead to speedups on GPUs when training multiple models in parallel, including implementation of certain methods like patience- -ased early stopping or loss-based learning rate schedules. While our ablations in in/pppendxx ].I]shor show the advantage of our multi-cycle schedule over decreasing learning rate schedules, the latter ones could potentially enable a faster average training time through low-patience early stopping. An interesting follow-up question could be whether the multi-cycle schedule still works well with pagger-ptience early stopping.
the benchmarks. On the other hand, it can hinder the
Regarding categorical embeddings, our meta-train benchmark does not contain many high-cardinality categorical variables, and we were not able to conclude whether categorical embeddings are helpful or harmful compared to one-hot encoding (see Appendix B.T). Our motivation to include categorical embeddings stems from Guo and Berkhahn [19] as well as their potential to be more efficient for high-cardinality categorical variables. However, in practice, faster on most datasets. Regarding the embedding size, we found that 4 already gave good results for numerical embeddings and decided to use 8 for categorical variables.
find pure one-hot encoding to be
Additionally, other speed-accurane accuracy tradeoffs are possible. Especially for regression, we observed that more epochs and larger hidden layers can be helpful. When faster networks are desired, the omission of numerical and categorical embedding layers as well as parametric activations from RealMLP-TD can be helpful, while the other omissions in SEaMMLP-TD-. do not considerably affect the training time. Of course, using larger batch sizes can also be helpful for larger datasets.
One caveat for classification is that cross-entropy with label smoothing is not a proper scoring rule, that is, in the infinite-sample limit, it is not minimized by the true probabilities Plylæ) [4. Hence, label smoothing might not be suitable when other classification error metrics are used, as demonstrated in Appendix B.5 for AUROC.
B More Experiments
In this section, we present more experimental results. Note that XGBoost results are affected by a bug where, if categorical value is not present in the training validation set, it could cause adjacent categorical values to be encoded differently during training. validation, and evaluation. This affects the results mainly the meta-test benchmarks, where the SGM scores for XGB-TD and XGB-D around 2% lower after fixing the bug. These differences are not large enough to affect our qualitative conclusions. Due to the large computational cost, we did not rerun XGB-HPO and XGB-PBB-D after fixing the bug, and we provide the old XGB-TD and XGB-D results for a fair comparison to XGB-HPO and XGB-PBB-D.
B.1 MLP Ablations
To assess the importance of different improvements in ReaIMLP-TD, perform ablation study. We perform the ablation study only on the meta-train benchmarks, first because they are considerably faster run, and second because tune the default parameters only the meta-train benchmarks. Since the hyperparameters of RealMLP-TD have been tuned on the meta-train benchmarks, the ablation scores not unbiased but represent some of the considerations that have been made when tuning the defaults. For each ablation, we multiply the default learning rate by learning rate factors from the grid {0.1, 0.15, 0.25, 0.35, 0.5, 0.7, 1.0, 1.4, 2.0, 3.0, 4.0} and pick the best one. Table B.] shows the results of the ablation study in terms of the relative increase of the benchmark score for each ablation.
In general, we observe that ablations often lead to much larger changes for regression than for classification. Perhaps this is because nRMSE is more sensitive to outliers compared to classification error. Another factor could be that the classification benchmark contains more datasets than the regression benchmark. For the specific ablations, observe few things:
For the numerical embeddings, we see that PBLD outperforms PI., PIR, and no numerical embeddings. Contrary to Gorishniy et al. [66, PL embeddings perform better than PLR
20

---

embeddings in our setting. While the configurations with PLR and no numerical embeddings appear extremely bad for regression, we observed that they can perform more benignly with lower weight decay values.
Using the Adam default value of f 0.999 instead of our default / = 0.95 leads to considerably worse performance, especially for regression. As for numerical embeddings, we observed that the difference is less pronounced at lower weight decay values.
Using cosine decay learning rate schedule instead of our multi-cycle schedule leads to small deteriorations. constant learning rate schedule performs even worse, especially for regression.
Not employing label smoothing for classification is detrimental by around 1.8%.
The learnable scaling layer yields improvements around 1.2% on both benchmarks.
The use of parametric activations results in a considerable 4.8% improvement for re- gression but is insignificant for classification. We observed that parametric activations can sometimes alleviate optimization difficulties with weight decay.
The differences between activation functions are rather small. For classification, Mish is competitive with SELU in this ablation but we found it to be worse in some other hyperparameter settings, we keep SELU the default. For regression, Mish performs best,
For dropout and weight decay, we observe that they yield comparable but not always sigmificant benefits for classification and regression. Scheduling dropout and weight decay parameters with the flat_cos cos schedule is helpful for regression, but not for classification in this setting.
When comparing the standard parametrization (SP) to the neural tangent parametrization (NTP), we disable weight decay for fair comparison. Moreover, for SP, we set the learning rate factors for weight and bias layers to 1/16 = 1/ 256 This is because. for the weights in NTP, the effective updates by Adam are damped by this factor in all hidden layers except the first one. Compared to NTP without weight decay, SP without weight decay performs insignificantly worse on both benchmarks. is unclear to us why the parametrization, which has a considerable influence on how the effective learning speed of the first linear layer scales with the number of features, is apparently of little importance.
When comparing the data-dependent initialization of ReaLMLP-TD to vanilla initialization with standard normal weights and zero biases, we see that the data-dependent initialization gains around 1% on both benchmarks.
For selecting the best epoch, we consider selecting the first best epoch instead of the last best epoch in case of tie. This is only relevant for classification metrics like classification error, where ties are somewhat likely to occur, especially on small and "easy" datasets. We observe non-significant 0.4% deterioration in the benchmark score.
We do not observe a significant difference when using one-hot encoding for all categorical variables, since our benchmarks contain only very few datasets with large-cardinality categorical variables.
B.2 MLP Preprocessing
we compare different preprocessing methods for numerical features. Since we want to compare these methods in a relatively conventional setting, we apply them to ReaIMLP-TD-S (without numerical embeddings) and before one-hot encoding. We compare the following methods:
In Table B.2
Robust scaling and smooth clipping, our method used in ReaIMLP-TD and RealMLP-TD-S and described in inecioon 3]
Robust scaling without smooth clipping.
Standardization, i.c. subtracting the mean and dividing by the standard deviation. If the standard deviation of feature is zero, we set the feature zero.
Standardization followed by smooth clipping.
(48) with normal output distribution, which is The quantile transformation from scikit-learn
popular in recent works [16 43
A variant of the quantile transform, which we call the RTDL version, used by Gorishniy et al. 15 and Gorishniy et al. [17]. This version uses a dataset-size-dependee number of quantiles and adds some noise before fitting the transformation. also uses a normal output distribution.
21

---

Table B.1: Ablation experiments for RealMLP-TD. We re-tume the learning rate (picking the one with the best SGM, benchmark score) for each ablation separately. For each ablation, we specify the increase in the benchmark score (SGM2) relative to RealML.--TD, with approximate 95% confidence intervals (Appendix C. and the best learning rate factor found. In the cases where values are missing, the corresponding option is already the default.
mcta-train-olass
meta-train-reg Errer increase %
Ablation
increase in
best factor
MLP-TD (without ablation)
(00, 0.0|
1.0
0.0 [0.0, 0.0]
1.0
Nam, embeddings: PL
0.7 [-0.0, 1.4]
1.0
0.5 [-0.5, 1.6] [13.7, 24.51
1.0
Num. embeddings:
5.27
1.0
0.25
Num. embeddings: None
2..7,.
1.0
20.6 119.4, 21.81
0.25
Adam 02 0.999 instend of 02 0.95
2.4]
2.0
22.8 [21.3, 24.41
0.35
I Learning schedule cosine decay Learning rate schodule = constant
[0.66
1.0
[-0.5, 1.2]
1.8 [0.9.,
0.25
(11,9, 15.0]
0.15
No label smoothing
[1.2,
4.0
No learnable scaling
2..7
2.0
[-0000.
2.0
Non-parametric activation
0.5 [-0.2, 1.2]
3.0
4.8 (3.4. 6.2]
0.35
Activation=Mish Activation-RellU Activatioa=SLLUU
-0.01-0.6,
0.5 [-0.1, 1.2]
2.0
0,7 [-0.1, 166]
1.0
2.3
No dropout Dropout prob. 0.15 (constant)
0.8 [0.2, 1.3]
3.0
0.8 [-0.5,5,
[-.00 0.8]
1.4
3.6 [3.04.4)
weight decay Weight decay 0.02 (constant)
[-0.2, 1.8]
0.5
(-01..11,
-0.3 [.1.
3.0
3.1 [1.7,44444]
1.4
Standard param weight decay
1.1 [0.2, 2.1]
0.5
1.3 [.37 L88
0.7
data-dependent
0.9 L.8
3.0
1.2 (0.2, 2.2]
First epoch instead
0.4 [-0.1, 1.0]
0.0[-0.0,00
Only one hot encoding
[-01.-..1 0.0]
1.0
0.0 0.00
The recent kernel density integral transform 42 with normal output distribution, which in- terpolates between the quantile transformation and min-max scaling, with default parameter
Table B.2 shows that on the meta-train benchmark, robust scaling and smooth clipping performs best for both classification and regression.
B.3 Bagging, Refitting, and Ensembling
In our benchmark, for each training-test split, we only train one model on one training-validation split for efficiency reasons. However, ensembling and cross-validation techniques usually allow additional improvements to models. Here, we study multiple variants for RealMLP-TD and LGBM-TD. Let D be the available data for training and validation, split into five equal-size subsets DI,..., Dz. (When (D) is not divisible by five, D2 U... U Ds Ç 'D since we need equal-size validation sets for vectorized NNs.) Let (X) the predictions on inputs X of the model trained on training set D after & E (1...., T epochs (for NNs) or iterations (for LGBM). For classification, we consider the class probabilities as predictions. Let LD ( be the loss of fD. on dataset D'. Then, we compare the test errors of an ensemble of M = 1 or M 5 models, trained using bagging refitting, with individual or joint stopping (best-epoch selection), which is formally given as follows:
M
Upred
Σ
S 17 (Xrest)
(M models)
M
22

---

Table B.2: Effects of different preprocessing methods for numerical features for RealMLP-TD-S. We report the relative increase in the shifted geometric mean benchmark scores compared to the standard method used RealMLP-TD and RealMI.P-TD-S, which robust scaling and smooth clipping. We also report approximate 95% confidence intervals. To have more common setting, we do not apply the preprocessing methods to one-hot encoded categorical features. each column, the best score is highlighted in bold, and errors whose confidence interval contains the best score are underlined.
Error increase relative robust scale smooth clip in % meta-train-reg
Method
meta-train-class
Robust scale smooth clip Robust scale
0.0 [0.0, 0.0]
0.0 [0.0, 0.0]
[-0.4, 1.4]
9.5 [4.4, 14.8)
Standardize + smooth clip
1.6 [0.9,
1.2 [0.6, 1.8]
Standardize
2.1 [1.2,
8.8 [3.9, 13.9]
Quantile transform (output dist. normal)
2.3 (1.5,
6.3 [5.5, 7.0]
Quantile transform (RTDL version)
2.6
[1.5,
2.6 [0.4, 4.81
KDI transform (a 1, output dist. normal)
4.9
[3.8, 6.0]
4.4 (2.6, 6.21
Table B.3: Improvements for LGBM-TD by bagging or (ensembled) refitting. We perform 5-fold cross-validation, stratified for classification, and 5-fold refitting. We compare compare bagging vs. refitting, one model vs. five models, and individual stopping vs. joint stopping. The table shows the relative reduction in shifted geometric confidence intervals (Appendix C.6). In each column, the best score is highlighted in bold, and errors whose confidence interval contains the best score underlined.
benchmark scores, including approximated 95% %
Error reduction relative to fold in % metn-test--clas
Method
ceen-train-clas.
mein-train-reg
LGBM-TD (bagging. model, indiv, stopping)
[-0.0-000, +0.0]
[-0.0 +0.0]
[0.00-0.0, -0.0]
-0.0 [-0.0, [0.0 -0.0]
LGBM-TD (bagging. model, joint stopping)
-0.2 [-2,4-0..0.1
-0.7 [-1.3,13, -0.21
0.0 [0.00, 0.0]
0.3 [-0.0 0.8]
LGBM- TD (bugging models, indiv, stopping)
3.4 3.71
[3.6, 4.51
[4.5, 6.0]
4.0 [3.6, 4.51
LGBM- TD (hagging, models, stopping)
3.2 (288 3.51
3.9, 3.6]
5.2 (4.5,
4.1 [357.4.5]
LGBM-T TD (refitting, model, indiv, stopping)
4.8 1.51
[-49-0.9
[2.0,
4.0 [3.3, 4.8]
LGBM-TD (refitting, model. joint stopping)
5..5
(4.1.4.6]
5.31.
LGBM-IU TD telitting. 5 models, indiv, stopping)
5.6 [5.2, 6.11
6.0 6.73
13.6..6 6.71
5.5 [4.7, 6.4]
LGBM-TD (refiting. models, joint stopping)
5.4 (5.0,
5.9 [5.6 6.1]
5.2 [3.6,, 6.71
5.5 [4.6, 6.3]
[ \ D. (bagging)
D =
D (refitting),
4 = aargmin LD, (Srrp..) arg.in EE=1 LD, (fD(D,))
(indiv. stopping)
(joint stopping).
Here, each model is trained with different random seed. For LGBM, since we an early stopping patience 300 for each of the individual models, the argmin in the definition of i can only go up to the minimum stopping iteration T across the considered models.
The results of our experiments can be found in Table B. for LGBM-TD and in Table B.4 for ReaIMLP-TD. As expected, five models are considerably better than one. We find that refitting is mostly better than bagging, although disadvantage available, and it is unclear how HPO would affect this comparison. Comparing individual stopping to joint stopping, we find that individual stopping has a slight advantage in five-model bagging, while joint stopping performs better for single-mon mooellloolll refitting. In the other two scenarios, joint stopping appears slightly better for ReaLMLP- and slightly worse for LGBM-TD, We also observe that the benefit of using five models instead of one appears be larger for RealMLP-TD than for LGBM-TD.
refitted models is that no validation scores are
B.4 Early stopping for GBDTs
Figure B. and Figure B.2 we study the influence of different early stopping patiences and metrics on the resulting benchmark performance of XGB-TD, LGBM-TD, and CatBoost-TD. While the regression results only deteriorate slightly for low patiences of 10 iterations, classification results are much more hurt by low patiences. In the classification setting, evaluate the use of
23

---

Table B.4: Improvements for RealMLP-TD by bagging or (ensembled) refitting. We perform 5-fold cross-validation, stratified for classification, and 5-fold refitting. We compare bagging vs. refitting, one model vs. five models, and individual stopping vs. joint stopping. The table shows the relative reduction in shifted geometric mean benchmark scores, including approximated 95% % confidence intervals (Appendix C.6). each column, the best score is highlighted bold, and errors whose confidence interval contains the best score underlined.
Broor reduction relative fold in
Method
meta train class
meta-test class
meta-train rey
meta- tenta-tes--rgggg
ReaiMLP:TD (bugging. model, indiv, stopping)
-0.0 [-0.0, 0.00
-0.0 [-0.0, 00.0
+0.0 [-0.0, -0.0]
-0.0 [-0.0, -0.0
RealMLLP TD (hagging. model, joint stopping)
1.6 [0.9, 2.4]
0.7 [0.0, 1.4]
[0.1 L.]
0.1 [-1.0, 0.7]
bbugging.5 models, indiv. stopping)
6.716.11.
6.7
ReaIMLP TD (hagging. models, joint stopping)
6.7 (6.16.11.6 7.4]
7.2, 8.3]
6.7 16.26.2.7 7.21
4.8 [3.7. 5.8]
P-TD trefiting. model, indiv, stopping)
2.8 2..
3.2.
28 38]
[-0.5 3.0]
ReaIMLP-TD (refitingg model, joint stopping)
(4.55 6.11
(39.
(35,5,5
2.6 [0.9, 4.2]
ReaIMLP-ID trefitiingS models, indiv, stopping]
7.6 8.51
8,8 1.97.9. 9.6]
8.5
5.3 13.9. 6.71
ReaMMLP TD trefitting. models, stopping)
(75.888.9)
8.6 [7.9. 9.3]
8.7 [8.0. 9.4]
5.7
XGB-TD
14
12
10
in
8
8-
Stopping putience
Stopping patience
Stapping
Stqgaai os classiicationn CITE
Eupped or Bricr Josa
wupppd on croe--criogg Josa
Figure B.1: Effeet stopping patiences and metrics on the performance of GBDTs on class We run the XGB-TD, LGBM-TD, and CatBoost- TD with different early stopping patiences (early_stogg soarly_stopping_rowe roundss) We compare three different metrics used for stopping and best-epoch selection: classification error, Brier loss, and cross-entropy loss. The y-axis reports the relative increase in the benchmark score relative stopping on classification error with patience 1000 (i.e.. never stopping early). The shaded areas are approximate 95% confidence intervals, ef. Appendix C.6
different losses for early stopping and for best-epoch selection: classification error, Brier score, and cross-entropy loss. In each case, cross-entropy loss is used the training loss, and classification error is used for evaluating the models on the test sets in the computation of the benchmark score. We observe that models stopped on classification error strongly deteriorate low patiences (S 100), while our default patience of 300 achieves close-to-optimal results. Models stopped on cross-entropy loss deteriorate much less low patiences, but achieve roughly 2% worse benchmark score high patiences. Stopping on Brier loss achieves very good high-patience performance and is still only slightly more sensitive the patience than stopping on cross-entropy loss. An interesting follow-up question would be if HPO can attenuate the differences between different settings.
B.5 Results for AUROC
For classification, there are many different metrics to capture model performance. In the main paper, we use classification error evaluate models. All TD configurations were tuned for classification eтror, early stopping and best-epoch selection were performed for classification eтror, and HPO was performed for classification error. Here, we evaluate models on the area under the ROC curve, also known as AUROC, AUC ROC, AUC. For the multi-class case, we use the one-vs-rest formulation of AUC, which is faster to evaluate than one-vs-one. Higher AUC values are better and the optimal value is 1. Since are interested in the shifted geometric mean error,
1 AUC instead.

---

XGB-TD
L.BM--TD
CatBoost-
0.7
0.0
0.6
in
0.3
0.2
0.1
10
20
50
100
300
1600
10
20
50
100
300
1000
10
20
50
100)
300
1000
Stopping patience
Stopping patience
Stopping patience
Figure B.2: Effect of stopping patiences on the performance of GBDTs on Bresin We run the TD configurations of XGB, LGBM, and CatBoost with different early stopping patiences Saarlysstopinng. rounds). As in the remainder of the paper, use RMSE for early stopping and best-epoch selection. The y-axis reports the relative increase in the benchmark score relative to stopping classification error with patience 1000 (i.e., never stopping early). The shaded areas approximate 95% confidence intervals, cf. Appendix C.6
We compare two settings:
(1) A variant of the original setting where early stopping and the selection of the best epoch/iteration is based on accuracy but HPO is performed on - AUC. (Thanks to using random search, we do not have re-run the HPO for this.)
(2) A setting where we use the cross-entropy loss for stopping and selecting the best epoch/iteration. While it would be possible to stop on AUC directly, this can be sig- nificantly slower since AUC is slower to evaluate. We do not perform HPO in this setting since it is expensive
In both settings, we also evaluate ReaLMLP without label smoothing (no Is), Figure shows the results optimized for accuracy and Figure B.3 shows the results optimized for cross-entropy. We make a few observations:
Stopping for cross-entropy generally performs better than stopping for classification error. Label smoothing harms RealMLP for AUC, perhaps because the stopping metric does not use label smoothing, or because it encourages near-constant logits in areas where the model is relatively certain.
Tuned defaults are mostly still better than the library defaults, except for XGBoost on Biest RealMLP without label smoothing still competitive with GBDT's on the meta-test bench- mark but does not perform better than GBDTs unlike what we observed for classification error.
B.6 Results Without Missing-Vale Value Datasets
To assess whether the results are influenced by our choices in missing value handling and exclusion, Figure B.4fpresents results on all meta-test datasets that originally did not contain missing values. Only six meta-lest datasets originally contain missing values: Three from Biest (kick, okcupid-stem, and porto-seguro) and three from Blest (fps bos__bnncmar,, house poniees_,,iaal,,,, regression). While RealMLP deteriorates slightly, especially due to the exclusion of fps_hbencharkk qualitative takeaways remain similar.
B.7 Comparing Preprocessing Methods for NNs
In the other sections of this paper, we run each NN using the preprocessing from the respective paper that introduced it. Specifically, we use robust scaling and smooth clipping RealMLPP and the RTDL version of the quantile transform for the other papers (see also Appendix B.2). Here,
25

---

oasn diasoifoation benshmank
classification benahmark
Grinsziajh (2002) classiiicaioon bonchmark
Sh gronergreeiceee meon L0
3014
LCOMG
0473
LOWU m
6.445
X33
Certaan-
PlauLa po
Caliecu
pandoeel UP
E:1.5
CofBcces7d
LOON-TD
boster
berter
Culfeeet
1030
banter
Jenso
Dest
aumano
[s] Figure B.3: Benchmark scores on classification benchmarks vs. average training time for AUC, optimized for cross-entropy. BestModel-TDD uses RealMLP-TD without label smoothing. The y-axis shows the shifted geometric mean (SGM,) 1 - AUC as explained in [Section 2.2] The z-axis shows average training times per 1000 samples (measured on gtrain for efficiency reasons), see Appendix C.7 The error bars are approximate 95% confidence intervals for the limit #splits oc, see Appendix C.6
giert without missing value datasets
giss without missing value datasets
0.50
Shifted peometric classification errors
0.14
GB
0.48
GGBMDD
Shitted geometric mean of nRMSEs
Pe207 3
MLP-D
ResNNlD
DesNer
XGB-5
M
11.46
XOB-TO
0.33
CalBocat- D
GGBM-TDD
UUP-PLR-O
ResNesHPQ
LGBM-TD
XOB-TD
Carleostt
TabR-S-D
MLP HPO
Red et-HPO
0.44
LGOM
MPP
MLPiPLADD
HPO
uco
CatBoost-TD
ReaMLP-TD
GBMHHPP
Mu PLR HO
0.42
CatBoost:To
Rea
Best-
Ensemple
onBoost-HPO
Bast-D
ReaIMLP-TD
Ensembic- D
RealTabRRD
11.00
ResiTabR-D
-LP-PLR-IPO
ReaIMLP-TD
HPO
better
Sost.TP
ReaIMLP-HPO
Beet HOW
0,38
better
Ensemble-TT
Ensemble
Ensemble HPO
Ensemble
0.30
20-1
10a Average training time (CPU) per samples [s]
10P
10f
10
10
10at
10a
Average training (CPU) per 1K samples [s]
Figure B.4: Benchmark scores on Blout and B Blest without missing value datasets vs. average training time. The y-axis shows the shifted geometric mean (SGM,) classification error (left) or nRMSE (right) as explained in in(eection 2.2 The æ-axis shows average training times per 1000 samples (measured on gtrain for efficiency reasons), see Aceledpii C.7 The error bars are approximate 95% confidence intervals for the limit #splits -> co, see Appendix C.6
reg
we evaluate if robust scaling and smooth clipping can improve MLP, ResNet, MLP-PLR, FTT, and TabR-S as well. This also yields a more direct comparison of the architectures, although the nets still differ in other aspects such as initialization and regularization.
Figure B.5 includes results with robust scaling and smooth clipping (RS+SC) for MLP, ResNet, MLP-PLR, FTT, and T. While the results look promising for some methods (MLP, TabR) and not so promising for others (MLP-PLR), at least without re-tuning their default parameters, our results also show that trying both preprocessing methods can already give considerable improvements on most benchmarks.
B.8 Results for Varying Architecture
Table B.5) shows the effects of including the preprocessing and architecture of RealMI.P within other models. In particular, study the benefits of our architectural changes, cf. Figure (c).
26

---

Meta-train classification benchmark
Meta-train regression benchmark
Shittd geometric meen of olass ficatton errore
Res
Reship
Reshles (RS+6C)
BesNet-O (oost bors)
MLP
Shita geomelifc mean of nRMSEE
0.000
McP-B-IRS 68
MLP O IRS
DUP-P (RS+SQ)
Reekven- (R5+
MUP
ResNer (Best bosk)
TTT
3,28
MUP- toest
8:008
NEP-G (bestrf poltt
MUP.PUR.D
M
FTTD (RS+SC)
MUP.PLRDD (RS-S0)
FTTD S+
MUP-PLR-D (puat
0:20
FITD Valh
MUP-PUR-D (best
0,004
T Reyus
Tab
0,082
better
foeat bot))
better
AexIMLP-TD
S-D (RS-SG)
TusTTebRDD
0.22
Roa MLP.
0,000
Average training (CPU) per samples (s)
Average training (CPU) per samples (4]
Meta-test classification benchmark
Meta-fest regression benchmark
00:15
Shitted geomettc meen of diessification errors
ResNAEDD (RS-SO)
0,:30
9- both)
Shifted geometri mean of nRMSEs
3.44
MLP
(RB.SC))
MUP-D (best
MLP MUP-PUR- A (RS
0.225
MUP-PUR (RS-SC)
BoR R-D
3,42
MUP-D (PS-50)
MORPOOO
eahios
MuP (bett both!
MUP-PU--O
MLP.D bost of boity
PLR-D (boss both)
0.:20
Tela S. iR
JabR- S
joest of
TabR (RS+SG)
0.15
Tab----D (best of boni)
better
Rea
3,38
better
R.u
0.:10
P-TT
R raneeer
Iff
Iff
Average training time (CPU) samples [s]
Average training (CPU) per samples
Grinsztajn (2022) cleesification benchmark
Grinsztajn (2022) regression benchmark
Shittd geomerrio mean of diesaitlcation enoor
0:000
MUP
MUP-D (RS-S0)
RasNa
NNi (68-80)
Shited geometric mean of nRMSES
0,00
ReeNwe- Ibwo bothy
SssNet (RS.SC)
0.:05
MLP-D fsest af both
aantee D jowst af both)
0,34
AE PXPER-D (C)
MUP-PUO-0
R-R-S-
MUP. D (RS+S0)
FITO (RS-SO)
9,03
0:000
MUP-PER-O SR-0
MUP-PUR- bstti
Fmil to maest
(best
FIT RS-S0)
TabR-9 s----p
bohh)
ReaMLP
SRsS (RS
0.35
BR.SDD (Saai ofaaor
better
3,31
better
Rea/babi-d Tobi-o
RearTabR
TabR-S-D (best bott
0.550
10 Average training (CPU) per samples [8]
10-1
10
10a
10
Average training (CPU) per samples [8]
D defaults
TD tuned defaults
HPO hyperparameter optimization
Best/Ensemble: out of XGB. LGBM, CatBoost, (Real)MLP
Figure B.5: Benchmark scores on all benchmarks vs. average training time. Compared to Figure 2] additional results for robust scale + smooth clip (RS+SC) preprocessing are included. The y-axis shows the shifted geometric mean (SGM)) classification eгror (left) or nRMSE (right) as explained in SSection 2.2 The æ-axis shows average training times per 1000 samples (measured on gtrain for efficiency reasons), see Appendix C.7, The error bars are approximate 95% confidence intervals for the limit #splits -> co, see Appendix C.6
27

---

Table B.5: Comparison of preprocessing and architecture for different models. We include variants with robust scaling and smooth clipping (RS+SC), as well as other modified aspects, cf. Appendix B.8 We report the relative decrease in the shifted geometric mean benchmark scores compared to MLP-D. We also report approximate 95% confidence intervals, cf. Appendix C.6
Erer reduction relative MLP-D in % meta-test-dlas
Methed
trii-trinn-rg reg
meda-ies-rreg
MI.P-D)
[-00-00 -0.0]
(-0.0000-440 -0.0)
MLP-D (RS+SC)
1.5 2.4]
-161--9, -121
-0.7 [-1.6. 0.2]
4.3 5.21
MLF-D (RS-SC, no wd. meti-tuned It)
2.5 [1.8 1.3]
-1.0 00.5
-1.6 -1.6 -0.6)
4.3.333 5.3]
MIF-D (RS+SC, wd. meta-tunal Pl. emhedlding))
4.6 44.4.0 5.2]
-1.5(-1.9.-100)
-10.9 [-12.3, .94]
5.4 6.9)
IRS+SC,
architecture)
(.7 8.5]
[20.0.444]]
96[86,
R.aDMLP-TDDS
[16. 13.2]
138. 14-4]
98 11.2]
13.2 14.3]
RealMI.P-1 P-TD
16.9 [16.1. 17.6)
22.1
22.92
182 16:5]
14. 15.8]
Tubl S-D
[8.2, 10.11
183 19.3]
43
8.9 9.8]
TubR-S-D (RS+SC)
(0(56.8.3
116[104.
ResNet-D
-.9 0.91
6.11-00 5.81
06 0.11
0.6 [-0.4. 166
ResNet-D (RS+SC)
2.0 2133
5.99 -.2)
-L5 0.4]
233 (1.4,,333
when applied directly to the setting of MLP-D. To this end, we approximately reproduce MLP-D in our codebase without weight decay (since the optimal value changes when including the NTP) and with marginally different early stopping thresholding logic. We also determine the best default learning rate on the meta-train benchmark, similar to Appendix A.4 Our reproduction achieves benchmark scores within 1% of the benchmark scores of the MLP-D (RS+SC) version. Adding the PL embeddings from Gorishniy et al. [T6] with our default settings sometimes gives good results but is sigmificantly worse on B indicating that they need more tuning. In contrast, incorporating the RealMLP architectural changes (including their associated learning rate factors) improves scores on all benchmarks by around 5% or more, although they alone do not match the results of TabR-S-D However, the non-architectural changes RealMLP-TTI make an even larger difference.
B.9
Comparing HPO Methods
In Figure B.6 we compare two different HPO methods for GBDTs:
Random search (HPO), used in the main paper, with 50 steps.
Tree parzen estimator (HPO-TPE) as implemented in hyperopt (2). with 50 steps. The first 20 of these steps random search.
While TPE often performs slightly better, the differences in benchmark scores are relatively small.
B.10 More Time-Error Plots
Here, we provide more time-vs-erro plots. Figure B.7 shows results for the arithmetic mean error, Figure shoss results for the arithmetic mean rank, and Figure B.9 shows results for the arithmetic mean normalized error. For the normalized error, the scores are affinely rescaled on each dataset split such that the worst score is 1 and the best score is O.
B.11 Critical Difference Diagrams
Figure 1010 analyzes the external validity differences average ranks between methods, i.e., whether they will generalize new datasets from distribution. While establishing external validity requires a large number of datasets, our meta-test benchmarks show at least the improvements of RealMLP-TD over ML.P-D be externaly valid.
B.12 Win-rate Plots
For pairs of methods, we analyze the percentage of (dataset, split) combinations on which the first method has lower error than the second method. We plot these win-rates marix plots: Figure B.1 shows the results on grrain class Figure 1. shows the results on Bust Figure B.13 shows the results
on B Grinsztajn class
Figure B. shows the results on Btrain Figure B.15 shows the results on Btest and
Figure B. shows the results @Grinszaaan
28

---

Meta-train classification benchmark
Meta-train regression benchmark
LGBMM
0.30
GBM-D
CalBoost-D
CalBook-D
RGB-D
0,066
Shifted geametric dessification
Shilted geometric nRIMSES
HPO
XGB HPO TPR
XGB-HPO-TPE
CalBooat HPO
0.004
XGB-TD
BolBoos:PPOTTPE
Canbe
LGBM-TD
0.26
LGBN-TD
EGBH-IPO
XGB XGB-TD
LGBMHIPOTPE
Gall
HPO-TPE
HPP
0.002
CatBooat.TD
ResIMLP-TD
0,060
better
better
ReaIMLP-TD
0.22
Rea HPP
ReaIMLP-HPO
10-1
1(P
10
102
10-1
10P
101
10F
Average training (CPU) per 1K samples
Average training time (CPU) per 1K samples
Meta-test classification benchmark
Meta-test regression benchmark
0.135
KBB-
GBM-D
Shifted geometric classification
0.130
XG8-0
Shifted geometric: of nRMSEs
0.44
XGB 101
LGBM-TD
0.125
SalBoost-D
CalBoost
LGBM-TD
JBB HPO
0.120
CalBoost-TD
XGB.HPO
CalBoool-TD
XOB+IRO-TRE
XGB-HPO
LBMW-HPO
GBV-HPO-TPE
CafB-
-P--P--PP
LGBM-IPO
aaroo HPD
ColBoost HPO
GatBoost HPO TPE
0.115
LGBV-HRP TPE
0.38
ReaMMLP-TD
0.110
better
better
ReaMLL+HPP
ReaIMLP-HPO
0.305
10-1
111-1
11f
Average training time (CPU) per 1K samples [s]
Average training time (CPU) per 1K samples
D defaults
TD tuned defaults
HPO hyperparameter optimization
Best/Ensemble: out of XGB, LGBM, CatBoost, (Real)MLP
Figure B.6: Benchmark scores of selected methods on gtrain girain Blest and Blest average training time. The y-axis shows the shifted geometric mean (SGM2) classification error (left) or nRMSE (right) as explained Section 2.2 The z-axis shows average training times per 1000 samples (measured on gtrain for efficiency reasons), see Apppendix C.7 The error bars are approximate 95% confidence intervals for the limit #splits - co, see Appendix C.6
29

---

Meta-train classification benchmark
Meta-train regression benchmark
D
0.40
GBM--D
CalBourrD
eosNet
Anthmetti mean ciassiication enors
8,095
MUP
MUR-PLR-O
Artinmetic mean nRMSSS
0:38
ResNet
8,090
DoH--S- S-D
Ret-HI
ResMLPP-
LGBMM-
NUP-PLR-
Calfocae-Hio
XGB-4
MUR-PUR-PPO
0:36
XGB-TB LGBM-
KeB HPO
BLO85
XAGPPBB-0
PTT
MealTabR-G
KOE-HO
Ensembree
Daspo
MLL-PLER HPOO
XGB
ReaIMLP-TD
CatBoost-TD
Ratlicas HPO
0.31
P-HPO
0r080
Ensembre-D
HHHMLP-HPOL
AuR.SDD
better
better
Besi-TD
TrommbleTT
TD
Ersembie
Bes--Pp
Ensembier
RealTaRR-D
Ensembis11P
Bes MRoc
101
1t"
TII
II
to
Average training time (CPU) 1K samples (s)
Average training time (CPU) per 1K samples [s]
Meta-test classification benchmark
Meta-test regression benchmark
0.58
0.190
RF
Anthmete maan diassiiiation orrots
L06-23
MuP
0GB
0.000
XGB.TO
0.50
XGB0
MI P-P. R.D
Ariimmetic mean nRMSEs
RF-O
Gaa-TD
0.185
abR-S-DD
Recket-PP
RPs
LGBM-TTLL
8:00
GenNee-O
0.180
Redla TD--S-
MLP PLR
bre Lerrg--Laaaerr Liaserd
CafBocat
CulBossa
MiL PLR
0.175
Co
ReaMML
giGa HPO
LGBM HPO
0.50
CaMBoosb-TD Ensembto
Towate P-TD-S
ReallMPP-
Jlet-D
LEN HPO
Calllonee Po
8.170
Reer-D
RearTabR
SarBoont HP
RealT
Entembte--
ReaMMLPHPP P
T
ReaML
0.165
better
Hest
better
Ensembre
0.38
0,060
IP
IdI
[G+ Average training time (CPU) per 1K samples [s]
I0-1
lif
Average training time (CPU) per 1K samples [6]
Grinaztajn et al. (2022) classification benchmark
Grinaztajn et al. (2022) regression benchmark
0.40
RF-
MUP-D
Arthmete mean of elassificatin error
MLPHIPO
0.68
RF CcB-h
MLP
0.015
Ra3MLP TD-S
MLP HDO
ARIMES
ResMet
KQB-D
SPaNe-HP
0:07
kGa-TD
TbR-B S
MIP
HP
0,910
LGBI
LOBM-TD
xBs-1
Men-
Tabl----
BeUIMPPTDD
TRF-PR
HPO
M LLP-LR-HPPP
PHMMPPHPP
Anirmeetic
8.46
CatBoool--T
BoUIMLP.TD.C
dawMLPP- Te
UGB #PO
HPoo
HHP
0.200
CotBoost:TD
Ensomble
TO
BMM LAMMA
CarBcaBBooss HPO
0.65
Ensomble D
Beee-TD
Caltooos upo
ReaTTaRR
Tob
better
Ensombio-TT
better
Ensemble- -Bett
Ensomblc HPO
0,300
TPR
Troome GTAP
11-11
1i
16-1
16f
Average training time (CPU) per 1K samples [6)
Average training time (CPU) per 1K samples [s]
defaults
TD tuned delaults
HPO hyperparameter optimization
Best/Ensemble: out of XGB, LGBM, CatBoost, (Real)MLP
Figure B.7: Benchmark scores (arithmetic mean) vs. average training time. The y-axis shows the arithmetic mean classification error (left) nRMSE (right). The z-axis shows average training times 1000 samples (measured on gtrain for efficiency reasons), see Appendix C.7 The error bars are approximate 95% confidence intervals for the limit #splits - oc, see Appendix C.6
30

---

Meta-train classification benchmark
Meta-train regression benchmark
25
RF-0
GenNee-D
D
VL P-H
XGB.D
M M.P-O
15
20 LGB
Antrmetic mean of ranks
G8B
MLP PLR.D
MLP-HPO
1.0BM
XIB-HP
Nes
P.
MLP PLR HP
Anihmmetic of
15
XGB-
Beet-
244
LGBM-PPO
CatBoost-T
LGBM-TD
Ensambee
ReaIMLPTTD
GBB HPO
MUP PUPHHPP
ReR-S-D
CalBooallTB TDP
LGBM-TD
BesiDD GeOTTER
5
CGEM HPO
ReaMMPPHHPP
ErsentiedD
ReaMLPPTP!
ReaIMLPHHPO
ReaTTabR-O
10
Bent: Insemble-11
better
better
HPO
Dest-1Pn
Ensombre 10
Ensemble HPO
1I
1t!
IPI
3tI Average training time (CPU) per 1K samples Meta-test regression benchmark
10
Average training time (CPU) per 1K samples [s]
Meta-test classification benchmark
20
RF-0
22.5
Besliet
LGDM-
RGB
VUP-PLI-D
P-HPO
20.0
M.P-D @
BasNie-D
BeNMMLRTD-S
Anihh matic mean of ranks
66
XtaB-TD
ALP-PLR-
Acith metheelic mean of ranke
MUP-PRRD
Res Hes-
Nes HPQ MEPAPPO
LGBM-TO
1.4
LGBM TD
PalBocsdp
KG--PP
CarBoost-D
XGa
CarBoost-TT To
CatBoosk-TD
ErowmbleD
De
LGIK-PP
ReaIMLP-HP
Eaihocar HPO
12.5
10.0
Ersembie-D
ReaIMLP
Best
KOB HPO
HPO
RealhabR-D D
ILP PLR
60
RealML PHHPP
Bose-TD
Ensemble TD
HPO
better
Ensemble
better
Ersenice
Ensemble HPO
10-1
Id2
101 Average training time (CPU) per 1K samples [s] Grinsztajn et al. (2022) regression benchmark
100
10P
Average training time (CPU) per 1K samples [8] Grinsztajn et al. (2022) cisesification benchmark
XGB-D
XSB G
ResNecHPP
MUP-D
XGB-TO
MLP MLP PLR
M APo
Arthmelic mean of tranns
buuM.P.TD HPO
Arthmmeiic of ranks
20
XGiB-T
g Destas-PPP
LGBM-T
PF+
FITHPO
LGBM-TD
MLP-PaSio
RoWMLL Prr
BeafTaen-D
PURRRPP
FITAPO
CarBoost
CalBooan-
Bea--
KGB HPO
LP-PLR
CatBoost
MPO
ReaIMLP Apo
CarBoos Haok
SeaMMP--HP HPO
Bwen-g
GalBoost HPO
IP
abR+PC
Bes--Po
Ensemb
Test- IPO
better
Ensemble
better
Ensemblc
Ensemblee
Ensambad PP
10-1
111"
1If
162
Average training time (CPU) per 1K samples [e]
Average training time (CPU) per 1K samples [s]
defaults
TD tuned defaults
HPO hyperparameter optimization
Best/Ensemble: out of XGB, LGBM, CatBoost, (Real)MLP
Figure B.8: Benchmark scores (ranks) vs. average training time. The y-axis shows the arithmetic mean rank, averaged over all splits and datasets. The æ-axis shows average training times per 1000 samples (measured on gtrain for efficiency reasons), see Appendix C.7 The error bars are approximate 95% confidence intervals for the limit #splits - oc, see Appendix C.6)
31

---

Meta-train classification benchmark
Meta-train regression benchmark
DeeNerDD
RF-0
Anthimelic of nomalized nRMSEs
D
8.6
Callleatt-
MUP-D
RM-5
LLP-
0-3
ResNar-HPO
C09-D
MUP MUP-PUR-D
TTD
0.5
MLP
MUP HPO
Rew
Hat-HPP
9.4
LGBM-
Galllongsh
XisB Pea D
0.
XGB
Hat +Po
RaxMUP-TD. P-T..
MUP-PUR-PPO
CatBooat
LGBM-TO
Ensemble
MLP-PURHPO
Arthiettc of
6-
CaliBoost-1
XGB-TD
Best-D
TalTabl-
R-uIMLP-TD
HPB
Talllonss
ReMIMLP-TO T-TE
TaIR-S-D
LRON-IPO
ReaMLPPHOO
Ersomble
RewIMLP-HPP
02
Best-To
better
HPO
befter
EnsembleTT
Tab--
$2
EntemblesiP
01
10t1
II
1grt
II
10
11f
Average training time (CPU) 1K samples
Average training time (CPU) 1K samples
Meta-test classification benchmark
Meta-test regression benchmark
Anthmatic mean of normallzzdd classilicatoon errors
Ariinmetic meun normalized nRIMSEs
0.50
MUP
ResVen-D
S D
0.43
GBMID
VLP HPO
8.5
MUP-D
BesNet
0.40
XiBB-T
Gool YoafyLPP LP--TD-
PLR-L
LabR-S-D
Red I-HPO
0.
XGB-TBU
MUP-PLR-O
Carthanhh
MUP HPO
LGBM-TT TD
RaaMMLP-TD
ResNet
MEP-ATO
LGBM-TDT!
XGB-HPO
833
CatBoost-To
XGPPPP
DbRR-S-D
0:30
CuBcoot
MariD
BM-H
Ensemble
Best
Ensembre
Rea
ReuMLPPHHO
LGEXA
0.35
Bast
better
Best-Poo
better
Ensemble-11
0.10
Ensemble-TD TD
Ensembre HPO
Ensemble HPO
0.05
10-1
II
16
101 Average training time (CPU) per 1K samples Grinsztajn al. (2022) regression benchmark
10r1
IF
108
Average training time (CPU) per 1K samples Grinsztajn et (2022) classification benchmark
RF
Ariinmetic normalized nRMSES
Arthmmllic of norallzzdd classitication
KGB
ULP-D
8.6
M.P.D
0.5
XG8
Reshie p
MLP-HPOL
XOB-TOL
MUPHPO
0.5
POaMLP.TD.S
Re M.P.TD.S
TTT
RF HPC
HPO
APH
LGBM
TaRR s
PUPPPA
GBM-0
F.bR..D
FTEHPO
(ona WEKHPO
833 CarBoost
LGBMM-7
LGBM-TT
FITHP
CaracrsA-TD
8.3
CalBaorn
Be T LGalHggo
QualTabito
Best
Bost
SuBorat-Po
MALP--PPP
CaliSoont-TD TD
Best
Callione--Hi
Sest+P heR-
82
ALarTaRR
TaRRHPO
better
Ensertiee
befter
Ersemble-Te
Best--ard
HPo
Engamble HPG
10-1
167
10
10"1
1P
1el
1P
10
19
Average training time (CPU) per 1K samples
Average training time (CPU) per 1K samples
defaults
TD tuned defaults
HPO hyperparameter optimization
Best/Ensemble: out of XGB, LGBM, CatBoost. (Real)MLP
Figure B.9: Benchmark scores (normalized errors) vs. average training time. The y-axis shows the arithmetic mean normalized error, averaged over all splits and datasets. Errors are normalized by rescaling the lowest error to zero and the largest error one. The z-axis shows average training times per 1000 samples (measured on gtrain for efficiency reasons), see Appendix C.7 The error bars approximate 95% confidence intervals for the limit #splits co, see Appendix C.6
32

---

ReMM
X05-0
LCaM
XCD-TD
LCaNCaoo
RacK
CGDM TD
30.0
XGB-7D
RaiM22 lif
9010
LCDM
SUAHIO
Celheee:tiP
Baauuac
lifo
Beald2
Xca-tto
Co
BuNo-3310
T
SEPPORIOO
ReaNes IIPO
LCOM11
CelDueao
XLNarr
Grar
0023 2233 2010
031
[20M EIG
Ta00220
FAR
Ca:DooaTDD
RnNe:
32-0
Cii
BegTaalk
T0
190
XG3-10
XGB-Itro
100M
XENFLD
BexMLSSP
Ceitbessot
Figure 1.10: Critical difference diagrams benchmarks. The plots show the average rank of methods on each benchmark. Horizontal bars indicate groups of algorithms that are not statistically significantly different at 95% confidence level according to Friedman and post-hoe Nemenyi test implemented in autorank (22].
33

---

Meta-train classification benchmark, percentage of wins
RF-O
453 45,8
35.7
8b.1
40.0
$4.2
2913
93.7
D4.0
35.8
23/8 59.0 $7.4 25.3 24.8 21 19.2 26.7
20 25.8 21.6 27.4 24.1 19.6 17.1 16A 16.5
ResNerD
M4 47.0
KGR
65.0
20.2
A5.0
30.7
K1.3
13-3
2015 PA
36.3
25.2
32
23.6
29.5
23
20-4
22
22.0
257
15.0 10.0
MLP-HPO
XGB-0
54.8
88.4
64.0
50.1 40.1
99.4
90.2
58.5
42.5
82.0
28.9
27.2
25.
27:2 94.2
51.1
24.4
21.1
20.8 21.5
FTTO
10.9
90.7
96
57. 19.9
90--
15.4
32.0
23.5
22.4
60.1
50.0
23.0
LGBMO
56.8
67.8
85.0
66.0 56.6
92.7
46-
49.0
46.4
46.7
94.2
90.9
50.6
32.0
27.4
38
38.6
29.3
94.5
94.7
25.8
21
22.4
HPO
20.3
64.2 01.3
83.1
33.3
48
20.4
A4.
43.2
44.5
18.0
38.66
38.4
38
30
30.5
27.5
25.6
26.7
23.1
26.5
XGB-BBB-
MUP-PUR-PO
67.1 66
82. 60
61
99.8
89
85.6
38.4
96
02.7 85
27.6
04.4
Caniloost-O
63
60
10.8
46.6
06.4
28.0
08.8
42.8
27.9
T&
228 67.8 20-4
68.2
94.8
59.3
99.9
61.5
90.7
44.8 40.7
42.8
43.4
44.0
42.3
42.5
30.4
31.5
28.5
HORMHPO
764 20. 63
K3.0
PM
60
50.8
90.7
$1.1
51.7
48-0488
44.5
43.5
40.0
34.7
33.3
98.7
96.7
LGBM-TD
XGB-TD
50. 21.1 67.0
F2.
94
66.1
68.0
08.085:
01.0
62.0
51.8
28.8 58.8
$1.8
48.0
40.7
48.0 47.0
46.2
45.2
43.9
36.4
32.1
50.2
To
783.20.7 66
74
60 91.1
40.7
K6.7.00
48.6
37.6
02.6
CPR
746.74.000
71.0
71.4 76.4 78.7
78.2 64.2 88.8
61
85.8
00.5
61.0
50.8
H0.0
88.0
84.0
53.2 83.0
83.064.
64.0
51.5 42.0 42.0
41.0
97.8
RedMLPPTD
726 7.6.77.31 72979
684
70.9
15-2
94.8
56.711
51.7
55.0
K032 20.3
43-
37.7
36.0
RPO
Hest-POO
50.4 80.8 78.0
78.0
15.
78.8
74.0
7411
73.7
78.4
69.9
48.9
70.8
84.0
69.3
66.9
98.6
88
48.0
50
$1.6
61.7
66.0
$1.8 48.0 8..8
Best- TO
029 81.
75.5
16-
103
P1
03.9
85
m144
TrewmEA-TO
51.
P21
8X-1
Enserrbie-HO
882 82.82.98
20.0
78.
774
774777677
776
75.0
78.6
73.3
78.7
P1.0
73.0
88.8 86.8
60.2
86.266622
03.2
68.8
50.2 55.1 02.6
MUP-D
Neb
ILP&PPO
XGB
MLF-PUL-D
LGBMD
XGB 16844P
PSB.
MUP MUP-PURWIN PPR
Calleosto
Rud P-TD-
TaaRe
BestD
LGBATTD
XGB TD
Cafboot-1
RethaRD
Encembe
hhaMUP-1D
Bon +PO
Best
mowontiee TD
TwaM
Figure 1:11: Percentages of wins of algorithms column algorithms gtrain Wins averaged over all datasets and splits. Ties count as half-wins. Methods are sorted by average win-rate (i.e., the average of the values in the row). When averaging, use dataset-dependent weighting as explained in Section C.3.T
34

---

Meta-test classification benchmark, percentage of row wins
RF-D
#2.0
41.5
46-4
K4.9
R2.6
24-5
37.7
25:1
20-4
27:3
22
25-7
22.8
30.7
20.2
20.0
23.5
22.2 20.5 2012 20.8 15-1 17.5
38.1
18.9
XGB-0
68
12.6
52:1
40.0
46
62.0
41.5
39.6
40.5
44.5
64.8
90.9
$4.5
27.0
067
22.9
25.5
29.1
23.
21.0 28.0
19.0
20.8 18.1
100
31.0
21.
MUP-HPO
91A
50.0
40.3
40.5
15.3
36.4
37.6
34-4
22.0
30.0
30.4 25.4 20.0
19.4
M.P-PR--O
07
95.
32.1
23.5
26.3
00.1
LGBM-D
66.0
511
45-4
34.3
30
20.5
24.
24.5
90
RD-M
629
42.4
98.8
59.8
57.7
99.0
90.8
52-7
46.1
45.8
43.4
95.4
39.2
52.0
94.7
32.7
25
20.2 23.
15.0
X0B-TD
64.0
40.0
48.8
51.5
21.0
MLP-PLR-HPO
R1 05.
397
20.2
82
66
53-4
54.0
51.4
53.11 52.5
45.0
43.5
41.6
N0.8
39.3
33.5
37.4
34.5
33.0
32.3
30
21
20.1
21.8
15.5
HesNet-HHP
80.99.9
99.9
654
72.
66.0
07
63.8
68.4
50.0
61.6
46.0
42.8
48.
38.0
86.8
88.2
36.6
37.0
85.7
34.8
31.6
26.0
34.0
2588
20.4
O
22-
94A
65.5
66.3
63
98.8
96.7
68.5
96.1
97.7
$1.2
48.4 43.9
42.1
37.0
45.3
40.5
40-4
40.1
42.5
31.5 00-0 20.2
XGR-HPO
63
10.9
$3.1
00.00
.7.4
40.0
P6.0
97.6
XGB-P8B-0
65.8
23.0
54
66.5
18.5
00-3
18.0
30.4
97.3
BestO
748
788
77.
71.4
67.6 69
81.8
68.8
00.2 61
88.8
57.9
49.8
45.6
40.0
40.7
42.4
43.
31.0
33.1
27.0
68.9
99.4
40.5
92.8
RedMLPPTD
P4.
09.5
V1.
10
84.2
06
61
57/
54.7
06
65.2
51.2
54.0 51.
47-4
34.7
30.33
20.65
2018
22.5
78
76
789
78.
70
74.0
6
R0.3
66
60.0
63.0
684
68.9
07
08.6
51.7
12.
11-7
14.9
45.0
22.1
01.8
25.4
LGBM-PO
778
11.8
780
72
80.0
79
67.3
79.8
87-
85
45.8
80.0
67
$7.8
54.4
48.0
62.8
96.1
91.2
47.4 52.2
53.1
31.4
23.2
RedMIP-HPO
715
88:1
83.
59.5
85.8
Besl-TD
78
0
77.7
66.4
71
17.8
56.
HPstHPO
84.5
06.
05
Encomble-11 TD
61
81
80.6
78:0
70.4
78.0
87.1
20.9
88.8
01.7
EnsembieeHH HPO
82
94.0
21.
84.0
12.0
02.5
12.0
$2.0
80.8
0313
n0.1
78.2
70.5
23.0
20.2
77.4
73.3
26.8
02.7 62.2
MLP-D
RodNen
ILP.PO
MUP-PURD.
(GBMA-D
Red T
XGB TD
VLP MLP.PLRMPP PLAMPO
TeF-S-D
Redlet4PPO
LS9M
X38
V08-9986
Best-D
CalBost: TD
Pic TD
ReaTaBB
Crventle0
LGBMHMF
RosMLP HPO
Beak-D
BeitHPh
Groemb TD
Ensomble HPO
Figure B.12: Percentages of wins of row algorithms column algorithms on class
Wins are
averaged over all datasets and splits. Ties count half-wins. Methods are sorted by average win-rate (i.e., the average of the values in the row).
35

---

Grinsztajn et al. (2022) classification benchmark, percentage of row wins
30.3 18.7 6
18
5.6
XGB-D
35-8 30.5 95 2014 2014
6 1-2 B272
15:1
14.2 12 15.
10.8 8
5.8
MLP-
S7.
42.5
25.0
SZ 25.5 28 20.0
2..3 3313 30
15.0
DesNet-
90
40.455
38.24
95.0
30.6
20
16
MPO
LGRM-D
ResNet-PP
FT1-
76
70.51
57
M.P-PURDD
26
20.0
32851
10.3
2214
HPO
75
3010
TWe--SO
73
045.8
62.851
LGRM-TD
13.44
FTTHPO
RedIMLP-TD
HPO
78
86.4444
4444
95.0
52.8
22:44 40
XG8-P88-0
01.
P42722
44.485.
433
LGRMHPO
18.7
TD
21.4 P3.
78.0
45.045.
Best-TD
O
73.9
988.
1.866.5
77.
77.844275
56
62.3
TPER-HPO
70.0
Best-HPO
676
Ensomble- D
20
01.1
487
2000000000
FL
PAT 8O.
Ensemble-TD 10
30
1
Ensemble HPO
44
24.266
097
98.7788.
23.80.33
079.278
778777
780.591
RFDD
XGB
Heehee--
NedIMPP-TD-
LGSMD
XOB-TD
FTFD
MLP PLR.D.....
LORATTD
ETTRRO
MINMPPTD
xes PEB
10R44Ni
Saltoosh TD
BeilTD
Bell
GoDresteing
TooR HPP
En Unnertlee7 TD
Eneemole IHPO
TooR
Figure B.13: Percentages wins of row algorithms vs column algorithms on class averaged over all datasets and splits. Ties count as half-wins. Methods are sorted by average win-rate (i.e., the average of the values in the row). When averaging. we use dataset-dependent weighting as explained in Section C.3.T
Grinsztaju, Wins
36

---

Meta-train regression benchmark, percentage of wins
PeaNerD
92.9
442
50.1
48.6
35.9
23,8
17.8
9.8
21.2
14.5
26.7
25.1
13.0
25.4
2888
25.2
9.7
12.0
10.7
122
198
8.5
7.1
4.2
5.8 5.8
45.4
10.5
52
LGBM-D
46.
50.5
45.3
37.4
12.5
17.6
96
23.5
23.5
11.
6.0
MLP-D
00
65.9
22.4
94.8
27.3
29.8
90.5
24.9
28
36.6
22.8
19.5
16.5
21.7
8.0
14.8
8.5
6.5
5.2
CarBoost-D
40.2
40.2
17.8
16.2
26.8
0.4
RFD
04.1
5633 33-4
54.9
50.0
45:0
46.1
47-3
60.0
19:4
34.9
32.5
17.9
123
20.5
22.5
34.8
201
22
5.5
27.4
18.7
9.5
6.0
4.0
3.8
MIP-HPO
60
05.1
56.
72.
53
90.8
49-2
43.1
25.55
30.8
08.9
30.9
38.6
20.0
05.5
99-3
04.7
331
97-4
10.5
17.6
12:1
11:44
6.3
5.1
98.8
30.8
13.5
FTTO
78
98.9
71.3
80.9
67.
67.2
88.9
62.6
61.9
43.0
46.4
48.5
45
48.8
08.1
26.0
28.5
99.8
11.5
20.4
14.4
7.4
$1.2
35.7
20.0
13.5
16.5
9.0
XSB-TD
23
M4A
213
872
19.3
50.22
45.2
47.0
45.0
43.6
41.2
22.0
32.1
24.7
14.0
10.3
Ho
04
838
68
65
61
59.6
63
96.8
61.8
54.0
34
87.8
89.0
38.8
52
20.9
17.8
11.7
BestO
06
98
06
46.0
48.
27.8
34.3
32.3
27.6
10.7
CatBocse
78
01.7
32.4
11.4
13.5
02
64.0
$1.4
54.2
58.7
98.0
45:1
66.4
96.0
50.5
47.7
59.8
40-4
35.2
29.0
22.5
5.5
14.8
XOR-HPO
764
18.
K7A
77.7
99
04.5
67-2
57.3
50.0
54.9
40.0
43.8
40.1
33.0
40.0
79
25.0
P0.1
0.9
Ro3IMLP-TD
88.0
68.7
80.7
66.7
66
75.4
75.3 78.2
74.0
72.0 62.1
83.2
66.9
62
66.0
83.8
64.1
56.0
48.6
28.3
41.8
29.8
26.8
17.6
73-
80.8
50-2
40.
14.0
TWB.SD
B78
P4.
15.
70-2
64
62.0
61
00.6
45-3 6A.1
00-33
40.8
19.0
37.33
34.8
LGBMAHHP
60
94.
92.
78.
72.8
99.0
710
69
61.
10.6
37.0
851
22.0
ReaMPP HPO
90.0
78.9
89.5
96
69
60.
71
64.3
10.6
07.66
R
81
8340
95.7
$1.4
91
20.4
94.7
79.
70
78.8
47.1
72.1
71.0
PCL1
71.5
58.7
62.9
61.0
43.0
40.2
$1.0
90.8
39.0
Freemble-TD
26.3
89.
93.5
18.0
09.0
12
82.2
A0
25.0
22.5
Ensemble HPO
00.096.
06
96.
85.7
96.0
12.0
94.9
98.8
663
863
06.6
94.22422
F1.6
64
94 91.1
929 8244 22.980
66.0668.6
96.8
92.0
84.3
61.0
77.6
XOB-D
MUP
CalBosstD
RF.D
MUP-PURD
MLPNPPO
RechetiP
FTTD
Rad M-T---DD-S
XGB-
Cattooniiing #PO
CaB TD
LGBM-T
Brsemble
RoaMLP TD
MLP. MUP-PLRAAPO
TabR--D
1390HHPP
Beat ReSTERD
Eesenbow TD
PastiPP
Figure 14:4: Percentages of wins of algorithms column algorithms Btrain Wins averaged over all datasets and splits. Ties count as half-wins. Methods are sorted by average win-rate (i.e., the average of the values in the row). When averaging, we use dataset-dependent weighting as explained in Section C.3.T
37

---

Meta-test regression benchmark, percentage of row wins
XGB-0
60.0
35.2
86.7
20.0
28.6
38-1
19.8
27.8
29.5
25.8
21-7
13.6
22.4
12.7
16.
10.0
6.0
17.8
19.
4.2
7.4
8.6
19.8
4.3
8.1
RF-O
35.5
24.0
27.4
16.0
10.2
17/
10.7
13.0
MLF-D
100
ResNat
21.3 64.5 493
47.4
42.1
39.5
39.3
22.6
35.6
36.
36.7
28.8
25.7
22.9
25.5
24.0
19.
21.0
22.4
25.5
192
13.
17.1
11.7
XGB-TD
75.5
16.0
26.
LGBM-D
MLP-PLRO
02. 733
16
00.3
82.6
46.2
46.0
37.0
61.2
28.7
25.3
00.3
29.5
22.9
21
29.8
23.5
18.1
15.9
LG
912 81
1e.
96.7
70.0
67.6
51.4
00.2
43.0
35.0
96.7
58
54.8
81.2
36.
21.2
88.8
24.6
11.1
15.7
P-TDS
RedNat-HPO
205 68.0
08.8
27.4
89.0
53.
5P-4
49-4
10.0
33.6
33.0
30.7
36.0
16.9
MLP-HPO
70.2 68.1
70.7
88.4
07.9
55.0
58.0
92.4
91-2
50.9
48.6
41.4
94.8
95.5
88.6
29.5
89.8
57.1
35.7
25.8
20.5
20.7
15.8
10.2
Galiloost-TD 10
66.0
72.8
06.0
56.4
$1.0
46
42.
06.2
58.6
29
14.5
TebR-S-D
77.8
78.2
74.5
21.2
02.4
61.9
63.3
98.3
58.4
61.7
$6.2
$1.3
B1.A
45.7
46.77
57.6
40.7
35.3
95.9
42.9
39.0
05-0
34.0
27.1
25.7
15.7
72.4
75.0
83.8
42.9
50.8
Re3IMLP-TD
77.8
72.1
XGB HPO
92.8
902
27.1
24.3
25.2
20
61-0
06.3
D21
$2.5
+1.A
44.0
46.7
46.0
48.6
29.0
41.2
60.2
3554
11.5
76.5
28.0
45.2
Real D
ME2
M4.
78.0
2R0
23.3
71
67
KT
66.4
61-
M6.
17.0
13.0
50.1
12.6 50.7
30.0
30.0
18.1
ErwenEAeO
87.6
79.0
99.9
49.3
31.0
HPO
TD
91A4 89.0
84.0
86.7
01
798
76.4
73.2
78
79.6
71.
99.7
88
98.6
68
08
960
66.8
07.6
83.6
27.6 27.4
25.6
12.1
PPPO
67.8
89.0
10
00
Ere
BPOI-HPO
91. 90.1
86.7
88.3
88.0
86.7
88:1
84.3
62.4
81.7
77.4
80.2
74.9
76.7
77.9
72.6
71.8
76.7
68.8
71.9
70.7
M1.4
on.1
56.0
1622
Ensembie-HPO
95.7
66.7
$1.7
00.1
66.0
91.0
94.0
00.2
88.8
67.9
54.9
36.2
88.5
88.1
57.7
86.8
81.8
86.8
88.6
87.8
09.6
60.0
RF-D
MLP
X06-1D
LGBM1
LG00-10
Red
ResNetPPO
NLPAPO
CalloonnO
CalBooe--1 TD
TaaR
Boon
ReaM
XG8 (G84PO
CaBoostHO
NUP-PURHPO
Ersamble
(GB4 HPO
Enzonba
TD
Gnemblee-1P0
Wins are averaged over all datasets and splits. Ties count half-wins. Methods are sorted by average win-rate (i.e., the average of the values in the row).
Figure B.15: Percentages of wins of row algorithms vs column algorithms on B
38

---

Grinsztajn et al. (2022) regression benchmark, percentage of row wins
2017 13 14. 15.4 10.0
X0B-D
56.1
43.5
23.6
26.00 P0
30.7
23 29 po PP.9 PP:1 1 10.0 11
96
211 61 25
144
M.P-D
22.3 88.0
42.
23.4
90.4
23.9 22.8 21
15.0 17.5 19.8
10.0
9/
9.0
8.6
7.5
RFD
100
MUP-PPO
58
48
4
13.6
11.81.1..11111
5.0
XGB-TD
50.0 96.0664
6
45.033 13
45.00.45.0 43.
40.0
22.1
84.6
38.6
20
14.8
19.3
4..0
13.6
0.0
M.P-PURI
77.5
10
36-4
19:3
50-4
56
20.00
1544
43
10.7
Re
LGBN-D
31 4.499 78 37
23.0
4:4
35.2
55
98
20.7
2.5
15.9
8.9
3.8
RF HPO
41.247551
42.8
37.53433
17
97
36.0
27.5
91.8
17.1
10.7
LGAM-TD
77
Am
10
31.2
23.0
10.
RedMi
TeR-8-D
FTTHPO
20. Pa
15
68.1
31 514
45.7 41.4
45-4
00.4
38
22.9
35.4
31.8
53.0
51
27
17.1
17.1
18.5
8.1
HPP
58 mamm
05
6
R6
62 14.
45.4
44.
40
52
32
25
29
29
20.0
17.1
6.4
Caniloost-O
35
92
64
06
38
50
$1.40
Best-D
13.75.0
#2.1 404
43.0
25.7
37
23
81
25.0
22.5
HPO
20.000 498
754788
69.766
67.
91 47.6
453
30.0
27.7
21.4
Best-T
87.992900
43.
72
$1.25
LORMHHP
40
0.7881200000
www.
198 26
TabR-HPO
22.000.000 4
84877.1
80.0
00.8
57.5
403
Ensomble
13897.955
61.4
97.
HPO
Eneemble-TD TO
33.887
74
Ensemble HPO
20.1 42.
24.000 5 weo.oo..
11 24 D0
92.0
12.0
co..a /o..
Nee
X0B-0
RFD
Reshee HPD
NLP4PP
XGB TD
MUP-PUR-D
Raa P-L.--D.-S
FTTD
LGBM
LORM-TD
ReeMLP-1 TD
TauR S.D
FTT#P
calkoott
Callboost11 TD
XGB-PP
CalBoodt PPO
Best TD
LOBMHPO
PPD
Envembe
Ente entembell 1D
EnowedeeHii HPO
Figure 1.16: Percentages of wins of row algorithms column algorithms on Beg
BCrisstaaju, Wins
are averaged over all datasets and splits. Ties count as half-wins. Methods are sorted by average win-rate (i.e., the average of the values in the row).
39

---

C Benchmark Details
C.1 Default Configurations
The parameters for RealMLP - and ReaIMLP-TD-S have already been given in Table A. Table C.] shows the hyperparameters of LGBM-TD and LGBM-D, Table C.2 shows the hyperparameters, of XGB-TD and XGB-D. Table C.3 shows the hyperparameters of CatBoost-TD and CatBoost-D, The parameters for LGBM-D D, XGB-D, and CatBoost- have been taken from the respective libraries the time of writing and are given here for completeness. We also provide tables for MLP-D Table C.4) ResNet-D Table C.6p. MLP-PLR-D (Table C.5), FTT-D (Table C.7). TabR-S-D Table C.8p. and RealTabRRDD (Table C.9 By "RTDL quantile transform", we refer to the version adding noise before fitting the quantile transform.
For XGB-PBB-D, we use the default parameters from Probst et al. [50]. with the following mod- ifications: We use hist gradient boosting since the new default in XGBoost 2.0. Moreover, since we have high-cardinality categories, we limit one-hot encoding to categories with less than 20 distinct values (not counting missing values) and use XGBoost's native categorical feature handling for the remaining categorical features. For RF-D, we use the default parameters from scikit- do not give RF-D access the validation set (to make more similar to other methods that do not use nested -ross-aaliat and encode categorical columns using ordinal encoding with a random shuffling of categories.
C.2 Hyperparameter Optimization
For all methods, we run 50 steps of random search with the search spaces presented in the following. The search spaces for LGBM-HPO (Table C.10), XGB-HPO Table C.IT), and CatBoost-HPO (Table C.12) are adapted from the "tree- "tree-friendly literature, using - The search space for RF-HPO (Table C.13) is taken from Grinsztajn et [[8]
in each case.
For ReaIMLP-HPO, we provide a custom search space specified in Table C.14 The search spaces for MIP-HPO (Table C.15), MIP-PLR-HHOO (Table C.16), ResNet-HPP (Table C.17), FTT-HPO Table C.18), and TabR-HPO Table C.19) are adapted from the literature, with minor modifications
decrease RAM usage.
Table C.I: Hyperparameters for LGBM-TD and LGBM-D. Italic hyperparameters have not been tuned.
Hyperparameter
LGBM-TD TD
LGBM-D
classif.
reg.
num leaves
50
100
31
learning_raa rrae
0.04
0.05
0.1
subsample colsample _ytteee min_data in leaf
0.75
0.7
1.0
1.0
1.0
1.0
40
3
20
min_sum hewp_heesiiiiiiinn lea
le-7
1e-7
le-3
n r_estimators bagging freq max bin darly_stopping stoppinggroundd rounds
1000
1000
100
1
255
255
255
300
300
1000
40

---

Table C.2: Hyperparameters for XGB-TD and XGB-D, Italic hyperparameters have not been tuned for XGB-TD.
Hyperparameter
XGB-TD
XGB-D
classif.
reg.
max max_depth
6
6
learning rate
0.08
0.05
0.3
subsample
0.65
0.7
1.0
shamppee_byree
1.0
1.0
1.0
colsample_bye bylevel
0.9
1.0
1.0
min_child__ weight
5e-6
2.0
1.0
lambda
0.0
0.0
1.0
tree method
hist
hist
hist
n_ estimators
1000
1000
100
max max_bin bin rounds
256
256
256
300
300
1000
Table C.3: Hyperparameters for CatBoost-TD and CatBoost-D, Italic hyperparameters have not been tuned for CatBoost-TD.
Hyperparameter
CatBoost-TD
CatBoost-D
classif.
reg.
boosting type
Plain Bernuuli
Plain Bernoulli
Plain Bayesian
bootstrap_tyy max_ _ex__ learning rate subsample Bagging_tempernen
6
0.08
0.09
automatic
0.9
0.9
1.0
12_leaf_reg random_ thhennh
1e-5
1e-5
3.0
0.8
0.0
1.0
one hwt_hhott_______eeeeee
15
20
2
wet__eestttttonieeeeeeeee jiterationn
20
None
estimators max bin od wait od rype
1000
1000
1000
254
254
256
300
300
None
Iter
Iter
Iter
Table C.4: Hyperparameters for MLP-D, adapted from McElfresh et al.
43
Hyperparameter
Value
Ir scheduler
None 3
n layers
Illaerrs
[128, 256, 128]
Dropout prob.
0.1
Ir
le-3
Optimizer
Adam W
d_ Lembeedding
8
batch_size
128
max_ max_epochs
1000
early stopping patience
20
Preprocessing
RTDL quantile transform ReLU
Activation function
Initialization
PyTorch default
Weight decay
0.01
41

---

Table C.5: Hyperparameters for MLP-PLR-D. The MLP hyperparameters taken from Table C.4 and the PLR embedding hyperparameters are taken as the defaults of the library associated with Gorishniy et al. [16].
Hyperparameter
Value
MLP hyperparameters Num. emb. type
same as in Table C.4
PLR
Num. emb. initialization
le-2
Num. emb. #frequencies
48
Num. emb. dimension
24
Table C.6: Hyperparameters for ResNet-D, adapted from McElfresh et
43
Hyperparameter
Value
scheduler
None
Activation
ReI.U
Normalization
BatchNorm
n_layers
2
dalees
[128, 128]
d_ __hhddd___aaaee factor
2
0.25
residual dropout Ir Beighh_ddeca
0.1
Ie-3
0.01
Optimizer
Adam W
Lembedding
8
batch_size
128
max_epochs
1000
early stopping patience Preprocessing
20
RTDL quantile transform
Table C.7: Hyperparameter search space for FTT-D, adapted from Gorishniy et al. 155 Differences
Gorishniy al. [15 are: We limit the number of epochs to 300 in Grinsztajn et al. 18] fix the batch size to 256 (Gorishniy et al. [15] use dataset-dependent batch sizes and Grinsztajn et al.
B uses 512). We do not adopt the larger patience from Grinsztajn al. [18]
Hyperparameter
Value
n_ layers
3
d t_token
192
d fin factor ffnn dropout attention_ crcppoo residual Seiiuual_rroouut Ir
4/3
0.1
0.2
0.0
le-4
le-5
batch size max_ max_epochs early stopping patience Preprocessing n heads
256
300
RTDL quantile transform
42

---

Table C.8: Hyperparameters for TabR-S-D, taken from Gorishniy et al. 17]. The criterion on batch sizes inferred to match the batch sizes used in the original paper.
Hyperparameter
Value
num_ cem__emeddings
None
d_ d_main context context_dropout multiplier encoder n blocks
265
1.389200715459444357
2.0
0
predictor blocks
1
mixer_noree
mixer_normalization dropout0 dropout
auto
0.38857797499169876
0.0
normalization
LayerNorm
activation
ReLU
batch_size
128 if Ntrain 10K else 256 if Nerain 30K
else 512 if Nerain < 200K else 1024
patience eeeppchs
16
100,000
context_size
96
optimizer
Adam W
0.00031212233641115566
weight_ decay
1. 1.2260552004046615---6
Preprocessing
RTDL quantile transform
Table C.9: Hyperparameters for RealTabR-D.
Hyperparameter
Value
embeddings num. emb. #frequencies num. emb. d embedding
PBLD
8
4
num, emb. frequency_ Preprocessing Add scaling layer Scaling layer Ir factor Label smoothing epsilon
0.1
robust scale + smooth clip yes 96
0.1 (for classification) in Table C.8
Other hyperparameters
Table 10 Hyperparameter seach space for LGBM-HPO, adapted from Prokhorenkova et al. with 1000 estimators instead 5000.
51
Hyperparameter
Space
n_estimators
1000
bagging_fre __ee
1
early_ hecollystoppngg___e __ssssppng_roundee rounds
300
num_leaves
LogUniforminu11, c7]
learning_ran rate
LogUniform{e 1]
subsample
Uniform(0.5, 1]
Seatureffractio
Uniform(0.5. 1]
min_ in leaf min sum hessian in leaf
LogUniformIn((1, e0
LogUniform(e 16 e
lambda_11 lambda
Random (0. LogUniform 16 e²1]
Random (0. LogUniform(c
43

---

Table C.11: Hyperparameter search space for XGB-HPO, adapted from Grinsztajn et al. 18] We the hist method, which is the new default in XGBoost 2.0 and supports native handling of categorical values, while the old auto method selection not available in XGBoost 2.0. We also increase roundds rounds 300.
Hyperparameter
Space
tree method n_estimators seppyysttopppgggooee
hist
1000
300
max_depil
Uniformint(1, 11]
learning rate
LogUniform[Ie-55 0.7]
subsample
Uniform[0.5, 1]
@asampple_btree
Uniform[0.5. 1]
colsample_bye byleeel
Uniform[0.5. 1]
min_child weight
LogUniformln((1, 100]
alpha lambda
LogUniformm 1e-2]
LogUniform[|, 4]
gamma
Log Uniforn(oee-S 7.0]
Table C.12: Hyperparameter search space for CatBoost-HPPO adapted from Shwartz-Ziv and Armon 58
who did not specify the number of estimators.
Hyperparameter
Space
boosting _oose
Plain
bootstrap_type n_estimators
Bayesian
1000
max__eppl od, wait od_type learning_ratt rate
6
300
Iter
LogUniform[c 1]
terppetttee 12 12_leaf_ gandom_strengee one one_hot wot_hooa__________e_eee_aa__oaat max fil_maxxsieeeee size
Uniform[O. 1]
LogUniform[1. 10]
UniformIn([1, 20]
UniformInt(O, 25]
leaf_ wet.eeettttooiteraaeeeee iiteationn
UniformIn(11, 20]
Table C.13: Hyperparameter search space for RF-HPO, taken from Grinsztaja et al. 18]
Hyperparameter
Space
estimators max depth criterion
250
Choice([None, 2, 4], 0=.0. 0.1, 0.1, 0.1,
Choice([gini, entropy]) if classification else Choice([squared error, absolute error)
max_ features
Choice([sqrt, sqrt, log2, None, 0.1, 0.2, 0.3, 0.4, 0.5. 0.6, 0.7, 0.8, 0.9)) Choice([2, 31, p=[0.95, 0.05])
min_ mnin_sample__s_
min_ cii__samples_ees bootstrap
LogUniformint| 1.5, 50.5)
Choice(True: False) Choice([O, 0.01, 0.02, 0.05), p=(0.85, 0.05, 0.05, 0.05))
min it__jjmpptt_ddeetee decrease
44

---

Table C.14: Hyperparameter search space for RealMLP-HPO. The remaining hyperparameters set as RealMLP-TD. For best performance, might be beneficial to use a larger search space for the init standard deviation of the first embedding layer, and to tune the embedding dimensions, as in Table C.16
Hyperparameter
classif.
Num. embedding lype
Choice([None, PBLD. PL. PLR))
Use scaling layer
Choice([Tuue,
p=[0.6, p= 0.4])
Same
Learning
LogUnnifor([[222, 3e-I])
Dropout proh.
Choice([O.0, 0.15, 0,33, p=[0.3, 0,5,000,.,,0.2. 0.2)
Same
for.
Choice([ReT.UI SEI.U, Mish))
same
Hidden layer sizes Weight decay
Choice([[56, 256, 256], [64, 64, 64, 64, 64), [512]). p=[0.6, 0.20
same
Cboice[[0.0, 2e-2])
same
to (1.4) init std. cup
LegUniformm[(00e 0.5])
Label smoothing
Choico([0.0. 0.11. p=[0.3
label smoothing
Table C.15: Hyperparameter search space for MLP-HPO, adapted from Gorishniy al. [15] We reduced the embedding dimension upper bound, and the maximum number of epochs to have more acceptable runtime on the meta-test benchmarks. As in the original paper, the size of the first and the last layers are tuned and set separately, while the size for "in-between" layers is the same for all of them.
Hyperparameter
Space
N < 100, 000
N 100, 000
n I_layers
UniformInt(1, 8]
UniformInt(1, 16]
d hhidde_llaeen d first layer last Jast_layer dropout
Uniformint(! 512]
UniformInt(1 1024]
UniformIn((1, 512]
UniformIn([1, 1024]
UniformInt[1, 512]
UniformInt[1, 1024]
Choice(O, Uniform[0. 0.5])
LogUniform| le-5, 1e-2]
weight decay embedding batch_size
Choice(O, Log LogUniform[1e-4 le-3))
UniformInt(1, 64]
128 if Nerain 10K else 256 if Nurain < 30K
else 512 if Nurain 100K else 1024
Ir sccheuulerrr Optimizer
None Adam
max #epochs
400
carly stopping patience Preprocessing
RTDL quantile transform
Table 1. Hyperparameter search space for MLP-PLR-HPP, adapted from Gorishniy et al. [16] Differences Gorishniy et al. (16] are: (1) For the MIPP part of the search space, we the same space as for MI.,, which includes categorical embeddings and slightly different ranges for some hyperparameters. (2) We shrank the search space for a, recommended by one of the authors in private communication. (3) We reduced the maximum embedding dimension from 128 to 64 to avoid RAM issues on datasets with many numerical features.
Hyperparameter
Space
MLP hy perpaarameeeess Num. emb. type Num. emb. initialization
Table C.15
PLR
LogUniform| le-2, lel]
Num. emb. #frequencies Num. emb. dimension
Uniform[1. [1, 64]
Uniform[1, 64]
45

---

Table C.17: Hyperparameter search space for ResNet-HPO, adapted from Gorishniy et al. 11 We reduced the embedding dimension upper bound, the maximum number of epochs, and the number of layers to have more acceptable runtime on the meta-test benchmarks, As in the original paper, the size of the first and the last layers are tuned and separately, while the size for "in-between" " layers is the same for all of them.
Hyperparameter
Space
N Σ 100, 000
N 100, 000
n_layers hidden layerss d hidden factor hidden hidden_dropou dropout residual_drew dropout
UniformInt(1. 8]
Uniformint(1, 16]
UniformInt11. 512]
UniformInt1, 1024]
hactor
UniformInt[1, 4]
Uniform[O. 0.5]
Choice(O, Uniform(O. 0.5)
Ir weight decay
Log Unifurro[m 1e-5, le-2]
Choice(O, LogUniform[Ie-6 le-3))
d_ d_embeddingg batch_size
UniformIn([1, 64]
128 if Ntrain 10K else 256 if Nirain 30K
else Nersin < 100K else 1024
activation
ReLU
normalization
BatchNorm None
Ir scheduler
Optimizer max #epochs early stopping patience Preprocessing
AdamW
400
16
RTDL quantile transform
Table C.18:
Hyperparameter search space for FTT-HPO, adapted from Gorishniy et al. [17]. Differ- ences to Gorishniy al. [17 are: We limit the number of epochs to 400, and the batch size choices might differ slightly since the criterion in Gorishniy et al. [7 is unclear to
H Hyperparameter
Space
layers
Uniformini[1, 4]
token d
UniformInt[2, 48]
Uniform[25 2/33, 8/3)
fîn ffn_dropout
Uniform[0. 0.5]
attention dropout residual dropout Ir weight_decay satch_size
Uniform[O, 0.5]
Choice(O, Uniform[O, 0.2])
UngUniform| 1e-5, 1e-3]
Choice(O, Log
le-4])
128 if Nerain 10K else 256 if Nurain < 30K
else 512 if Nuraia < 100K else 1024
max_epon epochs early stopping patience Preprocessing
400
16
RTDL quantile transform
n_ headd
8
46

---

Table C.19: Hyperparameter search space for TabR-HPO, taken from Gorishniy et al. 17.. Non- specified hyperparameters chosen in TabR-S-D Taale C.8). For the weight decay, we used an upper bound of le-4 as used in the original code, and not 1e-3 as specified in the paper.
Hyperparameter d main
Space
Uniformint[96, 384]
context dropout dropoutt
Uniform[0.0, 0.6]
Uniform[0.0, 0.6] 0.0
dropout Ir weight_decay encoder n_ blocks predictor n bloloc num. emb. type num. emb. frequencies num. emb. embedding num. emb. frequency_ scale num. emb. lite
meeUniform//e5,, Ic-3]
Choice(O, LogUniform|Ic- lc-4])
UniformInt(O, 1]
UniformIn(1, 2]
PLR
UniformInt[16, 16, 96]
UniformInt[16, 65]
LogUniform[le-2, 1e-2, le2)
True
47

---

C.3 Dataset Selection and Preprocessing
C.3.1
Meta-train Benchmarks
For the meta-train benchmarks, we adapt code from Steinwart 61 to collect all datasets from the UCI repository that follow certain criteria:
Between 2,500 and 50,000 samples.
Number of features at most 1,000.
Labeled classification regression task.
Description made it straightforward to convert the original dataset into numeric .csv format.
Uploaded before 2019-05-08.
We remove rows with missing values and keep only those datasets that still have at least 2,500 samples SamplesSSmme Some datasets are labeled both regression and classification datasets, in which case use them for both. Some datasets contain different versions (e.g., different target columns), which case all of them. To avoid biasing the results towards one dataset, we compute benchmark scores using weights proportional to 1 verssns. In total, we obtain 71 classification datasets (including versions) out of 46 original datasets, and 47 regression datasets (including versions) out of 26 original datasets. Tables C.20 and C.21 summarize key characteristics of these datasets. We count datasets with the same prefix (before the first underscore) being versions of the same dataset for weighting, except for the two "facebook" datasets in Bltsin, which we count as distinct because they are taken from different sources. For regression, we standardize the targets have mean zero and variance on the whole dataset. This does not introduce leakage since all neural networks standardize regression targets based on the training set, and tree-based methods are invariant to affine rescaling.
During earlier development of the MLP, the meta-train benchmark used to include an epileptic seizure recognition dataset, which has since been removed from the UCI repository, hence we do not report results on it.
C.3.2 Meta-test Benchmarks
The meta-test benchmarks consist of datasets from the AutoML Benchmark [3 and additional regression datasets from the OpenML-CTR23 benchmark [2], obtained from OpenML 65
We make the following modifications:
We brazilian houses from OpenML-CTRR3 and exclude Brazilian Brazilian_houses houses from the AutoML regression benchmark, since the latter contains three additional features that should
not be used for predicting the target.
We use another version of the sarcos dataset where the original test set is not included, since
the original test set consists of duplicates of training samples.
We excluded the following datasets because versions of them were already contained in the meta meta-trainine training set:
For classification: kr-vs-kp, wilt, ozone-level-8ie first-order-theoree piaeerr-tthorom--rrooioi provinng Ges-
uueePhaseSeemmnntiiPPreessss Phishing wine-quality-whiee white, nomao, bank- marketing, adult
wine_qualit abalone, OnlineNewsPopularity: Brazilian Brazilian_housess, physicochemical protein, red_wine grid_stability
For regression:
superconductivity, white_wine,
We preprocess the datasets as follows:
We remove with missing continuous values
We subsample large datasets to contain at most 500,000 samples. Since the dionis dataset particularly slow to train with GBDT models due its 355 classes, subsampled it to 100,000 samples.
We encode missing categorical values separate category.
For regression, we standardize the targets to have mean zero and variance 1. This does not introduce leakage since all neural networks standardize regression targets based on the training set, and tree-based methods are invariant to affine rescaling.
level_ 'e noticed later that the ozone level_eee_________________lle___ Ihr and ozone level_Shr 8hr datasets contain less than 2,500 samples, but decided to keep them since already used them for tuning the hyperparameters.
48

---

Table C.20: Datasets in the meta-train classification benchmark.
*samples
largest Acategorics
4177
ameran calls, families
7127
22
callk,
6073
22
species
avila
20867
10
bank marketing
41579
12
5
11
bank marketing additional
39457
19
11
chess
chess krek
28036
erod sourcal mopping
10494
28
default cnedit cand
30000
23
2
eng
14980
electrical gnd, siubiityy simulazed
10000
12
facebook, live sellers, thatland stalus
6622
9
firm tescher clave
10800
16
2
theorem, proving
gos senser drift class
13910
128
gestare phase segmentation me
9900
19
plase_ssgmentatiiee segmeniation_va
9873
32
humnn activity smartphone
10299
561
indoce_iit Joc_building
21048
470
50
indour loc. relative
21048
470
30
insurance benchmark
landeat satimage
6135
36
letter recogniion
20000
16
0
26
madelon
2600
500
magic gamma, velescupe
musrrumm
8124
12
musk
6398
166
0
34165
118
2
nursery
12960
occupancy detection
20560
online shoppers atsention
12330
16
3
2
optical_reogmittee handwwittonndeeeee
5620
Jevel, lhr
ozune level, Shr
1847
72
blocks
5473
10
paaJecogmition beneolititeen_hrrcttteee
10992
phishing
comphieossn bankruptcy__ Iyear
7027
64
polish companies, bankrupccy_ee.... 2year
10173
64
0
companics, bankrajccy
10503
64
polish wwwppaaissssaaaeeeeoeeee bankrupecy__yee
9792
polish compunies, bankrupecy_Syee seismic bumps
5910
64
2584
12
2
skilLcraft
3338
smarpphone humas activity
5744
561
smartphone haman wetttacctttteoeee postural
10411
361
spambase
4601
57
superoondactvityycom
21263
thyroid_ bp
thyroid all Jtypr
3621
17
thyrodd aedd_aaLlin htyo
3621
17
ByyrokLalleg
9621
thyroid_aee
thysoid_dis
3621
17
Sbyoid Jypo
2700
14
thyrod. sick
Bbyrodd
3163
mukiye_ studeat evalnation
3820
32
wall follow, rhhot
5456
wall, follow rohot, 2ahht_22
wiveform
5000
21
waveform_poose
5000
40
4830
wine. qunee_yuulyy.... all
vine_qualit__ype
6497
11
wine quality white
4898
49

---

Table C.21: Datasets in the meta-trai train regression benchmark.
Aramples
Brum festures
festures
largest Acategories
quality
8991
air_quality_coo co2
7624
10
0
oir quility mo2
7715
10
0
qur_quulity,
7718
appliances energy
hejing, pm225
41757
12
bike, sharing coetal
17579
bike sharing toral
sancrshex,
10721
cathon, sanorbee
10721
carbon sanotubes
10721
23055
eyele wewpppewerrrree plant
9568
electiiall grid sabiliiyy simulated
10030
12
facehook, comment volume
40949
38
facehook Jive selleellles thailand_shaee
cities beijing, pm25
19863
five. citics chengdi.
21074
five cities, gumgphou pm25
pm25
20074
14
cities shanglui pm25
shenying pm25
19038
gn sensor drift class
13910
128
sonsor drin cone
13910
128
indoor_Jnc_at Joc
21048
indur
21048
470
indoor loc loog
21048
470
incurnce benchmark
9822
volume
48204
metro interstate muflic volume
48704
noval, peopulpepllsion.comm cump
11934
14
naval, propulsio turb
11934
14
minsery
online news, pupularity
39664
parking birmingham
35717
parkinson moter
5875
2
5875
pressin tettary strictuee
45730
skill craft
3338
18
0
sm12000 dining
4137
smi2010
4137
supercondactivity
21263
travel review ralings
5456
23
wall follow robot 2 follow
5456
wall, follon robot
5456
wine quality all whhil
6497
11
2
1898
After preprocessing, we
exclude datasets with less than 1,000 samples, these were
for classification: albert, APSFailure, arcene, Australian, blooddtraasffusion-srvi--- center, eucalyptus, KDDCup09 aappeppccypppppycc..momm KDDCupO9-UUselling micro-mass, vehi-
cle
for regression: boston, cars, colleges, energy efficiency, forest fires, Moneyball, stwueent_performneeett performance por, tecator, us ssiiceemm exclude datasets that have more than 10,000 features after one-hot encoding. These were maazon_employee_nggm tion).
enerey__efiiiieeeeeeeeeeeeeeeeee QSAR fesh tisiiitty sensory, student
Click_ fhick_preeiction_sinn and if-poiice-nncidents (all classifica-
C.3.3 Grinsztajo et al. 188 Benchmarks
We select the datasets follows:
We use the newer version of the benchmark OpenML.
When a dataset is used both in benchmarks with and without categorical features, we use the version with categorical features.
We exclude the sve_movements dataset since leak in the dataset was reported by Gorishniy al. [17]
50

---

Table C.22: Datasets in the meta-test classification benchmark.
Name
Asamples
largest Reategories
Aclassss
OpenML
Bicrespunse
3751
1776
359967
Diabotest
789
Fastion MINST
70000
794
10
339976
Higgs
500000
28
3601 14
1555
359966
KDDDCup99
MiniBooNR:
130064
50
39990
Satelline
5300
36
389975
293
amizon maapoo--oomerce--eviieeen
1500
10000
50
10050
1778
0
4
359660
christine
1559
eme
1471
2
390959 9
enne-9
1080
856
389957
cowertyne
creditng
1000
7
13
10
168757
dilbert
10000
2000
168909
dionis
dia
fabert
R237
800
168910
gisa
3153
970
189222
guillemo
4296
heiens
27
jannis
83733
54
211979
josmine
2064
136
2
168911
choss. Ppes. endgame complete
kick
72600
14
18
1054
2
359991
mudeline
3140
259
100192 2
numeroa28.6
96320
167120
okcupid-stem
50758
2
17
7019
350903
pet
1458
37
359958
philiypiiee
phoneme
porto seguro
453016
26
31
102
359956 2
360113
quar-bisdog
1055
riceedda
4206
robert
10000
7200
vegment
2310
16
359963 7
shunte
58000
9
7
359987
plates-
sylvine
5124
20
volkert
58310
180
10
359985
1481
2073 10
51

---

Table C.23: Datasets the meta-test regression benchmark.
#iamples
festires
largest Rostegories
OpenMI tack
Airlines DepDelay__
Allanee Claim Severily
188318
14
116
326
233212
Buzzinocialmelia Twitter
50000
77
233213
MUP-200G-regessiie
1050
Merooles,
Manufacturing
368
QSAR-TID-10990
S766
1024
360933
QSAR-TID&11
5742
1024
560932
15
Sontander transaction,
Yolanda
400000
100
317614
sirfeil self noise
1503
361235
verification
friday
7
beazilian houses
10692
33
361 137
california honsing
20640
561255
compeessive strongth
epeRRuages
qpu autiviiiity
$192
21
361256
diamonés
53940
8
1257
399936
fifu Tps. benchmark
19178
27
163
361272
2592
29
14
24
361 768
poographical origin, of music health.
1059
116
56133 161
house 16H1
23784
16
house_ www.peccesseee nominal
1121
36
43
25
359951
huuse sales
21613
20
70
359949
kinSem
kings, county
21613
17
70
261266
miani housing
13932
15
36 1260
500000
9
259
399941
pol
pumadyn32nh
8192
32
361259
quake
2178
3
339920
44484
21
361011
164
solar, flare
1066
2
6
361244
space
3107
6
361623
1
8585
266
399939
transcoding
wave energy
72000
48
361253
ypcnp_4_1
5335
251
339940
Table C.24: Datasets in the Grinsztajn al. 18 classification benchmark.
Name
Asamples
Anum, features
Weat. features
largest Mentegorris
Aclasses
OpenML. tack
Ricresponse
71000
7
361279
Fiigs
500000
24
361069
MagieT:lesope
361066
MiniBooNE:
50
afbert
58252
21
10
14
2
361782
bank-marketing
10578
2
361066
california
2034
compas-tuo-yeams
4056
2
cowertype
423680
10
2
361113
unedit
16714
10
361055
default-of-en ceeei------eeiisecddeeeiens
13272
361 361753
electricity
7
heluc
10000
22
D
2
361278
house 16Ft
15488
16
361063
jennis
pol
10082
26
mosd-sufrry
111763
20
2
361785 2
52

---

Table C.25: Datasets in the Grinsztajn et al. [8 regression benchmark.
Name
Ssamples
Annm. features
Rear. fontures
largest fostegories
OpenMI. rask ID
Allecons
13750
33
361077
Aiflines DepDelay IM
50000
361201
Claims, ClvvsitSevvitty
Sharing Demand
Brazilian bonses
10692
5
361008
Mercades, Benz, Greener Manufacturing
4309
D
359
12
361097
MiamiHousing2001
19932
SCIMM
performance
abalone inllmndttaa
4177
361288 3
4053
361072 2
361093
epo
8192
delays_nurick
zurich, transport
diamonds
53940
8
361096
clevations
16999
16
361074
16H
huuse soles
21613
13
2
hnoses medical cltarges
20640
361078
163065
561294
nyc-taci-gnem--ko22011
pol seatllcrime6
15000
26
361073
53031
2
17
361 161
supercondact
21263
79
361088
topo
5535
252
361287 2
visualizing. soil
8641
2
361094
wine
sprup
53

---

C.4
option in Comparison with Standard Grinsztajn et al. [18] Benchmark
Here, we compare two versions of the Grinsztajn et al. benchmark:
(a) The "new" version, using our benchmarking setup with the datasets of the Grinsztajn et al. [18] benchmark. This version is used in all plots except Figure C.2 and Figure C.3]
(b) The "old" version, which is a slightly modified version of the original code, described in Appendix C.5
The corresponding results for the most comparable metrics are shown in Figure C.1 for the new paper version, and Figure C.2 for the old version. We decided to use the new version for multiple reasons, including a more realistic validation setting, having the exact same baselines, and having more options for evaluation and plotting, Here is list of differences in our adapted version:
In the new version, we removed the eye_movements dataset due to a leak reported in Gorishniy et al. [17].
We subsample datasets after downloading them to 500K samples (all train-test splits are performed on the same 500K samples).
We always standardize targets, to make our results independent of the scaling of the datasets. (In contrast, HPO methods on the original benchmark have standardization their tuning space.)
We limit training+validation set sizes 13333, such that at most 10K samples are used for training. Of these samples, we always use 25% for validation, unlike the original benchmark, which limits the training and validation set sizes separately to 10K and 50K samples.
The new version does not use separate validation sets for early stopping and for HPO, which avoids unfairly disadvantaging and TD methods compared HPO methods.
With the new version, we mostly report results using different aggregation strategies and using aRMSE instead of R2 for regression, but try to provide comparable aggregated metrics in Figure C.1
The new version uses different random hyperparameter configurations on different train-test splits, which should provide more accurate results and allows computing confidence intervals as in Appenddx C.6.
The new version uses ten train-test splits all datasets, instead of smaller dataset-size- dependent number.
The new version measures all runtimes on the CPU, while the old version measures NN runtimes on the GPU.
The old version uses slightly different baseline configurations:
The old version uses (in the code) simplified version of the MLP without dropout and without weight decay.
-
The old version sometimes replaces search spaces like Choice((O, LogUniform[1e-6,
le-3]]) with more simple spaces.
The new version doesn't I use as large categorical embedding sizes for ResNet and MLP models (up to 64 instead of [64, 512]).
The old version uses larger stopping patiences for default models than in the original literature 15].
In the old version, ResNet-HIP tunes the normalization, unlike the original paper [5]] In the old version, the batch size is tuned for some models.
In the old version, XGBoost uses the exact tree method with one-hot encoding, while in the new version, we use the hist method that supports native categorical feature handling. This makes XGBoost slower but also more accurate in the older version.
The old version uses different versions of quantile preprocessing for NN methods, while we use the RTDL quantile transform for all methods except ReaLMLP.
Both the new and the old version use early stopping and best-epoch selection on accuracy (for classification) / RMSE (for regression).
C.5
Closer-to-original Version of the Grinsztajn et al. (18) Benchmark
In the following, we document the benchmark settings for obtaining the results Figure C.2 and Figure C.3 The results were obtained using modification of the original code.
54

---

Grinsztain al. (2022) classification benchmark
Grinsztajn al. (2022) regression benchmark
Arihmelic custom--orrmllieed classification errors
ResNoc-D
0.7
RF
0..
Adthiellic mean of custom-normalizee 1
3.7
R
BasNot- HPO
0.6
XGB-TD
MLP-LR-D DLMLLPTDD
XGB-1
ReeNer-O
0.6
T
0.4
0.5
LGBMM
LGBM-TD
XGB-TD
MLLPPRRD
P-T-B
RLaMMLRTDD
Tan S
X
HPO
FTTC
HesNat-HFF
ALP-PLAHHPO MAILPPPPO FTT-HP
3.4
3.5
3BBM D
CafRon
LGBM
P.TT
Beel-D KGB APD
CaBBoog-TT To
Best T
DF-NPO
ReaIMLP-HPP
LP-PLR-HPO
++0
0.3
CatBoost- TD
CalBoesh
XCBIPR6-D
LGBM-HPO
BealTabRDD
0.
CalRone-
LGDM
RaalabR-D
HPO
Reat++PO
Ensenible
PPs
better
Ensemble-
HPO
0.2
better
Ensemble-To
02
Ersembie
Ens
Ensemble HPO
1Ca
LI
10a
10a
10-3
L0f
16a
10at
10
10
Average training time (CPU) per 1K samples [s]
Average training time (CPU) per 1K samples [s]
Figure C.I: Benchmark scores (custom normalized errors) average training time. The y-axis shows the arithmetic mean normalized error as described in Appendix C.5 averaged over all splits and datasets. Errors are normalized by rescaling the lowest error to zero and the largest error to one. The z-axis shows average training times per 1000 samples (measured on girain for efficiency reasons), see see Appendix C.7 The error bars are approximate 95% confidence intervals for the limit #splits + co, see Appendix C.6
Grinsztajn al. classification benchmark
Grinsztajn al. regression benchmark
0,3
BesNorD
BosNat-D
0.4
MLP-HPO
HPO
GBTD
MLP-D
SAINT-D
Normalzzed accuracy
0.5
RFF- XGB
Normalized R2 score
XG8
XG8-TD
BTT
0.4
RasNer.HP K
SAIT--
MLP-HPO& x SPINTHPO
0.6
Fa- ReaIMLP-TD
SAINTHPO
x
FT-Tranfommer
FIFTransdorme-1PP x
agR
Transtotrrr HBO
GBM-D LOIBM-TO
6.7
HPO
8.6
TabR-S-
ReaioLP.TD L.T.D
GETHPO
AMMPO
RE
H
HBT-PP
CatBoostt-
Carssar
ReaMMLPHPP
RealU P-HPO
0.8
better
CatBoost ID
CatBooss
LGRMM-HO
Best-
SanBoost HPO
088
better
Best TD
LGBM HPO
CalBorat-iP
GGFFPO
0.9
XGB-HPO
10
Average training time per 1K samples (s)
Average training time per 1K samples (8)
Figure C.2: Results on the benchmarks of Grinsztajn et al. [18), using
settings
AAppeddix C.5). The y-axis (inverted) shows the normalized accuracy / R2 score used in the original paper (see Appendix C.5). The z-axis shows average training times per 1000 samples, using GPUs for NNs as Grinsztajn et al. (18), see Appendix C.5
The datasets are taken from the benchmarks described in Grinsztajn et al. [18. When a dataset is used both in benchmarks with and without categorical features, we use the version with categorical features. We preprocess the datasets following the same steps as in Grinsztajn et al. [8]
For neural networks, we quantile- trantiil-ttnnsfomm the features to have a Gaussian distribution. For TabR 17,, we use the modified quantile transform from the TabR paper. For ReaIMLP, we use the preprocessing described in /SSecction 3, namely robust scaling and smooth clipping. For neural networks, we add as hyperparameter the possibility to normalize the target variable for the model fit and transform it back for evaluation (via scikit-leamn's 's Transformed-
55

---

Normalized test accuracy of best model (on valid set) up to this iteration
model (on valid set) up to this iteration
0.9
0.9
Normalized test score of best
CatBoost
0.8
CatBoost
LGBM
LGBM
ReaMMLP
0.7
XGB
TaDR Ferranfffore
FT TrrnnnsfommeRReee MeaM
GBT
0.6
XGB
TabR S SAINT
0.7
RF
0.5
MLP SAINT
GBT 111
RI MLP
0.6
ResNet
0.4
1
10
30
10 30
Number of random search iterations
Number of random search iterations
Figure C.3: Results the benchmarks of Grinsztajn et al. [18), for classification (left) and regression (right), using the closer-to-original settings (Appendix C.5). The plot similar to the one in the main part of Grinsztajn al. [18), with algorithms added. The y-axis shows the result of the best (on val, but evaluated on test) hyperparameter combination up to n steps of random step (x-axis). As in the original paper, normalize each score between the max and the 10% quantile (classification) or 50% (regression), and truncate scores below 0 for regression.
TergetRegressor and StandardScaler, which differs from the QuantileTransformer Transformer from the original paper, as we found it to work better). The same standardization is also applied to all deffult-parameterr versions of neural networks.
For models that do not handle categorical variables natively, we encode categorical features using OneHotEncoder from scikit-learn.
Train size restricted to 10,000 samples and test and validation size to 50,000 samples.
Note that the datasets from the original benchmark are already slightly preprocessed, e.g., heavy-tailed targets are standardized and missing values are removed. More details can be found in the original paper.
Results normalization For Figure C.2 as in the original paper, we normalize the R2 or accuracy score for each dataset before averaging them. We use an affine normalization between 0 and 1, corresponding the score of the best model for each dataset, and 0 corresponding to the score the worst model (for classification) and the 10th percentile of the scores (for regression). We use slightly different percentiles compared to original paper as we normalize across the scores of tuned and default models, and not all steps of the random search, which reduces the number of outliers. Other aggregation metries are shown in Appendix B.10
Time measurement We follow the original paper and run neural networks on a GPU and the other models 1 core of an AMD EPYC 7742 64-Core processor, and we average the time across all random steps (for each random step, the time is averaged across splits). To compute the runtime of neural networks,
restrict ourselves to steps ran on the same GPU model (NVIDIA A100-40GB), which means that we exclude datasets for which we have less than 15 steps of each model on this GPU (leaving us with 11 datasets for classification and 15 for regression). We then compute the average runtime per 1000 samples on each dataset and average them.
Other details We rerun classification results for neural networks compared to the original results to early stop on accuracy rather than on cross-entropy, to make results more comparable with the rest of this paper.
At Athttps://githh www/Leoorri/taaullaaa. htw.aaahh...bbcccmmmmmreereeeeer beettttrrr_bddffaulle we pro- vide code for the adapted original Grinsztajn et al. [18) benchmark.
56

---

C.6
Confidence Intervals
Here, we specify how our confidence intervals are computed. Let Xi, denote the score (error/rank) of method on dataset i and split j. with E {l...., n and j {l,..., m). Then, the benchmark score S can be written as
Σ WK
f(,))
S=9
(1)
where f= id for the arithmetic mean. For the shifted geometric mean, instead have exp and f(x) = log(x + e), E 0.01. We interpret the benchmark datasets as fixed, but the splits as random. For each dataset Xit..... Xim are ii.d. random variables. We first take the dataset averages
72
Zj
Σ wi (X)))
The random variables Xij..... X n. are independent but not identically distributed. Still, for lack of a better option, we assume that the 2, are normally distributed with unknown mean and variance. We know that the Zj are i.idd., hence normally distributed random variables with unknown mean and variance. This gives us a confidence interval [a, for دØ= Zj- Since g is increasing, we hence obtain confidence interval [g(a), g(b)]
the confidence intervals from the Student's t-distribution for
for S 9 j=1 Z3)
Comparison two methods We often compute the error increase in % the benchmark score of method A compared to method B with the shifted geometric mean, given by
S(A)
100
S(B) -1-11)
Here, we leverage that the shifted geometric mean uses g = exp to write
S(A) = 9 - ( f(x(3)))
S(B)
which is of the same form as Eq. @T. Hence, we obtain confidence intervals for this quantity using the same method.
C.7 Time Measurements
For our meta-train and meta-test benchmarks, report training times measured follows: We run all methods on a single compute node with 32-core AMD Ryzen Threadripper Pro 3975 WX CPU, using 32 threads for GBDTs and the PyTorch default settings for NNs. No method is run GPUs. We run methods sequentially on one split on each dataset of the meta cocta-rrain-oe class and meta-train benchmarks. For -artoom--saac--baae or two (other methods) random search steps and extrapolate the runtime to steps. Runtimes for combinations of models (Best and Ensemble) are computed as the of the individual runtimes. We compute the runtime per 1000 samples on each dataset and then average them. For simplicity, we do not use the dataset-dependen weighting employed otherwise on the meta-train benchmark.
HPO methods,
only
(TabR-HPO, FTT-HPO)
C.8 Compute Resources
While we did not measure compute resources precisely, our experiments required at least around 3000 hours on RTX 3090 GPUs and other GPUs, as well as roughly 10,000 hours on HPC CPU nodes (32-64 cores).
C.9 Used Libraries
Our implementation uses various libraries, out which we would like to particularly acknowledge Py [47] Scikit-learn 488, Ray 46), XGBoost [9]. LightGBM (31. and CatBoost [51]. For using XGBoost, LightGBM, and CatBoost, we adapted wrapping code from the CatBoost quality benchmarks [51].
57

---

D
Results for Individual Datasets
Here, provide and compare the results of central methods per dataset. Figures D.I scatterplot comparisons for different models.
D.7 show
Table D.1
and Table D.2
show results Barain
Table D.3 and Table D.4 show results B
Table D.5 and Table D.6 show results on Bleat
class
Table D.7 and
Table D.8 show results on Beet
Table D.9 and Table D.10 show results on B class
Griusztaju
Table D.11
and Table D.12 show results on
BCrinsztajn
reg
58

---

train classtticaiinn benchmark
Meta-train benchmarkk
846
1.0
Best-TD better
Best-TD better
0.4
nRMSE Callsoae-IPOO
0.4
0.6
Classiication cnvo for
CatBoost-HPO better
CatBoost-HH HPO better
0.0
Chasificaiio erner Best-TD
nRMSE for Bex:
Meta-lest classification benchmark
Meta-test benehmark
0.0
Best-TD better
Best-TD better
emvrtto Caffoont-HHO
0.5
BRMSE Cullooee-IIOOO IIPO
0.3
0.4
CatBoost-HPO better
CatBoost- HPO better
0.0
0.1
0.2
0.6
0.0
0.4
0.6
Chasiiiction erner Best (L)
nRMSC for Des: (D)
Grinsstajn (2022) elasifiiation benchmark
Grinsatajn (2022) regression benchmark
0.40
Best-TD better
Best-TD better
0.53
Cha-siScafivn ctror Cyf
0.90
nRMSE Calleone IPP
0.25
0.00
0.90
0.18
CatBoost-HPO hetter
CatBoost- HPO better
0.00
0.00
0.10
0.15
0.20
0.26
0.30
0.59
0.00
0.0
0.2
0.A
0.8
0.5
1.0
Classiscution
Best
aRMSE For Best: (D)
Figure D.1: Best-TD CatBoost-HPO individual datasets. Each point represents the errors of both models on a dataset, averaged across 10 train-valid-test splits. The black line represents equal emror (x v).
59

---

Meta-train classitcctionn benchmark
Meta-train regression beschmark
046
Best-TD better
Best-TD better
0.3
Clussifeation crne MPsHHHOO...
0.4
Best-HPO better
Bcst-HPO better
0.5
0.2
Classiiiation error for TD
oRMSE Beet TD (
Meta-4est classiflcation benchmark
Meta-test regression benehmark
Best-TD better
Best-TD better
0.1
crne
Best-HPO better
Best-HPO better
0.0
0.1
0.2
0.3
0.4
Classifictioon Best (1)
nRMSC Dest (
Grinertajo elassification benehmark
Grinsstaja regression benehmark
0.40
Best-TD better
Best-TD better
0.35
Classffiation etror Ear Beee (1)
0.8
0.00
(1)
0.37
0.20
0.18
0.10
0.2
Best-HPO hetter
Best-HPO better
0.00
0.06
0.10
0.15
0.20
0.20
0.30
0.35
0.40
0.0
0.2
0.4
0.5
0.8
Classifiation
Best (D)
ORMSE
Figure D.2: Best-TD Best-HPO individual datasets. Each point represents the errors of both models on a dataset, averaged across 10 train-valid-test splits. The black line represents equal errors (x y).
60

---

Meta-train classttcaaion benchmark
Meta-train regresiio benchmark
RealMLP-TD better
ReaIMLP-TD better
Cloasiication CarRows-TD
0.1
0.3
For
8.2
CatBoost- TD better
CatBoost- TD better
Classiiiation error RealMLP-TD
sRMSE ReaIMLP-TD
Meta-test classification benehmark
Meta-test regression benchmark
ReaIMLP-TD better
ReaIMLP-TD better
0.3
for CalBoont:TD
0.1
002
CatBoost TD better
CatBoost- TD better
0.0
0.1
0.8
0.5
0.0
0.0
Classificaiion Rew.MLP-TD
=RMSE Rea
Grinertajo dussification benchmark
Grinszzajo regression benehmark
0.40
ReaIMLP-TD better
RcaIMLP-TD better
0.37
erur for CatDoot--TD TTD (1)
0.5
0.90
ORMSE for (L)
0.25
Closiitcation crnr
0.20
0.18
0.10
0.3
CatBoost- TD hetter
CatBoost- TD hetter
0.00
0.06
0.10
0.15
0.20
0.26
0,30
0.40
0.0
0.2
0.4
0.8
0.5
Classitcation RealMLPP-DD
#RMSE Po
Figure D.3: RealMLP-TD CatBoost-TD on individual datasets. Each point represents the errors of both models on a dataset, averaged across 10 train-valid-tes splits. The black line represents equal errors (x y).
61

---

Mela-train classificatin benchmark
Meta-traia benchmark
RealMLP-HPO better
RealMLP-HPO better
nRMSE for Caltloce-HPd POres-PPO. (4)
CatBoost-HPO better
CatBoost- HPO better
0-1
0.5
0.0
0.6
Cassilication for ReaIMLP-HHO
nRMSE for ResIMLP-HIPO
Meta-lest classification benchmark
Meta-test benehmark
0.0
ReaIMLP-HPO better
ReaIMLP--HO better
emvorto Caffoont-HHO
ORMSE far Cuttocit-HO (J)
0.6
0.3
0.4
CatBoost-HPo better
CatBoost-HPP better
0.0
0.1
0.2
0.4
0.5
0.6
0.9
0.2
0.0
Classifiction for RealMLP LPP
oRMSE for ReslMLP (IPO
Grinsztaj dassifcation benehmark
Grinsztajo (2022) regression benchmark
0.40
RcaIMLP-HPO better
RcalMLP-HPO better
0.55
ClassiScotivn ctror Cyllrest-11OO0
0.00
IRMSB for Cuttoost-HI HID (J)
0.37
0.66
0.20
0.18
0.10
CatBoost-HPQ better
CatBoost- BoBoostt-HPOO better
0.00
0.00
0.10
0.15
0.20
0.25
0.30
0.36
0.40
0.0
802
0.8
0.5
Classitcation for RealMLP (4)
ORMSE for RexLMLP HPO
Figure D.4: RealMLP-HPO CatBoost-HPO individual datasets. Each point represents the errors of both models on a dataset, averaged across 10 train-valid-tesl splits. The black line represents equal errors (x y).
62

---

Meta-train classilication benchmark
Meta-train regression benehmmak
Ensemble-TD better
Ensemble-TD better
0.5
Clocaitiatio fo (3)
0.4
0.3
Eor
Best- TD better
Best-TD better
0.9
Classification ertur Emsemble-D
oRMSE Ensemble-TD (1)
Meta-test classification benelusark
Meto-4est regression benelmaakk
Ensemble-TD better
Ensemble-TD better
Clavsiticaioo for Ros-TD (3)
0.5
0.1
Ex
Best-TD better
Best-TD better
0.0
0.1
0.2
0.3
0.0
0.0
0.2
Classification Emsemble
nRMSC Ensemble TD
Grinsstajo classification henehmark
Grissrtajo (2022) regression benehmark
0.40
Ensemble-TD better
Ensemble-TD better
0.53
0.5
cror for Dest-TD ())
0.30
0.25
Oo Box--D
Clastiicatinn eiter
0.90
0.18
0.10
0.3
Best-TD TD better
Best-TD TDD better
0.00
0.06
0.10
0.15
0.20
0.26
0,00
0.55
0.40
0.0
0.6
0.01
0.
Classificaiion
Ensemble TD())
ORMSE Ensemble
Figure D.5: Ensemble-TDD Best-TD individual datasets. Each point represents the errors of both models on a dataset, averaged across 10 train-valid-test splits. The black line represents equal emror (x v).
63

---

Meta-train classification benehmarrk
Meta-train benehmark
RealMLP-TD better
ReaMMLP-TD better
Classffiatiion DDaMMM-PHPOO..
for
ReaIMLP-HPO better
ReaIMLP-HPO better
0-3
0-4
0.5
Classification ernur for RealMLP-DD
oRMSE RealMLPP-DD
Meta-test classtfication benelmmark
Meto-test regression benehmark
ReaIMLP-TD better
ReaIMLP-TD better
arnir S.G.MMPP-POO...
for CrGUMMPPHPHO1
0.1
0.4
0.2
ReaIMLP-HPP better
RcaIMLP-HPO better
0.0
0.1
0.2
0.5
0.0
0.0
0.2
Classification for ReaIMLP
nRMSE RealMLP
Grinsstajo clasdification benehmark
Grinsotajo repression benehmark
RcaIMLP-TD better
RcaIMLP-TD better
0.35
Classifcation erne for (11
0.5
0.00
0.87
0.20
ORMSE for
0.18
0.3
RealMI.P-HPO better
RealMI.-HHP better
0.00
0.05
0.10
0.15
0.20
0:25
0.30
0.36
0.40
0.0
0.2
0.64
0.0
0.
Classification ReatMLP ID()
ORMSE RealMLP
Figure D.6: RealMLP-TD RealMLP-HPO individual datasets. Each point represents the errors of both models on a dataset, averaged across 10 train-valid-tet splits. The black line represents equal errors (x y).
64

---

train clasttication benchmark
Meta-train benchmark
CatBoost-TD better
CatBoost-TD better
Classitiation HPO.stt-POOOO
nRMSE Callsoae-IPO...
0.1
0.3
82
CatBoost-HPO better
CatBoost-Hil better
0.5
0.6
0.0
6-4
Classification ertur CatBoost-TD (1)
sRMSE for CatBeost TD
Meta-lest classification benchmarkk
Meta-test benehmark
CatBoost-TD better
CatBoost-TD better
emorttx Caffoont-HRO
0.3
0.8
nRMSE Cullooxe-IIPO.U L1)
0.4
CatBoost-HPPO hetter
CatBoost-HH HPO better
0.0
0.1
0.3
0.8
0.0
0.4
0.5
Classification CatBoost (1)
sRMSE for Catllcott TD
Grinstaaj (2022) elasifiiation benchmark
Grinsatajn (2822) regression benchmark
0.40
CatBoost-TI better
CatBoost-TD better
0.55
ClassiScafivn ctror Callrost-1POO0
0.90
nRMSE Calleore IPP
0.25
0.00
0.90
0.18
CatBonst-HPO hetter
CatBoost-HPo better
0.00
0.05
0.10
0.15
0.20
0.28
0.30
0.50
0.00
0.0
0.2
0.8
0.5
1.
Classification
CatBoost
aRMSE for CalBeost (L)
Figure D.7: CatBoost-Tue D CatBoost-HPO individual datasets. Each point represents the errors of both models on dataset, averaged across 10 train-valid-test splits. The black line represents equal errors (x y).
65

---

Table D.1: Classification error of untumed methods on datasets in B averaged over ten train- validation-tes -eett splits. When we write a + b, a is the mean error on the dataset and [a - b, a + b] is an approximate 95% confidence interval for the mean in the #splits - oo limit. The confidence interval is computed from the l-distribution using normality assumption as in Anpppiix C.6 In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.
Datasel
ReaIMLP-TD RealTabR-D
T-bR-S-D
MLP-PLR-D
MLP-D
Ca
LGBM-TD TD
XGB-TD
RF-D
nbalonee
0:447:000014
0.45
0.440:000010
0,4534000100
0.418:00004
0458400009
0455:20022
0,51=20.013 0457=00.010
0.14020.004
0.33220.004
0.142:0.004
0.144
0.1441:0001
0.130::0000
0.331
0.31120.005
0.14620.015
unuran calls, families
0.006:00000
0.00
0.008::0.002
0.000
0.0091:0.002
0.00:1:0000
00091100022
0000:10.003
0011::0.003
anurin calls genus
0.007200002
0.00
0.008:20.002
0.011 000002
0.010:1:0002
0:00
0009::0002
0.09:000022 0001220.003
anmmnn calla, species
0.006-00000
0.07700.001
0.010:00:022
0.008==00007
0:002::0002
0.007400001
0008+00022
0008:000022
0.010=0.000
avila
0.00010.0000
0.000:0.000
0.001
0,00::000001
0.016:100022
0:001:100000
00001000001
0001100.000
0.01::0.001
bank marketing
6.089:00.002
0.887:0.003
0.088::0.002
0.089:00.001
0.001::0.001
0.088::00002
0000:1:00002
0.000:000022 0.00
bunk marketing additioal
0.085
0.08:20.002
0.086
2.006-1:0.003
0.005
0.086-1:0.022
0.06:11:0003
0.000000000
0.085:00.022
0.086.10.002
chees
0.005:0.0022
0.01::00003
0.015::0.003
0.010
0:008:100000
0.008.1:0.033
0.01
1:0000
0,00::0.001
0.015:0.003
choss, krek
0.00120.004
0.20:00.00
0.126::00004
0.121
0.141::0000
0.2
0.147:0000
0.14620.003 0 0.20320.017
crowd sourced_ mapping
0.032:1:0001
0.03::00.003
0.028
0.0001
0.007
0.054:1:0001
0.005:1:0003
0.031 1:0000
0.003:10.003
0.05
default credit card
0.179:10.004
0.101:20.004
0.102::00003
0.179
D. 0.1011:00004
0.
0.99:::00004
0.178:100004
0.180:20.004
0.18::200003
state
0.01:20.0022 0.011=0.001
0.107
0.
0.1762200014
D. 11102:00117
0.051 000
000
00002200002 0.08200001
electrical grd stability simulased
0.032:10.001 0.036:00.03
0.009:10.004
0.057-1:0.00
100000
0.053 0.053-10.001
0067:10.003
facebook live sellers thailind status
0.134:00.007
0.137:00005
0.139:200000
0.0
0.107
D. 139:1:0.007
0.131:1:0005
0. 0:
0.133:20.008
0.141:0.004
firm_teache _alve
0.129 0-,006
0.128 0.000
0.1 0,000
0.133
000007
0.134
0.150:10.006
0.149:+0.000
0.009:
0.191
fins order theurem proving
0.179:10.006 0.000
0,162:100.00
0,188
0,181 1:0.006
0,158 10000
O. 0.000:10.006
0.160:10.006
0.162:10.007
gos, sensee drift
0.0051:0.001 0.00620.001
0.004:20.001
0.00610.000
0.005:1:0001
0.0061:0001
0.006:10:000
0.06:20.001
0.006:0.000
gesture_ phase segmentation raw
0.087:1..0006
0.079:10.009
0.079
0.108 0.0000
0.106-1:0.077
1:000
000005
0.006:10.004
0.074:10.005
gesture phane, segmentation vu3
0.323:10.007
0,289 200008
0.293-1.0.006
0.347:0.001
0. 0.368:1:0:008
0.321 1:0007
0.112:1:000
0.316:10.009
0.255:20.006
hin?
0.001:0.002
0.019=0.002
0.020:20.001
0.000
0.020:1:00022
0.021 002
0.019:10002
0.000=0.0002
0.020-0.002
human, simetiyssmtpphoeee
0.000
0.099: 0.000
0.010
0.015-4-0.002
0.008
0.0002
0.008
500001
indoor loc, huildnng
0.002:1.0000
0.002:00.000
0.002:100000
0.000
0.000:1:0.00
0.002.10.000
0.003
1:000000001
0.002-0.0000
0.002:1.0000
indoor indoos_Joc_relatee Joc_relative
0,000-00000
00,00-000005
0.090-00.002
0.084+00.00
0.009:--0004
0000
0.059
0.003
0000-+0.007
0.077=000015
insurance benchmark
0.061 0-0,00
0.060:10.001
0,060 0.0004
0.004
0.00010..00
0.061
10.005
0.059
1.001
0.072.0.004
landaat, satimage
0.007:00.006
0.065
1:0.00
0,007
0.000110.005
0.00010.005
0.000110000
0180:10.004
0.090:0.004
0.019-0.001
0.019=0.0001
0.020--0.001
0.0002
6.000--00002
0.004+00000
0.002
0.033-+00002
0.045-5-.002
madelon
0,3001:0.016
0.401
0:0019
0.403-100001
0.349
0,435 1:0011
0.109-10.010
0.211 1:0000
0.210:10.014
0.220:20.011
magic Jelescupe
0,115200005
0.108
0.100
0,00
0.022::0.003
0.116.1:0.04
0. 0.0
0,118:20.003
0.12:10.003
musbroom
2.000-00.000
00000
0,000
00000
+.900+0.000
0,000
musk
0.003
0.002
0.006
0.005
0.0002
0J007
0.00
0.011 1:0000
0.001
0.002.10.002
0.012:10.002
0.022
nostao
0.022-0.002
0
0.022+00.001
000000
0:0026--000002
0001
00000
0.002
0.0200-000012
0.020-0.001
0.012-00003
0,011
0,022
0,024-1 40200--0.002
0,021
400001
0.006-00002
01024
0.034- 0,001
occupancy detection
0.000110.001
0:007:00 00000
0.0000
0,009 0,000
0,008 1.0000
100001
0.007
100001
100000
caline shoppers Jteention
0.99--0.004
0,101
000004
0.0004
0
+002
00.00+0.000
+0002
0.805
optical recognitson hingwittonddggis
0,011 000000
0.012-0.0022
0,015
000003
-.020-00.000
0.01S
+-0000
4.007
0,0ES
F.000
0
+.005
0.000
level_ Ihr
0.000
0.037
0.000
1.000
0.000
0,035
0.0000
0.006
0.035
0.0007
0.005
0.03
czane_ cease_Jevei_th Jevei_ttr
8071-00.010
0.001-000110
0.002-000012
0.067
0.001
+0100
OU0TD+00012
0007-+00011
page_blect blecks
0.028-10.004
00226=0.000
0.027
000004
0.026
0.000
0.028 +-0004
0,025
10000
0.05
+.0.03
OLOSS 400004
0.026-4-0000
pen oon_recogniion hwn.wwiisen_ehaaaaeeeee characters
0.000
0
1.000
0.007
0.0001
0.007
0.002
1.000
0.007
10.001
0.007
1:0001
0.006
00000
0.01110.000
phishing
0.031 +00.002
0.029--00000
0,029
0.0005
0.002
0.114
0-000Z
0,051
H0002
0000 GO0L
-0.00Z
0.052-000013
comppniesmmmmse____.m bankrupecy_
0,000
0.026
0-0000
0.026
0.003
0,025
0.001
100002
0X21 +0002
polish, companies, haskropacy_. Pyear
8017:-00002
0.017
0.041
000000
0.041
0.000
0.341
1:0.00
+000
0.025
00.000
0.006
polish companies__steeeeoooooo bankrupccy Syear
0,025 --0000 D
0.041
0-00004
0-366--00000
0.0EL
+000
0.00
0.003--0.003
0.040=-0.0E
0.029:00.000
0.030
0.000
0,051
0.0002
0.048:00.00
0,053
0.0002
0.001
10,002
0.006-1.000
0.005
0,0002
0048-0000
polish companies, buukropoy__Syeu
6.037-0.002
0.040)
0.064-00:004
0.056
0.366-2-0000
002
0
0.096-000004
0.050
selsmic bumps
0.068 -0000
0.065-000009
0.066
0.000P
0.065- 0.000
0.0KS
1-0.08
0006-+00009
+0007+00007
0.068-00000
Leraft
0,589 1,0010
0.562
0.601
0:0.012
0.574
0.597
1-0.015
0.500
0.607
10001
0.010
0.395
smartphnne Jumman octiviy
0.005000004
0.035
00005
0.00
0..04-00.000
1:0004
9.055
0.05
0.006-20.004
0.014
smartphone_ Juman, pesturtypostural
0.009-000002
0.008
0.010-00.001
0.011
0-0000
0.013-4-0.002
100001
+0000+00000
+0000+00001
spanslose
8852220.00
0.099-0.005
0.052-20.007
0.051
0.0006
0.054-100006
00
0.000-200008
0.022-0.008
superconducttvity_.dee
D E058800000
0.06320.0022
0.0002
+0.0K22
0:300---0000
0005
0.02
0000--0.005
01099-000012
tbyroid_all_ pp
0.025-0.004
0.027--00003
0,028
0.0003
+00000 0.027
0.029--0.005
9.024 0.024-00004
+:000
0U922 00,02-
0.22-
doyood hyper
6.015-0.0022
0.016=0.005
0.01
0.014-00000
0.018-20.00
0.004-00000
0.014-10000
0.014=0.0005
0.015-0.003
thyroid all_ Jyyo
0.008:10.002
0.012:000022
0.018:1000044
0.,00
0:020-1000002
0.003-200002
0.005
+0,002
0.005
20,002
0.002
dbyroid rrpp
1.009-00000
0.009
0.009
0.0002
0.0133-0.002
0.016---0.00
0.005 400002
0.00-
0.002
0J905 0.0002
0.009
thyrold
0.008200002
0.099-0.0022
0.012:00004
0.006:200000
0.010-00002
0,083-:00001
0000-00000
0.0052000002
0.00=20.001
thyvoid dis
8.013-00.002
0.013=0.0K12
0.014-20:002
0.015
0.016-2-0.00
0.010-1-0002
0L01L
+0.022
0L01D=000055
0.015-00.012
fhyroid hypo
6.11-20.003
0.01420.003
0.014:20.003
0.014-00.00
0:016::00001
0.099-10000
0000:110002
0.012-20.005
0012-00003
thyroid sick
001624000
0.020:000005
0.022-000044
0.019
0:029:2:0000
0,011-:000
0.018-20004
0.001-000033
0.07=00.015
myrood_ssck sick__u
0.000-00.000
00000-00000
0,001
+00000
000000000000
0,000-00:000
0,000-10000
0.000-100000
0.000=00000
0.000:000000
mrkiye stdent evaliation
8.06:00.002
0.01620.002
0.017:00003
0.016:000002
0.033-1:0005
0.016:::0022
0.0661000022
0.07--0.003
0.00350.005
wall follow, 2
00012-000011
0.00
0.00
00000
8:008.2:00002
00000
0.04+10002
0000-20.001
0.001 00000
wall, follow robot_20 24
00199-0.004
0042-000055
0:030-000007
0.006
0:342:40:005
0,002-00001
0102 +0002
0.004=00001
0.007
wall follow whot
0.002:000001
0009-00001
0.012200.03
0.007
0.011:00000
0:000100001
0004-00022
00002200001
0.001200001
waeeform
0.141
0.140:
0.047
0.145
0.145-20.000
0.146+000005
0.149:20000
0.147=0.005
0.145:00.005
woweform noise
0.130.10.006
0.14520.010
0.161
0..012
D. 0.000
0.141:1:0007
0. 0.5:::00077
0. 0:146:::000
0.145:00007 0.50220.008
wilt
6012240.003
0011200002
0:011 000002
0.014200.07
0.0112200022
0.014::002
00142000
0014200003
00162208812
wine wine_quuliy_aall all
0.577
0.853=0.0017
0.352
0.422
0,4124:0.015
00077 0.39::0.007
0.338=0.000
wine quality type
6.004-20.00
0.00220.002
0.001
0.00
0.004:100000
0:00-1:0001
0006100000
00001
0.00620.001
wine quality white
0.371
0.34220.007
0. 0.345.20.008
0.403-20:021
0.400-200008
g. 0:544:000008
0.341:2:000 0.398:2.008
0.300
66

---

Table D.2: Classification error of tuned methods on datasets in class averaged over ten train- validation-test splits. When we write + b a is the mean error on the dataset and [u - b, + bj is an approximate 95% confidence interval for the mean in the #splits - oo limit. The confidence interval is computed from the t-distribution using normality assumption as in Anppeddix C.6 In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error underlined.
Dalasel
ReaMMLP-HPO MLP-PLR-IIPO
MLP-HIP
CuBonsi-IIOO
LGBM-IIP
XGB-IIPO
abalone
0.441=00011
0.451-0.011
0.446 0.007
0.466-20.017
0.463-0,0133
0-59--00016
0.139=00003
0.136:0:005
0.145400004
0.46:20.003
0.130-00:00
0.129=0:003
0.131+000
amnran_calla, fallill
0.006:20.001
0.010-0.000
0.000
0000::0.001
0.00
0.009
0.011+0002
0.006-10.002
0.013-0.003
0.007 +00002
0001000.002
00000
0.008-0.002
0.012+00002
callk species
0.007=00002
0.010=0.005
0:00
0.050200003
0.008=000001
0.009=0:002
0.000200022
avila
0.001:00000
0.001 0:900
0.015-00.002
00144:0004
0.011 000000
0.001 00000
0,00--0000
bank marketing
0088:20.002
0.09020.002
0.091 0.0000
0.22::0.002
0.08:20.002
0.088=0.002
0.000:00002
bank_marken marketing additional
0.04400.003
0.084 0.002
0.00
0.007=0.002
0.033-000022
0.003=000002
0.004++00022
chess
0.00000.004
0.010=0.0014
0.005 0:00Z
0.009=00003
0.006=000002
0.007-
0.011+0000
chess brvk
0.900=:0.004
0.112=00008
0.106
0.100:000010
0.137=0:005
0:149=0:005
000
crowd sourced, siapping
0.031=20.005
0.086=00005
0:51 0.000
0.004
0.05L=20:005
0.0544000004
0000
defusit credir_dee
0.100=00.004
0.00
0.180=00000
0.001
0.179-00.005
0,179=00:034
O. 0.180-+00004
eye
0.015:200011
0.102:0:011
0000
0:0011
0.054:20000
0.050=00001
0.065:0000
electrical, grid stability simulated
0.029=:0.003
0.014
0.946:200005
00064
0.008:0:0044
0.054
0005
facebook live sellers thailand status
0.133:200077
0.137
0. 0.158:100000
0.138:20005
0.132:20.006
0. 136 0.006
0.1 0.138:2000
firm sencher clive
0.125=:0.006
0.15220.004
0.128=00006
0.006
O. 0.042000017
0.143=0.00%
141 2:0000000
pot_ooder_tae theorem proving
0.082:00000
0.082
000006
0.161 0.004
0.160=0.000
00164-00000
gin sensor drift
0.00::00001
0.006=0.001
0.004 001
0.00
0.006:0.001
0000=00000
0:007:00001
gesture plane segmentation row
0066:000000 004
0.007
0.000
0028:00007
0.071 000012000044
0.066-000004
0.000.00004
gesture phass pwt.ppttggggnetttiii.....o seggnemmttioo_eee va3
0.992:0.0099
0.333=00000
0 0,543=000009
0.333+0.008
0.323
0,307 0.005
0.331 000010
0.000200000
0.020=0.001 001
0.020000000
0.002
0.020:0.002
0.020=0.00
0.000 002
human, activity smantphne
0008=00000
0.012
0.011
0.01000000
0.00
0.00700:00
0.011-0000
indoor loc, building
0.002-00.000 000
0.002-00000
0.002100000
0.00210:000
0.002:000000
0.002=0.000
0.000:000000
indoor loe relative
0.60:200004 004
0.062
0.09400000
0.03
0.070:000003
0,156 000
0.062000004
insurence hendlaaak
0.01=000004
0.060/-0.014
00000
0.00
0.062=000005
0.061=00004
0001100004
landaat satimage
0.079::0.005
0.089 006
0.090 005
0.08:10.004
0.084:00004
0.000=0.003
0.088 00008
letter reengniion
0.018:000011
0.08
0.024-00000
0000:200002
0.032200002
0.034=0.002
0.4420000
modelon
0:19::0.019
0.31
0.14
0.20
0.158=00.015
0.176=00000
0. 0.0
magic gamma Ieleccpe
0.115:00003
0.11620.005
0.11510.004
0:20120.002
0.117:0.004
0.118:0.000
0.119:0000
mushroom
0.000:000000
0.000=00000
0.000
0.00010.000
0.000=000000
0.000=0:000
0,800:00000
musk
0.003:10.002
0.007=0.000
0.0091:0.000
0.008
0.010:00.003
0.010
0.0171100003
0.021::0.001
0,021 902
0.024
0005:000001
0.018:0.001
0
0.018:00000
marsery
0.0192000022
0.020
00000
0.22
0000220000
0.021=00007
0.03110003
occupancy detection
0.00:1000011
0.001
0.009110000
0000010.001
0.00
0.006:0.000
0.007:10001
online shoppers stiention
0:08::00006
0.095
0.097 0.005
0009
0.09
0.09R=0.003
0.098
0:9820000 004
optical rezagnition handwrittes digits
0.010:200002
0.03 0.005
000
0.016:200003
0 6000
0.030 004
ozune lhr
0.005:10.008
0.038
0.036
0006:10.007
0.03510.005
0.038=00000
16511000
arane level
0.071:10.011
0.01
0.068 000000 012
0000330001
0.07
0.077=0.014
0.073 007
page blokks
000
0.027-0.000
0.008-000000
0007
0.024:200004
0.036:00000
0,025 0D4
pen_ rewoomeiioooo hundwrilten churacters
0.004:00001
0,007 0 002
0,007 0.000
0.00:100.00
0.006:10.001
0.006 0.000
0.009 1.0 001
phishing
0.000:00002
0.033_0.005 003
00000
0003
2.0002
0,030
0.033 002
polish compunies bankrupecy__ Iyear
0.018:10.033
0.025
0.025 0.0000
000000
0.001
0.000
0.022:10.00
polich, compunies, bonkrppcy__.. 2year
0.016:20.022
0.035
0.041 1.0000
00000:000003
0.024:20.003
0.024 0.024
0.03
polish compunies bankrupicy Syear
0.22220.002
0.637 004
0.0432000004 004
0000
0.030=200002
0.031 00 004
0.002 004
polish_ companies_ cumpanies_bankcnppccyy
0000:100002
0.000 0.001
0003:10.002
0.00110.0002
0.052
0,004
polish compunies bankrupicy_ Syear
0008:10.003
0.055 005
0.064 1: 005
0000010000
0.000
0.053=0.000 0.004
0.006 004
seismic humps
0001-20.000
0.008
0.070 00S
0006-20.009 009
0.09 000
0.071
- 0.000
0.000 0O8
skill_craff
0:81:10.012
0.010
0,599
1.0013
0.007
0,609 0,0055
0.602
0.05
smurtyhone human, sctivity
0.0M1100006
0.06520.007
0.05110.000
0.02-10.005
0.040 0.000
004
smartphone_ human hwwanusctiiityya....mooooouamiee petvuityooootuurll
0002-+0.002
0.012=0.002
0.011 000002
0.014++0.002
00.000
0. 0.000
0.011 +-0002
spambuse
0.051:10.007
0.055
0.053-10.005
0003:100007
0.046:20.005
0 0.000
0.055 0.000
sopurconluctivity.. class
0.089:200003
0.060 902
0,061 2: 002
0.01
0.05
0.05k=00003
0.067 002
thyruid bbp
00004
0.026 0.005
0.025 0.0000
0.029 0.005
0.025 0.000
0.025
0.027 0-003
thyridd all hyper
0.014:00002
0.015 0.002
0.018:1:0000
10.004
0..00
1015-0.002
0.015 0.000
thyroid all hopo
0.00-10.002
0.010=000022
0.021 0.000
0.021 0:000
0.002
0.0002
0.005
1 002
sbyruid rt
0.010:1:0003
0.009 0.003
0.015 0.004
0.005 0.003
0.T 0.003
0.007
1.0002
ann
0.005:10.011
0,000
0.003
0.00200.002
0.000
0.000
0,002
0.001
thyroid_dis
0.03-+0.004
0.014
0.01
00000
0.010
0.005=
0.003
O-00S
thyroid Iypo
0.00
0.011 0.003
0.016
0.005
0,011
0.011 0.004
0,009 1 0.000
thyroid sick
0.011:20.003
0.000
0.026100000
0.005
0.003
0.005
0.017 00001
dhyrodd sick_ss cu
0.000-00.000
0.000
00000-00000
0.000
0000000:000
0.000--00000
narkiys studessteeeeeeee evaluation
0.016:10.002
0.052 1.0004
0,21 10,004
00.00
0.01900.00
0.0L7 0.0L7.1 0.0003
wall follow robot 2
0.002-10.022
0.003=0.000
0.011 0.0000
0.00510.000
0.002-00.001
0.003
0,001 010000L
folloooloow_ 24
00001+00004
0.012--00015
0,041 000006
000004
0.003
0.005
0L004 4-0002
follow 4
0.002-10.01
0.001
0.005
0.000110002
0.003
0.002
0.003
0.002 0.001
waveform
0.136-20.005
0.106
0.136.500000
0.140
0.145:20.005
0.152
0.148 0.148-00005
wnveform_aosee
01.01-+0.008
0.137 +000007
0.143-+0.007
0.L33---UD0S
0.145 -0,00
0.146-+-00077
wilt qinei_uuality all
0.003-10.003
0.02200.00
0
0.000
0012
0.016 0,003
0.015100000
0.367=00001
0.083-00012
0.0014
0.386-00.010
0.351 +00009
0:000
0.350-00001
wine wine_qualing _ualety jype
0.005 000002
0.005 0.001
0.004
100001
0.000+00001
10006-000000
0.000 0.0000
0-0002
wine quwltgullttyyyee white
0..008
0.001
0.378:10.005
0.775110.010
0.346 0.000
0,336 0.011
1..2110.016
67

---

Table D.3: nRMSE of untuned methods on datasets in B averaged over ten train-allidation-test splits. When write = b, a is the mean error on the dataset and Ia - b, + bj is an approximate 95% confidence interval for the mean in the #splits - oo limit. The confidence interval is computed from the i-distribution using normality assumption as in AAppendix C.6 In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.
Dutsset
ReaIMLP-TD
RealTabR-D
IaRRSSD
MLP-PLR-D
MLP-D
Cathooot:
LGBM-IL
XGB-TD
RF-D
sir, quality be
0.005
0.00
0.05520004
0.00000005
00434:0001
0.03120.007
0.030:00005
0.092:200007
0.013::00004
uir puullulllyyee sa2
0.305:10.018
0,23-00.010
0.1611000111
0.95200017
0:97:1:00017
0:40::000155
0.261:100017
0.278:100018
0.291:1:0006
air quality no2
0.315:20.004
0.261_0.009
0.281:2:0005
0.33520.006
0.3354:00008
0.284:2.005
0.2941:0000
0.294:20.066
0.323:2:0.06
air quality nox
0.287=00.16
0256=6.019
0.361:40.019
0.267=0.015
0.2882000014
02542000220
0.252::0019
0.256-20.017
0.257
appliances, energy
0,760:100115
0.604-0.11
0.651:100004
0.770:0.014
0.7961:00011
0.687:00007
0.699100007
0.67:110006
0.706110000
bejing pm25
0.310:20.006
0.256-0.008
0.279:1:0005
0.80
0440:100012
0.39420.007
0.577:1:0000
0.378:10.006
0.430100000
bike. sharsshhinngg.n casual
0.262:10.006
0.257-0.005
0.201:10.007
0.292:0.0066
0.2691:00005
0.276:10.006
0.278:100.07
0.286:10.008
0.306.1:0006
bike sharing total
0.213:10.005
0215=0.007
0.201:1:0.05
0.225
0.207:10.006 0.2341:0006
02111:00006
0,215:10.006
0.242.10000
carhon
0.010:00.000
0.01000.000
0.013::00001
0022200002
0.006100001
0.007:20.000
0.0001000000
0.010:0.0000
1.000000000
carbon, nanotbees_
0.010:10.000
0.010-0.000
0.014:1:0001
0.02::0.002
0.023:1:0.005
0.007:00.000
0.009110.000
0.010:10.000
0,011 1:0.000
carbon nanotubes_
0.000:000013
0.050-0.015
0.053:1:0001
0.054200011
0.06 1:001000010
0.052:100022
0.056:1:0000
0.08:10.000
0.060:1:0.00
choss tryk
0.09520.005
0.025=0.005
0.137::0000
0.22220.055
0:122::00006
0.26
0.236 004
0.207:20.004
0.300100014
cycle pockeppwer__tt plant
0,215:10.05
0.169:10.005
0.005
0:223-100000
0.02::0.001
0.104110.011
0.00:00000
1:0000
electrical gri, stability simulsted
0.143:00.03
0,149=0.003
0.178:1:0004
0.00
0.187110004
0.40
0.217:1:0000
0.251
200000
0,331 1:0.00
cooebonk_commett.. volume
0.225
0.646-0,034
0.007+000041
0
0-402+000025
0..03
0.0045
0.602 00000
0.599++0..04 F-0.045
facebook live seilers,_hhieee thailand shares
0,565:00.39
0,083=0,034
0,498
0:003
0.006
0.500.10.000
0
00038
0.83-1000
0.183-10.038
0.111100.000
five cities beijing, pm25
0,200:100020
0,244-0.009
0.254:1:0005
0.05620.015
0/181100008
0.000
0.345:100007
0.358
0.410100007
five_sities_ poe_ssites_ehennaee chengchh_ppmee pm25
0.269:10.010
0,205 0.006
0.214:1:0006
0.004
0
1.000
0.315
0.004
0 1.0006
0.326 1-0000
live_ oee_s guangehou pm25
0.401:10.014
0,317-0.015
0.3011100012
0.00
0
0.518110.011
0.050
0.151-1000011
0.157:10.012
0.ASS 1.0.000
live cities shanghai, pm25
0,318:0.014
0.229-0.007
0.2541:00088
0.432000031
0:451:0.011
0.38620.006
0.386:1:0006
0.398
0.40010.0100
five ive_cities, shenyung pm125
0,333:10.019
0.283 0.014
0:000-
0.020
0.520---.0118
0.015
0,4191-0014
0.130-10.015
0.519-10011
gas sensor, drif, clas
0.082:100012
0.079-0.009
0.003
0.00
0.079:100008
0.121
0.128.1:0.008
0.132:10.008
0.139:100.088
pai_sensor_ddme cane
0.162:00.13
0.169 00013
0.146
0.171
0.015
0.170+-0.019
0.168=00015
0.172+-0.016
0.173++0.015
0.172-+00014
indoct Joc_alt
0.009
0,128 0.006
0.181
0,121
0.004
0.187:10.005
0.066:10.003
0.108
1,000
0,171 1:0000
indoor
0.079-10.004
0.092-0.005
0.110:100001
0.112:1::001
0:105:10.000
indooe Joc Joc_oon
0.058:00.004
0.072 0.004
0.083++0002
0.000
0.0001000002
0.08
0.000-1-0000
+0.00
0.0744+-0000
insurance, benchmark
0.978:0.000
0.982-0,008
0.984 10.00
0.007
1.0007
00007
1.85-100.001
0.986 10,003
L.07 1.0.013
metro interstale Irallic volume long
0:463:0.003
0,289
0.007
100000
0 0.00
1:000100.00
0.389
0.300100000
0.41210.016
0.436:10.001
metro Jnterstaic www.hitcaaluuueeee cicii_yllmm__eeee
0.464:00003
0.286
0.004
0.299-0000
0
0.375-+-0001
0
000004
naval peopulsion comp
0.014:10.003
0.001 0.006
10000
0.086.0.004
0,059-10000
0.060:1.0002
0,063 1.0000
0.061:10.03
1.079-00.005
naval_ pevul_popution_tue tubb
0041-00.027
0.014
0.00
000000000
D0006-000006
000962000006
0:097:10:005
0.115 00057
maxery
0,085
0,079- 0.003
+0008
0000000004
0.000
0.111-20.03
0.105--0:001
00000
0.116-10.004
unline_news_ populatiyy
0.99310.003
0.989 0.003
100002
1:0000
0.000
0,995.10.000
0.909-10.001
1.0000
parking hirmingham
0.292:000044
0,295-0.00
0000
0:301
0.004
0.500---0004
0.283-00.004
0.288:10.004
0.293:20.004
0.333-+0.00
warkinson_motor
0,100-20.010
0.085
2.05-00000 0.009
0.095
10.008
0.002
0.408 +-0.012
0.182+0.011
0.168-F-..00 F-0.000
0.164-00.006
0.195 F-0011
parkinson
0.110:10.010
0,09400.00
0.105
10.010
0210100014
0.423-10.255
0.180:10.009
0,163 0:0000
0.108:000010
0,181 1.0007
protein_tettiey_iit structure
0.600--0.004
0.004
0.50240004
0.579+00004
0
0.576---00002
0.576
skill_sraft
0.627:000012
0.622-0.112
0.872
0
0.003
0.0021-0.010
000001
0.606-100.06
0.600-100010
0.645 1-0.009
sml2010, dining
0.030-10.003
0.030-0.00
0.000:10002
10.0000000
0:085:10.008
0.076:10.002
0.091 1:0.00
00000
0.102
smi2010_mmom
8,29--0.000
0.000000112
0.041 00004
0079-000044
0-002+-00005
0.026-20.005
0.089-10.04
0.094-200004
0,129--0.00
supencondactivity
0.293:10.007
0,295 0.005
0.294-100088
0,300
1-000-00006
0.252 4-0000
0.211
0.257 1-0004
travl seview nninngs
0.518220.018
0.528-0.012
0.533
0.11
0.530+0013
0.83-20.013
0.450-10.004
+1.8-+000111
wall fodlow robot_2
00377-0.000
0,101-00016
0.109-00013
0.01
00000-000177
0000-00.200
0:099--0:066
00244-00002
1.027 +0,000
0,199
0,313-0,013
0.307
10,025
0,722-0.025
0:30-100:017
0,103
0.000
0.091
0.095 100018
wall follow robot_4
0.057:00:225
0.115=0:024
0.136-+00021
0000000.013
0.141+00011
0.065
0.067--0.029
0.05--000024
40.000
mine_qualiy__uu
0.765-20.008
0,710
0.732-00011
0.777-0.008
0:797-00000
0.212-0.011
0.7001100002
0.013-000122
+0.022
wine qualityllyyyyyt whise
0.758:10.021
0.729 0,012
0.723 10.014
0.702
0.7741-0.011
0.710 0 0.013
1-0014
0.700 000014
0.714 1.0.11
68

---

Table D.4: nRMSE of tuned methods on datasets in Brgiin averaged over ten train-valdattion-ee splits. When we write = b, a is the mean error on the dataset and Ja - b, a + 6] an approximate 95% confidence interval for the mean in the #splits - oo limit. The confidence interval is computed from the t-distribution using normality assumption as in Appendix C.6 In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.
Datiset
R.HPMIP-HPO
MIP-PER-HPO
ResNei-HPO
MIPPHPO
CaB
1.GBM-HPO
XGB-HPO
air Aor.qualiy..u be
0.004:0.000
001240003
0.00-00.00
0.036-400006
0.020::0000
0.029-0.0044
0.026=00000
mit_quulit__coo
0.288-50.017
0:24400013
0.293:0.016
0.294
0.241
0:40-400016
0.245-0.014
tir ui_qqulliy qualitynn2
0.511 +-0006
0.225 0.005
0.322-
0.333 0.000-
0.286-10003
02207-0.007
0.287
0.260+00006
0.26720013
0.014
0.2515:0005
0.344:00115
0222:20.015
0.239-0.00
appliances energy
0.724+40.019
0.11+00011
0.77
0.791-40.00
0.002:00000
0.644:0.009
0.678-0.008
hejing pm25
0.309100007
0.343200009
0.391 000
0.42
0.5
03300:000000
0.386200007
bike, sharing casual
0.2724:0.00
0.284200077
0.287=00000
0.268:200008
0.20
0.200=0.008
0.264=0.00
bike_ shering
0.209
02177-0007
0.29900.006
0.228-100004
0.00
02130+0.006
0.215=00005
carbon menombee u
0.007 1000
0000:00003
0.02
0.011::00000
0.000:00000
0000000000
0.015=0.002
carbon masombeeyy
0.007 1.000
0.01
0.022000001
0:011
0:011::00000
0.000:00000
0009
0.014=00.01
sucbon_nnorabbe..
0.049::0.012
0.049 0003
0.02
0.05 4-00022
0.05 00051
0.0500-000122
0.0522000122
thes brok
0.090:100005
0.11710005
0.00
0.100:::0005
0:540::0002
0.6622000044
0.10200001
pellepowerr. plase
0.207:1:0004
0000
0.200
0.212:4:0005
0.154+0.004
0.88200005
0.1%==0.007
electrical gii stbiliit simulated
0.144120.03
0.152::0003
0.11
0. 1840200000
0.194:00003
0.226:20.000 0.0
frechook comment volume
0.6262:0005
061920049
0634
2.002
0.791
0. 0.595-0.044 0407
facebook live sellers, thailand stares
0.5662
05511-0056
0.00
00000
0
0.55
0.46
five cities. beijing pm25
0.277:10.009
0:300:100000
0.379
0431
0 008
0.000000000
0.367=00007
live cities chengda, pm25
0.261 2.000
0:20900011
0.342-0.007
0.574
0
000009
0 0.321=00014
tive_enies gunogzhoou pm225
0.392
0:001
0.502=00001
0.519:200008
0
0.58
0.47--00012
citie shanghai pm25
0.306:10.012
0:00110011
0.00
0.1111000122
000020
0.305:20.014
five shenyang pos25
0.300:10.022
0.00:00117
0.469 00006
0.50711:0001
011
0.27
0.442:00016
gas sensor drift class
0.079:1:0000
00067100009
0.073-0.000
0.078
0.008
0.120:1:0000
0.20::00000
0. 0.120:20.00
sensor drift
0.147::00014
165:20012
0. 0.002200122
0.00100001
0:
0. 063:2000014 0100200.0
0.100::00004
0000000006
0.172=0.00
0.0
000
0.0
0.150.0.006
indocr loc lal
0.079:10.004
0006:10004
0. 0.104:0.000 004
0.000:00000
0.122:1:0004 004
0. 0.00000000 0000010.004
indocr loc long insurance bendhwrark
0.860:000004
0.06:20000
0074400000 003
0.000
00007
0.084-20.000 0000
0:977:500008
0.999
0082-20.000
0.000
0
0.972
0.974-000077
metro, interstate traffic volume long
0.659100.001
0:18:00006
0.46710.004
0.46--10.001
0.39711.000
0.392
motro interstate trafficc volume shon
0.457:10.004
0018
0.466
0460
0.30711:000 004
0.386 0.000000000
naval wwwpppppppooooooooe comp
0.005:100001
0.003:100003
0.05910.003
0.036:1:0000
0.062.100002
0.088:10.001
0.062:00001
naval propulsion tubb
0.014::0.001
0.047:00007
0.78
0.054:00000
0.095:1:0002
0.096:00000 0000
marsery
0.0001100004
0000010002
0.00
0.0861100004
0.3
000
0.00000000
unline news, popularity
0.907:1:0.00
0.90110111
0.90 0.004
0.9001:0.001
0.990:1:0003
0.988 100004
0.990:00000
purking hirminghum
0.202:10.005
0228100000
0.299 109-00000 0.005
0.301
1.000
0.254:1:004
0.260200004
0.279
parkinson metor
25.015
1655200155
0. 0.072
0.
0.19
0.219::0001
0
0.214200.025
parkimsn
0.104100.013
0.17110017
0.38-10022
0.387:100008
0:1191000100
0.177:10.011
0.207
protein sertiry structure
0.967:1:0003
000007
0.566
0.577:100005
0.00
0.568:10.003
0.590 000
skill craft
0.625-2:0001
060710014
0661
0:662-10000
0
060
0.635-00011
2m1000 dining
0,029 100000
0.052.10004
0.06
0.066 0.000
0.076.1:000
0.00
0.069
sm12010
0.029:10.002
0.04110006
0.06
0.0641100001
0.07511:002
0003
0.067.0.006
superondaacivity
0.283 0:208-1...07
0.296:10007
0.293:10.008
0:0008
0.286:10.006
0.278 120007
0.281:00.005
travel review raings
0.499110.015
0.501
0.529-00.015
0.53 1:0016
0.475:100022
0.163:10.012
0.460:10.011
wall follow robot 2
0.044200.044
0.051 20022
0 0.01200012
0:092:100017
0.000:100000
0066:100025
0.211200005
wall follow robot_22
0:0020
0.126:10.29
0.002:00018
0.306110.021
0.095
000011100118
wall follow robut.4
0.047:1:00026
0.09910021
0211:10.012
0.126110.014
0.0051100019
0065 100024
0.043:00016
wine quility all
0.75 0:001
0.77
0.001
0.77
0.727:10000
0.002
0.000000010
wine quality white
0.736-100.01
0.775100115
0.768
0,775
0.722-10.013
0.704
0.000100013
69

---

Table D.5: Classification emrrr of unfuned methods on datasets in Blasss averaged over ten train- validation-test splits. When we write a + b, a is the mean error on the dataset and (a - b, a + bj is an approximate 95% confidence interval for the mean in the #splits -> co limit. The confidence interval is computed from the t-distribution using normality assumption as in Inpppeddix C.6 In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.
Dattset
ReaLMLP-TD
RealTabRRD
S.D
MEP-PLI-D
MLP-D
CutBoast: ID
LGBM-TD
XGB-TD
RF-D
ada
0.108:100012
0.146:
0.1511:0.009
0.116:10.013
0.119:000011
0.139 0.009:10.008
2.001
0.100:10::11
0.104:10.007
nirlines
0:335+00001
0.301-00.01
0.332+-0:001
0:307
0.337-00001
0332-+00001
0.353--0.001
0.337 00007
0.382+0.001
amazonn commerce ceviews
0.209
0.576-10.102
0.197--00017
0.285--0.020
0.407+00:022 02200400020
Bioresponse
0:38:100010
0.228:100013
0,228:100003
0.226:10.009
0.222:10.006
0.005100001
0.204:10.00
0.11 100007
0.205
0.008-100006
0013+00000
0:011 +0006
0003-+0.006
0.01+00.00
00019+000099
0.021 +-0.006
0:019:00004
00071
christine
0.293-10.012
0.203-0.009
0.209-00000
02.011000111
0.264
0200-10.012
0,266-3-0.00
02015
0.0013
0.014:100003
0.000
00.0:10.005
0.000
00000100004
0.008:10.005
00000000000
0.000
cimc
0,465=+0.019
0.005
400044
0.40150.015
0.400=000117
0
0.467+000018
0.472
cnae
0.068 1.001--0.010
0.055
00.000
0 1.0011
0.065
10.008
0.016:10.010
0.508-00.017
0.091 +00012
2.0971-0.012
0.130-+0..03
0.035-00003
00000
0.149=200002
0,149:00.000
0.343+00003
0.136-000003
0.
0.00
covertype
00029--00000
0026-+000022
0.06--00002
0.005-00000
00022+00000
0.055
0:300-
credit g
0.257:10.017
0.250
0.252
10.0
0.269 0.200100001
02500100018
0.252
0.255 1:0.13
0,256
1
0.402-+00003
0.090
0 +00003
0.90--20002
0:000-0000000
0.003-200002
0.300
0.
0.000
dilbert
0.010-+0.002
0.014-000022
0.020000000
0.009
0.024--0.000
0003+-0002
0.003-+0.002
00000
0.029-2000044
0.089 1:0.00
0.129:10.002
0.123
dna
0.044+00004
0.000000000
0.0077
0.066
0.00
00005
0.030 000004
0.01
+0,00
0.050
fobert
0.312-+00009
0.314:0.008
0,354 +00000
0.367-t00010
0,570-00.00
0.285
+00066
0.386--0.00
0.299 0.0006
0
MNIST
0.09::00000
00002
0.000
0.115
0.00
00000:00000
00001
0002--00002
0.122:200002
gina
0.052-+0.005
0000=00005
0.000-00.006
0.009
0.000-0.005
00077:00005
0.053--0.005
00005
0.077
guilermo
0.375-+00004
0209-00006
0,271 +00010
0210:
0.243---0006
0.168-10004
0.171 +-.000
0.179:40.004
0.197
heleno
0617=200002
00000
D 0.00
0634200022
0623-20.002
00000
0.63
0.718+0000
0.64
Higgs
0:250-$0.001
0020
00000
0.000
0.253--00000
0.25::00001
0.250
0:000-00001
0:000
Insernet Adveriisent
0.0244-0003
00000
00244-0005
0:025
0.020
jannis
0.273
0.262200002
0.0002
0.266
0.000
0.202400022
0.282:00.002
0 0.285
0.002
jasmine
0.207-
0.201
D. 0.20-200012
0.922-00011
0.207
0.187-200011
-1100-0.010
0 0.195 +0.022
0.190-000008
junele_chess 2pwtsss___aeeaae complees
0.004
0.004-0.000
0 +0,010
00000
0,107-00.00
0.133+40.022
0,134 0.0003
1.136-00.002
0.204 0.000
0.140-+-0077
0,143-00.000 00007
0.42-000077
0.147-0010
O.
0.141
KDIXCap99
0000=+00000
000000000000
0.000
0000
0:000-0.000
00000-
0.002
0:00
0.000-0000000
kick
0009:10000
00000000
0. 000000000
00000000
0.00-20000
0.0%620000
0.00000.00
0. 008
0.000
madeline
0:259:00:03
0002
0.022
0.005
0.415-0.012
0.136:20.088
0.19
0. 0.105
0.262:20:008
micar- factoes
0.016:1-0044
0023-00004
0.0005
00000100004
0.029-0.005
0021--00004
0:000-00005
0.031
Mini RooNF
0.650:10000
0001
0.000000000
0001
00522-0000
0051g0001
0.053-20.00
0:0
0.065-20000
mimera128.6
0.479:2-.004
0.414=0.003
0.421 000002
0.002
0480000.00
0.400220.003
0.481 20.000
0.0
0.009-000003
diccppi-stem
0:253--0.004
0200000003
0:000
0248-000004
0.289-00004
0223--00005
0.246--0.003
0.410-00016
0.262
pe4
0.095::0022
0000100014
0.101 100019
00000000000
0.94000000
0:00-10012
0.090-0:013
0.100-10013
0.104220:012
philippine
0.284:000011
0.00
0. 1.0012200012
0.70
0.500
0.249:100022
0.251 20.011
0 0.253400000
0.251
phoneme
0:007:100007
007
0,101 00000
0112:200008
01100:00013
00007:1:0007
0. 0.000
0. 007
0.0982000006
porto-seguro
000881000000
0000
0.000
0.00
0.00::000000
0008100000
0018-10000 0.000
0.008:00000
quar-bioceg
0.126++0001
0
0.133-200013
0.139:200117
0.121
0.109:00014
0001
0.1 0:3366
rieranlo
00002110000
0002100001
B 0.000000000
000000000
0.00610.000
0001100001
0.003-20000
0 0000000000
0.040
robent
0 0:488:200008
0:52240006
0.574
0.44:2022
0.579-0.007
0487:00006
0.0005
00
0.570:2000005
Sonellie
0006-+00002
0006:000002
0.0 0.000
0006:000002
0.000-00000
0006-00002
000002
00000
0.00
segment
0.077:10009
007410000
000007
0000:100000
0.000
00069110008
0.000
0070-100006
0.072-000006
shintle
0:000100000
0000
1:000000000
0000:00000
0.00
000010000
0.002-200001
00000:10000
0.000-200:00
steel ppatte fault
0:241:10011
0,22:00015
0230100111
0:27:100177
0250::00014
0233-00014
0.223:0.011
0:200100111
0:0003
sylvine
005410000
0035500000
00000000007
0058200006
0.075:0.006
005210005
0.057:20.006
005
0.067:00005
volkest
0.292400000
0.28
0.223-20.003
0.300:000004
0.271
0.29::0002
0.291
0.296:00002
0.341 20.002
yeast
0.403-1:0021
0 0.39:000055
0-0011000119
0400:20.020
0,411
0411110011
10.011
0:409-10002
0.301
70

---

Table D.6: Classification error of tuned methods on datasets in Bleat averaged over ten train- validation-test splits. When we write a + b, a is the mean error on the dataset and (a - b, a + bj is an approximate 95% confidence interval for the mean in the #splits -> co limit. The confidence interval is computed from the t-distribution using normality assumption as in inpppeeddxx C.6 In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.
Dutsset
ReuMMEP-HPO
MLP-PLR-HPO
ResNet-HPO
MLP-HPO
CaiBoost HPO
LGBM-HPO
XGB HPO
ada niclines
0,147:100008
0.140:100008
0.148:10.011
0.150110.010
0.108:10.012
0.001
0.1201000012
0.334
0.334+00001
0.354-20.01
0.354+-00001
0.001
0.329-00:01
0.329400001
amazon commerce- reviews
0.4774-0008
0.336--0.022
0216--00015
0.244:00222
0.300+00002
Diorespense car
0.219110.000
0233:1000000
0.225:100008
0.229:10.011
0:0001:00000
0.206:100012
0.2041.0001
0,004 000000
0.003++00007
0016+00013
0.012+00:007
0.017400008
0:017:00.000
0.022+00.307
christine chum
0.200100.01
1.2041
00201
0.272 2-0012
0.200-000009
0 0.012
0.200
1.0000
1:0.00
0.059:10.00
0.0001.0.006
0.000110.001
0.008:10000
0.0.0:00.000
cate cuae 9
0.472+-0.022
0.064+00004
0.000+
0.447
+:0000
0.411:00000
0.470:20.016
0.054-
10000-00.000
0:066
0066 00002
0.053 0.0011
0.066 0.000
0.079
0,095 +-0.015
connect-4
0.132
0.143:+0003
00
00.000
0.139
0.
0.145:00:001
covertype
0.028
0.020+00:00
credit g
0.262:1:0.03
0277:1000222
0.22240.028
0,271 1:0.08
0.234
0.268:10002
0,248 1:0011
Diahetes I BOUS
0.305-200.00
0.992=+00002
00000
0 +00003
0.00
0 0.900:200000
0.007=
dilbert
0,007 +-0000
0001900.003
0,026 0006-20.002
0.002
00040000002
0.022--00004
dionis
1-0.01
0.00010.002
0.101
0.122
dea
0.0.3-+00.005
0.056
00.00
0,054 0.0006
0.04 0.000
0.040::00003
0.022-
fabert
0,309-100.00
0.343+-0.004
0363--0.011
0 0.0000
0 -00006
0.298
0,313 0-0000
Fishicn-MNIST
0.093-40.003
0.107:00:002
000
0.002
0.001:00000
0.094:200002
gina
0.006-::0000
00077--006
0003+-00066
0.006
00000-20.05
0.05
guillermo
0.168+100000
0.202:-00006
0.228400006
0.242 000005
0.000
0.007-+0.002
0,169:000000
helenn
0.614-20.008
0427=20005
0000
0.630
0422400002
0002
0:
0.626
Higgs
0.347:00.002
0.252+00001
0.244200001
0.000
0.258 0000
00050000000
0.257
Advertisements
0.024--00002
00011-00000
00004
0.026 0:0004
janais
0.269::0.002
0.27830004
0.00
0.287
0.0000
0.281 000000
0.27
0.20
jasmine
0.213-2-0.12
0.205
20011
00001
0.200-00011
0 196 0116
0,188 0.0000
jungle_ chess Apeeesss__prrreeee
0.000
0000010.001
0.115
0,032 0:000
00002
+.000
0.134:000000
0.143:00.010
0.153+00010
0.000
0.040-00.000
1:40000000
KDDDCpp9
0.000-20.000
0:00:000000
00000
0.000
0,000 0000000000
0000-10.000
5.000-000000
Vick
0.098:20.001
000010000
0007
0.098 0.000
0.896
0 000
0.097:2 00000
madeline
0.166200012
0.215
0.411
0.00
0.150
015
0.222
miteat factoes
8.015--0.003
0,029-1-005
0009
0.000-00.00
0,022-+0003
0.029+00005
0,014
MiniRooNF.
0.049100.001
0.51 0051 1000
0000
0.051 10000
00001100000
00077400000
0.003-20:001
minera28..6
0,479:20.004
0.470-00000
0.41
0.000
0.400-00000
0.47
0:45) 0.0000
skcupid-stem
0.250+00.004
0:247:00008
0.48
0248-0.005
0.000
0.342 00000
01245 +00003
0.000
pe4
8.101:20.000
0.111:00177
0.013100111
0.101-10.00
0.004-:0011
0.104400:014 011
ghilippine
0.273:000015
0.272+00000
00001
0.200000000
0. 0.250:1:0000
00:007
phoneme
0.007
0:091100000
006
0.101:0000
0. 0.107 004
0 11100:000000 008
0. 1:200000
porto-saguro
0.03:000000
000001000000
0000
0.031
0
0.058:000000
quar-buddeg
0.129::0.14
0.134+00016
0026--0017
0.123-:0.066
0.135-200012
0.136:200022
0::000
riecadoo
8.002:10.001
00001000000
0005100001
1.005-00:00
0.001 00000000000
000
0.00:-20:000
robent
0.478200007
0:06-20007
0.54420006
0.555 00.000
0 008
0
0.475
Sttelitee
0.002-00001
40002
0006-200001
000000
0.007 0:007:00000
000
0.005
segment
6.0791:0.007
000010008
0.07710007
0.003:10.009
00000100000 007
00771100000 008
0.003
shuttle
8.000:10.000
0000
0001
0.001:00000
0000:100000
0:00:10000
0.000-00:000
steel flales-fault
0,239:1:0004
02410010
0,44:100122
0.248110.014
0,222:10002
0.233100008
0.252-:0.017
sylvine
1.0531000055
005810005
0074410004
0.071-100005
0.005
0.0
0.05k100000
volkest
0.272:20.004
0.288-00006
0.235
0.256.20.004
0.301::0000
0.585:20004
0.290=00.003
yeus
0,407
0:00-100044
0.399-10.026
0,410:10:000
0.391 0:3001:0.000
0 0.404:000022
0.301
71

---

Table D.7: nRMSE of untuned methods on datasets in reg averaged over ten train-validattio--esst splits. When we write a + b, a is the mean error on the dataset and (a - b, a + b] is an approximate 95% confidence interval for the mean in the #splits + oo limit. The confidence interval is computed from the l-distribution using a normality assumption as in [Appendix C.6 In each row, the lowest
mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.
Dataset
RealTablR TD
D
T-D
MLP-PLR-D
MLP-D
CatBoost TD
LGBM-TD
XGB-ID
RF.D
airfoil solff noise noise
0.180:10012
0.223:10.009
0.263-10:025
0.49:100020
0,308110.019
02100:100009
0,233
0.238
0.300-0.0116
Airliees_ Depibaee_DDiibyy_oo
0.979-100000
0.003-000000
0.903-00,000
0.984:00001
0.903--00001
0.581400000
0.985
1.00:1000000
1013=0.002
Allstate_ Claims Severit
0654:10006
0.665-10.088
0.662:100006
0.663-10.005
0.662
0.951100118
0.686-0.000
suction verification
0.197:10018
0065:10.013
0.107::0.029
0.160:1:0015
0.19610.0099
0.06411:001
0.206:000036
0.061100014
0.12
Black_tiday
0.092++0001
0.090-+0.002
0.688 +-03002
0.004:400002
0 00000-
0.679:000022
0.679
0.081
+-0002
0,245
bruzilian_ houses
1076:100:69
0.781:1.267
0.706
0.516:10..14
0.703:10.148
0.581:10.33
0,891
0.2281
0.818:10.114
0,539
0.074
Buzzinsochllmeiii. Twitter
0.141100014
0:201110012
0.24710.001
0.357:1:0020
0.337:10.028
0.323110016
0.29610.016
0.26111:011
0.234
califomnia bousing
0.200-40.007
0.302:10.008
+.0008
0.428-100007
0.432440.009
0.400:100007
0,802 0:000
0.409--00008
0.439-0.000
concrete
wowpett_smeeeesiieecooo streeghh
0.288 1.0.25
0.301-10.021
0,306
1.0029
0.303 1.00-1100019
0,325 0.0025
0.302 100017
0:022
0.298
1:0024
0.330
cpesswages
0314400016
0001--0005
0.835
00.15
0..6--0.0115
0.006
0.42-400013
0.055
0353--0014
0.928=0.029
cpu, activiiiity
0.121
100004
4.00004
0.143 0.0004
0.125
0.134
diamonds
0.000:1:0000
0.132
10.022
0.106.1.0.00
0.005:100002
0,144 0.144
0.000
0.008:10.002
0.143
0.138.1:0000
0,148 0.000
elevatoes
0124 0000
0.200
0.747 0
0
0.307
0.346-000077
0.421 00.011
0.400-1..223
0.007-10.01
0.006-100026
+.0.00
0,504 0.002
00022
0
0.548 1-0032
0 0.002
fpu, benchmark
0.006.10000
0032-100000
0,000
0.092
0.003
geograptical origin ofigin_of_music
0.074
0.901
0.002
09239+00009
0.006
0.81
+00.55
0:001
0.775 0075+00015
0.88-000044
Sralth_insuraace
0.778 00005
0.722
0.0004
0.706-100005
0.777
0.770 00000
000003
0.784+00000
0.827= 0.004
house_ LGH
01100-00000
0.73-+00011
0.564-0001
0.570-+0.015
0.51
0.500=+0001
0.001
0.54420012
0.008
house_ procppecceoolinnal
0.500+00044
0419--00041
0.005
0.4
0.372-0:038
jalse_
0.319-1:0007
0.336
0..06110.014
0.361 0.000
0.323-100013
0,334 0.010
0.321 +-0.13
0.324
0.2013
4-0011
0,360 0.009
kin8am
0.3
0.20
0,500
0 0.206=500005
0.30
0:347:00000
000005
0007+00007
0.560=00000
kings
0 0.200+00007
0.349:
0.325-00.011
0.336--0.000
03323-00013
0.001
0.333-0000
0.360=0.00
Mercedes_ Benz
Greener
Manuufctruua
0.622 100000
0,668
0,682
0.672
0.093
0.009
minmi housinn
000007
0.272::00008
0.200
0.266:00006
0.200
0:270:00000
0.204
0000
0.307=000008
map-2016-regresiin
0335 000
0313 +0.02
0
+00027
0.852--00087
0:009+000022
U.14 0:043
0.09
000036
0.33=00:025
my--axi-gren-aa-2266
0.05
0.006
0.660100117
0.722:20:19
0.705
0105
0.01
1.37
0.643=0.010
pol
0000++00003
0.06-110003
0.144500,005
0.74::00000
0:133-50:005
0.102:+00004
0:105
0007
0.120
pumadynšZaah
0.590-1-0007
0.590-20.007
-.604--00011
0.993-100006
0.602
0.593-00008
0.605
0:007
0.008 +0000
0.602-000008
QSAR-TID-IOS80
0192340012
0:400:20011
D. 0.000
0672220014
0.617=00000
01133-0000
0.509
0.9
2.612-00009
QSAR-TID--1
0122240016
0.511:20.114
0.11--00014
0.578:0.001
0.501
0527-40013
0.014
0.16--0013
0.038
quake
10000--0000
1000-+0.008
+00011
1004++00008
-.002-00.010
10000100001
LIKE
1001 +0.04
L050=0.020
Santander, hrnsscctionnooo value
0.00 10015
0937-2000111
0.000
0:07710001
0.00
0:66-100177
0.022
0.77
0.82--0:022
sareas
0.1
0.101:20002
0 000002
0.141 00.006
0.152
0.025
0.129-00002
0.344+00002
0.072=000055
AATII-HANDDAO cuntime regression
+00000
0.481-+0.00
0.465 000052
0.509-0.004
0.485 0.0031
0.492-4-0004
0
0.493 1-004
0.619-0.035
socmoh
0.4362200008
0.396-100034
0.996--00066 0.379-00005
0.481
0.378-1-002
028
0.056
0002-0002
0.490=0:055
solar_ftare
0.52240055
0976-
0.050-00000
09775+-0007
0.979-000057
0.595:10016
0.02
0.944200133
1.116=0.075
space ga
0.555::0004 034
0.496:20001
0. 0.04--0002
0 0250
0.503-000025
0:570:20001
0.565
0.571 0200
0.100=0.000
topo
0.0
0.200200004
0.900-0.009
0.733-00000
0.96
0075
0.977
0.978-20006
0.963=00009
video_ smansodding
0050-100004
0073-40006
00000-0.000
0.110:+0000
0.
0.052+00004
0.067
0.067-00006
0.115=0:005
wine energy
0000111:000
0006:20000
0.023-0.002
006-
0.110:20.005
0.078-1:0000
0.100
0.197:1:001
0.414_0.002
Yolanda
0:790-0000
0.754+00022
0 0.504-00002 0.754-00.01
0,796220.001
0.00420000
0.80620000
0:306-20001
0 0.84=00001
yprop_A_1
0503
09644-00009
0.950-20.007
0.967$0.009
0.965
0.500-200007
0.959
0596--0015
0.963
72

---

Table D.8: nRMSE of tuned methods on datasets in Bieg averaged over ten train-validation-test splits. When we write + b, a is the mean error on the dataset and [a - b, + bj is an approximate 95% confidence interval for the mean in the #splits - oc limit. The confidence interval is computed
from the t-distribution using normality assumption as in inApppendix C.6 In each row, the lowest
mean error is highlighted in bold, and errors whose confidence confidenceinterve interval contains the lowest error are
underlined.
Daiase
ReaMMLP-1IP
MLP-PLR-HIPO
ReNNe--IIO
MLP-1IPO
CuBoust-IIOO
LGBM-IIPO
XGB-IIPO
airfoil serfoll_eelf_nisee noise
0.174+00011
02210-0.:11
0.329+00015
0233++00117
0.227-00011
0:241-0:015
+0002
Airlines, DepDillay H0M
0.979:00000
0.900
0.96220:000
0.83
0.000
0.980=0:001
0:92200000
Allstate_ Claina, Severiy
0.651-00006
0.635-0.000
0.08
0.659+0005
0662-000055
0
060
verification
0.101:0016
0.067200025
0.178400015
0.162:0114
00000200000
0.08 0.100:0.050
friday
0.886:20003
0.607
0.000+00002
0,93+00022
0.679-00002
0.679=00000
0.611:0000
bezzilion houses
0.788-00335
0.14250.1099
L623+1.222
0.06::01117
0.21
0878-00.500
0.565::0.099
Ruzinnocialmeaia Twimer
0:266:20015
0.254200018
0. 0.256:000025
0:27:00017
0.30020.018
0.001
0.232 014
california housing
0.433-00000
0.027
0,010
000
0.402200007
0.400: 0.008
concrese
comprsssieeeeeeeeeeeeeeee strenghh
0.290:00.029
0:314
0.314:0029
0.201=0.030
0:279-0:021
0.272---0022
apshtaages
0.83420015
0.016
0.05
083420015
0.85200015
0.85
01814:10006
opa activityyy
0.125=00000
0.124
0.02
0.1
0.122
0.022
0.119:0 006
0.134::0:02
0.13::0.003
11411:0000
0:101:0002
0.30220002
0.
0.135:10
elevators
0.276=200005
0.306
0.315
0.007700077
0 0.018=0000
dife Tps benchmark
0.457:20.025
0:022
0.09
0,512200266
0.65
0 0:0027
0.004:200002
0.006
0.039100000
00000
0.00
0.018
0.002
0.031:0 007
peographical origin of music
0.800:20.088
0.934200053
0.919
0000
0.01
00024
0.561
health, incurance
0.775=+00004
0.775-00.055
0:000
0.775+0005
0.775=00005
0:775
0.774::00000
house 16H1
0.564::0014
0.05
0.001
0:570:20088
0.57320.001
0.013
0.57611:014
0
house prices, nominal
0.399:20005
0.378=00051
0.445
0
0444
0.002
0.37420 0.000
079
house sales
0.320-200013
0.020-00.010
0.00
00010:0011
0.310=00011
0.019=0.008
0.324+00007
kinSam
0.238:20.003
0.764:20.004
0.279
0000000000
0.37
0.424:000114
0.41
006
kings, county
0.327::00000
0.317=00008
0.341200013
0.390:00000
0.309
0. 0.325 0.303
Meroedes, Berz Groener Manufacturing
0.668:10003
0.670:10:000
0.680110.029
0677:00000
0.60
0,669
0.029
0.66110000 031
miami housing
0.267:10000
0.272:000077
0.287100010
022510013
0.257:00006
0.277:1:00 0.269
MIP-InI6-eerrssicc
0.833-0.077
0.82
0.00000000
0,44:00043
0.788=0.054
0.038
0.507-200000 030
nyc-laxi groen-dee-216 2016
0.13::000011
0.020
0.64310.019
0.692100066
0.66
0.655
0.661 1:0:016
0.659:200004
0.065=00005
0.069
0. 000
0.920.004
106 0.000
0. 0.108
006
0.586:200077
0.589=00001
0.00
0626--0010
0.99220.006
0.602:000000 0.009
QSAR-TID- 10930
0.598:10.014
0.636:00000
0.61410.000
061210011
0.596:0.011
0.000
0 0:500:1:0010
QSAR-TID-11
0.51420016
0.55
0. 0.524100017
0:52710011
0.52620.014
0,509
20.015
0.517:20015
quake
1002:100112
1.00:100010
1.006:00.007
10001000006
0.00
1001 0.007
1,005:: 0:0010
Santander Inmmaction
0.579:100024
0.856.200026
0.9341000116
0001200012
0.87110017
0,551:000 00000
streos
0.102::00022
0.100:0.002
0.10
0:03
0.622000022
0. 0.132:2000 0.22
SATII-IAAN runime regression
0.444:200053
0 0.035
0.4771:0.031
0.161100008
0.515:10.032
0,501 0.022
0.531 000111011
socmcc
0.363:200054
0.299
0.4591000055
0.12:10000
0.36420.054
0. 0.404 0.052
0.417.1: 054
flane
1017-10000
0 0:006
0.975
066
0.98420.046
1,009:00 0.05
space ym
0.4951000022
0.516:10.015
0.602)
0,19610011
0.00::00025
0,506
0.561:100025
tupo 2
0.968:200005
0.972:20.005
0.97010:005
0:968:100004
0.970:20.004
0.904
0 007
video_ jran__rrncooiing wave energy Yolunda Sptup___1
0.052.100005
0.05-40.000
0,068
0.063-10006
0.073:10.002
0.067
0,000
0,004
0.072:10.002
0.003:10000
0.007
0.0041:00002
0.029110004
0.019:0.001
0,081
0.000
0.095 009
0.786:200001
0.002
0.786:00000
0.79100002
0.800:0.001
0.795
0,500:10000 003
0.965 0.607100.07
0,965 0.000
0,963
0.965 0.000
0.963
0,949 0,000
0.951:00008
Table D.9: Classification error of untuned methods on datasets in B GGrinsztajn averaged over ten saiin-vaiiatton--est splits. When we write + b, a is the mean error on the dataset and (a- b, + bj is an approximate 95% confidence interval for the mean in the #splits -> oo limit. The confidence interval is computed from the i-distribution using a normality assumption as in Apppendix C.6 In
each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.
Dataset
ReaIMLP-TD
RealTabRDD
TabR
MLP-LRRD
MLP.
CatBoost ID
LGBM-12 -TD
XGB-ID
RF-D
albert
0348-10003
0.300110001
0.3491000011
0.36610.001
0.308:10.002
0.347:10002
0.367-10.000 0.347:10.002
0.353:10:001
bank-murketing
0.206100008
0000000000
0. 01961:0.004
0:201100006
0.20710.006
0.193-10009
0.196
0.005::0006
0.200=0.006
Bicresponse
02400100001
02007
0.36
0.219:10.013
0.200
0228-10.000
0.2010000100
0.229
10000
0.232--00008
california
0.114100003
0.00000.00
0,092 100000
0.113:10.03
0.122-10.00
0..0100002
0.000000.00
0.097 1.0003
0.011
compus-twp-year
03251100009
0.332-10.08
0.00
0.388:10007
0.306.00005
0.0
0.329-0.000
0.331 10111.0000
0.377:00000
covertype
0.12::10002
0.00010.001
1:0.00
0.145 10.004
0.000
00000100001
0.143-00.000
0.006
0.153-20:001
credit
0.27:110000
0224100.005
0.22610.007
0.201100007
0.225-10.006
0.233110007
0.226:10.00
0.227 000000
0.005
default of credit card-clients
0 400005
0224--0005
0.206--00008
0.204--00004
+00000
0.286-20.005
0.284 0.000
0.242 0.005
Disbetes 130US
0.97-100002
0.356
13.0110.00
1.397--0.00
0.386 0.000
03995-0.000
0.399 0.0002
0.000
0.438
electricity
0.170:10007
0.154:100155
0.110
1.0000
0.170
0.117:10002
0.115
0.111
0.137
heloc
0:244+-0.007
02260-0.009
0.284 +0000
0.276-200008
0.282 +.000
0 4-0008
0.282-00.00 0.0007
0.264-+-0006
0
Biggs
0.288 1.0002
0.00210.002
0.0001
0.888 100001
0.300
0.200-10001
0.269 0.000
0.200
0.300
0.116:10:044
0.119:10002
0.115:00.00
0.100:00003
0.116::0.005
0.116:00000
0..14
0.118.10004
0.020
jannis
02202+000002
0.226-10001
0.2072-00.002
02200100001
0.259-10.002
0.200-0000
0.200---..001
0.230-1:0001
0,235 0:000
Magic Telessope
0.137:1000004
0.131
0.100
0.100:00004
0.100
0.:06:10.004
0.139
0.108.10:005
0.104
MiniBooNE
0.063-+00001
0.065+00001
400000
0006-+000001
10040-00001
0004+000001
00000
00044200001
0.078:50:001
pol
0012-4-0.002
0.014-50.011
8.0:2-40.002
0.017400002
0:037
0.014000001
00300-
0.0144+0.001
0.016=00:012
road safety
0.233:10.002
0.22:10.002
0:2001
0228:10.004
0,241
1:0000
0.233:10.001
0.200
0.212-100000
0.243:10:001
73

---

Table D.10: Classification error of tuned methods on datasets in class cenn--vaadationteettt splits. When we write + b, O. is the mean error the dataset and (a - b, + b] is an approximate 95% confidence interval for the mean in the #splits - oo limit. The confidence interval is computed from the I-distribution using a normality assumption as in [Appendix C.6 In each row; the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.
G averaged over ten
Dutaset
RealMLP-UIPO
IEPO
MLP-PLR-JIO
PIT-HIP
ResNet-IIPO
MLP-HIPO
CatBoost IIO
LGBM-1PP
XGB HIP
RF-1IPO
albert
0.309:50.002
0.348:10.002
0.346:20.022
0.366:00002
0.348::0002
0,347=0.000
0.344:00001
0.347:1:0003
0.348:0.002
0.3461:00000
bank-morketing
0.302:200004
0.193:10.006
0.190:0.005
0.19920:003
0,302200077
0.207=0:007
0.104:00.008
0.10:::0005
0.10000:06
0.199::00009
Biccesponse
0.235:00.010
0.21:1000100
0,250:20.008
0.205:00001
0:2221000010
0.236=0.010
0.234 1:0009
0.229:1:000
0.229-0.013
0.203-100008
california
0.111
0.000110.002
0.114:20.033
0.000:00003
0.116:20.003
0.123=0.002
0.005
0:005:1:0004
0.097=0,002
0.108::0000
cxmpu--ru--yaar
0.025
0.330:10.000
0.329=20010
0.352-00011
0.330:200008
0.225=00006
0.327:10.008
0.327::00088
0.329=0.007
0.3294::0004
covertype
0.02:220.002
0.096:100000
0.100:10.002
0.002
0.
0.145=0.002
0.142:1:0.00
0.135 0:0000
0.007::00006
0.1441:0000
creditt
0.225:000066
0.225:10.007
0.225
0.224
0:22510:006
0.223=0.0055
0.0077
0.226:1:0007
0.224:0.007
0.2261100006
decault of credit curd-clients
0.285 0.000
0.264-10.007
0.286
0.005
0.285 4.0007
0.261 4-0,00
0.285 +-0.00
0,281= 000004
0.281
Diabetes 130US
0.397:1:0001
0.396:10.001
0.307
0:000:00000
0.395
0.395=0.000
0.005100:001
electricity
0.1:::00.003
0.063:1:0003
0.10::200004
0.100:00000
00
0.167=0.000
0.000
0.112:1:0000
0.000
0.126:1:0002
heloc
0.264-40.00
0.299 0.0000
0.207
0.279:00.008
02800:000009
0.263-0.008
0.272 0.000
0.250-1-0007
0.283
0.282-+00008
Diggs
0.000220.002
0 0.0001
0.290:10.003
0.291 0.0000
0 0929 0.000
0,303 0.002
0.290 0,000
0.289 1289:100.00
0.089
0.295 1:0.00
0.110:0.0044
0.115
0.114:2.003
0.005
0.115::0004
0.116=0.005
0.114-10.004
0.115:1:0005
0.115=00005
0.121
jannis
0.221 0.0002
0.230 100002
0.222.00.022
0.225 0.004
0234+000002
0.263
0.222 0.0001
0:202
0.22::
0.227
Magic Telessope
0.0055
0.100.10.005
0.137-20.008
0.004
0.137-100005
0.000
1:0007
0.138:1:0005
0,0
0.142100005
MiniBooNE:
0 0001-00.001
0.063
0003
0
0.061 +00000
0.002-0.001
0.000--00001
0,054++-0001
0.064=0.001
0.004++00001
pol
0.015 400002
0000
0.005
0.001
+000000
0.052:00.04
0.014 1.0.00
0.015 0.005:1:0001
0.016:000012
0.018:100002
roud-safely
0.209::0.002
0.233100.002
0.234:20.002
0.229:0.000
0.229:100002
0.200
0.0001
0.234:1:0000
0.237:00000
0:200
Table D.11: nRMSE of untumed methods on datasets in Brog
BCrinsstajj, averaged over ten train train-validation test splits. When we write a + 6, a is the mean error on the dataset and [a 6, a + 6] is an approximate 95% confidence interval for the mean in the #splits -> co limit. The confidence interval is computed from the t-distribution using normality assumption as in [pppendixx C.6) In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.
Dataset
ReaIMLP-ID
RealTabR-D
T-D
MEP-PLR-D
MLP.D
CaBBoastIDD
LGBM-TD
XGB-ID
RF-D
abalone
0:1
0.647:200012
0.699-000113
0:66:20012
0.6662000155
0/36-:0011
0.687
0:02+0011
0.688=00012
Ailenons
0.396:100006
0.394:10.07
0.00710.007
0.397:10006
0.403-10.007
0.333110007
0.38920:006
0,40810006
0.402:0.006
Airlines DepDelay
0:079110000
0.979110.001
0.00100.001
0:980:2000
0.000:0.000
0.99100000
0.90020:000
0.98410001
1.00100001
Allstate_ Claims Severiy
0 0.007:1:0003
0.697.10.001
0.690100.00
0.602:100001
0.698
0.605110001
0.691:2:000
0.809100026
0.728
amicandatn,
0.142:10055
0.136:10.011
0. 0,14::00:03
0.144200015
0,141:000014
0.10100013
0,144200013
0.145:1003
0.145=00015
Rike Shoring Demand
0.228.10000
0237:::0.006
0.25510.006
0.322110003
0.244120.005
0.288-00005
0.23120:004
0.24110016
0.258=0.005
Beazillan houses
0:068:10.266
0061:10.028
0.004-10.02
0.065110011
0.068:10.011
0.067:10.000
0.059:0:0021
0.067110021
0.075
qpu acl
0.129:1:000
0.001::0.044
0.125
0.127:100004
0.144:20.004
0.168:1:011
0.125
0.12711:011
0.134_0.005
dolays zurich teanoport
0966622000
0.066220000
0.967
0 0:967400000
0.969200000
0:067:20000
0.96820000
0.971
2000000000
1 0.06
dismonds
0.066.10.002
0.888 10.011
0.092110.001
0.002-10:00
0.102:10.000
0.092110.001
0.097:10:001
0.091110001
0.115:0.004
elevators
0.280-1:0005
0.00010.000
0.728110.013
0.667110.13 338
0.745:10.014
0.323:1:0006
0.334220007
0.345110006
0.420=0.011
IdEl
0.698 0:0024
0.701-10.019
0.685 1.0.21
0.002
0.680-10.011
0.682-+00014
00001
0.667 0:606100006
sales
0.221:100003
0.312-10.03
0.000
0.308
0,0000
0.3211100003
0.00
0.3341:0002
0.300
0.111100006
0.362-10.005
0,362:10.004
0.415100004
0.419
0:391110000 003
0.304
0.30011000
0.419:0.005
medical charges
0.144-100000
0.100:100000
0.145 0.000
0.045:10000
0.10
0.046:100000
0.000
0.154100000
0.000
Mercedes, Benz, Greemer Manufacturing
0.675:1:0000
0.672
10.030
0.675 0.009
0.633-100022
0..05
0.6771100099
0.602
1:0026
0.736-0.023
MiumiHonsing201e nyc taxi green dec 2016
0.363-1:0000 003
0.246:10.005
0.25
0 0369-100000
0.280
0 0:360:1:0000 004
0.367
0.27
1:000
0.295 006
0.701:10.009
0.664 10.002
1.607
0.750:10028
0.707
0.67-400.02
0.669 0:000
0.668
purticulate matter 2017
0.501:1:0002
0.566.10.044
1.568110.00 004
0.582-10002
0.587
0. 0.566:10.00
0.572-10.001
0.570
0.597
pol seutlecrimes
0.067
0002-+-0003
+0.000
0.074+-00004
0,141 0.0009
0. 0.002++00004
0.103==00000
0,109-40000
0.120=0.007
0.006110.002
0.00110.001
0.905
0.905 1.0001
0,906 0.000
0.903:10.01
0.905 0:001
0.910:10001
0.01
SGEMM GPU, kereel performnce
0,014:100000
0014:1000000
0.022-10.001
0002-100002
0.032110.003
0.017:100000
0.016100.000
0.017110000
0.015
sulfur superoondctt
0.422-40.063
0.376-10.038
4.602-40.001
0.438-+0.005 0.429
0.444--00066
0.055
0.05-000065
0.049
0.305 1:0.00
0.308 10.00
0.304
1.0004
03.00100000
0,308
0.291 0:0005
0. 0.200
0.289 100001
0.005
uopo.2. 2_1
0.969-100005
0.900-10.005
0.971
1.0001
0 0.90-000004
0.970
0000
0.07
00970100007
0.001
vasualizing_ao
0.009-+0.001
0007 00000
000099
0.000++0.001
0:027
01000+000000
0.005
0,005
0.004-0.00
wine quality
0.762 06001100014
0.736:10.010
0.737
0.883-100011
0.778:10.015
0.716:100010
0.001
0.711
0.0012
0.012
yprop_4. 4_1
0 0.966:100000 0:0
0.000-00.005
0.9075
0 0.966:10.033
0,969
0.962 000
0. 0.005
0.969-100004
0.968=00.007
74

---

Table D.12: nRMSE of tuned methods on datasets in BCriasztajaa averaged over ten train- train-validation: test splits. When we write + b, a is the mean error on the dataset and [a b, a + b] is an approximate 95% confidence interval for the mean in the #splits - oo limit. The confidence interval is computed from the t-distribution using a normality assumption as in Appenddix C.6 In cach row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error underlined.
Dotaset
ReaMMLP-11P
TubR-HIPO
MLF-PLR-HIO
FTT-HIO
ResNa-1IPO
MLP-EIPO
Caloost-IIPO
LGBM-HIPO
XGB-IIPO
RF-IIPO
abalone
0,561 +-001
0633-+0.009
0:04+00011
0.603 0.001
0606-40015
0.06-+0.0100
0679:40011
0.675400011
0672-40.000
06605+0.012
Allerons
0.3850000)
0.900:2.008
0.97700077
033
0007
0402220007
0.000
0335+0000
0401=0006
0.02
DeyilaayyD
0978:000011
0976-40001
0578-00001 0:000:00000
0982-20.011
0978:00000
097100000
0997:00000 0.09
Allizate Clainss Severity
10001
0.07::00000
0/99:10000
0:600::00022
06081000
0605:00000
0685:10000
0.685100001
0751:0012 0.13:200002
analeardatn supreme
0.14420018
0148:20.015
0.14240018
0.142:0017
0140:00166
0142:20.016
0142400014
0.144:00066
0.141:20118
0.146:20.013
Bike, Sharrin Demand
0:229+0005
0.227+0.004
0:377+00055
0.28:00007
0.26::00005
0:245-000077
0234:00004
0,244:00000
0241#00014
0:257-00005
Razilian
085110015
0.07220.0222
005740004
0.08:000014
00661000000
0062:10.014
0067:00016
0056:10002
0061:20001
0009:20.031
opu set
0.125:0004
0.115=00004
0.125::0044
0.130:00005
0.128:0.004 0:37::0.004
0123:00003
0.124:0007
0.130:20.005
0.32::0.005
Selayt_urichhtome transport
0966-00001
0.96:2200011
0966:00001
0:06:20001
0166::00001
0969:200001
0964:000001
0163+:00001
0.963:00001
0.965=20000
dianoeds
0000100001
0.000:000001
002510022
0005:00000
0 0.107=0000
0:100::00002
00220000
0000000
00
0.107:20.011
elevatoes
0.2762:0005
0.285:200005
0315
0.35::0.137
0.318:20.007
0.74
0330:200007
0.319::0000
0.32
0.423
house 1681
0.71410012
0.04:200013
0660:100055
0404:10006
01600:10014
0684:20018
0674:10.05
0/6001:00021
0671110016
0.065
house sales
0.330:00003
0.313:20.003
0:32110001
0.301
0:311:00001
0.38g20.002
03300:20000
0.3234:000
0:323:00003
0.351:20.003
houses
0.402200044
0.357=00005
0408
0.405:00005
0.200:00005
0.421=00.006
0.392:00004
0.391:00004
0.29 0.418::0.004
medical
0.143100000
0.144:00000
0.344110000
0.144200000
0:147::0002 0145:20001
0.44::00000
0,145:00000
0.147:00000 0:147:0.000
Maroadex, Greener Manufacturing
0671220012
0.672-0.000
0670-10000
0:88:0031
0.677:000122
0:60)=000000
0600=00022
0/66:10000
0.665:20001
0.688=000000
Mousingn
0.2003:0004
0.245:20.004
026240000
0.261::0004
0226:000000
0:200:20.0006
0254::00033
0125:::00000
0250:00000
0.200:20.005
nye-txii-gnen-do-2200 puniallst--mmmssskiiil--11
0670:200002
0656:200012
064
0.715::0022
0.001:00000
0.001
0668:000004
0:66:1:0000
0.60
0.655:20.004
0.578120002
0.54420.004
0.575
0.579::0003
0.56322.0022
0.586220001
0.53
0.563::0000
0.563-20002
0.57:220001
pal sestlerimme
0059100004
0.06::10004
0064100003
0.066:10000
0.166:10000
0.127:10008
0.100:10004
0.107:10000
0.109:10004
0.117:10.005
0:041:0001
0.90320.001
09041000
0394200001
0 0:00120001
09065.0.001
09001:00001
0:001:0000
0:901:00001
09042200001
SGEMM (PPM lernel, performance
0.11400000
0005:200001
0006:00001
0.017:00001
000920000222 0017::0.000
0017=00000
0,016::0000
0.016:00000 0.144:0.000
sulfur
03011
0.372:20.000
0.377110006
041001000566
0448:110005 0.404:0.0055
0.38:100036
0,423100000
0.416:00058
0.466:0.0044
sapensonduet
02001:0006
0:000:0.004
0330410000
0:315:00000
004
0:305:000004
0204:00003
0386:1:0004
0.385 0.006
lopo 1
0968:10006
0208:000004
0:70110004
0.08::00005
0501100004
0967-20.077
0972.00004
(1967::0000
0:666:20000
0,964:20.003
visulizing soll
0.001:00001
0.06::0.001
0009100001
00011:00001
0.026:00001
0010:20.001
0006:10.000
000611:0000
0.024:00000 006
0.006:10.022
quality
0.79:10000
0.730200011
0.77610012
0.781 008
000 0.700200012
0.737:00008
0770110001
0.000000000
0.709220.013
1_rop_11
0:000000007
0.953:10.004
0.967110000
0.968 10.00
0.966
0.960:10.004
0.982.10.030
0.953110000
0.55:110.05
0.909.10.06
75

---

E
Broader Impact
We present NN models with an improved speed-accuracy tradeoff and hope that this can reduce the resource consumption of tabular models in applications and further benchmarks. While tabular ML has many potential applications, we feel that none must be particularly highlighted here.
76

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims
made in the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide a paragraph on limitations in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We do have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
77

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide implementation details in the paper and the appendix, especially Appendix Cl We cannot provide all details dataset preprocessing, but these provided
with the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken
to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide code for the meta-train and meta-test benchmarks in the supplemen- tary material. For the camera-ready version, will provide more complete documentation, the code for the Grinsztajn et al. [18 benchmark, and the experimental data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc)
for more details.
78

---

reproducible, they While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
httac//gguuiee//eeuuuubiiise / //nips.cc/ wwtpppcicggguiuiaaacec des/ CoodeSubiisss settuubmissioooo ceipiiiissiooooociii.. onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: These details are provided in the paper and appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to level of detail
that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yss]
Justification: We provide error bars for fixed datasets, quantifying the uncertainty over the random splits, as described in the appendíx. We also provide critical-difference diagrams in Appendix B.I1
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
79

---

Question:
For each experiment, does the paper provide sufficient information on the com-
puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [No]
Justification: We did not track these resources in detail, but rough estimate for the total resources can be found Appenddi C.8
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httttp///.eeurrps.....o/obb///// cese
Answer: [NA]
Justification: The research appears to conform to the code of ethics but we did not check all 200 datasets used in this paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed? Answer: [Yes]
Justification: See Appendix E unclear.
but this work is foundational research and the impact is
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended
(e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
80

---

Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We only use publicly available tabular datasets without obvious safety risks and do not release pretrained models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [No]
Justification: We use Re 200 datasets from online repositories, so we cite the repositories benchmark curators but not the individual datasets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We only provide download code for existing datasets. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots. if applicable, as well as details about compensation (if any)? Answer: [NA]
81

---

Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
82


Reviews:
[{"review_id": "uT2t8wOvrN", "reviewer": "Reviewer_5ngr", "summary": "The paper proposes an enhanced version of the tabular MLP model -- RealMLP. By using multiple tricks over simple MLP, the proposed method becomes much more competitive with GBDTs than a simple MLP. Moreover, the authors provide strong \"tuned default\" configurations for GBDTs and RealMLP. These configs considerably reduce training time in exchange for arguably small test error increase compared to hyperparameter optimization procedures.", "questions": "1. Please, see weaknesses. \n2. Probably I missed it in the text but how many HPO iterations have you used to obtain -TD configs?\n3. Have you measured inference efficiency? I think MLP is usually a bit faster than GBDTs but I wonder if parametric activations or proposed numerical embeddings make MLP slower.", "limitations": "Limitations are addressed.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper addresses two important problems in tabular DL:  1) lack of an efficient strong DL baseline  2) high cost of extensive hyperparameter tuning for a good performance\n* Extensive evaluation of the proposed method. Appendix A and B are also provide important ablations on RealMLP. \n* Great experimental setup in terms of meta-train and meta-test splits that allow to find tuned-defaults.\n* RealMLP is considerably better than MLP. \n* Tuned-defaults for GBDTs.\n* Everything necessary to reproduce the results is provided.", "weaknesses": "1. One of the main contributions of the paper is the different DL techniques from Figure 1 that have been added to MLP. Although there are enough useful experiments and ablations on these techniques for RealMLP (Figure 1 and Appendix B), these tricks can be applied to any DL model. From Figure 3 we observe that TabR-S-D and FT-Transformer-D are much better than MLP-D and slightly worse than RealMLP-TD. So, I think it is essential to ablate the mentioned techniques to improve these models and compare the results with RealMLP. I expect that the techniques improve stronger models much less compared to MLP, and it would be a beneficial result. It will show that RealMLP-TD is a good baseline while TabR-S-D-with-tricks is only slightly better than TabR-S-D. Otherwise, if improved TabR or FT-T considerably outperform RealMLP, the results will be even closer to GBDTs' which is also a positive outcome. I think that the results in Table B.5, which show an advantage of RC+SF preprocessing for TabR-S-D , also support my idea. \n2. Table B.1. from Appendix: changing Adam's $\\beta_2$ from 0.95 to 0.999 significantly increases the error rate while changing Adam's $\\beta_2$ from 0.999 to 0.95 on Figure 1 has almost no effect. I would make a conclusion that the resulting RealMLP-TD architecture with all tricks is very sensitive to hyperparameters. \n3. Also Table B.1: Strangely, PLR embedding is much worse than PL one. The only difference is ReLU if I am not mistaken.\n4. No non-parametric DL baselines in Figure 2. Only TabR-S-D is presented. Also, it would be interesting to compare tuned MLP-PLR [1]. Authors from [1] use extensive HPO and do not fix embedding size, so a comparison with RealMLP-TD would be interesting to see if a properly tuned MLP-PLR is better/worse than RealMLP with tricks and fixed embedding size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback.\n\nWeaknesses:\n\n1. While we agree that such experiments would be interesting, they would be very costly and are not central to the research question we want to answer in the paper (how good can we make MLPs and GBDTs with default parameters?). We would argue that the existing results should already be interesting to readers and that our paper's ability to pose many interesting questions and potential follow-up studies should be viewed as a strength rather than a weakness.\n\n2. We can mention this in the main paper. We already mentioned in Appendix B that this is likely due to the very high weight decay value (we did an earlier ablation without weight decay for regression and the differences were much smaller). So we would argue that the sensitivity to some hyperparameters is not a big issue as it can be fixed with lower weight decay or by just using our provided defaults.\n\n3. We double-checked our implementation and we also checked MLP-PLR and MLP-PL with the original libraries and observed the same effect for defaults on meta-train (we didn’t run MLP-PL-HPO). So it appears that PLR is better than PL on the 11 datasets of its own paper with HPO, but is worse with fixed hyperparameters on our 118 meta-train datasets. The PLR library documentation also mentions that PL can allow for a lower embedding dimension (perhaps because there is no information loss due to the ReLU). So this seems to be a weakness of PLR embeddings and not of our paper.\n\n4. TabR-S-D is a non-parametric baseline. Unfortunately, TabR-HPO would be way too expensive to run on the meta-test benchmark (easily 5-20 GPU-months on RTX 3090 GPUs extrapolated from the runtime of TabR-S-D). We did also include TabR-HPO on the Grinsztajn et al. benchmark in the rebuttal. Since other non-parametric methods like SAINT are even more expensive, we cannot afford to run them in Figure 2 (but SAINT is in Figure 3). We did include MLP-PLR now. The defaults are on the level of MLP-D (the paper does not propose defaults, but the library does). MLP-PLR-HPO comes close to RealMLP-HPO on meta-test-reg but underperforms RealMLP-TD on the other three meta-benchmarks. Given the results, optimizing the embedding dimension could be interesting for RealMLP-HPO as well, although the larger embedding dimensions appear to considerably increase CPU runtimes. (MLP-PLR-HPO has about equal runtime to RealMLP-HPO, despite using early stopping.)\n\nQuestions:\n\n1. See weaknesses.\n\n2. We did not mention this in the paper since it is a one-time cost, but: Around 100-300 steps for GBDTs (could have probably matched this with 20-30 manual tuning steps). For RealMLP, we did not use automated tuning, since we repeatedly implemented new components and tried them, so it is hard to say but several hundred configurations were tried.\n\n3. We have measured inference efficiency but did not want to report it since it could be quite sensitive to implementation details (inference batch size, data conversions, etc.). For example, we heard from AutoGluon developers that they could greatly improve AutoGluon’s inference times just by running its preprocessing in numpy instead of pandas. Nonetheless, here are some of our measurements (inference time per 1K samples):\n- CatBoost-D_CPU: 0.00273057 s\n- XGB-D_CPU: 0.00297026 s\n- RF-SKL-D_CPU: 0.016169 s\n- RealMLP-TD_CPU: 0.00501239 s\n- RealMLP-TD-S_CPU: 0.00377973 s\n- MLP-D_CPU: 0.0187927 s\n- TabR-S-D_CPU: 0.125827 s\n\nAs you can see, RealMLP-TD is a bit slower than GBDTs but still fast, while for MLP-D there seems to be a suboptimal implementation (we don’t know why it would be so slow otherwise) and TabR is very slow.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NEnNNEEW6i", "reviewer": "Reviewer_RnHK", "summary": "The paper introduces RealMLP, an enhanced Multilayer Perceptron (MLP) designed for classification and regression tasks on tabular data. It also proposes optimized default parameters for both RealMLP and Gradient Boosted Decision Trees (GBDTs). Through benchmarking on diverse datasets, the authors demonstrate that RealMLP achieves a superior balance between time efficiency and accuracy compared to other neural networks, while remaining competitive with GBDTs. The integration of RealMLP and GBDTs using optimized defaults shows promising performance on medium-sized tabular datasets, alleviating the need for extensive hyperparameter tuning.", "questions": "The effective default performance achieved by MLP + GBDTs may not be suitable for all scenarios, such as those requiring tabular pre-training or zero-shot scenarios, where pure tabular models might be more appropriate. How does the proposed method address applications in tabular pre-training scenarios?", "limitations": "NA", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The study includes comprehensive benchmarking on a meta-train dataset comprising 71 classification and 47 regression datasets, and a meta-test dataset comprising 48 classification and 42 regression datasets.\n\n2. Detailed experimental settings and hyperparameters are explicitly specified in the paper and appendices, ensuring reproducibility.\n\n3. The experiments establish statistical significance, presenting error bars and critical-difference diagrams.\n\n4. RealMLP demonstrates superior efficiency by offering a better time-accuracy tradeoff compared to other neural networks.\n\n5. The optimized combination of RealMLP and GBDTs with improved default parameters achieves outstanding performance without the need for hyperparameter tuning.", "weaknesses": "1. The finding presented in this work is not entirely novel. Similar conclusions have been reached by Tree-hybrid MLPs `[1]`, although the methods are not quite the same. Moreover, ensemble different methods, including combinations like MLP + GBDTs, have long been recognized as effective approaches in Kaggle competitions and do not necessitate rediscovery.\n\n2. While mentioning good performance by default, it's worth noting that Excelformer achieves one of the best performances under default settings. However, a direct comparison was not included, which would enhance the clarity of the findings. Please include the comparison in the rebuttal.\n\n3. Despite emphasizing the applicability to medium-sized datasets, practical applications often involve diverse tabular data with varying feature lengths and training data sizes. The small application scope could potentially limit the impact of this study. Besides, most of tabular data sizes are often small.\n\n4. The compared methods are evasive. Since the authors want to claim the speed-performance balance, the effective approaches Net-DNF `[2]`, TabNet `[3]`, and TabCaps `[4]` are not compared. The compared models FT-Transformer and SAINT, especially SAINT, are heavy.\n\n**REF**\n\n`[1]` Team up GBDTs and DNNs: Advancing Efficient and Effective Tabular Prediction with Tree-hybrid MLPs\n\n`[2]` Net-DNF: Effective Deep Modeling of Tabular Data\n\n`[3]` Tabnet: Attentive interpretable tabular learning\n\n`[4]` Tabcaps: A capsule neural network for tabular data classification with bow routing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback.\n\nWeaknesses:\n\n1. **[1] was uploaded on 13th of July 2024**, so this paper cannot be used to question our novelty. (Besides, it also uses a mix of architectures, while we consider algorithm selection / ensembling.) Regarding ensembles, we do not claim to rediscover ensembles. Rather, we try to see how meta-learning better defaults affects the trade-off between ensembling / algorithm selection and HPO.\nWhile the ExcelFormer paper claimed to have very good defaults, ExcelFormer performed poorly in the experiments at https://github.com/pyg-team/pytorch-frame. So either the original results were problematic or the method is difficult to use correctly, both of which don’t make it attractive for us to compare to ExcelFormer. Moreover, as a transformer with inter-feature attention, ExcelFormer would likely result in out-of-memory errors on some high-dimensional datasets of the meta-test benchmark (as FT-Transformer does). We compared to TabR-S-D since their paper also claimed to have better defaults than GBDTs. We can, however, acknowledge the missing ExcelFormer comparison as a limitation.\n\n2. We consider dataset sizes of 1K to 500K samples and up to 10K features, which is quite a large range, larger than the Grinsztajn et al. benchmark, for example. Smaller datasets are also interesting but hard to evaluate due to noisy test scores, and might be much more affected by details in early stopping etc. In addition, they are easier to handle for TabPFN or LLM-based methods, necessitating different baselines and making MLPs less attractive.\n\n3. We agree that additional efficient baselines can help to substantiate our claim, but we think that the mentioned methods are not particularly promising based on recent results from the literature (see below). Instead, we evaluate MLP-PLR [5], an MLP with numerical embeddings that was presented as better than FT-Transformer from the group that created FT-Transformer, and performed well in recent benchmarks [6, 7]. MLP-PLR is faster than RealMLP but does not match RealMLP’s benchmark scores. Here are some reasons why we think that the mentioned NNs are not very promising:\n- TabNet performs poorly both in recent benchmarks as well as in our preliminary experiments. In McElfresh et al. (2023), TabNet performs much worse than MLP or FT-T while also being slower than both of them. In the FT-Transformer and SAINT papers, TabNet is also worse than MLP. The same holds for https://arxiv.org/abs/2407.09790 and https://arxiv.org/abs/2407.00956\n- Net-DNF performed slightly worse than TabNet in Shwartz-Ziv et al. (2022). It did perform a bit better than MLP on 4 datasets in Borisov et al. (2022). It was also outperformed by TabCaps in the TabCaps paper.\n- TabCaps performs worse than MLP in a recent large benchmark (https://arxiv.org/abs/2407.00956), see Tables 19 and 20 therein. The TabCaps paper also doesn’t contain a lot of datasets, so their own claims are probably not too reliable.\n\nQuestions:\n- We do not aim to address tabular pre-training or zero-shot scenarios. In cases where one can pre-train on a dataset with the same columns, RealMLP might be a good option.\n\n[5] https://proceedings.neurips.cc/paper_files/paper/2022/hash/9e9f0ffc3d836836ca96cbf8fe14b105-Abstract-Conference.html\n\n[6] https://arxiv.org/abs/2407.02112\n\n[7] https://arxiv.org/abs/2406.19380", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CjFxxI3AKf", "reviewer": "Reviewer_eZ6k", "summary": "The work investigates/provides better default hyperparameter configurations for MLPs and gradient-boosted decision trees. Additionally, it proposes an augmented neural network with a series of enhancements. Experimental results are provided showing that the neural network is on par with gradient-boosted decision trees. Moreover, the authors argue that the neural network and the gradient-boosted decision trees with the new defaults are competitive with their HPO-tuned per-dataset counterparts and motivate practitioners to use the different methods with the new defaults and perform ensembling.", "questions": "- Could the authors provide results for the missing baselines and for the baselines that do not feature HPO results?\n- How does this work fair in comparison to using a portfolio of configurations using dataset meta-features and selecting the default configuration per dataset based on a similarity distance between the portfolio of configurations and the context dataset?", "limitations": "- I believe the authors have adequately addressed the limitations.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The work is written well.\n- The work uses a wide collection of benchmarks that are well-known in the community.\n- The work includes results with default, tuned-per-benchmark, and tuned-per-dataset hyperparameter configurations for the main methods investigated.", "weaknesses": "- The novelty of the paper is limited in my opinion. The majority of the components that are proposed in Figure 1. C are not novel and are used in AutoML methods. Moreover, it is not clear what subset of the proposed components could actually benefit the other methods and not only the MLP. \n- The baselines used are not consistent, for example, FT-Transformer and SAINT are included for the Grinsztajn et al. benchmark, but not for the other benchmarks.\n- The protocol is not fully consistent, as for example, for the results of the meta-train and meta-test benchmark, HPO is not applied for the ResNet method or Tab-R.\n- Figure 2 shows results for accuracy, where RealMLP-TD strongly outperforms gradient-boosted decision tree methods for the meta-test classification and regression benchmark. However, AUROC is a metric that considers class imbalance and provides a better view of the method performances, Figure B.3 which reports results for AUROC shows a different picture, where, gradient-boosted decision trees outperform the RealMLP-TD. As such, I would propose to place Figure B.3 in the main paper and Figure 2 in the Appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback.\n\nWeaknesses:\n\n1. What is the source for your claim? Which AutoML tool would these be in? Are you referring to some of the color-coded improvements or only to the gray ones? There are some non-novel components in Figure 1 (c) since we wanted to start from a “vanilla MLP”, but as we color-coded in the figure, there are many new and unusual components. We also tried out the two NNs in AutoGluon and their default settings performed worse than MLP-D on the meta-train benchmark. Regarding your second point, please see the global response.\n\n2. While we understand the desire to have every method on every benchmark, unfortunately the meta-test benchmark is much more expensive to run than the other two benchmarks since it contains larger and higher-dimensional datasets. For example, FT-Transformer runs into out-of-memory issues on some datasets, trying to allocate, e.g., 185 GB of RAM. We did not try SAINT since it’s even more expensive.\n\n3. Continuing on the point above, TabR-HPO would potentially take 5-20 GPU-months (based on extrapolating the runtime of TabR-S-D) to run on meta-test on RTX 3090 GPUs. We are now running it on the Grinsztajn et al. benchmark to have it on at least one benchmark. We did not run ResNet-HPO before since in most papers and Fig. 3 it is quite similar to MLP-HPO while being slower (takes around two GPU-weeks on our benchmarks in Fig. 2). We are currently running it and provide preliminary results in the uploaded PDF, which are confirming our expectations.\n\n4. We included the AUROC experiments after the main experiments were finished to include an imbalance-sensitive metric. However, since the whole meta-learning and model development process was performed with accuracy as the target metric, and the best-epoch selection was also performed with accuracy as the target metric, we think that the results in Figure 2 are more suited for the main paper as they are more appropriate to demonstrate the effect of meta-learning default parameters. We do refer to the AUROC experiments in the main paper, though, and will try to emphasize these experiments more in the next version. \n\nQuestions:\n- See above.\n- This is an interesting question, but would deserve its own paper. This would also be challenging to do without too much overfitting, given the limited number of meta-train datasets. TabRepo, perhaps the most large-scale study on portfolio learning, still only considers static portfolios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h221YgtoxI", "reviewer": "Reviewer_J8gT", "summary": "In the paper \"Better by default: Strong pre-tuned MLPs and boosted trees on tabular data\", the authors make two major contributions: (i) they propose a multi-layer perceptron configurations that is tuned on a set of training datasets and (ii) investigate in a large scale empirical study how different default parameterizations (library vs tuned on a set of training datasets) compares to optimized hyperparameters on a set of test datasets. The study reveals interesting results regarding the performance of default parameterizations set in standard ML libraries, better default parameterizations exist (at least for the scope of the benchmark), and that the proposed MLP often performs competitive to gradient-boosted decision trees when it comes to a Pareto-optimal tradeoff between accuracy and training time.", "questions": "- How would a CASH approach compare here if we would first select the learning algorithm according to the (tuned) default parameterization and then optimize its hyperparameters? As it is done in Mohr, F., Wever, M. Naive automated machine learning. Mach Learn 112, 1131–1170 (2023). https://doi.org/10.1007/s10994-022-06200-0\n- Why was random search used for hyperparameter optimization?\n- Is there any intuition why using the last of tied epochs as the final model as stated on line 161?\n- Why are hyperparameter values rounded as stated on line 170?", "limitations": "Limitations have properly been described in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is very well written and easy to follow. Sufficient details on the architecture of RealMLP and the implementation of tuning the default parameterization or optimizing the hyperparameters are given.\n- Generally speaking, the authors are very eager to provide all the details to make their work reproducible which is of high importance for such a paper.\n- In principle, I like the 2D-comparisons but a summary plot how the D vs TD vs HPO behaves for the different methods would be very much appreciated.", "weaknesses": "- Intuitively one would expect that the performance of TD lies in between D and HPO but I could also imagine that TD is oftentimes very close to HPO or might even exceed HPO's performance due to overtuning. But this comparison is not considered as an overview.\n- It is not clear why random search has been used as HPO technique in the paper, especially, since the authors even mention more sophisticated techniques such as SMAC that are based on Bayesian optimization.\n- The style of writing in Section 3 could be improved as there are various content-wise enumerations in the form of sentences starting with \"We\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback.\n\nWeaknesses:\n- Thank you for the suggestion. We have something like this in Figures B.12-B.15, D.2, D.6, D.7, or do you have a specific suggestion for an overview plot?\n- Random search 1) is used in the Grinsztajn et al benchmark, 2) is more convenient to run since all steps can be run in parallel and resource constraints (RAM usage) can be adjusted per parameter configuration, 3) allows to change the optimization metric post-hoc as we do for AUROC in Figure B.4 since the sampled configurations do not depend on the optimization metric. Random search has also been used in the benchmark of McElfresh et al. (2023), for example. We have a comparison to TPE (in hyperopt) for GBDTs in Figure B.6 and the differences are rather small.\n- Thank you for the comment, we will try to improve this in the next version.\n\nQuestions:\n- Thanks for the question, we also looked at this. It seems that the results are about halfway between Best-TD and Best-HPO (whose difference is quite small already). We were not sure if including this would complicate the message too much.\n- See above.\n- Not a lot, but a reason could be that when the MLP continues to improve slowly but the last few validation accuracies are identical, the last one is still more promising for the test set.\n- To make them look nicer and more memorable in paper/code. (We checked that it doesn’t significantly hurt the performance.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7LFH3w2lZx", "reviewer": "Reviewer_vGM4", "summary": "The authors introduce RealMLP, an improved multilayer perceptron (MLP), alongside improved default parameters for GBDTs and RealMLP. The authors tune RealMLP on a meta-train benchmark with 71 classification and 47 regression datasets and compare them to hyperparameter-optimized versions on a disjoint meta-test benchmark with 48 classification and 42 regression datasets, as \nwell as the GBDT-friendly benchmark by Grinsztajn et al. (2022), claiming that RealMLP offers a better time-accuracy tradeoff than other neural nets and is competitive with GBDTs.", "questions": "QUESTIONS\n\n* Why was it necessary to create a new benchmark suite? Why is this the fairest choice of benchmark for these methods?\n* Why report only one non-standard metric?\n* Why are so few deep tabular methods included in the core experiments?\n\nSUMMARY\n\nWhile the detailed description of the authors method is helpful and the ideas contained therein could be of interest to the NeurIPS audience, the experimental results are not yet robust enough to warrant publication at NeurIPS.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The inclusion of the standard recent benchmark of Grinsztajn et al. (2022) is welcome, and gives more credence to the authors' claims.\n2. The description of the authors' improvements in Sec. 3 is unusually good, containing sufficient detail that, if needed, someone could most likely recreate the method from scratch.\n3. The goal of improving the performance of MLPs on tabular data is both useful and interesting.", "weaknesses": "MAJOR\n\n1. The support for a key claim (\"RealMLP offers a better time-accuracy tradeoff than other neural nets and is competitive with GBDTs\") is, at best, inconsistent. In Fig. 2, utilizing the authors' own benchmark suite, RealMLP is consistently less performant than the best GBDT. The claim that it outperforms other neural nets is not strongly supported by Fig. 2, since only a handful of the baseline NN methods from [1] are included. Particularly notable is the exclusion of [2], despite the fact that the authors reference it in their prior work. [2] requires less than a second to train and there exist methods in the literature to extend it to datasets of arbitrary size. As for Fig. 3, it seemingly contains no confidence intervals. However, the authors' method appears to perform comparably to FT-Transformer and Tab-R for classification tasks; on regression tasks, it is faster, but less performant, than FT-Transformer.\n2. Sec 2.1 explains how the new benchmark was assembled, but not why it was necessary to do so. [1], [2] and Grinsztajn et al. have already introduced robust tabular benchmark suites into the literature. The authors also mention, but utilize only parts of, the AutoML benchmark and the OpenML-CTR23 regression benchmark.\n3. While the concerns raised in Sec. 2.2 about some previously used metrics are reasonable, the choice of shifted geometric mean error, which adds noise to extremely low error scores, is also problematic. A better approach would have been to report multiple metrics.\n\nMINOR\n\n1. The 95% confidence intervals in the figures are very difficult to see, and should be made clearer.\n\n[1] When do neural nets outperform boosted trees on tabular data?\n[2] TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback.\n\nMajor weaknesses:\n\n1. \n- We meant to write “is competitive with GBDTs in terms of benchmark scores”, we will update this sentence. Of course, GBDTs are faster on most datasets with our settings. But note that lower is better in terms of benchmark scores in Figure 2 and RealMLP is better than GBDTs on all benchmarks in terms of scores in Figure 2.\n- The simple “TabPFN+subsampling” version from [1] is outperformed by TabR and FT-Transformer in the recent large benchmark in https://arxiv.org/pdf/2407.00956, both of which are included in our Figure 3 and partially in Figure 2. Regarding versions of TabPFN that scale to large datasets, many recent scaling works are concurrent works. There is TuneTables from February, but still it (1) will be substantially slower than 1 second when scaled to larger datasets, and (2) is limited to classification. TabPFN-based results will probably be outdated very soon anyways since TabPFNV2 is supposed to be out soon.\n- Figure 3 contains no confidence intervals because we used the codebase of Grinsztajn et al. to compute results and it does not provide confidence intervals with the same meaning as the confidence intervals in Figure 2 (in general, producing such confidence intervals for normalized benchmark scores as done in Figure 3 would be statistically questionable, as the normalization of the metric depends on the benchmark results itself.)\n- Your last claim is incorrect: RealMLP outperforms FT-Transformer in terms of speed and benchmark scores for regression on the Grinsztajn et al. benchmark (and performs slightly better for classification as well).\n\n2. \n- Regarding the meta-train benchmark: The benchmark consists of datasets that were available to us when we started experimenting. While it is not established, we cannot change it post-hoc since we already used it to design our methods.\n- Regarding the meta-test benchmark: The benchmark consists of the AutoML benchmark and the CTR23 benchmarks, which are well-curated tabular benchmarks. The AutoML benchmark is well-established in AutoML, and [1, 2] also use it for their dataset collection. The contained datasets span a large range of sizes and dimensionalities, allowing us to test meta-generalization of tuned defaults. We removed some datasets according to well-defined criteria in Appendix C.3.2, mainly datasets that are already included in the meta-train benchmark or are too small.\n- The benchmark suite of [1] is interesting but was released concurrently to when we started experimenting with the meta-test benchmark, and is mainly covering classification. As far as we can see, [1] doesn’t provide clear criteria by which they have selected their datasets, and their TabZilla subset has selection criteria prefering datasets that are hard for GBDTs, which would be unfairly favorable to neural networks such as our RealMLP.\n- [2] contains many datasets but mostly small ones, which we do not design our methods for, and which result in noisier evaluation due to small test sets. They also use some datasets from the AutoML benchmark, which we already use.\n- We use the Grinsztajn et al. benchmark datasets separately in Figure 3, to have a common benchmark where many baselines can run. We chose not to use these datasets for the meta-test benchmark since there are less of them (less than half of what we have now after excluding the meta-train datasets), and they are more limited in terms of number of features etc.\n\n3. We **do report alternative aggregation metrics** in Figure B.7 – B.15 and reference some of them from Section 5.2. We also report normalized accuracies on the Grinsztajn et al. (2022) benchmark (Figure 3), to conform with the original benchmark. We agree that all aggregation strategies have some upsides and downsides.\n\nMinor weaknesses:\n- We increased the linewidth of the error bars, see the rebuttal PDF.\n\n\nQuestions:\n- See above.\n- See above, we do report multiple aggregation metrics.\n- Because the meta-test benchmark is so expensive to run, see the global response. We already have the Grinsztajn et al benchmark for this purpose. Another issue is the lack of easily usable interfaces for most other published deep learning methods, including details such as how to pass a separate validation set, how to change the early stopping metric, how to pass categorical data, how to select the correct GPU device, etc. In contrast, we make scikit-learn interfaces available that allow configuring all of this easily.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,
thank you for the constructive feedback. We identified two main points raised by multiple reviewers:
- 4/5 reviewers asked for more baselines. In total, ten baselines were mentioned (MLP-PLR, ExcelFormer, Net-DNF, TabNet, TabCaps, versions of TabPFN, TabR-HPO, ResNet-HPO, SAINT, FT-Transformer). In the **attached PDF**, we **add MLP-PLR** as a related baseline with good speed/accuracy trade-off, as well as **TabR-HPO** on the Grinsztajn et al. benchmark and **ResNet-HPO** on the meta-benchmarks. However, we hope that the reviewers understand that we cannot run all possible baselines for reasons of computational cost, implementation/verification effort and cluttering of the paper/figures. In particular, we should have emphasized more that the meta-test benchmark is very expensive to run (ca 4 GPU-weeks for RealMLP-HPO, ca 2 GPU-weeks for ResNet-HPO, probably >5 GPU-months for TabR-HPO given that the runtime of TabR-S-D is >4 GPU-days). In addition, some methods (Transformer-based/TabPFN-based) fail due to the high numbers of features (up to 10K). The diversity in size and dimensions of the meta-test datasets allow for a broad study of meta-generalization. We have already included the Grinsztajn et al. benchmark in Figure 3 in order to have a common and less expensive benchmark with more deep learning baselines. Details on our decisions can be found in our individual replies.
- 2/5 reviewers asked whether our tricks would help other architectures as well. While we agree that this is an interesting question, we would argue that it is not relevant to our central question, namely how well MLPs and GBDTs can perform with tuned default settings. Moreover, due to the much higher training cost of models like TabR or FT-Transformer, such experiments would require a lot of extra effort. We therefore do not want to explore this question further in this paper beyond the preprocessing experiments that we already have in Table B.5.

We kindly ask the reviewers to consider raising their score, given that our benchmarking efforts are already more extensive than what can be found in most other method-papers.

Meta-review: While the work is not entirely novel, there are extensive benchmarks, and the methods provided are easily applicable for practitioners.
The main concern of the reviewers seems to be the breadth of the comparison. However, the reviewers do provide a wide array of benchmarks and baseline methods.
Decision: Accept (poster)