Better by default: Strong pre-tuned MLPs and boosted trees on tabular data

Authors: david  holzmüller
léo  grinsztajn
ingo  steinwart
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 21:01:55

Abstract: For classification and regression on tabular data, the dominance of gradient-boosted decision trees (GBDTs) has recently been challenged by often much slower deep learning methods with extensive hyperparameter tuning. We address this discrepancy by introducing (a) RealMLP, an improved multilayer perceptron (MLP), and (b) strong meta-tuned default parameters for GBDTs and RealMLP. We tune RealMLP and the default parameters on a meta-train benchmark with 118 datasets and compare them to hyperparameter-optimized versions on a disjoint meta-test benchmark with 90 datasets, as well as the GBDT-friendly benchmark by Grinsztajn et al. (2022). Our benchmark results on medium-to-large tabular datasets (1K--500K samples) show that RealMLP offers a favorable time-accuracy tradeoff compared to other neural baselines and is competitive with GBDTs in terms of benchmark scores. Moreover, a combination of RealMLP and GBDTs with improved default parameters can achieve excellent results without hyperparameter tuning. Finally, we demonstrate that some of RealMLP's improvements can also considerably improve the performance of TabR with default parameters.

Main paper: 
 introduction: 
 Perhaps the most common type of data in practical machine learning (ML) is tabular data, characterized by a fixed number of features (columns) that can take different types such as numerical or categorical, as well as a lack of the spatiotemporal structure found in image or text data. The moderate dimension and lack of symmetries make tabular data accessible to a wide variety of machine learning methods. Although tabular data is very diverse and no method is dominant on all datasets, gradient-boosted decision trees (GBDTs) exhibit excellent results on benchmarks [58, 18, 43, 69] , although their superiority has been challenged by a variety of deep learning methods [3] .While many architectures for neural networks (NNs) have been proposed [3] , variants of the simple multilayer perceptron (MLP) have repeatedly been shown to be good baselines for tabular NNs [30, 15, 16, 54] . Moreover, in terms of training time, MLPs are often slower than GBDTs but still considerably faster than many other architectures [18, 43] . Therefore, we study how MLPs can be improved in terms of architecture, training, preprocessing, hyperparameters, and initialization. We also demonstrate that at least some of these improvements can successfully improve TabR [17] .The problem of finding the best fixed hyperparameters is a meta-learning problem [4, 64] . Although we do not introduce or employ a fully automated method to find good defaults, we use a meta-learning benchmark setup to properly evaluate them. Wistuba et al. [66] and Pfisterer et al. [49] learn portfolios of configurations and van Rijn et al. [63] learn symbolic defaults, but neither of these papers considers GBDTs or NNs. Salinas and Erickson [55] learn large portfolios of configurations on an extensive benchmark, without studying the best defaults for individual model families. Such portfolios are successfully applied in modern AutoML methods [10, 11] . At the other end of the meta-learning spectrum, TabPFN [23] meta-learns a (tuning-free) learning method on small synthetic datasets. Unlike TabPFN, we only meta-learn hyperparameters and can therefore use fewer but larger and more realistic meta-train datasets, resulting in methods that scale to larger datasets. 
methodology: 
 To evaluate a fixed hyperparameter configuration H, we need a collection B train of benchmark datasets and a scoring function that computes a benchmark score S(B train , H) by aggregating the errors attained by the method with hyperparameters H on each dataset. However, when optimizing H on B train , we might overfit to the benchmark and therefore ideally need a second benchmark B test to get an unbiased score for H. We refer to B train , B test as meta-train and meta-test benchmarks and subdivide them into classification and regression benchmarks B train class , B train reg , B test class , and B test reg . We also use the Grinsztajn et al. [18] benchmark B Grinsztajn , which allows us to run more expensive baselines, since it limits training set sizes to 10K samples and contains fewer datasets due to more strict dataset inclusion criteria. Since B train contains groups of datasets that are variants of the same dataset, for example by using different columns as targets, we use weighting factors inversely proportional to the group size.Table 1 shows some characteristics of the considered benchmarks. The meta-test benchmark includes datasets that are more extreme in several dimensions, allowing us to test whether our default parameters generalize "out of distribution". For all datasets, we remove rows with missing numerical values and encode missing categorical values as a separate category.The meta-train set consists of medium-sized datasets from the UCI Repository [32] , adapted from Steinwart [61] . The meta-test set consists of the datasets from the AutoML Benchmark [13] as well as the OpenML-CTR23 regression benchmark [12] with a few modifications: we subsample some large datasets and remove datasets that are already contained in the meta-train set, are too small, or have categories with too large cardinality. More details on the datasets and preprocessing can be found in Appendix C.3.To optimize the default parameters, we need to define a single benchmark score. To this end, we evaluate a method on N splits = 10 random training-validation-test splits (60%-20%-20%) on each dataset. As metrics on individual dataset splits, we use classification error (100% -accuracy) or 1-AUROC(one-vs-rest) for classification and nRMSE := RMSE standard deviation of targets = 1 -R 2for regression. There are various options to aggregate these errors into a single score. Some, such as average rank or mean normalized error, depend on which other methods are included in the evaluation, hindering an independent optimization. We would like to use the geometric mean error because arguably, an error reduction from 0.02 to 0.01 is more valuable than an error reduction from 0.42 to 0.41. However, since the geometric mean error is too sensitive to cases with zero error (especially for classification error), we instead use a shifted geometric mean error, where a small value ε := 0.01 is added to the errors err ij before taking the geometric mean:SGM ε := exp   N datasets i=1 w i N splits N splits j=1 log(err ij + ε)   .Here, we use weights w i = 1/N datasets on the meta-test set and Grinsztajn et al. [18] benchmark. On the meta-train set, we make the w i dependent on the number of related datasets, cf. Appendix C. 3 . In Appendix B.10, we present results for other aggregation strategies. 
improving neural networks: 
 The following section presents RealMLP-TD, our improved MLP with tuned defaults, which was designed based on experiments on the meta-train benchmark. A simplified version called RealMLP-TD-S is also described. To demonstrate that our improvements can be useful for other architectures, we introduce RealTabR-D, a version of TabR that includes some of our improvements but has not been tuned as extensively as RealMLP-TD.Data preprocessing In the first step of RealMLP, we apply one-hot encoding to categorical columns with at most eight distinct values (not counting missing values). Binary categories are encoded to a single feature with values {-1, 1}. Missing values in categorical columns are encoded to zero. After that, all numerical columns, including the one-hot encoded ones, are preprocessed independently as follows: Let x 1 , . . . , x n ∈ R be the values in column i, and let q p be the p-quantile of (x 1 , . . . , x n ) for p ∈ [0, 1]. Then, x j,processed := f (s j • (x jq 1/2 )), f (x) := x 1 + ( x 3 ) 2 , s j :=      1 q 3/4 -q 1/4 , if q 3/4 ̸ = q 1/4 2 q1-q0, if q 3/4 = q 1/4 and q 1 ̸ = q 0 0 , otherwise. In scikit-learn [48] , this corresponds to applying a RobustScaler (first case) or MinMaxScaler (second case), and then the function f , which smoothly clips its input to the range (-3, 3). Smooth clipping functions like f have been used by, e.g., Holzmüller et al. [24] and Hafner et al. [20] . Intuitively, when features have large outliers, smooth clipping prevents the outliers from affecting the result too strongly, while robust scaling prevents the outliers from affecting the inlier scaling.NN architecture Our architecture, visualized in Figure 1 (a), is a multilayer perceptron (MLP) with three hidden layers containing 256 neurons each, except for the following additions and modifications:• RealMLP-TD employs categorical embedding layers [19] to embed the remaining categorical features with cardinality > 8. • For numerical features, excluding the one-hot encoded ones, we introduce PBLD (periodic bias linear DenseNet) embeddings, which concatenate the original value to the PL embeddings proposed by Gorishniy et al. [16] and use a different periodic embedding with biases, , where the best default learning rate is found separately for each step. The vanilla MLP uses categorical embeddings, a quantile transform to preprocess numerical features, default PyTorch initialization, ReLU activation, early stopping, and is optimized with Adam with default parameters. For more details, see Appendix A. 4 . The error bars are approximate 95% confidence intervals for the limit #splits → ∞, see Appendix C.6.inspired by Huang et al. [26] and Rahimi and Recht [52] , respectively. PBLD embeddings apply separate small two-layer MLPs to each feature x i asEQUATIONemb cos(2πw(1,i)emb x i + b (1,i) emb ) + b (2,i) emb ∈ R 4 .For efficiency reasons, we use 4-dimensional embeddings with w(1,i) emb , b (1,i) emb ∈ R 16 , b (2,i) emb ∈ R 3 , W (2,i) emb ∈ R 3×16 .• To encourage (soft) feature selection, we introduce a scaling layer before the first linear layer, which is simply a matrix-vector product with a diagonal weight matrix. In other words, it computes x i,out = s i • x i,in , with a learnable scaling factor s i for each feature i.We found it beneficial to use a larger learning rate for this layer. • Our linear layers use the neural tangent parametrization (NTP) as proposed by Jacot et al. [28] , i.e., they compute z (l+1) = d -1/2 l W (l) x (l) + b (l) , where d l is the dimension of the layer input x (l) . The motivation behind the use of the NTP here is that it effectively modifies the learning rate for the weight matrices depending on the input dimension d l , hopefully preventing too large steps whenever the number of columns is large. We did not observe improvements when using the Adam version of the maximal update parametrization [68] .• RealMLP-TD uses parametric activation functions inspired by PReLU [21] . In general, for an activation function σ, we define a parametric version with separate learnable α i for each neuron i: σ αi (x i ) = (1α i )x i + α i σ(x i ) . When α i = 1, this recovers σ, and when α i = 0, the activation function is linear. As activation functions, we use SELU [35] for classification and Mish [45] for regression.• We use dropout after each activation function. We do not use the Alpha-dropout variant originally proposed for SELU [35] , as we were not able to obtain good results with it. • For regression, at test time, the MLP outputs are clipped to the observed range during training. (We observed that this is mainly helpful for suboptimal hyperparameters.)The parameters s i of the scaling layer are initialized to 1, making it an identity function at initialization. Similarly, the parameters α i of the parametric activation functions are initialized to 1, recovering the standard activation functions at initialization. We initialize weights and biases in a data-dependent fashion during a forward pass on the (possibly subsampled) training set. We rescale rows of standard-normal-initialized weight matrices to scale the variance of the output pre-activations over the dataset to one. For the biases, we use the data-dependent he+5 initialization method [61] .Training Like Gorishniy et al. [15] , we use the AdamW optimizer [40, 34] . We set its momentum hyperparameters to β 1 = 0.9 and β 2 = 0.95 instead of the default β 2 = 0.999. The idea to use a smaller value for β 2 is adopted from the fastai tabular MLP [25] . RealMLP is optimized for 256 epochs with a batch size of 256. As a loss function for classification, we use softmax + cross-entropy with label smoothing [62] with parameter ε = 0.1. For regression, we use the MSE loss and affinely transform the targets to have zero mean and unit variance on the training and validation set.Hyperparameters We allow parameter-specific scheduled hyperparameters computed in each iteration using a base value, optional parameter-specific factors, and a schedule, as base_value • param_factor • schedule iteration #iterations , allowing us, for example, to use a high learning rate factor for scaling layer parameters. Because we do not tune the number of epochs separately on each dataset, we use a multi-cycle learning rate schedule, providing multiple valleys that are usually preferable for stopping the training, while allowing high learning rates in between. Our schedule is similar to Loshchilov and Hutter [39] and Smith [59] , but with a simpler analytical expression:coslog k (t) := 1 2(1cos(2π log 2 (1 + (2 k -1)t))) .We set k = 4 to obtain four cycles as shown in Figure 1 (b). To allow stopping at different levels of regularization, we schedule dropout and weight decay using the following schedule, cf. The detailed hyperparameters can be found in Table A .1.Best-epoch selection Due to the multi-cycle learning rate schedule, we do not perform classical early stopping. Instead, we always train for the full 256 epochs and then revert the model to the epoch with the lowest validation error, which in this paper is based on classification error, or RMSE for regression. In case of a tie, we found it beneficial to use the last of the tied best epochs.RealMLP-TD-S Since certain aspects of RealMLP-TD are somewhat complex to implement, we introduce a simplified (and faster) variant called RealMLP-TD-S in Appendix A. Among the simplifications are: omitting embedding layers, using non-parametric activations, using a simpler initialization method, and omitting dropout and weight decay.RealTabR-D For RealTabR-D, we adapt TabR-S-D by using our numerical preprocessing, setting Adam's β 2 to 0.95, using our scaling layer with a modification to amplify the effective learning rate by a factor of 96, adding PBLD embeddings for numerical features, and adding label smoothing for classification. More details can be found in Appendix A.3. 
gradient-boosted decision trees: 
 To find better default hyperparameters for GBDTs, we employ a semi-automatic approach: We use hyperparameter optimization libraries like hyperopt [2] and SMAC3 [38] to explore a reasonably large hyperparameter space, evaluating the benchmark score of each configuration on the meta-train benchmarks, and then perform some small manual adjustments like rounding the best obtained hyperparameters. To balance efficiency and accuracy, we fix the number of estimators to 1000 and use the hist method for XGBoost. We only consider the libraries' default tree-building strategies since it is one of their main differences. The tuned defaults (TD) for LightGBM (LGBM), XGBoost (XGB), and CatBoost can be found in Table C .1, C.2, and C.3, respectively.While some of the obtained hyperparameter values might be sensitive to the tuning and benchmark setup, we observe some general trends. First, row subsampling is used in all tuned defaults, while column subsampling is rarely applied. Second, trees are generally allowed to be deeper for regression than for classification. Third, the Bernoulli bootstrap in CatBoost is competitive with the Bayesian bootstrap while also being faster. 
experiments: 
 In the following, we evaluate different methods with library defaults (D), tuned defaults (TD), and hyperparameter optimization (HPO). Recall that TD uses fixed parameters optimized on the meta-train benchmarks, while HPO tunes hyperparameters on each dataset split independently. All methods except random forests select the best iteration/epoch on the validation set of the respective dataset split based on accuracy / RMSE. All NN-based regression methods standardize the labels for training.We provide methods in the following variants:• D: Default parameters, taken from the original library if possible (Appendix C.1).• TD: Tuned default parameters from Section 3 and Section 4.• HPO: Hyperparameters optimized separately for every train-test split on every dataset, using 50 steps of random search. Search spaces are specified in Appendix C.2 and are usually adapted from original or popular papers.As tree-based methods, we use XGBoost (XGB), LightGBM (LGBM), and CatBoost from the respective libraries, as well as random forest (RF) from scikit-learn. The variant XGB-PBB-D uses meta-learned default parameters from Probst et al. [50] . For neural methods, we compare to MLP, ResNet, and FT-Transformer (FTT) from Gorishniy et al. [15] , MLP-PLR from Gorishniy et al. [16] , as well as TabR and TabR-S (without numerical embeddings) from Gorishniy et al. [17] . We compare these methods to RealMLP and RealTabR from Section 3. In addition, we investigate Best, which on each dataset split selects the method with the best validation score out of XGB, LGBM, CatBoost, and MLP-PLR (for Best-D) or RealMLP (for Best-TD and Best-HPO). Ensemble builds a weighted ensemble out of the same methods as Best, using the method of Caruana et al. [5] with 40 greedy selection steps as in Salinas and Erickson [55] .We do not run FTT, RF-HPO, and TabR-HPO on all benchmarks since some benchmarks (especially meta-test) are more expensive to run and these methods may run into out-of-memory errors.Figure 2 shows the results of the aforementioned methods on all benchmarks, along with their runtimes on a CPU. How good are tuned defaults on new datasets? To answer this question, we compare the relative gaps between TD and HPO benchmark scores on the meta-test benchmarks to those on the meta-train benchmarks. The gap between RealMLP-HPO and RealMLP-TD is not much larger on the meta-test benchmarks, indicating that the tuned defaults transfer very well to the meta-test benchmark. For GBDTs, tuned defaults are competitive with HPO on the meta-train set, but not as good on the meta-test set. Still, they are considerably better than the untuned defaults on the meta-test set. Note that we did not limit the TD parameters to the literature search spaces for the HPO models (cf. Appendix C.2); for example, XGB-TD uses a smaller value of min_child_weight for classification and CatBoost-TD uses deeper trees and Bernoulli boosting. The XGBoost defaults XGB-PBB-D from Probst et al. [50] outperform XGB-TD on B test class , perhaps because their benchmark is more similar to B test class or because XGB-PBB-D uses more estimators (4168) and deeper trees.RealMLP and RealTabR perform strongly among NNs. On most benchmarks, RealMLP-TD and RealTabR-D bring considerable improvements over MLP-PLR-D and TabR-S-D, at slightly larger runtimes, respectively. Similarly, RealMLP-HPO improves the results of MLP-PLR-HPO. TabR and FTT are notably slower than MLP-based methods on CPUs, while the difference is less pronounced on GPUs (Figure C.2). While RealMLP-TD beats TabR-S-D on many benchmarks, RealTabR-D performs even better on four out of six benchmarks, especially all regression benchmarks. On the Grinsztajn et al. [18] benchmark where we can afford to run more baselines, TabR-HPO performs best according to many aggregation metrics. It performs especially well on the electricity dataset, where MLPs struggle to learn high-frequency patterns [18] .RealMLP and RealTabR are competitive with tree-based models. On the meta-train and metatest benchmarks, RealMLP and RealTabR perform better than GBDTs in terms of shifted geometric mean error, while also being comparable or slightly better in terms of other aggregations like mean normalized error (Appendix B.10) or win-rates (Appendix B.12). On the Grinsztajn et al. [18] benchmark, RealMLP performs worse than CatBoost for classification and comparably for regression, while RealTabR-D performs comparably to CatBoost-TD for classification and better for regression.Among GBDTs, CatBoost defaults are better and slower. Several papers have found CatBoost to perform favorably among GBDTs while being more computationally expensive to train [51, 43, 8, 33, 69] . We observe the same for our tuned defaults on most benchmarks.Simply trying all default algorithms is faster and very often better than (naive) single-algorithm HPO. When comparing Best-TD to 50-step HPO on RealMLP or GBDTs, we notice that Best-TD is faster on average, while also being competitive with the best of the HPO models. In comparison, Best-D is often outperformed by RealMLP-HPO. We also note that ensemble selection [5] usually gives 0-3% improvement on the benchmark score compared to selecting the best model, and can potentially be further improved [6] . Unlike McElfresh et al. [43] , who argue in favor of CatBoost-HPO over trying NNs, our results favor model portfolios as used in modern AutoML systems [10] .Analyzing NN improvements Figure 1 (c) shows how adding the proposed RealMLP components to a simple MLP improves the meta-train benchmark performance. However, these results depend on the order in which components are added, which is addressed by a separate ablation study in Appendix B. For example, the large weight decay value makes RealMLP-TD sensitive to changes in some other hyperparameters like β 2 . We also show in Appendix B.8 that our architectural improvements alone are beneficial when applied to MLP-D directly, although non-architectural aspects are at least as important. In particular, our numerical preprocessing is easy to adopt and often beneficial for other NNs as well (Appendix B.7). The scaling layer and PBLD embeddings are easy to use and turned out to be effective within RealTabR-D as well. If affordable, larger stopping patiences and the use of (cyclic) learning rate schedules can be useful, while label smoothing is influential but can be detrimental for metrics like AUROC (Figure 3 , Appendix B.5).We observe that choices in benchmark design can affect the interpretation of the results. The use of different aggregation metrics than the shifted geometric mean reduces the advantage of TD methods (Appendix B.10). For classification, using AUROC instead of classification error (Figure 3 , Appendix B.5) favors GBDTs. Different dataset selection and preprocessing criteria on different benchmarks lead to large differences between benchmarks in the average errors, as indicated by the y-axis scaling in Figure 2 .In Appendix B, we present additional experimental results. We compare bagging and refitting for RealMLP-TD and LGBM-TD, finding that refitting multiple models is often better on average. We demonstrate that GBDTs benefit from high early stopping patiences for classification, especially when using accuracy as the stopping metric. When considering AUROC as a stopping metric, we show that stopping on cross-entropy is preferable to accuracy (Appendix B.5).Limitations While our benchmarks cover medium-to-large tabular datasets in standard settings, it is unclear to which extent the obtained defaults can generalize to very small datasets, distribution shifts, datasets with missing numerical values, and other metrics such as log-loss. Additionally, runtimes and the resulting tradeoffs may change with different parallelization, hardware, or (time-aware) HPO algorithms. For computational reasons, we only use a single training-validation split per train-test split. This means that HPO can overfit the validation set more easily than in a cross-validation setup. While we extensively benchmark different NN models from the literature, we do not attempt to equalize non-architectural aspects, and our work should therefore not be seen as a comparison of architectures. We compared to TabR-S-D as a recent promising method with good default parameters [17, 69] . However, due to a surge of recently published deep tabular models [e.g., 7, 8, 57, 41, 33, 67, 29] , it is unclear what the current "best" deep tabular model is. In particular, ExcelFormer [7] also promises strong-performing default parameters. For GBDTs, due to the cost of running the benchmarks, our limits on the depth and number of trees are on the lower side of the literature. 
conclusion: 
 In this paper, we studied the potential of improved default parameters for GBDTs and an improved MLP, evaluated on a large separate meta-test benchmark as well as the benchmark by Grinsztajn et al. [18] , and investigated the time-accuracy tradeoffs of various algorithm selection and ensembling scenarios. Our improved MLP mostly outperforms other NNs from the literature with moderate runtime and is competitive with GBDTs in terms of benchmark scores. Since many of the proposed improvements to NNs are orthogonal to the improvements in other papers, they offer exciting opportunities for combinations, as we demonstrated with our RealTabR variant. While the "NNs vs GBDTs" debate remains interesting, our results demonstrate that with good default parameters, it is worth trying both algorithm families even with a moderate training time budget.  
 
 Appendix: The detailed hyperparameter settings for RealMLP-TD and RealMLP-TD-S are listed in Table A .1.Architecture To make the binary and multi-class cases more similar, we use two output neurons in the binary case, using the same loss function as in the multi-class case.Initialization We initialize categorical embedding parameters from N (0, 1). We initialize the components of w For weights and biases of the linear layers, we use a data-dependent initialization. The initialization is performed on the fly during a first forward pass of the network on the training set (which can be subsampled adaptively not to use more than 1 GB of RAM). We realize this by providing fit_transform() methods similar to a pipeline in scikit-learn. For the weight matrices, we use a custom two-step procedure: First, we initialize all entries from N (0, 1). Then, we rescale each row of the weight matrix such that the outputs 1√ d l W (l) x (l)j have variance 1 over the dataset (i.e. when considering the sample index j ∈ {1, . . . , n} as a uniformly distributed random variable). This is somewhat similar to the LSUV initialization method [44] . For the biases, we use the data-dependent he+5 initialization method [61] .Training We implement weight decay as in PyTorch using θ ← θ -lr • wd • θ, which includes the learning rate unlike the original version [40] .For RealMLP-TD-S, we make the following changes compared to RealMLP-TD:• We apply one-hot encoding to all categorical variables and do not apply categorical embeddings. • We do not apply numerical embeddings.• We use the standard non-parametric versions of the SELU and Mish activation functions. • We do not use dropout and weight decay.• We use simpler weight and bias initializations: We initialize weights and biases from N (0, 1), except in the last layer, where we initialize them to zero. • We do not clip the outputs, even in the regression case.• We apply a different base learning rate in the regression case.To obtain RealTabR-D, we modify TabR-S-D in the following ways:• We replace the standard numerical preprocessing (a modified quantile transform) with our robust scaling and smooth clipping. • We set Adam's β 2 to 0.95 instead of 0.999.• We use our scaling layer, but modify it to obtain a higher effective learning rate. We do this by modifying the forward pass tox i,out = γ • s i • x i,in , while initializing s i to 1/γ. This will multiply the gradients of s i by γ, which will be ignored by Adam's normalization (when neglecting Adam's ε parameter). It will also multiply the optimizer updates by γ, leading to approximately the same effect as multiplying the learning rate by γ. However, a difference is that multiplying the learning rate by γ will also lead to stronger weight decay updates in PyTorch's AdamW implementation, while the introduction of γ does not increase the relative magnitude of weight decay updates. We chose the version with γ for simplicity of implementation. While RealMLP-TD uses a learning rate factor of 6 for the scaling layer, it uses a higher base learning rate due to the use of the neural tangent parametrization. For all layers except the first one, which have width 256, the neural tangent parametrization in RealMLP-TD uses a factor similar to γ, which is set to 1/16 = 1/ √ 256. Hence, RealMLP-TD without NTP should use a base learning rate for these layers that is smaller by a factor of 1/16, and therefore use a learning rate factor of 6 • 16 = 96 for the scaling layer. Consequently, we set γ := 96 for RealTabR-D without further tuning, noting that it performed significantly better than γ = 6 on the meta-train benchmarks.• We use our PBLD embeddings for numerical features before the scaling layer, instead of no numerical embeddings in TabR-S-D. In order to make every experiment run on a GPU with 24GB RAM, we decrease the dimension of the hidden embedding layer from 16 to 8, although using 16 would have performed slightly better in our experiments on the meta-train benchmarks. • For classification, we use label smoothing with parameter ε = 0.1.Since we adapted hyperparameters like learning rate and weight decay from TabR-S-D without meta-learning them, we refer to the resulting method as RealTabR-D and not RealTabR-TD. We did not include other tricks from RealMLP-TD for various reasons:• Brief experiments with NTP and the Mish activation deteriorated the performance.• Parametric activations and increased stopping patience showed small improvements but were excluded due to a larger runtime. • Other tricks were not tried due to limited time of experimentation, expected increases in the already somewhat large runtime, and/or implementation complexity.Here, we provide more details on the vanilla MLP and the ablation steps from Figure or Mish (regression). • Parametric act. fn.: We use parametric versions of the activation functions, with a learning rate factor of 0.1 for the parameters. • Scaling layer: We use a scaling layer with a learning rate factor of 6 before the first linear layer. • Num. embeddings: PL: We apply the PL embeddings [16] to numerical features. • Num. embeddings: PBLD: We apply our PBLD embeddings instead.• Dropout p = 0.15: We apply dropout with probability 0.15.• Dropout sched: flat_cos: We apply the flat_cos schedule to the dropout probability.• Weight decay wd = 0.02: We apply weight decay (as in AdamW, PyTorch version) with value 0.02. • wd sched: flat_cos: We apply the flat_cos schedule to weight decay. • Bias init: he+5: We apply the he+5 bias initialization method from Steinwart [61] .• Weight init: data-driven: We apply our data-driven weight initialization method.Here, we discuss some of the design decisions behind RealMLP-TD and possible trade-offs. First, our implementation allows us to train RealMLP-TD in a vectorized fashion on multiple train-validationtest splits at the same time. On the one hand, this can lead to speedups on GPUs when training multiple models in parallel, including on the benchmarks. On the other hand, it can hinder the implementation of certain methods like patience-based early stopping or loss-based learning rate schedules. While our ablations in Appendix B.1 show the advantage of our multi-cycle schedule over decreasing learning rate schedules, the latter ones could potentially enable a faster average training time through low-patience early stopping. An interesting follow-up question could be whether the multi-cycle schedule still works well with larger-patience early stopping.Regarding categorical embeddings, our meta-train benchmark does not contain many high-cardinality categorical variables, and we were not able to conclude whether categorical embeddings are helpful or harmful compared to one-hot encoding (see Appendix B.1). Our motivation to include categorical embeddings stems from Guo and Berkhahn [19] as well as their potential to be more efficient for high-cardinality categorical variables. However, in practice, we find pure one-hot encoding to be faster on most datasets. Regarding the embedding size, we found that 4 already gave good results for numerical embeddings and decided to use 8 for categorical variables.Additionally, other speed-accuracy tradeoffs are possible. Especially for regression, we observed that more epochs and larger hidden layers can be helpful. When faster networks are desired, the omission of numerical and categorical embedding layers as well as parametric activations from RealMLP-TD can be helpful, while the other omissions in RealMLP-TD-S do not considerably affect the training time. Of course, using larger batch sizes can also be helpful for larger datasets.One caveat for classification is that cross-entropy with label smoothing is not a proper scoring rule, that is, in the infinite-sample limit, it is not minimized by the true probabilities P (y|x) [14] . Hence, label smoothing might not be suitable when other classification error metrics are used, as demonstrated in Appendix B.5 for AUROC.To assess the importance of different improvements in RealMLP-TD, we perform an ablation study. We perform the ablation study only on the meta-train benchmarks, first because they are considerably faster to run, and second because we tune the default parameters only on the meta-train benchmarks. Since the hyperparameters of RealMLP-TD have been tuned on the meta-train benchmarks, the ablation scores are not unbiased but represent some of the considerations that have been made when tuning the defaults. For each ablation, we multiply the default learning rate by learning rate factors from the grid {0.1, 0.15, 0.25, 0.35, 0.5, 0.7, 1.0, 1.4, 2.0, 3.0, 4.0} and pick the best one. Table B .1 shows the results of the ablation study in terms of the relative increase of the benchmark score for each ablation.In general, we observe that ablations often lead to much larger changes for regression than for classification. Perhaps this is because nRMSE is more sensitive to outliers compared to classification error. Another factor could be that the classification benchmark contains more datasets than the regression benchmark. For the specific ablations, we observe a few things:• For the numerical embeddings, we see that PBLD outperforms PL, PLR, and no numerical embeddings. Contrary to Gorishniy et al. [16] , PL embeddings perform better than PLR embeddings in our setting. While the configurations with PLR and no numerical embeddings appear extremely bad for regression, we observed that they can perform more benignly with lower weight decay values. • Using the Adam default value of β 2 = 0.999 instead of our default β 2 = 0.95 leads to considerably worse performance, especially for regression. As for numerical embeddings, we observed that the difference is less pronounced at lower weight decay values. • Using a cosine decay learning rate schedule instead of our multi-cycle schedule leads to small deteriorations. A constant learning rate schedule performs even worse, especially for regression.• Not employing label smoothing for classification is detrimental by around 1.8%.• The learnable scaling layer yields improvements around 1.2% on both benchmarks.• The use of parametric activations results in a considerable 4.8% improvement for regression but is insignificant for classification. We observed that parametric activations can sometimes alleviate optimization difficulties with weight decay. • The differences between activation functions are rather small. For classification, Mish is competitive with SELU in this ablation but we found it to be worse in some other hyperparameter settings, so we keep SELU as the default. For regression, Mish performs best. • For dropout and weight decay, we observe that they yield comparable but not always significant benefits for classification and regression. Scheduling dropout and weight decay parameters with the flat_cos schedule is helpful for regression, but not for classification in this setting. • When comparing the standard parametrization (SP) to the neural tangent parametrization (NTP), we disable weight decay for a fair comparison. Moreover, for SP, we set the learning rate factors for weight and bias layers to 1/16 = 1/ √ 256. This is because, for the weights in NTP, the effective updates by Adam are damped by this factor in all hidden layers except the first one. Compared to NTP without weight decay, SP without weight decay performs insignificantly worse on both benchmarks. It is unclear to us why the parametrization, which has a considerable influence on how the effective learning speed of the first linear layer scales with the number of features, is apparently of little importance.• When comparing the data-dependent initialization of RealMLP-TD to a vanilla initialization with standard normal weights and zero biases, we see that the data-dependent initialization gains around 1% on both benchmarks. • For selecting the best epoch, we consider selecting the first best epoch instead of the last best epoch in case of a tie. This is only relevant for classification metrics like classification error, where ties are somewhat likely to occur, especially on small and "easy" datasets. We observe a non-significant 0.4% deterioration in the benchmark score. • We do not observe a significant difference when using one-hot encoding for all categorical variables, since our benchmarks contain only very few datasets with large-cardinality categorical variables.In Table B .2, we compare different preprocessing methods for numerical features. Since we want to compare these methods in a relatively conventional setting, we apply them to RealMLP-TD-S (without numerical embeddings) and before one-hot encoding. We compare the following methods:• Robust scaling and smooth clipping, our method used in RealMLP-TD and RealMLP-TD-S and described in Section 3. • Robust scaling without smooth clipping.• Standardization, i.e. subtracting the mean and dividing by the standard deviation. If the standard deviation of a feature is zero, we set the feature to zero. • Standardization followed by smooth clipping.• The quantile transformation from scikit-learn [48] with normal output distribution, which is popular in recent works [18, 16, 17, 43] . • A variant of the quantile transform, which we call the RTDL version, used by Gorishniy et al. [15] and Gorishniy et al. [17] . This version uses a dataset-size-dependent number of quantiles and adds some noise before fitting the transformation. • The recent kernel density integral transform [42] , which interpolates between the quantile transformation and min-max scaling, with default parameter α = 1.Table B .2 shows that on the meta-train benchmark, robust scaling and smooth clipping performs best for both classification and regression.In our benchmark, for each training-test split, we only train one model on one training-validation split for efficiency reasons. However, ensembling and cross-validation techniques usually allow additional improvements to models. Here, we study multiple variants for RealMLP-TD and LGBM-TD. Let We report the relative increase in the shifted geometric mean benchmark scores compared to the standard method used in RealMLP-TD and RealMLP-TD-S, which is robust scaling and smooth clipping. We also report approximate 95% confidence intervals. To have a more common setting, we do not apply the preprocessing methods to one-hot encoded categorical features. In each column, the best score is highlighted in bold, and errors whose confidence interval contains the best score are underlined.Activation=Mish -0.0 [-0.6, 0.6] 3.0 Activation=ReLU 0.5 [-0.1, 1.2] 2.0 0.7 [-0.1, 1.6] 1.0 Activation=SELU 2.3 [1.2, 3.6] 1.0 No dropout 0.8 [0.2, 1.3] 3.0 0.8 [-0.5, 2.1] 1.4 Dropout prob. 0.15 (constant) -0.1 [-1.0, 0.8] 1.Error LGBM -0.0 [-0.0, -0.0] -0.0 [-0.0, -0.0] -0.0 [-0.0, -0.0] -0.0 [-0.0, -0.0] RealMLP-TD (bagging, 1 model, joint stopping) 1.6 [0.9, 2.4] 0.7 [0.0, 1.4] 0.6 [0.1, 1.0] -0.1 [-1.0, 0.7] RealMLP-TD (bagging,y pred := 1 M M i=1 f Di,t * i (X test ), (M models) Di := D \ D i (bagging) D (refitting), t * i := argmin t∈{1,...,T } L Di (f D\Di,t ) (indiv. stopping) argmin t∈{1,...,T } 5 j=1 L Dj (f D\Dj ,t ) (joint stopping).Here, each model is trained with a different random seed. For LGBM, since we use an early stopping patience of 300 for each of the individual models, the argmin in the definition of t * i can only go up to the minimum stopping iteration T across the considered models.The results of our experiments can be found in Table B .3 for LGBM-TD and in Table B . 4 for RealMLP-TD. As expected, five models are considerably better than one. We find that refitting is mostly better than bagging, although a disadvantage of refitted models is that no validation scores are available, and it is unclear how HPO would affect this comparison. Comparing individual stopping to joint stopping, we find that individual stopping has a slight advantage in five-model bagging, while We run the XGB-TD, LGBM-TD, and CatBoost-TD with different early stopping patiences (early_stopping_rounds). We compare three different metrics used for stopping and best-epoch selection: classification error, Brier loss, and cross-entropy loss. The y-axis reports the relative increase in the benchmark score relative to stopping on classification error with patience 1000 (i.e., never stopping early). The shaded areas are approximate 95% confidence intervals, cf. Appendix C.6. joint stopping performs better for single-model refitting. In the other two scenarios, joint stopping appears slightly better for RealMLP-TD and slightly worse for LGBM-TD. We also observe that the benefit of using five models instead of one appears to be larger for RealMLP-TD than for LGBM-TD.In We observe that models stopped on classification error strongly deteriorate at low patiences (≲ 100), while our default patience of 300 achieves close-to-optimal results. Models stopped on cross-entropy loss deteriorate much less at low patiences, but achieve roughly 2% worse benchmark score at high patiences. Stopping on Brier loss achieves very good high-patience performance and is still only slightly more sensitive to the patience than stopping on cross-entropy loss. An interesting follow-up question would be if HPO can attenuate the differences between different settings.For classification, there are many different metrics to capture model performance. In the main paper, we use classification error to evaluate models. All TD configurations were tuned for classification error, early stopping and best-epoch selection were performed for classification error, and HPO was performed for classification error. Here, we evaluate models on the area under the ROC curve, also known as AUROC, AUC ROC, or AUC. For the multi-class case, we use the one-vs-rest formulation of AUC, which is faster to evaluate than one-vs-one. Higher AUC values are better and the optimal value is 1. Since we are interested in the shifted geometric mean error, we use 1 -AUC instead.We compare two settings:(1) A variant of the original setting where early stopping and the selection of the best epoch/iteration is based on accuracy but HPO is performed on 1 -AUC. (Thanks to using random search, we do not have to re-run the HPO for this.) (2) A setting where we use the cross-entropy loss for stopping and selecting the best epoch/iteration. While it would be possible to stop on AUC directly, this can be sig- We run the TD configurations of XGB, LGBM, and CatBoost with different early stopping patiences (early_stopping_rounds). As in the remainder of the paper, we use RMSE for early stopping and best-epoch selection. The y-axis reports the relative increase in the benchmark score relative to stopping on classification error with patience 1000 (i.e., never stopping early). The shaded areas are approximate 95% confidence intervals, cf. Appendix C.6. nificantly slower since AUC is slower to evaluate. We do not perform HPO in this setting since it is expensive to run.In both settings, we also evaluate RealMLP without label smoothing (no ls). Figure 3 shows the results optimized for accuracy and Figure B.3 shows the results optimized for cross-entropy. We make a few observations:• Stopping for cross-entropy generally performs better than stopping for classification error.• Label smoothing harms RealMLP for AUC, perhaps because the stopping metric does not use label smoothing, or because it encourages near-constant logits in areas where the model is relatively certain. • Tuned defaults are mostly still better than the library defaults, except for XGBoost on B test class . • RealMLP without label smoothing is still competitive with GBDTs on the meta-test benchmark but does not perform better than GBDTs unlike what we observed for classification error. To assess whether the results are influenced by our choices in missing value handling and exclusion, Figure B .4 presents results on all meta-test datasets that originally did not contain missing values. Only six meta-test datasets originally contain missing values: Three from B test class (kick, okcupid-stem, and porto-seguro) and three from B test reg (fps_benchmark, house_prices_nominal, SAT11-HAND-runtimeregression). While RealMLP deteriorates slightly, especially due to the exclusion of fps_benchmark, qualitative takeaways remain similar.In the other sections of this paper, we run each NN using the preprocessing from the respective paper that introduced it. Specifically, we use robust scaling and smooth clipping for RealMLP and the RTDL version of the quantile transform for the other papers (see also Appendix B.2). Here, we evaluate if robust scaling and smooth clipping can improve MLP, ResNet, MLP-PLR, FTT, and TabR-S as well. This also yields a more direct comparison of the architectures, although the nets still differ in other aspects such as initialization and regularization. and not so promising for others (MLP-PLR), at least without re-tuning their default parameters, our results also show that trying both preprocessing methods can already give considerable improvements on most benchmarks.Table B .5 shows the effects of including the preprocessing and architecture of RealMLP within other models. In particular, we study the benefits of our architectural changes, cf. Figure 1 (c) , when applied directly to the setting of MLP-D. To this end, we approximately reproduce MLP-D in our codebase without weight decay (since the optimal value changes when including the NTP) and with marginally different early stopping thresholding logic. We also determine the best default learning rate on the meta-train benchmark, similar to Appendix A.4. Our reproduction achieves benchmark scores within 1% of the benchmark scores of the MLP-D (RS+SC) version. Adding the PL embeddings from Gorishniy et al. [16] with our default settings sometimes gives good results but Comparison of preprocessing and architecture for different models. We include variants with robust scaling and smooth clipping (RS+SC), as well as other modified aspects, cf. Appendix B.8. We report the relative decrease in the shifted geometric mean benchmark scores compared to MLP-D. We also report approximate 95% confidence intervals, cf. Appendix C.6. is significantly worse on B test class , indicating that they need more tuning. In contrast, incorporating the RealMLP architectural changes (including their associated learning rate factors) improves scores on all benchmarks by around 5% or more, although they alone do not match the results of TabR-S-D. However, the non-architectural changes in RealMLP-TD make an even larger difference.In Figure B .6, we compare two different HPO methods for GBDTs:• Random search (HPO), as used in the main paper, with 50 steps.• Tree parzen estimator (HPO-TPE) as implemented in hyperopt [2] , with 50 steps. The first 20 of these steps use random search.While TPE often performs slightly better, the differences in benchmark scores are relatively small.Here, we provide more time-vs-error plots. Figure B.10 analyzes the external validity of differences in average ranks between methods, i.e., whether they will generalize to new datasets from a distribution. While establishing external validity requires a large number of datasets, our meta-test benchmarks show at least the improvements of RealMLP-TD over MLP-D to be externally valid.For pairs of methods, we analyze the percentage of (dataset, split) combinations on which the first method has a lower error than the second method. We plot these win-rates in marix plots: The parameters for RealMLP-TD and RealMLP-TD-S have already been given in Table A.1. Table C .1 shows the hyperparameters of LGBM-TD and LGBM-D. Table C .2 shows the hyperparameters of XGB-TD and XGB-D. Table C .3 shows the hyperparameters of CatBoost-TD and CatBoost-D. The parameters for LGBM-D, XGB-D, and CatBoost-D have been taken from the respective libraries at the time of writing and are given here for completeness. We also provide tables for MLP-D (Table C C .9). By "RTDL quantile transform", we refer to the version adding noise before fitting the quantile transform.For XGB-PBB-D, we use the default parameters from Probst et al. [50] , with the following modifications: We use hist gradient boosting since it is the new default in XGBoost 2.0. Moreover, since we have high-cardinality categories, we limit one-hot encoding to categories with less than 20 distinct values (not counting missing values) and use XGBoost's native categorical feature handling for the remaining categorical features. For RF-D, we use the default parameters from scikit-learn, do not give RF-D access to the validation set (to make it more similar to other methods that do not use nested cross-validation), and encode categorical columns using ordinal encoding with a random shuffling of categories.For all methods, we run 50 steps of random search with the search spaces presented in the following. The search spaces for LGBM-HPO (Table C .10), XGB-HPO (Table C .11), and CatBoost-HPO (Table C .12) are adapted from the "tree-friendly" literature, using n_estimators=1000 in each case. The search space for RF-HPO (Table C .13) is taken from Grinsztajn et al. [18] .For RealMLP-HPO, we provide a custom search space specified in Table C .17: Hyperparameter search space for ResNet-HPO, adapted from Gorishniy et al. [15] . We reduced the embedding dimension upper bound, the maximum number of epochs, and the number of layers to have a more acceptable runtime on the meta-test benchmarks. As in the original paper, the size of the first and the last layers are tuned and set separately, while the size for "in-between" layers is the same for all of them. For the meta-train benchmarks, we adapt code from Steinwart [61] to collect all datasets from the UCI repository that follow certain criteria:• Between 2,500 and 50,000 samples.• Number of features at most 1,000.• Labeled as classification or regression task.• Description made it straightforward to convert the original dataset into a numeric .csv format.• Uploaded before 2019-05-08.We remove rows with missing values and keep only those datasets that still have at least 2,500 samples. 3 Some datasets are labeled both as regression and classification datasets, in which case we use them for both. Some datasets contain different versions (e.g., different target columns), in which case we use all of them. To avoid biasing the results towards one dataset, we compute benchmark scores using weights proportional to 1/#versions. In total, we obtain 71 classification datasets (including versions) out of 46 original datasets, and 47 regression datasets (including versions) out of 26 original datasets. Tables C. 20 and C.21 summarize key characteristics of these datasets. We count datasets with the same prefix (before the first underscore) as being versions of the same dataset for weighting, except for the two "facebook" datasets in B train reg , which we count as distinct because they are taken from different sources. For regression, we standardize the targets to have mean zero and variance 1 on the whole dataset. This does not introduce leakage since all neural networks standardize regression targets based on the training set, and tree-based methods are invariant to affine rescaling.During earlier development of the MLP, the meta-train benchmark used to include an epileptic seizure recognition dataset, which has since been removed from the UCI repository, hence we do not report results on it.The meta-test benchmarks consist of datasets from the AutoML Benchmark [13] and additional regression datasets from the OpenML-CTR23 benchmark [12] , obtained from OpenML [65] .We make the following modifications:• We use brazilian_houses from OpenML-CTR23 and exclude Brazilian_houses from the AutoML regression benchmark, since the latter contains three additional features that should not be used for predicting the target. • We use another version of the sarcos dataset where the original test set is not included, since the original test set consists of duplicates of training samples. • We excluded the following datasets because versions of them were already contained in the meta-training set:-For classification: kr-vs-kp, wilt, ozone-level-8hr, first-order-theorem-proving, Ges-turePhaseSegmentationProcessed, PhishingWebsites, wine-quality-white, nomao, bankmarketing, adult -For regression: wine_quality, abalone, OnlineNewsPopularity, Brazilian_houses, physicochemical_protein, naval_propulsion_plant, superconductivity, white_wine, red_wine, grid_stabilityWe preprocess the datasets as follows:• We remove rows with missing continuous values • We subsample large datasets to contain at most 500,000 samples. Since the dionis dataset was particularly slow to train with GBDT models due to its 355 classes, we subsampled it to 100,000 samples. • We encode missing categorical values as a separate category.• For regression, we standardize the targets to have mean zero and variance 1. This does not introduce leakage since all neural networks standardize regression targets based on the training set, and tree-based methods are invariant to affine rescaling. After preprocessing, we• exclude datasets with less than 1,000 samples, these were for classification: albert, APSFailure, arcene, Australian, blood-transfusion-servicecenter, eucalyptus, KDDCup09_appetency, KDDCup09-Upselling, micro-mass, vehicle for regression: boston, cars, colleges, energy_efficiency, forest_fires, Moneyball, QSAR_fish_toxicity, sensory, student_performance_por, tecator, us_crime • exclude datasets that have more than 10,000 features after one-hot encoding. These were Amazon_employee_access, Click_prediction_small, and sf-police-incidents (all classification).We select the datasets as follows:• We use the newer version of the benchmark on OpenML.• When a dataset is used both in benchmarks with and without categorical features, we use the version with categorical features. • We exclude the eye_movements dataset since a leak in the dataset was reported by Gorishniy et al. [17] . Here, we compare two versions of the Grinsztajn et al. [18] benchmark:(a) The "new" version, using our benchmarking setup with the datasets of the Grinsztajn et al. [18] benchmark. This version is used in all plots except • In the new version, we removed the eye_movements dataset due to a leak reported in Gorishniy et al. [17] . • We subsample datasets after downloading them to 500K samples (all train-test splits are performed on the same 500K samples). • We always standardize targets, to make our results independent of the scaling of the datasets.(In contrast, HPO methods on the original benchmark have standardization as an option in their tuning space.) • We limit training+validation set sizes to 13333, such that at most 10K samples are used for training. Of these samples, we always use 25% for validation, unlike the original benchmark, which limits the training and validation set sizes separately to 10K and 50K samples. • The new version does not use separate validation sets for early stopping and for HPO, which avoids unfairly disadvantaging D and TD methods compared to HPO methods. • With the new version, we mostly report results using different aggregation strategies and using nRMSE instead of R 2 for regression, but try to provide comparable aggregated metrics in -The new version doesn't use as large categorical embedding sizes for ResNet and MLP models (up to 64 instead of [64, 512] ). -The old version uses larger stopping patiences for default models than in the original literature [15] . -In the old version, ResNet-HPO tunes the normalization, unlike the original paper [15] . -In the old version, the batch size is tuned for some models.-In the old version, XGBoost uses the exact tree method with one-hot encoding, while in the new version, we use the hist method that supports native categorical feature handling. This makes XGBoost slower but also more accurate in the older version. -The old version uses different versions of quantile preprocessing for NN methods, while we use the RTDL quantile transform for all methods except RealMLP.Both the new and the old version use early stopping and best-epoch selection on accuracy (for classification) / RMSE (for regression).In the following, we document the benchmark settings for obtaining the results in The datasets are taken from the benchmarks described in Grinsztajn et al. [18] . When a dataset is used both in benchmarks with and without categorical features, we use the version with categorical features. We preprocess the datasets following the same steps as in Grinsztajn et al. [18] :• For neural networks, we quantile-transform the features to have a Gaussian distribution. For TabR [17] , we use the modified quantile transform from the TabR paper. For RealMLP, we use the preprocessing described in Section 3, namely robust scaling and smooth clipping. • For neural networks, we add as a hyperparameter the possibility to normalize the target variable for the model fit and transform it back for evaluation (via scikit-learn's Transformed- [18] , for classification (left) and regression (right), using the closer-to-original settings (Appendix C.5). The plot is similar to the one in the main part of Grinsztajn et al. [18] , with our algorithms added. The y-axis shows the result of the best (on val, but evaluated on test) hyperparameter combination up to n steps of random step (x-axis). As in the original paper, we normalize each score between the max and the 10% quantile (classification) or 50% (regression), and truncate scores below 0 for regression.TargetRegressor and StandardScaler, which differs from the QuantileTransformer from the original paper, as we found it to work better). The same standardization is also applied to all default-parameter versions of neural networks.• For models that do not handle categorical variables natively, we encode categorical features using OneHotEncoder from scikit-learn. • Train size is restricted to 10,000 samples and test and validation size to 50,000 samples.Note that the datasets from the original benchmark are already slightly preprocessed, e.g., heavy-tailed targets are standardized and missing values are removed. More details can be found in the original paper.Results normalization For Figure C.2, as in the original paper, we normalize the R2 or accuracy score for each dataset before averaging them. We use an affine normalization between 0 and 1, 1 corresponding to the score of the best model for each dataset, and 0 corresponding to the score of the worst model (for classification) and the 10th percentile of the scores (for regression). We use slightly different percentiles compared to the original paper as we normalize across the scores of the tuned and default models, and not all steps of the random search, which reduces the number of outliers. Other aggregation metrics are shown in Appendix B.10.We follow the original paper and run neural networks on a GPU and the other models on 1 core of an AMD EPYC 7742 64-Core processor, and we average the time across all random steps (for each random step, the time is averaged across splits). To compute the runtime of neural networks, we restrict ourselves to steps ran on the same GPU model (NVIDIA A100-40GB), which means that we exclude datasets for which we have less than 15 steps of each model on this GPU (leaving us with 11 datasets for classification and 15 for regression). We then compute the average runtime per 1000 samples on each dataset and average them.Other details We rerun classification results for neural networks compared to the original results to early stop on accuracy rather than on cross-entropy, to make results more comparable with the rest of this paper.At https://github.com/LeoGrin/tabular-benchmark/tree/better_by_default, we provide code for the adapted original Grinsztajn et al. [18] benchmark.Here, we specify how our confidence intervals are computed. Let X ij denote the score (error/rank) of a method on dataset i and split j, with i ∈ {1, . . . , n} and j ∈ {1, . . . , m}. Then, the benchmark score S can be written asEQUATIONwhere f = g = id for the arithmetic mean. For the shifted geometric mean, we instead have g = exp and f (x) = log(x + ε), ε = 0.01. We interpret the benchmark datasets as fixed, but the splits as random. For each dataset i, X i1 , . . . , X im are i.i.d. random variables. We first take the dataset averagesZ j := n i=1 w i f (X ij ) .The random variables X 1j , . . . , X nj are independent but not identically distributed. Still, for lack of a better option, we assume that the Z j are normally distributed with unknown mean and variance. We know that the Z j are i.i.d., hence we use the confidence intervals from the Student's t-distribution for normally distributed random variables with unknown mean and variance. This gives us a confidence interval We often compute the error increase in % in the benchmark score of method A compared to method B with the shifted geometric mean, given by 100 • S (A) S (B) -1 . Here, we leverage that the shifted geometric mean uses g = exp to writeS (A) S (B) = g   n i=1 w i m m j=1 (f (X (A) ij ) -f (X (B) ij ))   ,which is of the same form as Eq. ( 1). Hence, we obtain confidence intervals for this quantity using the same method.For our meta-train and meta-test benchmarks, we report training times measured as follows: We run all methods on a single compute node with a 32-core AMD Ryzen Threadripper Pro 3975 WX CPU, using 32 threads for GBDTs and the PyTorch default settings for NNs. No method is run on GPUs. We run methods sequentially on one split on each dataset of the meta-train-class and meta-train-reg benchmarks. For random-search-based HPO methods, we only run one (TabR-HPO, FTT-HPO) or two (other methods) random search steps and extrapolate the runtime to 50 steps. Runtimes for combinations of models (Best and Ensemble) are computed as the sum of the individual runtimes. We compute the runtime per 1000 samples on each dataset and then average them. For simplicity, we do not use the dataset-dependent weighting employed otherwise on the meta-train benchmark.While we did not measure compute resources precisely, our experiments required at least around 3000 hours on RTX 3090 GPUs and other GPUs, as well as roughly 10,000 hours on HPC CPU nodes (32-64 cores).Our implementation uses various libraries, out of which we would like to particularly acknowledge PyTorch [47] , Scikit-learn [48] , Ray [46] , XGBoost [9] , LightGBM [31] , and CatBoost [51] . For using XGBoost, LightGBM, and CatBoost, we adapted wrapping code from the CatBoost quality benchmarks [51] .Here, we provide and compare the results of central methods per dataset. Figures D.1 -D.7 show scatterplot comparisons for different models. Table D .1: Classification error of untuned methods on datasets in B train class , averaged over ten trainvalidation-test splits. When we write a ± b, a is the mean error on the dataset and [ab, a + b] is an approximate 95% confidence interval for the mean in the #splits → ∞ limit. The confidence interval is computed from the t-distribution using a normality assumption as in Appendix C.6. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.RealMLP-TD RealTabR-D TabR-S-D MLP-PLR-D MLP-D CatBoost-TD LGBM-TD XGB-TD RF-D abalone 0.447±0.014 0.445±0.006 0.440±0.010 0.453±0.010 0.448±0.014 0.458±0.009 0.455±0.012 0.451±0.013 0.457±0.010 adult 0.140±0.004 0.134±0.004 0.142±0.004 0.144±0.003 0.144±0.003 0.130±0.003 0.131±0.003 0.131±0.003 0.146±0.003 anuran_calls_families 0.006±0.001 0.006±0.001 0.008±0.002 0.009±0.002 0.009±0.002 0.007±0.002 0.009±0.002 0.008±0.003 0.012±0.003 anuran_calls_genus 0.007±0.002 0.008±0.002 0.008±0.002 0.011±0.002 0.010±0.002 0.008±0.002 0.009±0.002 0.009±0.002 0.012±0.003 anuran_calls_species 0.006±0.001 0.007±0.001 0.010±0.002 0.008±0.002 0.009±0.002 0.007±0.001 0.008±0.002 0.008±0.002 0.010±0.001 avila 0.000±0.000 0.000±0.000 0.001±0.000 0.003±0.001 0.016±0.002 0.001±0.000 0.001±0.001 0.001±0.000 0.011±0.001 bank_marketing 0.089±0.002 0.087±0.003 0.088±0.002 0.089±0.001 0.091±0.001 0.088±0.002 0.090±0.002 0.090±0.002 0.091±0.002 bank_marketing_additional 0.085±0.003 0.084±0.002 0.086±0.003 0.085±0.002 0.086±0.002 0.084±0.003 0.084±0.002 0.085±0.002 0.086±0.002 chess 0.005±0.002 0.013±0.003 0.015±0.003 0.010±0.004 0.008±0.003 0.008±0.003 0.011±0.002 0.005±0.001 0.015±0.003 chess_krvk 0.081±0.004 0.120±0.005 0.128±0.004 0.121±0.006 0.141±0.009 0.153±0.002 0.147±0.003 0.146±0.003 0.293±0.017 crowd_sourced_mapping 0.032±0.004 0.031±0.003 0.028±0.001 0.037±0.003 0.034±0.004 0.035±0.003 0.031±0.003 0.033±0.003 0.058±0.004 default_credit_card 0.179±0.004 0.181±0.004 0.182±0.003 0.179±0.003 0.181±0.004 0.179±0.004 0.178±0.004 0.180±0.004 0.183±0.003 eeg_eye_state 0.016±0.002 0.011±0.001 0.107±0.024 0.176±0.014 0.120±0.017 0.051±0.002 0.052±0.001 0.051±0.002 0.083±0.003 electrical_grid_stability_simulated 0.032±0.004 0.036±0.003 0.048±0.004 0.039±0.004 0.057±0.003 0.051±0.004 0.053±0.004 0.057±0.003 0.086±0.005 facebook_live_sellers_thailand_status 0.134±0.007 0.137±0.005 0.139±0.007 0.137±0.008 0.139±0.007 0.131±0.005 0.135±0.006 0.133±0.008 0.141±0.004 firm_teacher_clave 0.129±0.006 0.128±0.005 0.130±0.005 0.133±0.007 0.134±0.005 0.150±0.006 0.149±0.003 0.149±0.005 0.191±0.006 first_order_theorem_proving 0.179±0.006 0.180±0.007 0.182±0.007 0.188±0.008 0.181±0.006 0.158±0.008 0.160±0.008 0.160±0.006 0.162±0.007 gas_sensor_drift_class 0.005±0.001 0.006±0.001 0.004±0.001 0.006±0.001 0.005±0.001 0.006±0.001 0.006±0.001 0.006±0.001 0.008±0.001 gesture_phase_segmentation_raw 0.087±0.006 0.079±0.009 0.079±0.006 0.108±0.006 0.106±0.007 0.071±0.005 0.067±0.005 0.066±0.004 0.074±0.005 gesture_phase_segmentation_va3 0.323±0.007 0.289±0.008 0.293±0.006 0.347±0.011 0.368±0.008 0.321±0.007 0.312±0.006 0.316±0.009 0.355±0.006 htru2 0.020±0.002 0.019±0.002 0.020±0.001 0.020±0.002 0.020±0.002 0.021±0.002 0.019±0.002 0.020±0.002 0.020±0.002 human_activity_smartphone 0.008±0.001 0.009±0.001 0.010±0.001 0.011±0.002 0.015±0.002 0.008±0.002 0.008±0.002 0.008±0.001 0.024±0.002 indoor_loc_building 0.002±0.000 0.002±0.000 0.002±0.000 0.002±0.000 0.002±0.000 0.002±0.000 0.003±0.001 0.002±0.000 0.002±0.000 indoor_loc_relative 0.070±0.002 0.081±0.005 0.090±0.002 0.084±0.003 0.099±0.004 0.077±0.003 0.059±0.003 0.066±0.007 0.077±0.003 insurance_benchmark 0.061±0.005 0.060±0.004 0.060±0.004 0.060±0.004 0.060±0.004 0.061±0.005 0.059±0.004 0.061±0.004 0.072±0.004 landsat_satimage 0.077±0.006 0.083±0.006 0.094±0.006 0.085±0.007 0.090±0.005 0.087±0.005 0.080±0.004 0.080±0.004 0.090±0.004 letter_recognition 0.019±0.001 0.019±0.001 0.020±0.001 0.039±0.002 0.030±0.002 0.034±0.003 0.032±0.002 0.033±0.002 0.045±0.002 madelon 0.340±0.016 0.401±0.019 0.443±0.011 0.349±0.027 0.435±0.019 0.139±0.010 0.211±0.010 0.210±0.014 0.320±0.011 magic_gamma_telescope 0.115±0.005 0.108±0.004 0.109±0.004 0.119±0.005 0.122±0.003 0.116±0.004 0.118±0.004 0.118±0.003 0.122±0.003 mushroom 0.000±0.000 0.000±0.000 0.000±0.000 0.000±0.000 0.000±0.000 0.000±0.000 0.000±0.000 0.000±0.000 0.000±0.000 musk 0.003±0.002 0.004±0.002 0.005±0.002 0.007±0.001 0.011±0.003 0.011±0.002 0.012±0.002 0.012±0.002 0.028±0.002 nomao 0.022±0.002 0.020±0.001 0.022±0.001 0.024±0.001 0.026±0.002 0.018±0.001 0.017±0.001 0.017±0.002 0.020±0.002 nursery 0.020±0.001 0.012±0.003 0.011±0.002 0.022±0.002 0.024±0.002 0.021±0.001 0.026±0.002 0.024±0.002 0.034±0.001 occupancy_detection 0.006±0.001 0.007±0.001 0.008±0.001 0.009±0.001 0.008±0.001 0.007±0.001 0.007±0.001 0.007±0.000 0.006±0.001 online_shoppers_attention 0.098±0.004 0.101±0.004 0.098±0.004 0.095±0.003 0.099±0.004 0.097±0.003 0.096±0.005 0.097±0.004 0.096±0.003 optical_recognition_handwritten_digits 0.011±0.003 0.012±0.002 0.015±0.003 0.020±0.003 0.018±0.003 0.017±0.003 0.015±0.003 0.016±0.003 0.020±0.004 ozone_level_1hr 0.036±0.007 0.037±0.008 0.035±0.008 0.035±0.008 0.035±0.008 0.036±0.009 0.035±0.007 0.035±0.007 0.035±0.007 ozone_level_8hr 0.071±0.010 0.070±0.010 0.072±0.012 0.067±0.011 0.072±0.010 0.071±0.011 0.070±0.012 0.067±0.011 0.070±0.009 page_blocks 0.028±0.004 0.026±0.005 0.027±0.004 0.026±0.004 0.028±0.004 0.025±0.003 0.025±0.003 0.025±0.004 0.026±0.003 pen_recognition_handwritten_characters 0.004±0.001 0.004±0.001 0.007±0.001 0.007±0.002 0.008±0.002 0.007±0.001 0.007±0.001 0.006±0.001 0.011±0.001 phishing 0.031±0.002 0.029±0.002 0.029±0.003 0.032±0.003 0.034±0.002 0.031±0.002 0.032±0.002 0.031±0.002 0.032±0.003 polish_companies_bankruptcy_1year 0.018±0.003 0.019±0.001 0.026±0.003 0.026±0.003 0.028±0.002 0.021±0.002 0.022±0.003 0.021±0.002 0.027±0.003 polish_companies_bankruptcy_2year 0.017±0.002 0.017±0.001 0.041±0.002 0.041±0.002 0.041±0.002 0.025±0.003 0.025±0.003 0.026±0.003 0.035±0.002 polish_companies_bankruptcy_3year 0.023±0.002 0.025±0.002 0.041±0.004 0.038±0.002 0.046±0.003 0.031±0.003 0.033±0.003 0.033±0.003 0.040±0.003 polish_companies_bankruptcy_4year 0.029±0.003 0.030±0.002 0.051±0.002 0.048±0.004 0.053±0.002 0.031±0.002 0.036±0.002 0.035±0.002 0.048±0.002 polish_companies_bankruptcy_5year 0.037±0.002 0.040±0.002 0.064±0.004 0.056±0.004 0.066±0.002 0.031±0.004 0.033±0.003 0.036±0.004 0.050±0.005 seismic_bumps 0.068±0.009 0.065±0.009 0.066±0.009 0.065±0.009 0.065±0.008 0.070±0.007 0.066±0.009 0.067±0.007 0.068±0.008 skill_craft 0.589±0.010 0.582±0.014 0.601±0.012 0.574±0.019 0.597±0.015 0.600±0.014 0.607±0.013 0.610±0.014 0.593±0.014 smartphone_human_activity 0.045±0.004 0.035±0.007 0.042±0.004 0.064±0.006 0.071±0.004 0.035±0.003 0.035±0.005 0.036±0.004 0.079±0.004 smartphone_human_activity_postural 0.009±0.002 0.008±0.002 0.010±0.001 0.011±0.002 0.013±0.002 0.006±0.001 0.007±0.001 0.007±0.001 0.025±0.002 spambase 0.052±0.008 0.059±0.005 0.052±0.007 0.051±0.006 0.054±0.006 0.048±0.008 0.049±0.008 0.048±0.008 0.052±0.008 superconductivity_class 0.058±0.003 0.063±0.002 0.059±0.002 0.067±0.002 0.063±0.003 0.057±0.003 0.058±0.003 0.058±0.003 0.059±0.002 thyroid_all_bp 0.026±0.004 0.027±0.003 0.028±0.003 0.027±0.006 0.029±0.005 0.024±0.004 0.025±0.003 0.022±0.003 0.027±0.003 thyroid_all_hyper 0.015±0.002 0.016±0.003 0.017±0.003 0.014±0.002 0.018±0.003 0.014±0.003 0.014±0.003 0.014±0.003 0.015±0.003 thyroid_all_hypo 0.008±0.002 0.012±0.002 0.018±0.004 0.013±0.002 0.020±0.002 0.003±0.002 0.005±0.002 0.005±0.002 0.007±0.002 thyroid_all_rep 0.009±0.002 0.009±0.003 0.009±0.002 0.013±0.002 0.016±0.005 0.005±0.002 0.006±0.002 0.005±0.002 0.009±0.002 thyroid_ann 0.008±0.002 0.009±0.002 0.012±0.004 0.006±0.002 0.010±0.002 0.003±0.001 0.003±0.001 0.005±0.002 0.003±0.001 thyroid_dis 0.013±0.002 0.013±0.002 0.014±0.002 0.015±0.003 0.016±0.003 0.010±0.002 0.011±0.002 0.010±0.003 0.013±0.002 thyroid_hypo 0.014±0.003 0.014±0.003 0.014±0.003 0.014±0.005 0.016±0.003 0.009±0.004 0.010±0.002 0.012±0.003 0.012±0.003 thyroid_sick 0.016±0.003 0.020±0.003 0.022±0.004 0.019±0.004 0.029±0.003 0.011±0.003 0.018±0.004 0.011±0.003 0.017±0.005 thyroid_sick_eu 0.000±0.000 0.000±0.001 0.001±0.001 0.000±0.000 0.000±0.000 0.000±0.000 0.000±0.000 0.000±0.000 0.000±0.000 turkiye_student_evaluation 0.016±0.002 0.016±0.002 0.017±0.003 0.016±0.002 0.033±0.005 0.016±0.002 0.016±0.002 0.017±0.003 0.113±0.005 wall_follow_robot_2 0.002±0.001 0.008±0.003 0.009±0.002 0.004±0.001 0.008±0.002 0.002±0.001 0.004±0.002 0.002±0.001 0.001±0.001 wall_follow_robot_24 0.029±0.004 0.042±0.003 0.040±0.007 0.016±0.004 0.042±0.005 0.002±0.001 0.005±0.002 0.004±0.001 0.007±0.001 wall_follow_robot_4 0.002±0.001 0.009±0.003 0.012±0.003 0.007±0.002 0.011±0.003 0.002±0.001 0.004±0.002 0.003±0.001 0.001±0. , averaged over ten trainvalidation-test splits. When we write a ± b, a is the mean error on the dataset and [ab, a + b] is an approximate 95% confidence interval for the mean in the #splits → ∞ limit. The confidence interval is computed from the t-distribution using a normality assumption as in Appendix C.6. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined. , averaged over ten train-validation-test splits. When we write a ± b, a is the mean error on the dataset and [ab, a + b] is an approximate 95% confidence interval for the mean in the #splits → ∞ limit. The confidence interval is computed from the t-distribution using a normality assumption as in Appendix C. 6 . In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined. , averaged over ten train-validation-test splits. When we write a ± b, a is the mean error on the dataset and [ab, a + b] is an approximate 95% confidence interval for the mean in the #splits → ∞ limit. The confidence interval is computed from the t-distribution using a normality assumption as in Appendix C.6. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined. , averaged over ten trainvalidation-test splits. When we write a ± b, a is the mean error on the dataset and [ab, a + b] is an approximate 95% confidence interval for the mean in the #splits → ∞ limit. The confidence interval is computed from the t-distribution using a normality assumption as in Appendix C.6. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined. Table D.6: Classification error of tuned methods on datasets in B test class , averaged over ten trainvalidation-test splits. When we write a ± b, a is the mean error on the dataset and [ab, a + b] is an approximate 95% confidence interval for the mean in the #splits → ∞ limit. The confidence interval is computed from the t-distribution using a normality assumption as in Appendix C.6. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined. , averaged over ten train-validation-test splits. When we write a ± b, a is the mean error on the dataset and [ab, a + b] is an approximate 95% confidence interval for the mean in the #splits → ∞ limit. The confidence interval is computed from the t-distribution using a normality assumption as in Appendix C.6. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined. , averaged over ten train-validation-test splits. When we write a ± b, a is the mean error on the dataset and [ab, a + b] is an approximate 95% confidence interval for the mean in the #splits → ∞ limit. The confidence interval is computed from the t-distribution using a normality assumption as in Appendix C.6. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined. , averaged over ten train-validation-test splits. When we write a ± b, a is the mean error on the dataset and [ab, a + b] is an approximate 95% confidence interval for the mean in the #splits → ∞ limit. The confidence interval is computed from the t-distribution using a normality assumption as in Appendix C. 6 . In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined. , averaged over ten train-validationtest splits. When we write a ± b, a is the mean error on the dataset and [ab, a + b] is an approximate 95% confidence interval for the mean in the #splits → ∞ limit. The confidence interval is computed from the t-distribution using a normality assumption as in Appendix C.6. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguardsinspired by a similar schedule in https://github.com/lessw2020/ Ranger-Deep-Learning-OptimizerWe noticed later that the ozone_level_1hr and ozone_level_8hr datasets contain less than 2,500 samples, but we decided to keep them since we already used them for tuning the hyperparameters. 


Decision: While the work is not entirely novel, there are extensive benchmarks, and the methods provided are easily applicable for practitioners.
The main concern of the reviewers seems to be the breadth of the comparison. However, the reviewers do provide a wide array of benchmarks and baseline methods.

Reviews: review: paper_summary: paper_summary: The paper proposes an enhanced version of the tabular MLP model -- RealMLP. By using multiple tricks over simple MLP, the proposed method becomes much more competitive with GBDTs than a simple MLP. Moreover, the authors provide strong "tuned default" configurations for GBDTs and RealMLP. These configs considerably reduce training time in exchange for arguably small test error increase compared to hyperparameter optimization procedures.
strength_weakness: strength_weakness: 1. One of the main contributions of the paper is the different DL techniques from Figure 1 that have been added to MLP. Although there are enough useful experiments and ablations on these techniques for RealMLP (Figure 1 and Appendix B), these tricks can be applied to any DL model. From Figure 3 we observe that TabR-S-D and FT-Transformer-D are much better than MLP-D and slightly worse than RealMLP-TD. So, I think it is essential to ablate the mentioned techniques to improve these models and compare the results with RealMLP. I expect that the techniques improve stronger models much less compared to MLP, and it would be a beneficial result. It will show that RealMLP-TD is a good baseline while TabR-S-D-with-tricks is only slightly better than TabR-S-D. Otherwise, if improved TabR or FT-T considerably outperform RealMLP, the results will be even closer to GBDTs' which is also a positive outcome. I think that the results in Table B.5, which show an advantage of RC+SF preprocessing for TabR-S-D , also support my idea. 
2. Table B.1. from Appendix: changing Adam's $\beta_2$ from 0.95 to 0.999 significantly increases the error rate while changing Adam's $\beta_2$ from 0.999 to 0.95 on Figure 1 has almost no effect. I would make a conclusion that the resulting RealMLP-TD architecture with all tricks is very sensitive to hyperparameters. 
3. Also Table B.1: Strangely, PLR embedding is much worse than PL one. The only difference is ReLU if I am not mistaken.
4. No non-parametric DL baselines in Figure 2. Only TabR-S-D is presented. Also, it would be interesting to compare tuned MLP-PLR [1]. Authors from [1] use extensive HPO and do not fix embedding size, so a comparison with RealMLP-TD would be interesting to see if a properly tuned MLP-PLR is better/worse than RealMLP with tricks and fixed embedding size.
questions: questions: 1. Please, see weaknesses. 
2. Probably I missed it in the text but how many HPO iterations have you used to obtain -TD configs?
3. Have you measured inference efficiency? I think MLP is usually a bit faster than GBDTs but I wonder if parametric activations or proposed numerical embeddings make MLP slower.
limitations: limitations: Limitations are addressed.
paper_summary: paper_summary: The paper introduces RealMLP, an enhanced Multilayer Perceptron (MLP) designed for classification and regression tasks on tabular data. It also proposes optimized default parameters for both RealMLP and Gradient Boosted Decision Trees (GBDTs). Through benchmarking on diverse datasets, the authors demonstrate that RealMLP achieves a superior balance between time efficiency and accuracy compared to other neural networks, while remaining competitive with GBDTs. The integration of RealMLP and GBDTs using optimized defaults shows promising performance on medium-sized tabular datasets, alleviating the need for extensive hyperparameter tuning.
strength_weakness: strength_weakness: 1. The finding presented in this work is not entirely novel. Similar conclusions have been reached by Tree-hybrid MLPs `[1]`, although the methods are not quite the same. Moreover, ensemble different methods, including combinations like MLP + GBDTs, have long been recognized as effective approaches in Kaggle competitions and do not necessitate rediscovery.

2. While mentioning good performance by default, it's worth noting that Excelformer achieves one of the best performances under default settings. However, a direct comparison was not included, which would enhance the clarity of the findings. Please include the comparison in the rebuttal.

3. Despite emphasizing the applicability to medium-sized datasets, practical applications often involve diverse tabular data with varying feature lengths and training data sizes. The small application scope could potentially limit the impact of this study. Besides, most of tabular data sizes are often small.

4. The compared methods are evasive. Since the authors want to claim the speed-performance balance, the effective approaches Net-DNF `[2]`, TabNet `[3]`, and TabCaps `[4]` are not compared. The compared models FT-Transformer and SAINT, especially SAINT, are heavy.

**REF**

`[1]` Team up GBDTs and DNNs: Advancing Efficient and Effective Tabular Prediction with Tree-hybrid MLPs

`[2]` Net-DNF: Effective Deep Modeling of Tabular Data

`[3]` Tabnet: Attentive interpretable tabular learning

`[4]` Tabcaps: A capsule neural network for tabular data classification with bow routing
questions: questions: The effective default performance achieved by MLP + GBDTs may not be suitable for all scenarios, such as those requiring tabular pre-training or zero-shot scenarios, where pure tabular models might be more appropriate. How does the proposed method address applications in tabular pre-training scenarios?
limitations: limitations: NA
paper_summary: paper_summary: The work investigates/provides better default hyperparameter configurations for MLPs and gradient-boosted decision trees. Additionally, it proposes an augmented neural network with a series of enhancements. Experimental results are provided showing that the neural network is on par with gradient-boosted decision trees. Moreover, the authors argue that the neural network and the gradient-boosted decision trees with the new defaults are competitive with their HPO-tuned per-dataset counterparts and motivate practitioners to use the different methods with the new defaults and perform ensembling.
strength_weakness: strength_weakness: - The novelty of the paper is limited in my opinion. The majority of the components that are proposed in Figure 1. C are not novel and are used in AutoML methods. Moreover, it is not clear what subset of the proposed components could actually benefit the other methods and not only the MLP. 
- The baselines used are not consistent, for example, FT-Transformer and SAINT are included for the Grinsztajn et al. benchmark, but not for the other benchmarks.
- The protocol is not fully consistent, as for example, for the results of the meta-train and meta-test benchmark, HPO is not applied for the ResNet method or Tab-R.
- Figure 2 shows results for accuracy, where RealMLP-TD strongly outperforms gradient-boosted decision tree methods for the meta-test classification and regression benchmark. However, AUROC is a metric that considers class imbalance and provides a better view of the method performances, Figure B.3 which reports results for AUROC shows a different picture, where, gradient-boosted decision trees outperform the RealMLP-TD. As such, I would propose to place Figure B.3 in the main paper and Figure 2 in the Appendix.
questions: questions: - Could the authors provide results for the missing baselines and for the baselines that do not feature HPO results?
- How does this work fair in comparison to using a portfolio of configurations using dataset meta-features and selecting the default configuration per dataset based on a similarity distance between the portfolio of configurations and the context dataset?
limitations: limitations: - I believe the authors have adequately addressed the limitations.
paper_summary: paper_summary: In the paper "Better by default: Strong pre-tuned MLPs and boosted trees on tabular data", the authors make two major contributions: (i) they propose a multi-layer perceptron configurations that is tuned on a set of training datasets and (ii) investigate in a large scale empirical study how different default parameterizations (library vs tuned on a set of training datasets) compares to optimized hyperparameters on a set of test datasets. The study reveals interesting results regarding the performance of default parameterizations set in standard ML libraries, better default parameterizations exist (at least for the scope of the benchmark), and that the proposed MLP often performs competitive to gradient-boosted decision trees when it comes to a Pareto-optimal tradeoff between accuracy and training time.
strength_weakness: strength_weakness: - Intuitively one would expect that the performance of TD lies in between D and HPO but I could also imagine that TD is oftentimes very close to HPO or might even exceed HPO's performance due to overtuning. But this comparison is not considered as an overview.
- It is not clear why random search has been used as HPO technique in the paper, especially, since the authors even mention more sophisticated techniques such as SMAC that are based on Bayesian optimization.
- The style of writing in Section 3 could be improved as there are various content-wise enumerations in the form of sentences starting with "We".
questions: questions: - How would a CASH approach compare here if we would first select the learning algorithm according to the (tuned) default parameterization and then optimize its hyperparameters? As it is done in Mohr, F., Wever, M. Naive automated machine learning. Mach Learn 112, 1131–1170 (2023). https://doi.org/10.1007/s10994-022-06200-0
- Why was random search used for hyperparameter optimization?
- Is there any intuition why using the last of tied epochs as the final model as stated on line 161?
- Why are hyperparameter values rounded as stated on line 170?
limitations: limitations: Limitations have properly been described in the paper.
paper_summary: paper_summary: The authors introduce RealMLP, an improved multilayer perceptron (MLP), alongside improved default parameters for GBDTs and RealMLP. The authors tune RealMLP on a meta-train benchmark with 71 classification and 47 regression datasets and compare them to hyperparameter-optimized versions on a disjoint meta-test benchmark with 48 classification and 42 regression datasets, as 
well as the GBDT-friendly benchmark by Grinsztajn et al. (2022), claiming that RealMLP offers a better time-accuracy tradeoff than other neural nets and is competitive with GBDTs.
strength_weakness: strength_weakness: MAJOR

1. The support for a key claim ("RealMLP offers a better time-accuracy tradeoff than other neural nets and is competitive with GBDTs") is, at best, inconsistent. In Fig. 2, utilizing the authors' own benchmark suite, RealMLP is consistently less performant than the best GBDT. The claim that it outperforms other neural nets is not strongly supported by Fig. 2, since only a handful of the baseline NN methods from [1] are included. Particularly notable is the exclusion of [2], despite the fact that the authors reference it in their prior work. [2] requires less than a second to train and there exist methods in the literature to extend it to datasets of arbitrary size. As for Fig. 3, it seemingly contains no confidence intervals. However, the authors' method appears to perform comparably to FT-Transformer and Tab-R for classification tasks; on regression tasks, it is faster, but less performant, than FT-Transformer.
2. Sec 2.1 explains how the new benchmark was assembled, but not why it was necessary to do so. [1], [2] and Grinsztajn et al. have already introduced robust tabular benchmark suites into the literature. The authors also mention, but utilize only parts of, the AutoML benchmark and the OpenML-CTR23 regression benchmark.
3. While the concerns raised in Sec. 2.2 about some previously used metrics are reasonable, the choice of shifted geometric mean error, which adds noise to extremely low error scores, is also problematic. A better approach would have been to report multiple metrics.

MINOR

1. The 95% confidence intervals in the figures are very difficult to see, and should be made clearer.

[1] When do neural nets outperform boosted trees on tabular data?
[2] TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
questions: questions: QUESTIONS

* Why was it necessary to create a new benchmark suite? Why is this the fairest choice of benchmark for these methods?
* Why report only one non-standard metric?
* Why are so few deep tabular methods included in the core experiments?

SUMMARY

While the detailed description of the authors method is helpful and the ideas contained therein could be of interest to the NeurIPS audience, the experimental results are not yet robust enough to warrant publication at NeurIPS.
limitations: limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.

Comments: title: Response to Authors
Author Rebuttal Response
Response to Authors
Response to Rebuttal
comment: Thank you for your answer and your suggestions!

> Regarding the use of random search: I am well aware of the benefits of random search in empirical studies, but it should be mentioned in the paper to justify the choice and explain the benefits to the reader.

We will discuss this choice in more detail in the revised manuscript.

>Regarding the overview plot, I was rather thinking of some spider chart where the axes are given by the learner. TD, D, and HPO are then data rows for that kind of chart so that one can see the area spanned by the respective types for each base learner or something in that direction. Does that make sense?

Thank you for this insightful suggestion, we will experiment with spider charts and add it to the revised version if it gives a clear picture of the situation. We are not sure the D versions of different algorithms are directly comparable, but for TD and HPO at least it should be interesting.
We thank the reviewer for the response and the score increase. 

We now ran TabR-S-D with the above modifications (except label smoothing) on our meta-train regression benchmark as well (took ~5 GPU-hours) and see the exact same picture as for classification. We are wondering whether we should try a version of TabR-HPO in Figure 3 that includes some of our tricks into the search space. If this helps, then of course it is unclear which tricks help and what this means for defaults.

Regarding the inconsistent baseline usage, most other papers evaluate their methods on a single benchmark. Therefore, we would argue that the fact that we have an additional very diverse meta-test benchmark, even though we cannot run all methods due to large and high-dimensional datasets, should not be seen as a weakness but a strength of our paper.
I thank the authors for the response.

- Regarding: **"The label smoothing is highlighted as “unusual”, and we would argue that it is indeed unusual in the tabular context. We tried TabR-S-D with our preprocessing + beta_2=0.95 + scaling layer + parametric Mish + label smoothing on meta-train-class and it was slightly better than with just the preprocessing, which is in turn better than with the original preprocessing (as we showed in the Appendix). At this point the message of this investigation is unclear and would require more ablations, tuning, etc., opening a big rabbit hole."**

   I am not sure about the practitioners, but personally, I have used label smoothing in my experiments. Nevertheless, this is a paper for the community and I would need to review the other papers in the literature to confirm this very minor point. A recipe from the authors could be beneficial for the community. I agree that it is more complicated in the scenario the authors are pointing out for other baselines. One could maybe frame it as a general recipe or set of defaults for deep learning methods but it is not trivial and I am not sure if that is the message the authors want to convey.

- Regarding: **"TabR-S-D was run without numerical embeddings, since this is the version that was proposed in the paper for default hyperparameters. While this might deteriorate performance, it also improves the training time, which we also report. The new TabR-HPO results accidentally didn’t use numerical embeddings, we will rerun them."**

   I understand, in that case, I would agree with the authors on using TabR-S-D for the default case and the one with numerical embeddings for the HPO case.

In my perspective, the work is in a good position, it needs a bit more work regarding the inconsistent baseline usage, but overall it has extensive results. Based on this, I will increase my score from 3 to 4.
We thank the reviewer for acknowledging our response, but are surprised to see no change of score given that we addressed most concerns:
- Weakness 3 was incorrect.
- Weakness 2 is unjustified in our opinion since we do include a standard benchmark and additional benchmarks *on top of this* with well-defined dataset inclusion criteria. Again, we used all datasets from the AutoML and CTR23 benchmarks except those that were excluded for a good reason based on exclusion criteria defined in Appendix C.3.2.
- RealMLP performed better in terms of benchmark scores than claimed by the reviewer in Weakness 1.
- We did add more baselines in the rebuttal, and we already have most of the well-performing baselines from [1] in Figure 3 except TabPFN, and in addition to [1] we have TabR and MLP-PLR which we believe to be more relevant since we are improving standard NNs and not PFNs, which are currently hard to compare against (How do you fairly compare to a method that doesn’t use a validation set? What do you do about limitations in #features/#classes? Which of the recent methods do you use to scale it to large datasets?)
We thank the reviewer for the detailed response. 

- The label smoothing is highlighted as “unusual”, and we would argue that it is indeed unusual in the tabular context. We tried TabR-S-D with our preprocessing + beta_2=0.95 + scaling layer + parametric Mish + label smoothing on meta-train-class and it was slightly better than with just the preprocessing, which is in turn better than with the original preprocessing (as we showed in the Appendix). At this point the message of this investigation is unclear and would require more ablations, tuning, etc., opening a big rabbit hole.
- We understand that some readers might be concerned by the non-matching baselines, and we will explicitly mention the reasons for this in the next version of the paper. For baselines that fail on some datasets, we have the additional problem that not all aggregation metrics support missing results.
- In case of acceptance, we will try to use the extra page to add Figure B.3 or B.4, which contain the AUC results, to the main paper.
- TabR-S-D was run without numerical embeddings, since this is the version that was proposed in the paper for default hyperparameters. While this might deteriorate performance, it also improves the training time, which we also report. The new TabR-HPO results accidentally didn’t use numerical embeddings, we will rerun them.
We thank the reviewer for taking the time to consider our response.

Regarding the fairness of our comparison between our MLP and other neural networks, we do not strive to be fair between different architectures, and state this in the paragraph on limitations. This paper is not about comparing architectures, but rather offering a good model, and measuring the importance of meta-learning good default hyperparameters.

We tried some of our tricks on top of TabR-S-D on our meta-train-class benchmark. The result is slightly better than the TabR-S-D version with only our preprocessing, which is already better than TabR-S-D. We are not sure how to turn this into a fair comparison with a reasonable amount of effort.
I would like to thank the authors for the reply. I have carefully read the other reviews and all the responses from the authors.

- Regarding: **"What is the source for your claim? Which AutoML tool would these be in? Are you referring to some of the color-coded improvements or only to the gray ones? There are some non-novel components in Figure 1 (c) since we wanted to start from a “vanilla MLP”, but as we color-coded in the figure, there are many new and unusual components. We also tried out the two NNs in AutoGluon and their default settings performed worse than MLP-D on the meta-train benchmark. Regarding your second point, please see the global response."**

   The gray components would be an example, but additionally, the highlighted label smoothing is not something surprising. As mentioned by another reviewer, the paper provides a set of results that give intuition on the different method performances. As such, it would be beneficial for techniques that can be applied to the MLP to be added to the other baselines. It might be that the components hurt performance, it might be that the components improve performance. 

   One simple test would be to reuse the defaults for the MLP for another deep baseline (for what component can be applied), while this might not be conclusive as the hyperparameters probably need to be tuned per method, it would provide some insights.

- Regarding: **"While we understand the desire to have every method on every benchmark, unfortunately, the meta-test benchmark is much more expensive to run than the other two benchmarks since it contains larger and higher-dimensional datasets. For example, FT-Transformer runs into out-of-memory issues on some datasets, trying to allocate, e.g., 185 GB of RAM. We did not try SAINT since it’s even more expensive."**

   In my perspective, the already considered list of methods is extensive. For example, running TabNet from my perspective is not interesting, the method is outperformed significantly from many methods. However, the already included baselines in the comparisons should be present in all results, since they leave "holes" in the results.

   One might worry that some methods might achieve more competitive performance and that is the reason why the authors do not include them. I personally do not think the authors are hiding results, however, the concerns should be addressed, especially considering the work lies towards the benchmarking and experimental side.

   If a method fails on certain datasets, the authors can point it out in the work.

- Regarding: **"We included the AUROC experiments after the main experiments were finished to include an imbalance-sensitive metric. However, since the whole meta-learning and model development process was performed with accuracy as the target metric, and the best-epoch selection was also performed with accuracy as the target metric, we think that the results in Figure 2 are more suited for the main paper as they are more appropriate to demonstrate the effect of meta-learning default parameters. We do refer to the AUROC experiments in the main paper, though, and will try to emphasize these experiments more in the next version."**

   While that is true, the main attention is invested in the main manuscript, as such, the results can be potentially misleading. AUROC is a better metric that considers class imbalance. Personally, I find the work interesting, whether it beats CatBoost or not.

- As a last question, was TabR run with embeddings or without? My understanding is the latter. If that is the case, it is questionable, since the main version of TabR contains embeddings. Additionally, the MLP is run with embeddings.
I carefully read the rebuttal and would like to thank the authors! Also, I apologise for the late reply.

1. In my opinion, the idea of making MLP as strong as possible is really valuable for the field but the lack of "fair" comparison is a big weakness of the paper. ("fair comparison" = other models with the proposed techniques). Though I understand the high cost of the tuning, I think even results for TabR-S-D+some_tricks would be beneficial for the paper.
2. Thanks for the answer. It should be added to the main text since stability to hyperparameters is important.
3. I think this "issue" could also be related to optimization stability (or sensitivity to hyperparameters).
4. Sorry, I made a typo, TabR is the non-parametric baseline. I meant that the paper lacks a strong parametric baseline since FT-T is quite expensive, and ResNet is just bad. MLP-PLR is a good baseline thanks for the provided results.
5. Inference time seems reasonable, thanks for the results.

Overall, in my opinion, the main weakness (the first one) remains and, thus, I keep my score unchanged.
We thank the reviewer for the insights on various NN models. Does the reviewer have insights on the performance of Net-DNF vs. more modern / well-studied methods like MLP/ResNet from RTDL that are used in many more recent benchmarks?

Regarding the main point “a paper should aim to present new scientific findings”:
- The general benefits of combining GBDTs and NNs are of course widely known both on Kaggle as well as established scientifically through AutoML tools as well as Shwartz-Ziv et al. (2022). However, we specifically study whether (1) ensembling is preferable to HPO under time constraints and whether (2) meta-learning defaults shifts this balance. While some Kagglers may have intuitions about this, we doubt that there would be a consensus, for example, by looking at the posts of Kaggle Grandmaster Bojan Tunguz (“XGBoost is all you need”). We invite the reviewer to provide concrete evidence to the contrary if they disagree.
- Even though practitioners and Kaggle experts have some important intuitive knowledge not found in the literature, we believe that verifying this knowledge on large scale benchmarks and making it accessible to the scientific community is important. 
- Even if the reviewer does not find our results on this aspect particularly interesting, there are other contributions of our work: we also provide an easily accessible improved MLP which performs very well both with default HP and after HPO, find better hyperparameters for standard GBDT libraries, and provide numerous insights in our ablations.
Thank you very much for the on-point responses and the clarification.

Regarding the use of random search: I am well aware of the benefits of random search in empirical studies, but it should be mentioned in the paper to justify the choice and explain the benefits to the reader.

Regarding the overview plot, I was rather thinking of some spider chart where the axes are given by the learner. TD, D, and HPO are then data rows for that kind of chart so that one can see the area spanned by the respective types for each base learner or something in that direction. Does that make sense?
I would like to thank the authors for their response; I will not change my score at this time.
Thank you for your reply.

I’m aware that [1] was uploaded on July 13th, 2024, and that it is completely different from your paper. As I mentioned in my initial comment, it was only an example to illustrate that this paper revisits an old conclusion. I’m not questioning the novelty of your work based on this example.

Regarding the performance of Excelformer, I believe you should use the official code. I’ve tried using the code from PYG, but it seems to have many bugs, and the hyperparameters were clearly misused.

Additionally, in my experience, Net-DNF often outperforms both TabNet and NODE. 

But, the performance of previous works and the similarity to [1] are not the main points! We all know that a paper should aim to present new scientific findings.

You said: "we try to see how meta-learning better defaults affects the trade-off between ensembling / algorithm selection and HPO". Essentially, integrating different methods (such as MLP + GBDT) has long been recognized as an effective approach in Kaggle tabular prediction competitions. Besides, achieving good results in these competitions typically does not require extremely rigorous HPO (that may leads to over-fitting), which are well understood by tabular data prediction experts.
We just noticed that our pdf uses an older version of the meta-train/meta-test plots where ResNet-HPO is not included. The results for ResNet-HPO are similar to the Grinsztajn et al. benchmark: a bit better than MLP-HPO for classification, a bit worse for regression, and about twice as slow.