Well-tuned Simple Nets Excel on Tabular Datasets
Authors: Arlind Kadra
Marius Lindauer
Frank Hutter
Josif Grabocka
Venue: NeurIPS 2021 Poster (2021)
Keywords: tabular datasets
neural networks
multilayer perceptrons
deep learning
regularization cocktails
combined algorithm selection and hyperparameter optimization.

Abstract: Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures. In this paper, we hypothesize that the key to boosting the performance of neural networks lies in rethinking the joint and simultaneous application of a large set of modern regularization techniques. As a result, we propose regularizing plain Multilayer Perceptron (MLP) networks by searching for the optimal combination/cocktail of 13 regularization techniques for each dataset using a joint optimization over the decision on which regularizers to apply and their subsidiary hyperparameters. We empirically assess the impact of these regularization cocktails for MLPs in a large-scale empirical study comprising 40 tabular datasets and demonstrate that (i) well-regularized plain MLPs significantly outperform recent state-of-the-art specialized neural network architectures, and (ii) they even outperform strong traditional ML methods, such as XGBoost.

Well-tuned Simple Nets Excel on Tabular Datasets
Arlind Kadra
Marius Lindauer
Department of Computer Science
Institute for Information Processing
University of Freiburg
Leibniz University Hannover lindauerStnt. uni hannovernrv..... de
kadraaßes uni-freiburg
Frank Hutter
Josif Grabocka
Department of Computer Science University of Freiburg & Bosch Center for Artificial Intelligence fhêcs. uni umi-freiburg.de
Department of Computer Science
University Freiburg
grabocka@informatikk uni mmii-fribuugg.cm
Abstract
Tabular datasets the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures. In this paper, we hypothesize that the key to boosting the performance of neural networks lies in rethinking the joint and simultaneous application of large set of modern regularization techniques. As a result, we propose regularizing plain Multilayer Perceptron (MIP) networks by searching for the optimal combination/cocktail of 13 regularization techniques for each dataset using joint optimization over the decision on which regularizers apply and their subsidiary hyperparameters.
We empirically assess the impact of these regularization cocktails for MLPs in a large-scale empirical study comprising 40 tabular datasets and demonstrate that (i) well-regularized plain MLPs significantly outperform recent state-of-the-art specialized neural network architectures, and (ii) they even outperform strong traditional ML methods, such as XGBoost.
Introduction
In contrast to the mainstream in deep learning (DL), in this paper, we focus on tabular data, domain that we feel is understudied in DL. Nevertheless, is of great relevance for many practical applications, such as climate science, medicine, manufacturing, finance, recommender systems, etc. During the last decade, traditional machine learning methods, such Trees (GBDT) [5], dominated tabular data applications due to their superior performance, and the success story DI. has had for raw data (e-g., images, speech, and text) stopped short of tabular data.
Cradient-Boosted Boosted Decision
Even in recent years, the existing literature still gives mixed messages on the state-of-the-ee status of deep learning for tabular data. While some recent neural network methods 46 claim outperform GBDT, others confirm that GBDT are still the most accurate method on tabular data [48] (2)). The extensive experiments 40 datasets report indeed confirm that recent neural networks 46 do not outperform GBDT when the hyperparameters of all methods are thoroughly tuned.
We hypothesize that the key improving the performance of neural networks tabular data lies in exploiting the recent DL advances on regularization techniques (reviewed in Section 3), such as data augmentation, decoupled weight decay, residual blocks and model averaging (e.g., dropout snapshot ensembles), or on learning dynamics (e.g., look-ahead optimizer or stochastic weight averaging).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Indeed, we find that even plain Multilayer Perceptrons (MLPs) achieve state-of-the thrtt-r-tt-atttt- results when regularized by multiple modern regularization techniques applied jointly and simultaneously.
Applying multiple regularizers jointly is already a common standard for practitioners, who routinely mix regularization techniques (e.g. Dropout with early stopping and weight decay). However, the deeper question of "Which subset of regularizers gives the largest generalization performance on particular dataset among dozens of available methods?" remains unanswered, currently combine regularizers via inefficient trial-and-error procedures. In this paper, we provide simple, yet principled answer to that question, by posing the selection of the optimal subset of regularization techniques and their inherent hyperparameters, as a joint search for the best combination of MLP regularizers for each dataset among a pool of 13 modern regularization techniques and their subsidiary hyperparameters (Section 4.
practitioners
From empirical perspective, this paper is the first to provide compelling evidence that well- regularized neural networks (even simple MLPs!) indeed surpass the current state-of-the-at models in tabular datasets, including recent neural network architectures and GBDT (Section 6). In fact, the performance improvements are quite pronounced and highly significant | potentially have far-reaching implications, and to open up a garden of delights of new applications on tabular datasets for DL.
We believe this finding to
Our contributions are as follows:
We demonstrate that modern DL regularizers (developed for DL applications on raw data, such as images, speech, or text) also substantially improve the performance of deep multi- layer perceptrons on tabular data.
2. We propose simple, yet principled, paradigm for selecting the optimal subset of regulariza- tion techniques and their subsidiary hyperparameters (so-called regularization cocktails).
3. We demonstrate that these regularization cocktails enable even simple MLPs to outperform both recent neural network architectures, as well as traditional strong MI. methods, such as GBDT, on tabular data. Specifically, we are the first to show neural networks to significantly (and substantially) outperform XGBoost in a fair, large-scale experimental study.
2 Related Work on Deep Learning for Tabular Data
Recently, various neural architectures have been proposed for improving the performance of neural networks on tabular data. TabNet [ introduced a sequential attention mechanism for capturing salient features. Neural oblivious decision ensembles (NODE (46)) blend the concept of hierarchical decisions into neural networks. Self-normalizing neural networks [29] have neuron activations that converge to zero mean and unit variance, which in turn, induces strong regularization and allows for high-level representations. Regularization learning networks train regularization strength on every neural weight by posing the problem as a large-scale scale hyperparameter tuning scheme [48]. The recent NET-DNF technique introduces a novel inductive bias in the neural structure corresponding to logical Boolean formulas in disjunctive normal forms (26]. An approach that is often mistaken as deep learning for tabular data is AutoGluon Tabular (11) which builds ensembles of basic neural networks together with other traditional ML techniques, with its key contribution being a strong stacking approach. We emphasize that some of these publications claim to outperform Gradient Boosted Decision Trees (GDBT) [[46,, while other papers explicitly stress that the neural networks tested do not outperform GBDT on tabular datasets [48 26 In contrast, we do not propose new kind of neural architecture, but a novel paradigm for learning a combination of regularization methods.
3 An Overview of Regularization Methods for Deep Learning
Weight decay: The most classical approaches of regularization focused on minimizing the norms of the parameter values, e.g., either the L1 [5T], the L2 [52]. or a combination of L1 and L2 known as the Elastic Net 633. A recent work fixes the malpractice of adding the decay penalty term before
'I that sense, this paper adds to the growing body of literature on demonstrating the sensitivity of modem ML methods to hyperparameter settings [19), demonstrating that proper hyperparameter tuning can yield substantial improvements of modern ML methods (6], and demonstrating that even simple architectures can
obtain state-of of the performance with proper hyperparameter settings (38).

---

momentum-basee adaptive learning rate steps (e.g., in common implementations of Adam (27 by decoupling the regularization from the loss and applying it after the learning rate computation 36
Among the augmentation regularizers, Cut-Out [0] proposes to mask a subset of input features (e.g., pixel patches for images) for ensuring that the predictions remain invariant to distortions in the input space. Along similar lines, Mix-Up [60] generates new instances as linear span of pairs of training examples, while Cut-Mix [58] suggests super-positions of instance pairs with mutually-exclusive pixel masks. A recent technique, called Aug-Mix (20], generates instances by sampling chains of augmentation operations. On the other hand, the direction of reinforcement learning (RL) for augmentation policies was elaborated by Auto-Augment [7] followed by a technique that speeds up the training of the RL policy [34.. Recently, these complex and expensive methods were superseded by simple and cheap methods that yield similar performance (RandAugment [8] or even improve on it (TrivialAugment (4T). Last but not least, adversarial attack strategies (e.g.. FGSM [17) generate synthetic examples with minimal perturbations, which are employed in training
Data Augmentation:
robust models (37].
Ensemble methods: Ensembled machine learning models have been shown to reduce variance and act regularizers 45 A popular ensemble neural network with shared weights among its base
models is Dropout 49 which was extended to a variational version with a Gaussian posterior of
the model parameters 28 As follow-up. Mix-Out 32 extends Dropout by statistically fusing
the parameters of two base models. Furthermore, so-called snapshot ensembles [21] can be created using models from intermediate convergence points of stochastic gradient descent with restarts [35]. In addition to these efficient ensembling approaches, ensembling independent classifiers trained in separate training runs can yield strong performance (especially for uncertainty quantification), be based on independent training runs only differing in random seeds (deep ensembles [II,, training runs differing in hyperparameter settings (hyperdeep ensembles, (55)), or training runs with different neural architectures (neural ensemble search (59].
Structural and Linearization: In terms of structural regularization, ResNet adds skip connections across layers 18), while the Inception model computes latent representations by aggregating diverse convolutional filter sizes [50]. A recent trend adds a dosage of linearization deep models, where skip connections transfer embeddings from previous less non-linear layers [T8 22]. Along similar lines, the Shake-Shake regularization deploys skip connections in parallel convolutional blocks and
aggregates the parallel representations through affine combinations [15] while Shake-Drop extends
this mechanism to larger number of CNN architectures (56).
The last family of regularizers broadly encapsulates methods that do not directly propose novel regularization techniques but have an implicit regularization effect as virtue of their 'modus operandi' 2 The simplest such implicit regularization is Early Stopping (57). which limits overfitting by tracking validation performance over time and stopping training when validation performance no longer improves. Another implicit regularization method is Batch Normalization, which improves generalization by reducing internal covariate shift (24. The scaled exponential linear units (SELU) represent an alternative to Datch-normalization through self- ooffmaizmlizingg activation functions 30 On the other hand, stabilizing the convergence of the training routine is another implicit regularization, for instance by introducing learning rate scheduling schemes [35) The recent strategy of stochastic weight averaging relies on averaging parameter values from the local optima encountered along the sequence of optimization steps 25 while another approach conducts updates in the direction of few 'lookahead' steps [61].
Implicit:
4 Regularization Cocktails for Multilayer Perceptrons
4.1 Problem Definition
A training set is composed of features (Trin) and targets y (TTaii) while the test dataset is denoted by X (Test), (Tes), A parametrized function f. i.e., neural network, approximates the targets as y f(X: 0), where the parameters are trained to minimize a differentiable loss function L as arg ming L (y(laain) (X(rai))) ()). To generalize into minimizing L [ 0)). the parameters of I are controlled with a regularization technique Ω that avoids overfitting to the peculiarities of the training data. With a slight abuse of notation we denote I (X; 52 (0; to be the predictions of the model whose parameters 0 are optimized under the regime of the regularization
3

---

method [ A), where À E A represents the hyperparameters of S2. The training data is further divided into two subsets as training and validation splits, the later denoted by X X y such that À can be tuned on the validation loss via the following hyperparameter optimization objective:
X E arg min L (y-Van,
of (x (Val) 02)),
(1)
dEA
0 arg min (y(Thaim) f (Tmin), Ω (0:
0
After finding the optimal (or in practice at least a well perfforng))g)ng) configuration X", we re-fit @ on the entire training dataset, i.e., (Trrin) X and y-Train) Uy(Val)
While the search for optimal hyperparameters À is an active field of research in the realm of AutoMI. [23], still the choice of the regularizer Ω mostly remains ad-hoc practice, where practitioners select a few combinations among popular regularizers (Dropout, I.2, Batch Nor- malization, etc.). In contrast to prior studies, we hypothesize that the optimal regularizer is a cocktail mixture of a large set of regularization methods, all being simultancously applied with different strengths (i.e., dataset-specific hyperparameters). Given a set of K regularizers (S2( (::())) km (2(1) (1: ) ..... ,..., (2(K) (:: X ((K)) }. each with its own hyperparameters ( E (R) vk fl....,AA K}. the problem of finding the optimal cocktail of regularizers is:
X E
arg min
C (y (x(aa);2)))
A= ((..........A
S.L. 0 E arg min L (ymmain) ( { (0.2(*))3 (K=1)
)
0
The intuitive interpretation of Equation 2 is searching for the optimal hyperparameters (i.e., strengths) of the cocktail's regularizers using the validation set, given that the optimal prediction model parameters 0 are trained under the regime of all the regularizers being applied jointly. We stress that, for each regularizer, the hyperparameters (1) include a conditional hyperparameter controlling whether the k-th regularizer is applied or skipped. The best cocktail might comprise only a subset of regularizers.
4.2 Cocktail Search Space
To build our regularization cocktails we combine the 13 regularization methods listed in Table which represent the categories of regularizers covered in Section 3 The regularization cocktail's 's search space with the exact ranges for the selected regularizers' hyperparameters is given in the same table. In total, the optimal cocktail is searched in a space of 19 hyperparameters.
While we can in principle use any hyperparameter optimization method, we decided to use the multi-fidelity Bayesian optimization method BOHB [12] since it achieves strong performance across a wide range of computing budgets by combining Hyperband [3 and Bayesian Optimization (40).
and since BOHB can deal with the categorical hyperparameters
for enabling or disabling regularization techniques and the corresponding conditional structures. Appendix [A describes the implementation details for the deployed HPO method. Some of the regularization methods cannot be combined, and we, therefore, introduce the following constraints to the proposed search space: (i)
and Shake-Drop are not simultaneously active since the latter builds on the former; (ii)
Only one data augmentation technique out of Mix- Up. Cut-Mix, Cut-Out, and FGSM adversarial learning can be active once due to a technical limitation of the base library we use 62
Experimental Protocol
5.1 Experimental Setup and Datasets
We use large collection of 40 tabular datasets (listed in Table Appendix D). This includes 31 datasets from the recent open-source OpenML AutoML Benchmark [16] In addition, we added
The remaining 8 datasets from that benchmark were too large to effectively cluster.

---

Group
Regularizer
Hyperparameter
Type
Range
Conditionality
BN
BN-active
Bonlean
(True, False]
Implicit
SWA
SWA-active
Boolean
(Tre, False)
LA-active
Bonlean
(True, False]
LA
Step size
Continuous
0.8) 0.8)
LA-active LA-active
Num. steps
Integer
[5, 10)
W. Decay
WD-active
Boolean
(Thue, False) 100",..11
WD
Decay factor
Continuous
WD-active
DO-active
Bookan
(True, Falsc)
Ensemble
(funnel, long funnel, diamond, hexugon, brick, triangle, stairs) [0.0, 0.8)
DO
Dropout shape
Nominal
DO-active
Drop rate
Continuous
DO-active
SE
SE-active
Bookaan
(Tree, Falsc)
SC-active
Booleun
(True, False) {SS, SI>, Standard)
SC
Structural
MB choice
Nominal
SC-active
SD
Max. probability
Continuous
[0.0, 1.0)
SC-active A MB choice = SD
S$
SC-active A MB choice = S8
Augment
Nominal
{MU, CM, CO, AT, None)
Augmentation
MU
Mix. magnitude
Continuous
[0.0. 1.0j
Augment MU
CM
Probability
Continuous
[0.0, 1.0]
Augment CM
Probability
Continuous
0.00, 1.0)
Augment CO
CO
Patch ratio
Continuous
[0.0, 1.0)
Augment CO
AT'
Augment AT
Table 1: The configuration space for the regularization cocktail regarding the explicit regularization hyperparameters of the methods and the conditional constraints enabling or disabling them. (BN: Batch Normalization, SWA: Stochastic Weight Averaging, LA: Lookahead Optimizer, WD: Weight Decay, DO: Dropout, SE: Snapshot Ensembles, SC: Skip Connection, MB: Multi-branch choice, SD: Shake-Drop, SS: Shake-Shake, MU: Mix-Up, CM: Cut-Mix, CO: Cut-Out, and AT: FGSM Adversarial Learning)
9 popular datasets from UCI 3 and Kaggle that contain roughly 100K+ instances. Our resulting benchmark of 40 datasets includes tabular datasets that represent diverse classification problems, containing between 452 and 416 188 data points, and between 4 and 2 2001 features, varying in terms of the number of numerical and categorical features. The datasets
retrieved from the OpenML. repository (54] using the OpenML-Pyythn connector [14] and split as 60% training, 20% validation, and 20% testing sets. The data is standardized to have mean and unit variance where the statistics for the standardization are calculated on the training split.
We ran all experiments on CPU cluster, each node of which contains two Intel Xeon E5-2630v4 CPUs with 20 CPU cores each, running at 2.2GHz and total memory of 128GB. We chose the PyTorch library 43
deep learning framework and extended the AutoDL-framework 62 with our implementations for the regularizers Table [ We provide the WellfuunddSimleeNet..
Auto-Pytorch [39
code for implementation at the following link: https: https://github. http://itttu..ooeeeeeeeeeeeeeeee www./reeeauiirrrrbbrrg
To optimally utilize resources, we ran BOHB with 10 workers in parallel, where each worker had access to 2 CPU cores and 12GB of memory, executing one configuration at time. Taking into account the dimensions D of the considered configuration spaces, we BOHB for at most days, or at most 40 x D hyperparameter configurations, whichever came first. During the training phase, each configuration was run for 105 epochs, in accordance with the cosine learning rate annealing with restarts (described in the following subsection). For the sake of studying the effect on more datasets, only evaluated single trait-val-test split. After the training phase is completed, we report the results of the best hyperparameter configuration found, retrained on the joint train and validation set.
5.2 Fixed Architecture and Optimization Hy Hyperparameters
In order to focus exclusively on investigating the effect of regularization we fix the neural architecture to simple multilayer perceptron (MLP) and also fix some hyperparameters of the general training
5

---

procedure. These fixed hyperparameter values, specified in Table of Appendix B.I have been tuned for maximizing the performance of an unregularized neural network on our dataset collection (see Table in Appendix D). We use 9-layer feed-foree forward neural network with 512 units for each layer, a choice motivated by previous work 42
Moreover, we set a low learning rate of 10 after performing grid search for finding the best value across datasets. We use AdamW W (36), which implements decoupled weight decay, and cosine annealing with restarts [5 as a learning rate scheduler. Using learning rate scheduler with restarts helps in our case because we keep fixed initial learning rate. For the restarts, we use initial budget of 15 epochs, with a budget multiplier of 2, following published practices [62]. Additionally, since our benchmark includes imbalanced datasets, we use a weighted version of categorical cross-entropy and balanced accuracy [ as the evaluation metric.
5.3 Research Hypotheses and Associated Experiments
Hypothesis I:
Regularization cocktails outperform state-of-the-art deep learning architectures on tabular datasets.
Experiment 1: We compare our well-regularized MLPs against the recently proposed deep learning architectures Node [46] and TabNet M. Additionally, we compare against two versions of AutoGluon Tabular [T]. a version that features stacking and a version that addition- ally includes hyperparameter optimization. Moreover, we add an unregularized version of our MLP for reference, as well as version of our MLP regularized with Dropout (where the dropout hyperparameters are tuned on every dataset). Lastly, we also compare against neff-normalizing neural networks 22 by using the same MLP backbone as with our regularization cocktails.
Hypothesis 2: Regularization cocktails outperform Gradient-Boostie Decision Trees (GBDTs), the most commonly used traditional ML method and de-facto state-of-the-ar for tabular data.
Experiment 2:
We compare against three different implementations of GBDT: an implementation from scikit-learn 44 and optimized by Auto-skleamn [13, the popular XGBoost [5], and lastly, the recently proposed CatBoost [47).
Hypothesis 3: Regularization cocktails are time-efficient and achieve strong anytime results.
Experiment 3: We compare our regularization cocktails against XGBoost over time.
5.4 Experimental Setup for the Baselines
All baselines the same train, validation, and test splits, the seed, and the same HPO resources and constraints as for our automatically-consttuuce with 128GB of memory). After finding the best incumbent configuration, the baselines are refitted on the union of the training and validation sets and evaluated on the test set. The baselines consist of two recent neural architectures, two versions of AutoGluon Tabular with neural networks, and three implementations of GBDT, as follows:
regularization cocktails (4 days on 20 CPU cores
TabNet:
This library does not provide an HPO algorithm by default; therefore, we also used BOHB for this search space, with the hyperparameter value ranges recommended by the authors [I].
Node: This library does not offer an HPO algorithm by default. We performed a grid search among the hyperparameter value ranges proposed by the authors [46]; however, we faced multiple memory and runtime issues in running the code. To overcome these issucs we used the default hyperparameters the authors used in their public implementation.
AutoGluon Tabular:
This library constructs stacked ensembles with bagging among diverse neural network architectures having various kinds of regularization [II]. The training of the stacking ensemble of neural networks and its hyperparameter tuning are integrated into the library. Hyperparameter optimization (HPO) is deactivated by default to give more resources to stacking, but here we study AutoGluon based on either stacking or HPO (and HPO actually performs somewhat better). While AutoGluon Tabular by default uses broad range of traditional MI. techniques, here, order to study as a "pure" deep learning method, we restrict to only use neural networks as base learners.

---

ASK-GBDT:
The GBDT implementation of scikit-learn offered by Auto-sklearn [3] uses SMAC for HPO, and we used the default hyperparameter search space given by the library.
XGBoost:
The original library [5] does not incorporate an HPO algorithm by default, so we used BOHB for its HPO. We defined a search space for XGBoost's s hyperparameters following the best practices by the community; we describe this in the Appendix B.2
CatBoost:
Like for XGBoost, the original library 47 does not incorporate HPO algorithm, so we used BOHB for its HPO, with the hyperparameter search space recommended by the authors.
For in-depth details about the different baseline configurations with the exact hyperparameter search spaces, please refer to Appendix B.2
Dataset
Feat
MLP
MLP+D
XGB.
ASK-G.
TabN.
Node
AutoGL.
MLP+C
anneal
898/ 39
84.131
86.916
85,416
90.000
84,248
20.000
80,000
89. 39.270
kr-vs.kp
3196 37
99.701
99,850
99,850
99.850
93.250
97.264
99.687
99.850
arrhythmia
452 / 280
37.991
38.70
48.779
46.850
43,562
N/A
48 934
61.461
mfcat.
2000/ 217
97.750
98,000
98.000
97.500
97.250
97.250
98,000
98.000
credit-g
1000 / 21
69:405
68.095
68 929
71.191
61.190
73.095
69.643
74.643
vehicle
83.766
206
74.973
83.793
kel
2109 / 22
70.274
72,980
66 846
63.353
52.517
55,803
67.270
74.381
adult
48842/ 15
76.893
78.520
79.824
79,830
77.155
78.168
80.557
82.443
walking.
149332
60,997
63.754
61.616
62.764
56,801
N/A
60,800
63.923
phoneme
5404 6
87.514
88.387
87.972
131
86.824
82.720
83.943
86.619
skin-sgg.
245057 /4
99.971
99,962
99 968
99.967
99.961
N/A
99,973
99.953
ldpa
164860 /
62.831
67.035
99.008
68.947
54.815
N/A
$3.023
68. 107
34465 11
95.917
96.232
96.872
97.217
95.425
96.217
96.420
96.826
cnac
1080 857
87 570
90.741
94.907
93,519
89,352
96.759
92,593
95.833
blood.
67.836
64.327
67.251
bank.
45211 / 17
78.076
83.145
72.658
72.283
70.639
74.607
79.483
85,993
connect.
67557/
73.627
76.345
72.374
72.645
N/A
$0.073
shuttle
58000 10
99.475
99,892
98.563
98.571
88.017
42,805
83,433
99.948
higgs
98050 29
67.752
66.873
72.944
72,926
72.036
N/A
73.798
73.546
australian
690
86,268
86.268
89.717
88.589
85,278
83.468
88.248
$7.088
car
1728
97.
99.690
92.376
100.000
98.701
16.119
99.675
99.587
segment
2310 / 20
94,805
94.589
93.723
93.074
91,775
90.043
91,991
93.723
fashion.
70000/ / 785
90.464
90,507
91.243
90.457
89.793
N/A
91.336
91,950
jungle.
44819
97.061
97.237
87.325
83,070
73.425
N/A
93.017
97.471
numerai
963.2 22
50.262
50,301
$2. 22.363
$2,421
$1.599
52.364
51,706
52.668
devmugrri
92000/ 1025
93.310
77,897
N/A
97.734
helena
65196 / 28
16.836
23.983
21 994
21.144
19.032
N/A
27.115
27.701
jannis
83733 55
51.505
55.118
55 225
55,593
56,214
N/A
58,526
65.287
volkert
58310/ 181
65 15.088
66.996
64.170
63,428
$9.409
N/A
70.195
71.667
minihoone
130064/ 51
90.639
94.099
94.024
94.137
62.173
N/A
94.978
94.015
apsfailure
76000 / 171
87.759
91.194
88 825
91.797
51,444
N/A
88,890
92.535
christine
5418 1637
70.941
70.756
74.815
74.447
69 649
73.247
77.
74 262
dilbert
10000/ 2001
96.930
96.733
99.106
98.704
97.608
N/A
98.758
99.049
fabert
8237 801
63.707
64.814
70.098
70.
62.277
66.00
68, 142
69.183
jasmine
2984/ 145
78.048
76.211
80.546
78.878
76.690
80.053
79.217
sylvine
5124 21
93.070
93.363
95.509
95.119
83.595
93.852
93.753
94.045
dionis
416188
91.905
92.724
N/A
aloi
108000/ / 129
92.331
93.852
95.338
13.534
93.589
N/A
97,423
97.175
oefraud
284807 31
50.000
50.000
90.303
92,514
85,705
N/A
91.831
92.531
clickpred.
399482 12
63.125
64.367
58.361
58,201
50,163
N/A
54,410
64,280
Wins/LossessTies
MI.+ vs...
35/5/D
30/8/2
26/12/2
38/2/0
19/2/0
30/9/1
Wilcoxon p-vaiue
MLP+C vs
5.3 10-7
8.9
10-5
6 10
2.8
10
4.5 x 10-
8.2 10
4 10-5
Table 2: Comparison of well-regularized MLPs vs. other methods in terms of balanced accuracy. N/A values indicate failure due to exceeding the cluster's memory (24GB per process) or runtime limits (4 days). The acronyms stand for MLP+D: MLP with Dropout, XGB.: XGBoost, ASK-G.: GBDT by Auto-sklearn, AutoGL. S: Autogluon with stacking enabled, TabN.: TabNet and MLP+C: our MLP regularized by cocktails.
Experimental Results
We present the comparative results of our MLPs regularized with the proposed regularization cocktails (MLP+C) against ten baselines (descriptions in Section (.4)) (a) two state-of-the-arr architectures (NODE, TabN.); (b) two AutoGluon Tabular variants with neural networks that features stacking (AutoGL. S) and additionally HPO (AutoGL. HPO); (c) three Gradient-Boosted Decision Tree
7

---

0.8
MLP + C EErorr Rate
MLP + C Error Rate
0.6-
MLP + +.C Error
0.6
0.4
0.4
0.4
0.2
0.0
0,2
0.4 0.6
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
AutoGL Error Rate
ASK-G. Error Rate
Figure I: Comparison of our proposed dataset-specific cocktail (MLP+C) against the top three baselines, Each dot the plot represents dataset, the y-axis our method's errors and the x-axis baselines' errors.
(GBDT) implementations (XGB., ASK-G., and CatBoost); (d) well as three reference MLPs (unregularized (MLP), and regularized with Dropout (MLP+D) 49 or SELU (MLP+SELU) (30
Table shows the comparison against subset of the baselines, while the full detailed results involving all the remaining baselines are located in Table in the appendix. is worth re- emphasizing that the hyperparameters of all the
Ranks
7
3
TabN. + ES 7.022
2:2125
MLP + C
presented baselines (except the unregularized MLP, which has no hyperparameters and Au-
NODE 6,1805
TabN.
3,6750
ALtooL HPO
AutoGL.
MLP 5,5326
4,770
MLP + D
MLP SELUA2290
toGL. S) are carefully tuned on a validation set
detailed in Section 3 the appendices ref-
CD of MLP+C neural networks
erenced therein. The table entries represent the
Ranks
test sets' balanced accuracies achieved the
5
3
1
described large-scale collection of 40 datasets. Figure visualizes the results showing substan- tial improvements for our method.
XGB + ES 1,0000
2.0750 MLP + C
C 22051
XGB.
ASK-G. 3.0875
To assess the statistical significance, lyze the ranks of the classification accuracies across the 40 datasets. We use the Critical Difference (CD) diagram of the ranks based the Wilcoxon significance test, standard metric for comparing classifiers across multi- ple datasets (9] The overall empirical compar- ison of the elaborated methods is given in Fig- ure The analysis of neural network baselines in Subplot 2a reveals a clear statistical signifi- cance of the regularization cocktails against the other methods. Apart from AutoGluon (both versions), the other neural architectures are not competitive even against an MLP regularized only with Dropout and optimized with our stan- dard, fixed training pipeline of Adam with co- sine annealing. To be even fairer to the weaker
(b) CD of MLP+C vs. GBDT
Ranks
7
4
TabN, 6,000
2.4625
MLP + C AutoGL. HPO
NODE
CatBoost" 3590
$,9000 AutoGL.
ASK-G.
XGB.
(c) CD of MLP+C vs. all baselines
Figure 2: Critical difference diagrams with Wilcoxon significance analysis on 40 datasets. Connected ranks via bold bar indicate that perfor- mances are not significantly different (p > 0.05).
baselines (TabNet and Node) tried boosting them by adding early stopping (indicated with "+ES"), but their rank did not improve. Overall, the large-scale experimental analysis shows that Hypothesis in Section 5.3 is validated: well-regularized simple deep MLPs outperform specialized neural architectures.
Next, we analyze the empirical significance of our well-regularized MLPs against the GBDT im- plementations in Figure (bb The results show that our MLPs outperform all three GBDT variants (XGBoost, auto-sklearn, and CatBoost) with statistically significant margin. We added early stop- ping ("+ES") to XGBoost, but it did not improve its performance. Among the GBDT implementations, XGBoost without early stopping has non-significant margin over the GBDT version of auto-sklearn

---

100
Grouped Cocklail Ingrediente
Frequercies (%%
SE
BN
WD
CM 40%
DO
72.5%
62.5%
57.5%
32.5%
30%
60
BN
50%
40%
37.5%
Implist
0
Weight Decay
Model Averaging
Data Augmentation
Strustural
Figure 3: Left: Cocktail ingredients occurring in at least 30% of the datasets. Right: Clustered histogram (union of member occurrences) with the acronyms from Table [ SWAJ, M. Averaging: (DO, SE), Structural: {SC, SS, SD), D. Augmentation: (MU, CM, CO, AT).
Implicit: {BN, LA,
as well as CatBoost. We conclude that well-regularized simple deep ML.s outperform GBDT, which validates Hypothesis 2 in Section 5.3
The final cumulative comparison in Figure 2c provides further result: none of the special- ized previous deep learning methods (TabNet, NODE, AutoGlun Tabular) outperforms GBDT significantly. To the best of our awareness, this paper is therefore the first to demonstrate that neural networks beat GEDT with a statistically significant margin over large-scale experimen- tal protocol that conducts a thorough hyperpa- rameter optimization for all methods.
Time (Hours)
Wins
Ties
Losses
p-value
0.25
21
1
17
0.8561
0.5
25
0.0145 13
1
24
1
0.0120 14
2
27
1
0.0006 11
4
28
1
0.0006 11
8
28
0.0004 11
16
28
1
0.0005 11
32
29
10
0.0003
Figure g provides a further analysis on the most prominent regularizers of the MLP cocktails, based on the frequency with which our HPO procedure selected the various regularization methods for each dataset's cocktail. In the left plot, we show the frequent individual regulariz- ers, while the right plot the frequencies are grouped by types of regularizers. The grouping reveals that cocktail for each dataset often has least one ingredient from every regularization family (detailed in Section 3). highlighting the need for jointly applying diverse regularization methods.
64
9
0.0002
96
30
9
0.0002
Table 3: Comparing the cocktails and XGBoost over different HPO budgets. The statistics are based the test performance of the incumbent configurations over all the benchmark datasets.
Lastly, Table sshows the efficiency of our regularization cocktails compared to XGBoost over increas- ing HPO budgets. The descriptive statistics calculated from the hyperparameter configurations with the best validation performance for datasets during the HPO search, however, taking their respective test performances for comparison. A dataset is considered in the comparison only if the HPO procedure has managed to evaluate at least one hyperparameter configuration for the cocktail baseline. As the table shows, regularization cocktails achieve better performance in only 15 minutes for the majority of datasets. After 30 minutes of HPO time, regularization cocktails are statistically significantly better than XGBoost. As more time is invested, the performance gap with XGBoost increases, and the results get even more significant; this is further visualized in the ranking plot over time in Figure Based on these results, conclude that regularization cocktails are time-efficient and achieve strong anytime results, which validates Hypothesis 3 in Section 5.3
7 Conclusion
Summary. Focusing on important domain tabular datasets, this paper studied improvements to deep learning (DL) by better regularization techniques. We presented regulurization cocktails, per-dataset-optimizee combinations of many regularization techniques, and demonstrated that these improve the performance of even simple neural networks enough to substantially and significantly

---

surpass XGBoost, the current state-of- ofte--of-the-art method for tabular datasets. We conducted large- scale experiment involving 13 regularization methods and 40 datasets and empirically showed that (i) modern DI. regularization methods developed in the context of raw data (e.g.. vision, speech, text) substantially improve the performance of deep neural networks on tabular data; (ii) regularization cocktails significantly outperform recent neural networks architectures, and most importantly iii) regularization cocktails outperform GBDT on tabular datasets.
Limitations. To comprehensively study basic principles, we have chosen an empirical evalu- ation that has many limitations. We only stud- ied classification, not regression. We only used somewhat balanced datasets (the ratio of the minority class and the majority class is above 0.05). We did not study the regimes of extremely few or extremely many data points (our smallest data set contained 452 data points, our largest 416 188 data points). We also did not study datasets with extreme outliers, missing labels, semi-supervised data, streaming data, and many more modalities in which tabular data arises. An important point worth noticing is that the re- cent neural network architectures (Section (.4) could also benefit from our regularization cock- tails, but integrating the regularizers into these baseline libraries requires considerable coding efforts.
1.7
Average Rank
1.6
MLP rank
1.5
XGBoost rank
1.4
1.3
10
10 Time (Hours)
104
Figure 4: Ranking plot comparing XGBoost and regularization cocktails over time.
Future Work. This work opens up the door for a wealth of exciting follow-up research. Firstly, the per-dataset optimization of regularization cocktails may be substantially sped up by using meta- learning across datasets [53]. Secondly, as we have used a fixed neural architecture, our method's performance may be further improved by using joint architecture and hyperparameter optimization. Thirdly, regularization cocktails should also tested under all the data modalities under "Limitations" above. In addition, it would be interesting to validate the gain of integrating well-regularized MLPs into modern AutoML. libraries, by combining them with enhanced feature preprocessing and ensembling.
Take-away.
Even simple neural networks can achieve competitive classification accuracies on tabular datasets when they are well regularized, using dataset-specific regularization cocktails found via standard hyperparumeter optimization.
Societal Implications
Enabling neural networks to advance the state-of-the-art on tabular datasets may open up a garden of delights in many crucial applications, such as climate science, medicine, manufacturing, and recommender systems. In addition, the proposed networks of data science for social good, such as the realm of fair machine learning where the associated data are naturally in tabular form. However, there are also potential disadvantages in advancing deep learning for tabular data. In particular, even though complex GBDT ensembles are also hard to interpret, simpler traditional ML methods are much more interpretable than deep neural networks; we therefore encourage research on interpretable deep learning on tabular data.
backbone for applications
Acknowledgements
We acknowledge funding by the Robert Bosch GmbH, by the Eva Mayr-Stihl Foundation, the the Baaiininns-BrinnTT CoE, the European
MWK of the German state of Baden- Research Council (ERC) under the European Union's 's Horizon 2020 programme, grant no. 716721, and the German Federal Ministry of Education and Research (BMBF, grant RemormalizedFlows 01IS19077C7 The authors acknowledge support by the state of Baden-Wüirternd Württemberg through bwHPC and the German Research Foundation (DFG) through grant no INST 39/963-1 FUGG.
10

---

References
[1] S. Arik and T. Pfister. Tabnet: Attentive interpretable tabular learning. In AAAI Conference on Artificial Intelligence, 2021.
[2] S. Arora, N. Cohen. W. Hu, and Y. Luo. Implicit regularization in deep matrix factorization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alche Buc, E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems 32, pages 7413-7424. Curran Associates, Inc., 2019.
[3] A. Asuncion and D. Newman. Uci machine learning repository. 2007.
[4] K. Henning Brodersen, C.,, and E. Klaas Stephan J. M Buhmann. The balanced accuracy and its posterior distribution. In 2010 20th International Conference on Pattern Recognition, pages 3121-3124. IEEE, 2010.
[5] Chen and C. Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785-794, 2016.
[6] Yutian Chen, Aja Huang, Ziyu Wang, Ioannis Antonogiou, Julian Schrittwieser, David Silver, and Nando de Freitas. Bayesian optimization in alphago. CoRR, abs/1812.068555 2018.
[7] E. Cubuk, B. Zoph, D. Mane, and Vasudvaanand Q Le. Autoaugment: Learning augmentation strategies from data. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
[8] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
[9] J. Demšar. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res., 7:1-30, December 2006.
[10] T. Devries and G. Taylor. Improved regularization of convolutional neural networks with cutout. ArXiv,
2017.
[11] N. Erickson, J. Mueller, A. Shirkov, H. Zhang, P. Larroy, M. Li, and A. Smola. Autogluon- 2020.
tabular: Robust and accurate automl for structured data. CORR,
[12] S. Falkner, Aleiein, and F. Hutter. BOHB: Robust and efficient hyperparameter optimization at scalae. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018), pages 1436-1445, July 2018
[13] M. Feurer, Klein, K. Eggensperger, Springenberg, M. Blum, and Hutter. Efficient and robust automated machine learning. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume page 2755 2763. MIT Press, 2015.
[14] M. Feurer, JN. van Rijn, A. Kadra, P. Gijsbers, N. Mallik, Ravi, A. Müller, J. Vanschoren, and F. Hutter. Openml-python: extensible python api for openml. Journal of Machine Learning Research, 22(100):1-5, 2021.
[15] X. Gastaldi. Shake-shake regularization of 3-branch residual networks. In 5th International Conference on Learning Representations, ICLR. OpenReview.net, 2017.
[16] P. Gijsbers, E. LeDell, S. Poirier. J. Thomas, B. Bischl, and J. Vanschoren. An open source automl benchmark. arXiv preprint urXiv:1907.000 [cs.LG], 2019. Accepted at AutoML Workshop at ICML 2019.
[17] I. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR, 2015.
[18) K. He, Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
11

---

[19] P. Henderson, R. Islam. P. Bachman, J. Pineau, D. Precup. and D. Meger. Deep reinforcement learning that matters. In S. Mcllraith and K. Weinberger, editors, Proceedings of the Conference on Artificial Intelligence (AAAI'18) AAAT Press, 2018
[20] D. Hendrycks, N. Mu, E. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan. Augmix: simple method to improve robustness and uncertainty under data shift. In International Conference on Learning Representations, 2020.
[21] G. Huang, Y. Li, G. Pleiss, Z. Liu, J. Hopcroft, and K. Weinberger. Snapshot Ensembles: Train
1, Get M for Free. International Conference Learning Representations, November 2017.
[22] G. Huang, Z. Liu, L. van der Maaten, and K. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, 2017.
[23] F. Hutter, L. Kotthoff, and J. Vanschoren, editors. Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019. In press, available at http://automml.rr/bbrr
[24] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 448-456. PMLR, 07-09 Jul 2015.
[25] P. Izmailov. D. Podoprikhin, T. Garipov, D. Vetrov, and A. Wilson. Averaging weights leads to wider optima and better generalization. In Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI, pages 876-885. AUAI Press, 2018.
[26] L. Katzir, G. Elidan, and R. El-Yaniv. Net-(dnf): Effective deep modeling of tabular data. In International Conference on Learning Representations, 2021.
[27] D. Kingma Jnd J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
[28] D. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparameterization trick. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 2, NIPS' 15, page 2575-2583. MIT Press, 2015.
[29] G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter. Self Self-normalizing neural networks In Proceedings of the 31st international conference on neural information processing systems, pages 972-981, 2017.
[30] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[31] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, page 6405-6416, Red Hook, NY, USA, 2017. Curran Associates Inc.
[32] C. Lee, K. Cho, and W. Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In International Conference on Learning Representations, 2020.
[33] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn, Res., 18(1):67656681 January 2017.
[34] S. Lim, I. Kim, T. Kim, C. Kim, and S. Kim. Fast autoaugment. In H. Wallach, H. Larochelle, A. Beygelzimer, d Alche-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 6665-6675. Curran Associates, Inc., 2019.
[35] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. In In- ternational Conference on Learning Representations (ICLR) 2017 Conference Truck, April 2017.
12

---

[36] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[37] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[38] Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. In International Conference Learning Representations, 2018.
[39] H. Mendoza, A. Klein, M. Feurer, Tobias Springenberg. M. Urban, M. Burkart, M. Dippel, M. Lindauer, and F. Hutter. Towards automatically-turat deep neural networks. In F. Hutter, L. Kotthoff, and Vanschoren, editors, AutoML: Methods, Sytems, Challenges, chapter 7, pages 141-156. Springer, December 2019.
[40] Mockus. Application of bayesian approach numerical methods of global and stochastic optimization. J. Glob. Optim., 4(4):347-365, 1994.
[41] Samuel G. Müller and Frank Hutter. Trivialaugmet:: Tuning-free yet state-of-the-art data augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 774-782, October 2021.
[42] A. Emin Orhan and X. Pitkow. Skip connections eliminate singularities. arXiv preprint arXiv:1701.09173 2017.
[43] A. Paszke, S. Gross, Massa, A. Lerer, Bradbury, G. Chanan, T. Killeen Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, library. In Advances neural information processing systems, pages 8026-80 222-8037,11 2019.
deep learning
[44] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[45] R. Polikar. Ensemble Learning. In C. Zhang and Y. Ma, editors, Ensemble Machine Learning: Methods and Applications, pages 1 1-34. Springer US, 2012.
[46] S. Popov, Morozov, and A. Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In International Conference on Learning Representations, 2020.
[47] L. Prokhorenkova, G. Gusev, A. Vorobev, AV. Dorogush, and A. Gulin. Catboost: unbiased boosting with categorical features. Advances in Neural Information Processing Systems, 31, 2018.
[48] I. Shavitt and E. Segal. Regularization learning networks: Deep learning for tabular datasets. In Proceedings of the 32nd International Conference Neural Information Processing Systems, page 1386-1396. Curran Associates Inc., 2018.
[49] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Mach. Learn. Res., 15(1):1929-1958, January 2014.
[50] C. Szegedy, S. Ioffe, V. Vanhoucke, and Alemi. Inception-v4, inception reeentt and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence, 2017.
[51] R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society (Series B), 58:267-288. 1996.
[52] A. Tikhonov. On the stability of inverse problems. In Doklady Akademii Nauk SSSR, 1943.
[53] J. Vanschoren. Meta-learning. In F. Hutter, L. Kotthoff, and J. Vanschoren, editors, Automated Machine Learning Methods, Systems, Challenges, The Springer Series on Challenges in Machine Learning, pages 65-61. Springer, 2019.
13

---

[54] J. Vanschoren, J. Van Rijn, B. Bischl, and L. Torgo. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49 60, 2014.
[55] Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 65 15227 Curran Associates, Inc., 2020.
[56] Y. Yamada, M. Iwamura, and K. Kise. Shakedrop regularization, 2018.
[57] Y. Yao, L. Rosasco, and A. Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26(2):289-315, August 2007.
[58] S. Yun, D. Han, S. Joon, S. Chun, J. Choe, and Y. Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In International Conference on Computer Vision (ICCV), 2019.
[59] Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris Holmes, Frank Hutter, and Yee Whye Teh. Neural ensemble search for uncertainty estimation and dataset shift. In Advances in Neural Information Processing Systems, 2021.
[60] H. Zhang, M. Cisse, Y. Dauphin, and D. Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Reprexentations, 2018.
[61] M. Zhang, J. Lucas, J. Ba, and G. Hinton. Lookahead optimizer: k steps forward, step back. H. Wallach, H. Larochelle, A. Beygelzimer, Buc, Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, pages
2019.
[62] L. Zimmer, M. Lindauer, and Hutter. Auto-pytorch tabular: Multi-fidelity metalearning for efficient and robust autodl. IEEE TPAMI, 2021. IEEE Early Access.
[63] H. Zou and T. Hastie. Regularization and variable selection via the clastic net. Journal of the royal statistical society: series B (statistical methodology), 67(2):301-320 2005.
14


Reviews:
[{"review_id": "r13FteYnRPn", "reviewer": "Reviewer_V9FE", "summary": "The paper supplies an empirical existence proof that an \"appropriately regularized\" multi-layer perceptron (MLP)-like model can outperform gradient-boosted decision trees (GBDT) on tabular data. The paper's proposed approach defines a set of parameterized techniques which includes ensembling, architectural changes, and data augmentation, and performs search over combinations from this set. The extensive experimental comparison across 40 tabular datasets shows that MLP-like models can outperform GBDTs and other neural network architectures.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I appreciated the simplicity of the proposed approach. Showing that Bayesian optimization can select the right combination from such a disparate set of techniques (\"soup\" came to my mind) is novel, at least to my knowledge. The paper also seems to have addressed some concerns raised in previous submissions with a thorough evaluation performed across a competitor set. However, I have some major concerns on whether the comparison makes sense. The computational cost associated with the proposed approach could be considered impractical wrt/ the standard GBDT solution, but perhaps more work could show this can be reduced. More fine-grained conclusions about why certain techniques, e.g., snapshot ensembles, are necessary and/or sufficient for identifying a high-performing MLP-like model would also have strengthened the submission.\n\nThe use of \"regularization\" to describe the set of techniques over which the proposed method optimizes is misleading. For example, take ensembling: just b/c ensembling has regularizing properties does not make it a regularizer in the traditional sense of a technique to mitigate against overfitting of a single model during training. Conversely, just b/c dropout has ensembling properties does not make it an ensembling technique in the traditional sense of combining the predictions of multiple trained models. Another example is what are termed structural/linearization techniques which include adding skip connects. These really fall under model selection. \n\nA more accurate title would be \"Traditional regularization, data augmentation, model selection, model averaging, and deep learning tricks are all you need...\" \nThe value of the contribution comes down to how surprising the community thinks the news that some specific combination of these techniques can yield an MLP-like model or ensemble which, when searched for using 4 days of compute, outperforms GBDT on tabular datasets with <500K rows. If there was evidence that the search could be performed efficiently, that would be hugely beneficial to the message. Unfortunately, there is no finer timing analysis included in the submission.\n\nThe overloading of \"regularization\" in the paper isn't a pedantic issue. It makes fair comparisons wrt/ hypothesis 1 difficult. For example, AutoGluon tabular was run with only neural networks but is designed with general ML models in mind (as this paper notes). This seems artifically limiting if AutoGluon tabular natively, i.e., with AutoGluon's own model and techniques, outperforms GBDT(-like) models; since, if it does, then GBDT should no longer be considered the correct baseline of comparison. More generally, a discussion on why this paper's set of techniques (listed in Table 1) is the \"right\" set would help in framing which competitors are appropriate vs. unappropriate in a comparison.\n\n* A finer analysis would be helpful to the community. Is there a method rank vs. time plot? method mean+-err balanced accuracy vs. time?\n\n* Are there obvious efficiencies, such as training data-subsampling, in which the computational cost of the proposed approach is retained w/o compromising accuracy?\n\n* How does the absence of one (or multiple) technique(s) from the set of techniques affect performance? E.g., if snapshot ensembles are excluded, how do the results of Table 2 change? Note, this is not a question about which techniques are selected; it is a question about which techniques are generally important/unimportant for the tabular datasets problem.\n\n* It isn't clear how AutoGluon tabular performs natively. If it does not outperform GBDT, then this fact should be included in the paper as it will substantially increase the case for MLP-like solutions.\n\n* Please motivate the selection for the specific set of techniques listed in Table 1.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed both of these.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5-7"}, "extra_text": {}}, {"review_id": "qM3vNVmuSYn", "reviewer": "Reviewer_VweU", "summary": "This paper proposes that properly optimized MLPs can perform just as well if not better on tabular data. This is an important topic as it's well know that MLPs have not achieved similar performance as GBDT on data with categorical variables. The paper is well written and reasonably comprehensive in the experiments. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes that properly optimized MLPs can perform just as well if not better on tabular data. This is an important topic as it's well know that MLPs have not achieved similar performance as GBDT on data with categorical variables. The paper is well written and reasonably comprehensive in the experiments. \n\nPositives:\n- This paper is written clearly and have considered a large set of regularization techniques as well as tested on a comprehensive tabular dataset\n- The results is surprising and very useful and relevant to the community\n- Real world Kaggle datasets are included as well, in addition to the tiny UCI datasets.\n- The discussions on limitations is also interesting and useful.\n\nNegatives:\n- the optimization being fixed is a negative. Optimizers typically have a big impact on the performance and generalization of MLPs.\n- What about the different types of weight initialization for MLPs?\n\nGiven that the results are re-generated by the authors, what is not super clear is that are the numbers for non-MLP results consistent with other published papers?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "mJasJIRLDe", "reviewer": "Reviewer_BhRe", "summary": "On a wide selection of tabular data datasets, this paper showed empirically that well regularized MLPs outperform recent tabular data specific architectures as well as classical ML methods on tabular data classification tasks. In particular, an array of regularizes was studied experimentally, and a principled choice of methods and hyperparameters leads to  high performing MLPs.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper makes a compelling case that MLPs have potential to outperform domain specific architectures. The results and figures presented show careful assessment of the experimental design. While there is not much motivation provided, the fact that MLPs  are capable, albeit with some help, to beat older methods is maybe not surprising, but the fact that a well-regularized MLP can outperform TabNet and NODE is interesting. It begs the question: Could the TabNet and NODE be regularized, or otherwise have their hyperparameters tuned and perform better yet?\n\nStrengths: Thorough and principled experimental demonstration of the main point of the paper. Informative and legible plots, in particular, Figure 1 is easy to digest. Clear and concise writing style.\n\nWeaknesses: There is little mentioned about why a given \"regularization cocktail\" might be a good combination. Also, the comparison to recent neural architectures is limited in that the recent methods are not as optimized as the MLPs. To the authors' credit, this limitation is noted in the final section of the paper and it may not harm the significance of these findings.\n\nIn summary, the main finding is important for the community to understand, even if it is not wildly surprising. Intuition and explanation behind the best performing combinations of regularizers would be nice.\n\n___ \nReaction to authors' response: I went back to reread existing work on NNs for tabular data. It was correctly pointed out by the authors that even works like TabNet (Arik and Pfister 2020) and TabTransformer (Huang et al. 2020) do not make an entirely compelling case that NNs outperform XGBoost. For that reason, I think additional empirical works like this paper are valuable to the community.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, both the \"Limitations\" and \"Societal Impact\" are adequate.", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "DvCJWK8Nxh", "reviewer": "Reviewer_9cio", "summary": "This paper aims to study how regularizing basic MLP models can make them as effective as the best available models for tabular data (GBDT and more sophisticated/recent NN architectures like TabNet/NODE). The main contribution is to propose to search among 13 regularizers to find a combination that maximizes validation score, where the search is done via standard hyperparameter optimization techniques, and they find the resulting regularized MLPs outperform GBDT and more recent NN architectures across many tabular datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Through a careful empirical study, this paper provides a number of important results with practical significance. Namely that MLPs can outperform XGB across a wide dataset range and that \nIn particular, the conclusion that \"well-regularized simple deep MLPs outperform specialized neural architectures\" is an important one for the community to be informed of. However the main weakness of this paper in my eyes is it does not actually answer the question of how I should use MLPs for tabular data to get the best performance. This is because the authors did not consider how to split a limited HPO budget into tuning regularizers vs other hyperparameters like the network architecture, learning rates, etc. I also still have questions regarding the details of baselines in the empirical  comparisons.\n\n\n- What makes these \"regularization cocktails\" different than just a hyperparameter-tuned MLP?\nIt seems the authors are considering implicit bias aspects of the NN as \"regularization\" as well, in which case wouldn't also the activation function and other hyperparameters that affect implicit bias also fit this definition of regularization?\nIn fact the authors say their fixed hyperparameters \"have been tuned for maximizing the performance of an unregularized neural network on our dataset collection\"; more details about this process would be nice to see. \nWhy not tune hyperparameters like number of layers, learning rate, hidden layer size, etc? Is there some practical advantage to tuning only the regularization cocktail in MLP+C vs other hyperparameters? Given a limited HPO budget, it's still not clear to me how I should design a MLP search space and whether I should only search over regularizers or also the hyperparameters I just mentioned.\n\n\n- Tuning num_round as a hyperparameter of XGBoost is a bit odd. Normally one just sets this large and  relies on early-stopping. Tuning num_round might lead to the effective number of meaningful XGB HPO trials completed within the budget much smaller than for other models. The authors state they also tried XGB + ES, but did not clarify precisely what was done there. In particular did you remove num_round from the search space and if so to what value was it set?\n\n\n- The authors are missing reference to SeLU paper:\n\nSelf-Normalizing Neural Networks\nKlambauer et al. (2017)\nhttps://arxiv.org/abs/1706.02515\n\nAnd an empirical comparison against this technique, which seemed to also perform well on tabular data in the original paper.\n\n\n- More details of MLP+D baseline would be good to have. Eg. Was the DO-active hyperparameter fixed = True and the same number of HPO trials / total compute run for this baseline?\n\n\n- Is multifidelity HPO method like BOHB able to accurately assess effect of something like Dropout? Wouldn't dropout appear to be harmful in the early stages of MLP training since it slows down the optimization (although allowing the optimization to eventually reach a better-generalizing solution)?\n\n\n- Table 1 caption should include references for each regularization method. Some are not fully clear in their details such as \"Multi-branch choice\".\n\n\n- If the authors claim \"MLP with Regularization is all you Need\", then why only compare against the NN model in AutoGluon Tabular, why not compare against the whole automl system which as far as I know combines NN with GBDT models?\n\n- The authors compared against AutoGluon without HPO and with only NN models, claiming its stacking is superior to HPO. However to me, it seems stacking is not nearly as useful with only NN models and is intended for settings where many different models are trained. It would be good to see AutoGluon + HPO as well if only its NN models are being used.\n\n\n- It would be good to compare plot the performance vs number of HPO trials on a dataset where MLP+C is better than XGB for both methods. Is it the case the XGB performance plateaus after certain number of HPO trials, while MLP+C performance continues to increase with additional HPO?  If so, this could indicate the XGB hyperparameter search space is too small relative to the allotted compute/runtime.\nRelatedly it would be good to know how sensitive the results are to the specific compute/runtime limits chosen for on all methods here.\n\n\n- Were any of the regularizers never selected in the cocktail? These could be helpful to point out.\n\n\n- It would be interesting if the authors could investigate what characteristics of a dataset make it such that MLP+C >>> XGB versus the reverse. For example XGB crushes MLP on the ldpa dataset, and is crushed by MLP+C on arrhythmia. \n\n\n- It may also be interesting to produce a figure like sklearn's classifier comparison:\nhttps://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html\nshowing the effects of each regularizer on the MLP decision boundary on a toy 2d dataset. \n\n\n### Update after reading author response to my original review ###\nI find the paper stronger after the additional experiments the authors have run and have slightly increased my score. That said, I still find this a borderline paper. The biggest remaining concerns I have that do not seem to have been addressed in the authors' response include:\n\n- the title needs to be changed in my opinion. It's clearly not close to \"all you need\" if the authors acknowledge autogluon with all of its different models will be stronger than just their regularized-MLP. \n\n- the authors should more clearly explain what criteria they use to decide whether a hyperparameter is regularization-related or not. Again, to me almost every hyperparameter affects inductive bias in some way. There should be a more systematic explanation for why these particular hyperparameters were considered in the search space but not other standard hyperparameters that are typically tuned in MLPs.\n\n- the early-stopping experiment of XGBoost with num_round = 4000 could still be made more convincing. In particular, num_round could be set to a much greater value no, such that early-stopping is guaranteed to hit?  I feel at the very least num_round should be set to the maximal value such that XGBoost training runtimes do not exceed the MLP training runtimes.\n\n- autogluon with just its MLP model should be run with HPO. I believe the autogluon recommendation of \"use stacking instead of HPO\" only applies when the full system with all of its models are being utilized. If you are only utilizing MLP models, then I believe HPO would outperform stacking.  To reiterate: stacking seems a bit odd to me when the ensemble only contains one type of model as in your usage.\n\nThat said, I believe the authors can easily address these in a revision, and I do think this is a high-impact paper with an important conclusion for the ML community. Papers like this are difficult to write and sometimes harshly judged since there are uncountably many comparisons/analyses that could've been run, but I think the choices made by the authors appear reasonable to me and sufficient to convincingly present some important conclusions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe the authors have adequately addressed societal impact & limitations of their work.\nOne unaddressed limitation is that given limited training time, it's unclear whether training both MLP+D and XGBoost followed by model-selection would be competitive to MLP+C (since XGBoost is presumably much quicker to train and both MLP+D and XGBoost presumably require less HPO trials than MLP+C). Also the paper didn't consider more advanced boosting variants like CatBoost and used questionable label-encoding of categoricals for XGBoost.\n", "review_text": "", "extra_scores": {"ethical_concerns": "I don't have remaining omitted ethical concerns regarding this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found this paper somewhat "unsurprising", but potentially high-impact and useful to the community. They also noted that papers like this one can be judged harshly, subjecting the authors to a potentially infinite number of experimental comparison. In this case, the consensus was that the revisions/responses from the authors did a good job addressing the concerns of the reviewers. We recommend adopting the suggestions arising from the discussion phase, particularly around the choice of title. 
Decision: Accept (Poster)