Regularization Cocktails for Tabular Datasets
Authors: Arlind Kadra
Marius Lindauer
Frank Hutter
Josif Grabocka
Venue:  (2021)
Keywords: deep learning
regularization
hyperparameter optimization
benchmarks.

Abstract: The regularization of prediction models is arguably the most crucial ingredient that allows Machine Learning solutions to generalize well on unseen data. Several types of regularization are popular in the Deep Learning community (e.g., weight decay, drop-out, early stopping, etc.), but so far these are selected on an ad-hoc basis, and there is no systematic study as to how different regularizers should be combined into the best “cocktail”. In this paper, we fill this gap, by considering the cocktails of 13 different regularization methods and framing the question of how to best combine them as a standard hyperparameter optimization problem. We perform a large-scale empirical study on 40 tabular datasets, concluding that, firstly, regularization cocktails substantially outperform individual regularization methods, even if the hyperparameters of the latter are carefully tuned; secondly, the optimal regularization cocktail depends on the dataset; and thirdly, regularization cocktails yield the state-of-the-art in classifying tabular datasets by outperforming Gradient-Boosted Decision Trees.

Under review as a conference paper at ICLR 2021
REGULARIZATION COCKTAILS FOR TABULAR DATASETS
Anonymous authors Paper under double-blind review
ABSTRACT
The regularization of prediction models is arguably the most crucial ingredient that allows Machine Learning solutions to generalize well on unseen data. Several types of regularization are popular in the Deep Learning community (e.g., weight decay, drop-out, early stopping, etc.), but so far these are selected on an ad-hoc basis, and there is no systematic study as how different regularizers should be combined into the best "cocktail". In this paper, we fill this gap, by considering the cocktails of 13 different regularization methods and framing the question of how to best combine them as a standard hyperparameter optimization problem. We per- form a large-scale empirical study on 40 tabular datasets, concluding that, firstly, regularization cocktails substantially outperform individual regularization meth- ods, even if the hyperparameters of the latter are carefully tuned; secondly, the optimal regularization cocktail depends the dataset; and thirdly, regularization cocktails yield the state-of-the-ar in classifying tabular datasets by outperforming Gradient-Boosted Decision Trees.
INTRODUCTION
In most supervised learning application domains, the available data for training predictive models is both limited and noisy with respect to the target variable. Therefore, is paramount to regularize machine learning models for generalizing the predictive performance on future unseen data. The concept of regularization is well-studied and constitutes one of the pillar components of machine learning. Throughout this work we use the term "regularization" for all methods that explicitly or im- plicitly take measures that reduce the overfitting phenomenon; we categorize these non-exhaustively into weight decay, data augmentation, model averaging, structure and linearization, and implicit reg- ularization families (detailed in Section 2). In this paper, we propose a new principled strategy that highlights the need for automatically learning the optimal combination of regularizers, denoted as regularization cocktails, via a hyperparameter optimization procedure.
Truth be told, combining regularization methods is of course far from being novel practice per As a matter of fact, most modern deep learning models use combinations of a number of regularizers. For instance, EfficientNet (Tan & Q.Lee, 2019} mixes components of structural regularization and lincarization via ResNet-style skip connections (He et al., (2066), learning rate scheduling, Drop- Out ensembling (Srivastava et al. 2014) and AutoAugment data augmentation (Cubuk al. 2019 However, even though each of those regularizers is motivated in isolation, the reasoning behind a specific combination of regularizers largely based on accuracy-driven manual trdal-and-error iterations, mostly on image classification benchmarks like CIFAR (Krizhevsky et al.) 2009 and ImageNet (Deng et al. 2009).
Unfortunately, the manual search for combinations of regularizers is sub-optimal, unsustainable, and essence consists of an example of manual hyperparameter tuning, which in turn easily outperformed by automated algorithms (Snock et al. 2012 Thorton et al.) 2013 Feurer et al. 2015 Olson & Moore 2016, et al., 2019: Erickson et al. 2020 Zimmer et al. (020).
Following the spirit of AutoML (Hutter et al. 2018), the optimal dataset-specifii regularization cocktail by means of a modern hyperparameter optimiza- tion (HPO) method. To the best of our knowledge, there exists study providing empirical evidence that a mixture of numerous regularizers outperforms individual regularizers, this paper fills this gap. More precisely, the research hypothesis of this paper is that properly mixed regularization cocktail
therefore, propose strategy for learning

---

Under review as a conference paper at ICLR 2021
outperforms every individual regularizer in it, in terms of accuracy under the same run-time budget, and that the best cocktail to use depends on the dataset. To validate this hypothesis, we executed a large-scale experimental study employing 40 diverse tabular datasets and 13 prominent regularizers with a thorough hyperparameter tuning for all regularizers. We focus on tabular datasets because, in contrast to large image datasets, thorough hyper-parameter search procedure is feasible. Neural networks are high variance models for tabular datasets, therefore improved regularization schemes can provide a relatively higher generalization gain on tabular datasets compared to other data types.
Thereby, we make the followings contributions:
1. We demonstrate the empirical accuracy gains of regularization cocktails in a systematic manner via a large-scale experimental study on tabular datasets;
2. We challenge the status-quo practices of designing universal dataset-agnostic regularizers, by showing that an optimal regularization cocktail is highly dataset-dependent:
3. We demonstrate that regularization cocktails achieve state-of-the-art classification accuracy performance on tabular datasets and outperform Sradient-Boossee Decision Trees (GBDT) with a
margin;
4. As an overarching contribution, this paper provides previously in-depth empiri- cal evidence better understand the importance of combining different mechanisms for regularization, one of the most fundamental concepts in machine learning.
2 RELATED WORK
The classical approaches of regularization focused on minimizing the norms of the parameter values, concretely either the LI (Tibshirani] 1996). the L2 (Tikhonov) 1943). or a combination of LI and L2 known as the Elastic Net (Zou & Hastie 2005). recent work fixes the malpractice of adding the decay penalty term before momentum based adaptive learning rate steps (e.g., in common implementations of Adam (Kingma & Ba 2015 ), by decoupling the regularization
Weight decay:
from the loss and applying it after the learning rate computation [Loshchilov & Hutter 2019).
Data Augmentation: A different treatment of the overfitting phenomenon relies on enriching the training dataset via instance augmentation. The literature on data augmentation is vast, especially for image data, ranging from basic image manipulations (c.g., geometric transformations, mixing images) up to parametric augmentation strategies such as adversarial and controller-based meth- ods (Shorten & Khoshgoftaar 2019). For example, Cut-Out (Devries & TTylorr 2017) proposes to mask subset of input features (e.g., pixel patches for images) for ensuring that the predictions remain invariant distortions in the input space. Along similar lines, Mix-Up (Zhang et al. 2018) generates new instances as a linear span of pairs of training examples, while Cut-Mix (Yun et al. 2 suggests super-pon super-positions of instance pairs with nutually-exclusivv pixel masks. recent tech- nique, called Aug-Mix (Hendrycks et al. 2020), generates instances by sampling chains of augmen- tation operations. On the other hand, the direction of reinforcement learning (RI.) for augmentation policies was elaborated by Auto-A Auto-Augment (Cubuk et al. 2019), followed by technique that speeds up the training of the RL policy (S.Lim et al. 2019).. Last but not least, adversarial attack strategies (e.g., FGSM (Gooddellow et al. 2015)) generate synthetic examples with minimal perturbations, which are employed in training robust models (Madry et al. 2018).
Model Averaging: Ensembled machine learning models have been shown to reduce variance and act as regularizers (Polikar 2012). A popular ensemble neural network with shared weights among its base models is Drop-Out (Srivastava et al. 2014), which was extended to a variational version with a Gaussian posterior of the model parameters (Kingma et al. 2015). A follow-up work that is known as Mix-Out (Lee et al., 2020) extends Drop- Drop-Out by statistically fusing the parameters of two base models. Furthermore, ensembles can be created using models from the local optima discovered along a single convergence procedure (Huang et al. 2016).
Structural and Linearization: One strategy of regularizing deep learning models is to discover dedicated neural structures that generalize on particular tasks, such as image classification or Nat- ural Language Processing (NLP). In that context, ResNet adds skip connections across layers (He) jet al. 2016). while the Inception model computes latent representations by aggregating diverse con- volutional filter sizes (Szegedy et al. 2017). The attention mechanism gave rise to the popular
2

---

Under review as a conference paper at ICLR 2021
Transformer architecture in the realm of NLP (Vaswani et al. 2017). Recently, EfficientNet is an architecture that easily scales deep convolutional neural networks by controlling only a few hyper- parameters (Tan & Q.Le 2019).
Besides the aforementioned manually designedd architectures, the stream of Neural Architecture Search (Elsken et al. 2019) focuses on exploring neural connectivity graphs for finding the optimal architectures via reinforcement learning (Zoph & Le 2017). black-box search (Real et al.] 2019) or differentiable solvers (Liu al. 2019). A recent trend adds dosage of linearization to deep models, where skip connections transfer embeddings from previous less non-linear layers (He et al., 2016 Huang al., 2017). Along similar lines, the Shake-Sha Shake regularization deploys skip con- nections in parallel convolutional blocks and aggregates the parallel representations through affine combinations (Gastaldi 2017). while Shake- Shake-Drop extends this mechanism to a larger number of CNN architectures (Yamada et al. 2018).
The last family of regularizers broadly encapsulates methods which do not directly pro- pose novel regularization techniques but have an implicit regularization effect as a virtue of their 'modus operandi' (Arora et al. 2019). For instance, Batch Normalization improves generalization by reducing the internal covariate shifts (loffe & Szegedy) 2015). while early stopping of the opti- mization procedure also yields similar generalization effect (Yao et al. 2007). On the other hand, stabilizing the convergence of the training routine another implicit regularization for instance by introducing learning rate scheduling schemes (Loshchilov & Hutter 2017). The recent strategy of stochastic weight averaging relies on averaging parameter values from the local optima encountered along the sequence of optimization steps (lzmailov et al. (018)), while another approach conducts updates in the direction of few 'lookahead' steps (Zhang et al. 2019
Implicit:
Positioning in the realm AutoML: In contrast to the prior literature, we not propose new individual regularization method, but empirically identify the superiority of learning regularization cocktails among a set of existing regularizers from the aforementioned categories. We train dataset- specific cocktails regard, our work is positioned in the realm of AutoML and is a special case of a combined algo- rithm selection and hyperparameter optimization (Thornton et al. 2013). We learn the regularization cocktails and optimize the joint hyperparameter configuration space by means of BOHB (Falkner) cet al.) 2018b), which is a variation of Hyperband (Li al. 2017) with model-based surrogates and is one of the current state of the art approaches for efficient HPO.
hyperparameter optimization (HPO) task (Feurer & Hutter 2019). In that
3 MIXING THE REGULARIZATION COCKTAIL
3.1 THE RECIPE AS PROBLEM STATEMENT
training set is composed of features X Train) and targets (Traii)), while the test dataset is denoted by (Test)) y(Tss)) A parameterized function approximates the targets as ÿ = f(X; whereas the parameters are trained to minimize a differentiable loss function as:
0 E arg min L (y (Trai)) f (Train) ;0))
(I)
To generalize into minimizing L (y ( (Te())));; 0). the parameters of f are controlled with a regularization technique J2 that avoids overfitting to the peculiarities of the training data. With slight abuse of notation we denote f (X; Ω (0: X)) to be the predictions of the model f whose parameters are optimized under the regime of the regularization method 20:; where À A represent the hyperparameters of Ω. The training data is further divided into two subsets as training and validation splits[ the later denoted by X (Val), such that À can be tuned on the validation loss via the following hyperparameter optimization objective:
x
arg min AEA
L (yy(Val) (Va), S (0": :)))
(2)
A* E arg min L (y (Tain) (Tainn), (0) A))
S.t.
'oor simplicity, we only discuss hold-out validation scheme here, but in principle any other validation scheme, such as cross validation and bootstrap sampling, would be possihle.
3

---

Under review as a conference paper at ICLR 2021
While the search for optimal hyperparameters À is active field of research in the realm of AutoML (Hutter et al., 2018), still the choice of the regularizer Q mostly remains an ad-hoc practice, where practitioners select few combinations among popular regularizers (Dropout, L2, Batch Normalization, etc.). In contrast to prior studies, we hypothesize that the optimal regular- izer is cocktail mixture of a large plied with different strengths (i.e., datasct-specific hyperparameters). Given a set of K regularizers {(220) (=>())) ] {2(1) (=:(0) ...... ((()) (-: (
of regularization methods, all being simultaneously ap-
each with its own hyperparameters ( E A (R), vk f1. fl...... K}. the problem of finding the optimal cocktail of regularizers is:
C yynaa f x {2(UA) (0r.11))
K
X E (0) A00)
arg min
(4)
k=1
S.L. 0" arg min L C (Traii)) X(Tramn) {D())) (0, K
(5)
The intuitive interpretation of Equations strengths) of the cocktail's regularizers using the validation set (Equation given that the optimal prediction model parameters 0 are trained under the regime of all the regularizers being applied jointly (Equation 5). We stress that the hyperparameters controlling whether the k-th regularizer applied at all, or skipped. Therefore, the best cocktail might consist of combinations of subset of regularizers.
is searching for the optimal hyperparameters À (i.e.,
(k) include conditional hyperparameter
3.2 REGULARIZATION INGREDIENTS AND THE SEARCH SPACE
To build the regularization cocktails we combine the 13 methods shown in Table [ which are se- lected among the categories of regularizers covered in Section 2 each having its own hyperparam- eter search space. We set the other hyperparameters regarding the architecture and the optimizer as detailed in Table 2 in Appendix [. The regularization cocktails introduce 9 non-conditional hyperparameters in the search space, which, in turn, can add up to 9 conditional hyperparameters. In total, our regularization cocktails can add up to 18 hyperparameters in the search space.
In the defined search space some of the combinations are not technically feasible, therefore, we introduce the following constraints to the proposed search space: (i) Shake-Shake and Shake-Drop are not simultaneously active since the latter builds on the former. (ii) Only one data augmentation technique out of Mix-Up, Cut-Mix, Cut-Out, and FGSM adversarial learning can be active at once due to a technical limitation of the base library (Zimmer et al. 2020).
As an optimizer, we decided to use BOHB (Falkner et al. 2018a) since it achieves a strong any- time performance by combining Hyperband (Li et al.) 2017 and Bayesian Optimization (Shahriari eet al. (2166), and still has the convergence guarantees of Hyperband. Furthermore, BOHB can deal with the categorical hyperparameters for enabling or disabling regularization techniques and the corresponding conditional structures. In Appendix A we provide a brief description of how BOHB
works.
EXPERIMENTAL PROTOCOL
4.1 EXPERIMENTAL SETUP
We use a collection of tabular datasets (listed in Table 4 in Appendix D from the recent open- source OpenML AutoML Benchmark (Gijsbers et al. 2019), as well as popular online repositories, such as UCI (Asuncion & Newman 2007) and Kaggle. Our benchmark of 40 datasets includes tabular datasets that represent diverse classification problems, containing between 452 and 416188 instances, and between 4 and 2001 features, varying in terms of the number of numerical and cat- egorical features. The datasets are retrieved from OpenML (Vanschoren et al. 2014) and split as 60% training, 20% validation and 20% testing sets. All data is standardized unit variance. For datasets with missing values, the median value on the known values is used as an imputation strategy.
have O mean and
We ran all experiments on a CPU cluster, each node of which contains two Intel Xeon E5-2630v4 2.2GHz with 20 CPU cores and total memory of 128GB. We chose the PyTorch library (Paszke
4

---

Under review as a conference paper at ICLR 2021
Regularizer
Hyperparameter
Type
Range
Conditionality
BN
BN-active
Boolean
(True, False)
WD-active
Boolean
(True, False)
WD
Decay factor
Continuous
[10-9, 1009,0.11 0.1]
WD-active
DO-active
Boolean
(True, False)
(funnel, long funnel, diamond, hexagon, hrick, triangle, stains) [0.0, 0.8)
DO
Dropout shape
Nominal
DO-active
Drop rale
Continuous
DO-active
SC-active
Boolean
(True, False)
SC
MB choice
Nominal
{SS, SD, Standard)
SC-active
SD
Max. probability
Continuous
[0.0, 1.0)
SC-active MB choice SD
LA-active
Boolean
(True, False) [0.5, 0.8)
LA
Step size
Continuous
I-Aactivee LA-active
Num. steps
Integer
[1,
SWA
SWA-active
Boolean
(True, False)
SE-active
Boolean
(TTuue, False)
SE
Num. models
Constant
{3}
SE-active
Augment
Nominal
MU CM, CO, AT, None)
MU
Mix. magnitude
Continuous
[0.0, 1.0)
Augment MU
CM
Probability
Continuous
[0.0, 1.0)
Augment CM
Probability Putch ratio
Continuous
[0.0, 1.0)
Augment = CO
CO
Continuous
[0.0, 1.0)
Augment CO
Table The configuration space for the regularization cocktail regarding the explicit regularization hyperparameters of the methods and the conditional constraints enabling or disabling them. (BN: Batch Normalization, WD: Weight Decay, DO: Drop-Out, SC: Skip Connection, MB: Multi-branch choice, SS: Shake-Shake, SD: Shake-Drop, LA: Lookahead Optimizer, SWA: Stochastic Weight Averaging, SE: Snapshot Ensembles, MU: Mix-Up, CM: Cut-Mix, CO: Cut-Out, and AT: FGSM Adversarial Learning)
ctt al. 2019) as the main deep learning framework for our work and we extended the AutoDL- framework Auto-Pytorch (Mendoza al. 2018 Zimmer et al. 2020) with our implementations for the regularizers, as shown in Table
To optimally utilize resources, ran BOHB with 10 workers in parallel, where each worker had access to 2 CPU cores and 12GB memory, executing one configuration at a time. In view of limited computational resources and taking into account the dimensions D of the considered config- uration spaces, we ran BOHB for at most 4 days, or at most 40 x D hyperparameter configurations, whichever came first. During the training phase, each configuration sake of studying the effect on more datasets, we only evaluated single train-val-test split. After the training phase is completed, we report the results the best hyperparameter configuration found retrained on the joint train and validation set for 105 epochs.
for 105 epochs. For the
4.2 FIXED ARCHITECTURE AND OPTIMIZATION HYPERPARAMETERS
In order to focus exclusively on investigating the effect of individual regularization methods, we fix the hyperparameters that are related to the model architecture and general training procedure in the search-space, as specified in Table 2 of Appendix B.] These hyperparameter values are tuned for maximizing the performance of an unregularized neural network on dataset collection (see Table in Appendix D). Moreover, we use a 9-layer feed-forward neural network with 512 units for each layer, choice based previous related work (Orhan & Pitkow 2017). We emphasize that the network has a sufficiently large capacity, to ensure that the effect of regularization methods would be noticeable.
5

---

Under review as a conference paper at ICLR 2021
Moreover, low learning rate of 10-3 after performing grid search for finding the value performing best across all datasets. We use the AdamW implementation (Loshchilov & Hutter, 2019), which implements decoupled weight decay, and cosine annealing with restarts (Loshchilov & Hutter, 2019) as a learning rate scheduler. Using a learning rate scheduler with restarts helps in keep fixed initial learning rate. For the restarts, epochs, with a budget multiplier 3, following published practices (Zimmer et al., (020). Addition- ally, since our benchmark includes imbalanced datasets, we use weighted version of categorical cross-entropy and balanced accuracy (Brodersen et al. 2010) as the evaluation metric.
case because
initial budget of 15
4.3 HYPOTHESES AND ASSOCIATED EXPERIMENTS
Hypothesis 1:
The regularization cocktails achieve better generalization performance compared to the individual regularization methods over all datasets.
Experiment 1: We regularize the plain neural network (Section 4.2) with each method from Ta- ble [ one at time. For every regularizer, we tune its hyperparameters on each dataset, then finally measure the regularized network's performance ing the best dataset, compare the results against the results of cocktail optimized the same way as each of the individual ingredient regularizers.
the test set after retrain- configuration on the joint train and validation set. For each
Hypothesis 2: The optimal regularization cocktails are datasct-dependent.
Experiment 2:
We study the best-found regularization cocktails of every dataset and frequencies of the regularizers that were chosen to be activated by BOHB, to demonstrate that no com- bination of regularizers is frequent. Furthermore, we regularize the plain network with the most frequent regularizers and compare it against proposed method of Section 3
Hypothesis 3:
The regularization cocktails achieve state-of-the-arr classification accuracies in tab- ular datasets.
Experiment 3: We compare against GBDT. the state-of-the-ar classifier for tabular data. For fair comparison, we optimized the hyper-parameters of GBDT on every dataset using the pop- ular AutoSklear library, by following the exact hyper-paramette search protocol (same train, validation, and test splits) and provided GBDT with the same HPO budget our proposed method. The search space for the hyperparameters GBDT further detailed in Section 5
EXPERIMENTAL RESULTS
Ranks
15
14
12
11
10
8
7
3
2
Shake-Shake
1,2755
Cocktail
Stochastic Weight Averaging
1.8500
5.2375
CutMix
Adversarial Training
6,790
MixUp
Plain network
6.7250
CutOut
Lookahead
8.5750
5.9 5.9875
Dropout
Weight Decay
2000
7. 7,22
Shake-Drop
Skip Connection
3000
Batch Normalization
Snapshot Ensembling 26265
Figure 1: Critical difference diagram generated with the Wilcoxon- post-hoc analysis on 40 datasets. The diagram shows the ranks and the statistical significance the results for every individual regularization technique and our regularization cocktails.
Regularization cocktail performance (Experiment 1): Figure presents the critical difference diagram of the ranks, which demonstrates that the cocktail outperforms each individual regular- izer. The critical difference diagram is generated by performing a posthoc analysis based on the Wilcoxon-Holm method (Wilcoxon) 1992 Holm 1979) with p value of 0.05 as the threshold for
httpss://uutmm..giin o/aato- sklearn

---

Under review as a conference paper at ICLR 2021
statistical significance. Observing the results, the regularization cocktail manages to outperform all individual cocktail ingredients with a statistically significant margin. This confirms our hypothesis that well-tuned regularization cocktails outperform well-tuned individual regularization techniques across a diverse suite of tabular datasets. In addition, Figure additional information on the rank distributions of the different compared methods, while Figure Appendix [ Cfffers detailed descriptive statistics for each one-on-one comparison against basclines.
Ranks individual regularizers and the cocktail
12.5
10.0
Rank
7.5
5.0
2.5
0.0
PN
BN
LA
SE
SWA
SC
AT
SS
SD
MU
CO
CM
WD
DO Cocktail
Figure 2: Rank distribution for the individual regularization methods and the regularization cocktail. The rank distribution for each method is calculated on the over all datasets.
Pataset-dependent optimal cocktails (Experiment 2): Figure shows that the optimal regular- ization cocktail depends on the dataset at hand since no combination of regularizers was active on the majority the datasets. The plot depicts all frequent singular regularizers and combinations of pairs of regularizers occurring in at least 30% of the datasets, based on how often they were part of the per-dataset cocktails. The most frequent pair of regularizers (BN SE) selected only on 50% of the datasets, which highlights the fact that regularization cocktails are dataset-specific and there is no frequent universal combination.
Grouped Cocktail Ingredients
100
Frequencies (%)
80
60
SE
BN
WD
CM
DO
LA
CO
40
72.5%
20
WD 40%
BN 37.5%
Batch Normalization
Dropout
Weight Decay
Data Augmmentation
Structural
Weight aggregation
50%
Figure 3: Left: Individual and pairwise cocktail ingredients occurring in at least 30% of the datasets. Right: Clustered histogram of cocktail ingredients. Data Augmentation: (CutMix, Cutout, Mixup, Adversarial Training). Structural: (Skip connection, Shake-Shake, Shake-Drop). Weight aggrega- tion: [Lookahead Optimizer, Stochastic Weight Averaging, Snapshot Ensembling).
Moreover, the results presented in FigureBprovide insights into frequent cocktail ingredients with regards to the regularization types. For instance, although Snapshot Ensembling individual

---

Under review as a conference paper at ICLR 2021
method ranks 7-th among 13 regularizers in Figure 1 it is nevertheless present in the regulariza- tion cocktails of 72.5% datasets. The finding hints that optimal cocktails are composed of weaker regularizers, whose dataset-dependen combination enhances the regularization effect. For more in-depth summary the frequencies that correspond to the individual regularization methods, we refer Figure 7 in Appendix
Our Dataset-specific vs. Top-5 Most Frequent (22 wins, 15 losses, 3 draws, p-value: 0.08143)
1.050
1.050
1.050
Gain
Gain
Gain
1.025
1.025
1.025
1.000
1.000
1.000
10
10
10
10
10
10
1.0
Number examples
Number of features
Test accuracy
Our Dataset-specific vs. Top-5 Highest Ranks (32 wins, 6 losses, 2 draws, p-value: 0.00004)
1.2
1.2
Gain
1.1
Gain
1.1
Gain
1.1
1.0
1.0
1.0
10"
10"
10
10
10"
10"
0.4
0.6
0.8
1.0
Number examples
Number of features
Test accuracy
Figure 4: Comparison of our proposed dataset-specific cocktail against the cocktail of the top-5 most frequent regularizers (top row), and the cocktail of the top-5 regularizers with the highest performance ranks (bottom row). Each point represents a dataset and the gain is defined as the test accuracy of our method divided by the test accuracy of each baseline. We illustrate the gain with three ablations: the number of samples (left), number of features (middle) and test accuracy (right).
Lastly, further validate that good performing regularization cocktails dataset-dependent, conducted another experiment by creating 2 baselines consisting of the following top-5 cocktails:
1. The top-5 most frequent regularizers of Experiment 2 (Snapshot Ensembling, Batch Nor- malization, Dropout, Weight Decay, and dataset-specific augmentation):
2. The top-5 regularizers with the highest ranks from Experiment (Dropout, Shake-Droo Batch Normalization, Snapshot Ensembling, and dataset-specific augmentation).
In both top-5 cocktails "dataset-specific augmentation" signifies having data augmentation activated, however, the choice between CutMix, CutOut, and Mixup dataset-specific and tuned during the HPO process. This design decision taken to make the baselines even more competitive. The aforementioned regularizers in the top-5 baselines are always applied jointly (i.c. no subset those methods are selected per dataset basis), however, tune the hyper-parameters of all regulariz- ers in each top-5 baseline jointly for each dataset. We observe that both top-5 baselines underperform against our proposed dataset-specific cocktail indicated in Figure 4 Additionally, we measured the statistical significance between the top-5 baselines and our method using the Wilcoxon signed- rank test a 10% significance level. For the top-5 highest ranks variant, the result confirms that the difference is significant with p-value of 0.00004. Similarly, the results show significance against the top-5 most frequent variant with p-value of 0.08143. For detailed summary of all the results for every method we refer to Appendix D Table
Regularization cocktails achieve state-of-the-art classification accuracy in tabular datasets (Experiment 3): To investigate whether the regularization cocktails achieve ofte-offtheeart clas- sification accuracy, we compare our method against the GBDT method, which is the de-facto state- of-the-art in tabular datasets. The results, presented in Figure 5 show the superiority of the
8

---

Under review as a conference paper at ICLR 2021
regularization cocktails in terms of the predictive performance under the same time and resource constraints compared to the GBDT algorithm. We used the GBDT implementation of Auto- aato-Skleam, GBDT
popular automated tool in the realm of AutoML. For ensuring fair comparison with the same setup (same training, validation, testing splits) and the same hyper-parameter search experiments with the same computational hardware as for the cocktail. More details on the experimental setup are presented in Appendix B.2
time. In addition,
Wins: 28, Losses: 12, Draws: 0 p-value: 0.0003
2.0
GBDT wina
Reg Cocktail wins
Frequencies
Gain
1.0
0.92
0.96
1.00
1.04
1.08
1.12
1.16
1.20
1.24
1.28
10
10"
10"
Gain over GBDT
Number examples
Cocktail gain over the GBDT algorithm calculated the test set for every dataset (The gain is calculated by dividing the cocktail accuracy with the GBDT accuracy). Right: The distribution the cocktail gain.
Figure 5: Left:
The regularization cocktails achieve higher accuracies compared to GBDT on 28 of 40 datasets (70% win ratio) and the difference is statistically significant. Figure 5 further illustrates that the performance gain of the regularization cocktails is invariant the dataset size. In the left subplot of Figure 5 we observe that our method outperforms GBDT in both small and large datasets. Fur- thermore, the right subplot of F depicts the fact that our gain is not marginal and in certain datasets, we achieve to 30% increase in test accuracy. The full per-dataset accuracies of GBDT are found Appendix DJ Table Lastly, we computed the statistical significance between the cock- tail and GBDT using the Wilcoxon signed-rank test, which resulted in a p-value of 0.0003. Based on the empirical results, we conclude that the regularization cocktails yield state-of-the- prediction models for classifying tabular datasets.
CONCLUSIONS AND FUTURE WORK
Even though combining regularizers is relatively frequent practice by researchers, date, there exists no prior work that systematically studies the effect of optimally combining regularization methods. This paper presented the first step in empirically studying regularization cocktails, by pos- ing the problem as standard hyperparameter optimization challenge. We conducted a large-scale experiment involving 13 regularization methods and datasets, with thorough hyperparameter optimization procedure for each technique. The findings of this study can be summarized as three simple take-home messages for practitioners:
Instead of applying a single regularization technique, we recommend exploiting the com- plementary effects of regularization cocktails.
2. To make neural networks achieve -fte-of-theeart classification accuracy in classifying tab- ular datasets the regime regularization cocktails should applied.
3. To obtain a well perfooerfor,ing. dataset-specific regularization cocktail, using state-of-the-ar hyperparameter optimization techniques is recommended.
As future work, would like to combine regularization cocktails for neural networks with auto- mated data preprocessing pipelines and architecture search, order to advance the performance gain of deep learning small tabular data.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Arora, Cohen, Hu, and Luo. Implicit regularization deep matrix factorization. H. Wallach, H. Larochelle, A. Beygelzimer, F. Alche Buc, E. Fox, and R. Gameett (eds.), Advances Neural Informa- tion Processing Systems 32, pp. 7413-7424. Curran Associates, Inc., 2019.
A. Asuncion and D. Newman. Uci machine learning repository, 2007.
K. Henning Brodersen, C., and E. Klaas Stephan J. M Buhmann. The balanced accuracy and its posterior distribution. In 2010 20th International Conference Pattern Recognition, pp. 3121-3124, IEEE, 2010.
E. Cubuk, B. Zoph, D. Mane, and Vasudevvaaadd Q. Le. Autoaugment: from data. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Learning augmentation strategies
J. Deng, W. Dong, R. Socher, L.. Li, K. Li, and L.. Fei-Fei. ImageNct: Database. CVPR09, 2009.
Large-Seale Hicrarchical Image
Devries and G. Taylor. Improved regularization of convolutional neural networks with cutout. ArXiv, abs/1708.00452 2017.
Elsken, J. H. Metzen, and F. Hutter. Neural architecture search: A survey. Journal of Machine Learning Revearch, 20(55):1-21. 2019.
N. Erickson, J. Mueller, A. Shirkov, H. Zhang, P. Larroy, M. Li, and A. Smola. Autogluon-tabuiee Robust and accurate automl for structured data. CoRR, abs/2003 066505 2020.
S. Falkner. Klein, and Hutter. BOHB: Robust and efficient hyperparameter optimization at scalae. In Proceedings of the 35th International Conference Machine Learning (ICML. 2018), pp. 1436-1445, July 2018a.
Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. J. Dy and A. Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1437-1446, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018b. PMLR.
M. Feurer and F. Hutter. Hyperparameter optimization. In Hutter, L. Kotthoff, and J. Vanschoren (eds.), Au- tomated Machine Learning Methods, Systems, Challenges, Springer Series on Challenges Machine Learning, pp. 3-33. Springer, 2019. doi: 10. 11007797883-000-053338338
M. Feurer, A. Klein, K. Eggensperger. J. Springenberg. M. Blum, and F. Hutter. Efficient and robust automated machine learning. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 2, NIPS' 15, pp. 2755-2763, Cambridge, MA, USA, 2015. MTT Press.
X. Gastaldi. Shake-shake regularization of 3-branch residual networks. In 5th International Conference Learning Representations, ICLR. OpenReview.net, 2017.
Gijshers, F. LeDell, S. Poirier, J. Thomas, B. Bischl, J. Vanschoren. An open automl henchmark. arXiv preprint arXiy: h0t0:::9900000000000 [cs. 2019. htttt//s://aaav.orr///bbbaa 00909 Ac- cepted at AutoMI. Workshop at ICMI. 2019.
I. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. 2015. URL http://arxis org/ abs 111/21112.66 6572
K. He, X. Zhang.5 S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
D. Hendrycks, N. Mu, E. Cubuk, Zoph, J. Gilmer, and Lakshminarayanan. Augmix: simple method improve robustness and uncortainty under data shift. In International Conference on Learning Representa- tions, 2020. URL
net wttp://foooddsiimemm 1d=51m1rx=1111111
Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pp. 65-70, 1979.
G. Huang, Y. G. Pleiss, Z. Liu, J. Hopcroft, and K. Weinberger. Snapshot Ensembles: Train 1, Get M for Free. International Conference on Learning Representations, November 2016.
G. Huang, 7. Liu, L. van der Maaten, and K. Weinberger. Densely connected convolutional networks. In Proceedings of the IEFF Conference Computer Vision and Pattern Recognition, 2017.
10

---

Under review as a conference paper at ICLR 2021
E. Hutter. L. Kotthoff, and J. Vanschoren (eds.). Automated Machine Learning; Methods, Systems, Challenges. Springer, 2018, press, available http://automm.org/boor
loffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, In F. Bach and D. Blei (eds.). Proceedings of the 32nd International Conference on Machine Learning. volume 37 of Proceedings of Machine Learning Research, pp. 448 448-456, Lille, France, 07-09 Jul 2015. PMIRR
Izmailov, D. Podoprikhin, Garipov, D. Vetrov, and A. Wilson. Averaging weights leads to wider optima and better generalization, 2018. cite arxiv: 1rxiv:1803.055. 050.055477 Appears at the Conference on Uncertainty in Artificial Intelligence (UAI), 2018.
H. Jin, Q- Song, and X. Hu. Auto-keras: An efficient neural architecture search system. In A. Teredesai, V. Kumar, Y. Li, R. Rosales, E. Terzi, and Karypis (eds.). Proceedings of the 25th ACM SIGKDD termational Conference on Knowledge Discovery & Data Mining, KDD, pp. 10. 10.1145/32925003333648
ACM, 2019, doi:
D. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.
D. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparamcterization trick. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 2, NIPS 15, pp. 2573, Cambridge, MA, USA, 2015. MIT Press.
A. Krizhevsky, G. Hinton, al. Learning multiple layers of features from tiny images. 2009.
C. Lec, K. Cho, and Kang. Mixout: Fffective regularization to finetune large-scale pretrained language models. In International Conference on Learning Representations, 2020. URL URR.hhtts:///penttt ex. net bet/forum 2 d=t.idkkttKaa
L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit-based approach hyperparameter optimization. J. Mach. Learn, Res., 18(1)6676568866, January 2017. ISSN 1532-4435.
H. Liu. S. Karen, and Y. Yiming. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019. URL wwllaattss//pppeereeee...t www.eeeeeeoooou....eeeeeeeeooooo
Loshchilov Hutter. Sgdr: Stochastic gradient descent with warm restarts. International Conference on Learning Representations (ICLR) 2017 Conference Truck, April 2017.
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URLL/tttp:://ppeeeoooo...oo http://wwwwwwwtttttoooooe--aa-a-
Madry, A. Makelov, Schmidt, D. Tsipras, and Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. URL httpp::7
openrevies net. forum? 1 d- z. BfZAb
H. Mendoza, A. Klein, M. Feurer, J. Tobias Springenberg, M. Urban, M. Burkart, M. Dippel, M. Lindauer, and Hutter. Towards automatically-tumg deep neural networks. F. Hutter, L. Kotthoff, and J. Vanschoren (eds.), AutoML: Methods, Sytems, Challenges, chapter 7, pp. 141-156. Springer, December 2018. To appear.
R. Olson and J. Moore. TPOT: A tree-based pipeline optimization tool for automating machine learning. In F. Hutter, L. Kotthoff, and J. Vanschoren (eds.), Proceedings of the 2016 Workshop Automatic Machine Learning, AutoML 2016, co-located with 33rd International Conference Machine Learning (ICML 2016), volume of JMLR Workshop and Conference Proceedings, pp. 66-74 74, JMLR.org, org. 2016.
A. Emin Orhan and X. Pitkow. Skip connections eliminate singularities. 2017.
arXiv preprint arXiv:
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen Z. Lin, N. Gimelshein, L. Antiga, al. Pytorch: imperative style, geigh-erformance deep learning library. In Advances in neural informa- tion processing systems, pp. 8026 8037, 2019.
R. Polikar. Ensemble Learning. In C. Zhang and Y. Ma (eds.), Ensemble Machine Learning: Meth- ods and Applications, pp. 1-34. Springer US, Boston, MA, 2012. ISBN 978- 978-1-4419-9926-1 doi: 10. 10.1007/978-1 1-41978---44199926----
11

---

Under review as a conference paper at ICLR 2021
E. Real, A. Aggarwal, Y. Huang, and Q. Le. Regularized evolution for image classifier architecture search.
In The Thirty-Thite Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposiun on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, AAAI Press, 2019. doi: 1a.1669/aaa...333...30 333014780. URL wtt.http////// //dii. org/1C. 1609/aaa v33101 33014780
pp. 4780-4789.
B. Shahriari, K. Swersky, Z. Wang. Adams, and N. de Freitas Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148-) 2016.
C. Shorten and T. Khoshgoftaar. A survey on image data augmentation for deep learning, Journal of Big Data, 6:1-48, 48, 2019.
S.Lim, I. Kim, T. Kim, C. Kim, and S. Kim. Fast autoaugment. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d Alche-Buc, E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Sys- tems 32, pp. 6665-6675. Curran Associates, Inc., 2019. URL http://paperss wwp.perr.niss................... ca.paperr//rrr 8892- wet-ffffttuuuuuuuumune
pdf
J. Snock, H. Larochelle, and R. Adams. Practical bayesian optimization of machine learning algorithms. In P. Bartlett, F. Pereira, C. Burges, L. Bottou, and K. Weinberger (eds.), Advances in Neural Information Pro- cessing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings, pp. 2960-2968, 2012.
N. Srivastava, G. Hinton, Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent ncural networks from overfitting. J. Mach. Learn. Res., 15(1):1929-199) January 2014. ISSN 1532-4435.
C. Szegedy, S. loffe, V. Vanhoucke, and A. Alemi. Inception v4, inception recptt resnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence, 2017.
M. Tan and Q. EfficientNet: Rethinking model scaling for convolutional neural networks. In K. Chaud- huri and R. Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114, 14, Long Beach, Califomia, USA, 09-15 Jun 2019. PMLR. URL
ngs ml wrw.sr.rresvvvvv wtt.tt//n///aaaaa/ wwt.tatt.a hhmll
C. Thornton, F. Hutter, and H. Hoos K. Leyton- Brown. Auto weka: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 847-855, 2013.
R. Tibshirani. Regression shrinkage and selection via the lasso. Journal the Royal Statistical Society (Series B), 58:267-288 1996.
A. Tikhonov. On the stability inverse problems. In Doklady Akademii Nauk SSSR, 1943.
J. Vanschoren, J. Van Rijn, B. Bischl, and I.. Torgo. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49-60)) 60, 2014.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5998-6008. Curran Associates, Inc., 2017. WtRLbttt://pppes: http:/pppppp..ooo.ppppppppp///..
-olss---l-ooo------ pdf
Wilcoxon. Individual comparisons by ranking methods. Breakthmughs in statistics, pp. 196-202. Springer, 1992.
Y. Yamada, M. Iwamura, and K. Kise. Shakedrop regularization, 2018 URL https::/ net / wow.mmidddee NHHMWDD
Y. Yao, I.. Rosasco, and A. Caponnetto. On early stopping in gradient descent learning. Constructive Ap- proximation, 26(2):289-115, August 2007. ISSN 1432-0940. doi: 10.1007/-003665-006-6666- URL.
https://dci wrgg 10. 1007/ 00066/50055---// 006-0663-2
Yun, D. Han, S. Joon, S. Chun, J. Choe, and Y. Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In International Conference on Computer Vision (ICCV), 2019. to appear.
H. Zhang, M. Cisse, Y. Dauphin, and D. Paz. mixup: Beyond empirical risk minimization. In Interna- tional Conference on Learning Representations, 2018 URI httpps://on htttpp::///ppeeeiiii ew. nwt.//ee/...... forum?
d [idpp1 Rb
12

---

Under review as a conference paper at ICLR 2021
M. Zhang, J. Lucas, J. Ba, and G. Hinton.. Lookahead optimizer: k steps forward, 1 step back. In H. Wallach, H. Larochelle, A. Beygelzimer, Buc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 9593 9604, 2019.
Hutter. Auto-pytorch tabular: Multi-fidelity metalcaring for efficient and robust autodl. CoRR, abs/2006. 13799, 2020.
Zimmer, Lindauer,
B. Zoph and Q. Le. Neural architecture search with reinforcement learning. In 5th International Conference on Learning Representations, ICLR Conference Track Proceedings. OpenReview.net, 2017.
H. Zou and Hastic. Regularization and variable selection via the clastic net. Journal of the royal statistical sociery: series B (statistical methodology), 67(2):301-320, 2005.
13

---

Under review as a conference paper at ICLR 2021
A DESCRIPTION OF BOHB
Algorithm BOHB
Input:
bmins Umar and !]
b stands for budget, while v stands the downsampling rate
Initialization: Smas log, Uman
bmaz
for s in (Smar, Smaz 1.....0) .,... do
Smaz +1
Number of configurations
Call SH routine with rj bmaz initial budget
SH Successing Halving routine
end for
Output: Return hyperparameter configuration with the smallest loss
BOHB Falkner et al. 2018a hyperparameter optimization algorithm that extends Hyperband (Li et al. by sampling from model instead of sampling randomly from the hyperparameter search space.
2017
Initially, BOHB performs random search and favors exploration. As iterates and gets more observations, builds models over different fidelities and trades off exploration with exploitation to avoid converging in bad regions of the search space. BOHB samples from the model of the highest fidelity with probability p and with 1 from random. A model is built for fidelity only when enough ohservations exist, by default the criteria set equal S 1, where S the dimensionality of the search space.
BOHB achieves strong anytime results by combining Random Search Bayesian optimization helps deep neural networks in achieving faster convergence compared with traditional Bayesian Optimisation methods.
B CONFIGURATION SPACES
B. METHOО IMPLICIT SEARCH SPACE
Category
Hyperparameter
Type
Range
Conditionality
Iterations multiplier
Continuous
{2.0)
Scheduler = COS
Cosine Annealing
Max. iterations
Integer
{15}
Scheduler COS
Activation
Nominal
[ReLU}
Bias initialization
Nominal
YYes)
Blocks in group
Integer
(2)
Embeddings
Nominal
(Nome)
Network
Number of groups
Integer
(2)
Resnet shape
Nominal
[Brick]
Type Shaped-Resnet Resnet
Type
Nominal
(Shaped- (Shaped-Resnet) Resnet) {512)
Units in layer
Integer
Preprocessing
Preprocessor
Nominal
(None}
Target size
Nominal
(Median, Upsample) (Random, None)
Resampling
Under sampling
Nominal
Batch size Imputation
Integer
{128} [Median]
Nominal
Initialization method
Nominal
[Default)
Learning rate
Continuous
(10-3)
Training
Loss module
Nominal
[Weighted Cross-Entropy)
Normalization strategy
Nominal
(Standardize)
Optimizer
Nominal
{AdamW)
Scheduler
Nominal
(COS)
Seed
Integer
(11)
Table 2: The configuration space of the training and model architecture hyperparameters.
Table presents the implicit search space used in all our experiments. hetween all the individual regularizers and the regularization cocktail.
The implicit search space is shared
14

---

Under review as a conference paper at ICLR 2021
B.2 AUTO-SKLEARNY GRADIENT BOOSTED DECISION TREE SEARCH SPACE
Experiment we set up the search space Auto-Sklearn as follows:
Hyperparameter
Type
Range
Conditionality
Early Stopping
Nominal
{Off, Train, Valid]
Estimator = Gradient Boosting
L2 Regularization
Continuous
[le - 10, 1]
Estimator = Gradient Boosting
Learning Rate
Continuous
[0.01, 1]
Estimator Gradient Boosting
Max Leaf Nodes
Integer
[3, 2047]
Estimator = Gradient Boosting
Min Samples Leaf
Integer
[1, 200]
Estimator = Gradient Boosting
# Iterations No Change
Integer
20]
Estimator Gradient Boosting
Validation Fraction
Continuous
[0.01, 0.4]
Estimator = Gradient Boosting
The search space of the training and model hyperparameters for the gradient boosting estimator of the Auto-Sklearn tool.
Table 3:
Furthermore, the estimator for Auto-Sklcarn is restricted to only include GBDT, for the sake of fully comparing against the algorithm as a baseline. We do not activate any preprocessing since also our regularization cocktails do not make use of preprocessing algorithms in the pipeline. The time left is always selected based on the time it took BOHI find the hyperparameter with the best validation accuracy from the start of the hyperparameter optimization phase. The ensemble size kept to since our method only features one classifier and not multiple ones. The seed set to 11 as it was set in the experiments with the regularization cocktail, so we can have the same data splits. To keep the comparison fair, there is no warm start for the initial configurations with meta-leane meta-leaming, since, method also does not make of meta-learning Lastly, the number of workers in parallel is 10 match the parallel resources that given the experiment with the regularization cocktails.
C PLOTS
C.1 EXPERIMENT REGULARIZATION COCKTAIL PERFORMANCE
Cocktail WD
DO
SD
SS
SC
SWA
Cocktail
3:010 30
5000
38
7
34
36 1.3
38.18
8 6.6
8 34
29 811 346.6 8. 35
Wilcoxon signed-rank test p-value
6213
0.05
R 0.00
0O0E
O 0.00 0.00 00000
0076 p.0054 0510
DO
0LAG8 P 0.379
00000 2:035 D B.01 P 0072
30
1828 1238
& D & CARE
OE
A 0013 P 6,02 8.000
0,8
BN
01.0 12
0.00
22 1925, O 14.0 2
& 12 27 T4. 25 1824 1429 6.14
0.11 0006 101
SD
52
25 18
TO
29
2013 11
0.19
SS
2000000
DOO DIZE 2011
8C
BODG
U310
& DUT
00000 P 0.129 2.2
SWA
0.00
P 000
SE
14 1.25 B.
5.
27 12.2. 31,
.0.4
CO
17
0018
1, 0.011
O
063
0001
6,00 0.0
MU
0.00 8.00 nm
2
.0.
LA
27, 1289
964 0:211
AT
16.13
Figure 6: Pairwise statistical significance and comparison. For every entry, the first row show- cases the wins, draws and losses of the horizontal method with the vertical method on all datasets, calculated on the test set; the second row presents the p-value for the statistical significance test.
15

---

Under review as a conference paper at ICLR 2021
In Figure 6 present the results of each pairwise comparison. The results presented calculated the set after the refit phase is completed on the best hyperparamettr configuration. The p-value is generated by performing the Wilcoxon signed-ram rank test. As can be seen from the results, the regularization cocktail is the only method that has statistically significant results compared to all the other methods.
C.2 EXPERIMENT 2: DEPEEDET OPTIMAL COCKTAILS
In Figure 7 we present the occurrences of every regularization method over all datasets. The occurrences are calculated by analyzing the best-found hyperparameter configuration for each dataset and observing the number of times the regularization method chosen to be activated by BOHHB
Occurences of Cocktail Ingredients
40
30
Occurences
20
BN
DO
5C
SS
SD
LA
SWA
SE
WD
CO
CM
MU
AT
Figure 7: Frequency of the regularization techniques.
The occurrences of the individual reg-
ularization techniques in the best hyperparameter configurations found by the cocktail across 42 datasets.
16

---

Under review as a conference paper at ICLR 2021
D TABLES
In this section, at Table 4 provide information about the datasets that considered in experiments. Coneretely, we provide descriptive statistics and the identifiers for every dataset. The identifier (the task id) can be used to download the datasets from OpenML.
Tink
Dataset Name
Number Instancees
Number Features
Majority Class Percentage
Minority Class Percentage
233090
anneal
898
0.89
233091
kr-vs-kp
3196
37
52.22
47.78
233092
anhhythmia
280
233093
mifeatfisce infeat-factors
2000
217
10.00
10.00
233088
credit-g
1000
70.00
30,00
233094
vehicle
846
25.77
23.52
233096
kel adult
2109
84.54
15:46
233099
48842
76.07
23.93
233102
walking activity
149332
5
14.73
0.61
233103
phoneme
5404
6
70.65
29.35
233104
skin segmentation
245057
79.25
20.75
233
Idpa nomao
164860
8
33.05
0.84
233107
34465
119
71.44
28.56
233106
enac-9
1080
857
11.11
11.11
233109
blod-transfusion
748
5
76,20
23,80
233110
bank-marken murketing
11.70
233112
connect 4
67557
43
65.83
9.55
233113
shuttle
0.02
233114
higgs
98050
29
$2.86
47.14
233115
Australian
233116
car
1728
7
70.02
3.76
233117
segment
2310
14.29
233118
Fashion-MNIST
70000
785
10.00
10.00
233119
Jungle- 2pcs
44819
7
$1.46
9.67
233120
oumerai28.6 6
96320
22
50,52
49,48
233121
Deviagari-Serig
92000
1025
2.17
2.17
233122
helena
65196
6.14
0.17
233123
jannis
83733
55
46.01
2.01
233124
volkert
58310
181
21,96
2.33
233126
MiniBooNF
130064
71.94
28.06
233130
APSFailure
76000
171
98.19
1.81
233131
christine
5418
1637
50,00
50,00
233132
dilben
10000
2001
20.49
19.13
233135
fabett
8257
801
23.39
6.09
233134
jasmine
2984
233135
sylvine
$124
21
50,00
50.00
233137
diunis
416188
233142
aloi
108000
129
0.10
0.10
233143
C.C.FrandD. Click prediction
284807
99.83
0.17
233146
399482
12
83.21
16.79
Table 4: Datasets. The collection of datasets used in our experiments, combined with detailed information for each dataset.
Moneover, Table 5 shows the results for the comparison between the Regularization Cocktail and the Top-5 cocktail variants described in Experiment 2. The results are calculated on the test set for all datasets, after retraining on the best datasct-specifie hyperparameter configuration.
Task Id
Cockt.
Top-5 F
Top-5 R
Task
Cockt.
Top-5
Top-5
Task Id
Cockt.
Top-5 F
Top-5 R
233090
89.27
89.71
88.54
233091
99.85
99.85
98.20
233092
61.46
$7.21
233093
98.00
98.75
98.75
233088
74.64
71.43
74.76
233094
82.58
82.01
80.33
233096
74.38
78.03
73.96
233099
82.44
82.35
82.24
233102
63.92
62,21
$4.10
233103
86.62
85.90
82.33
233104
99.95
99.96
99.85
233106
68.11
68.81
55.45
233107
96.83
96.67
96.59
233108
95.83
95.83
95.83
233109
67.62
67.32
68.20
2331
85.99
86.35
86.06
233112
80.07
79,57
77.49
233113
99.95
97.95
85.34
2331 14
73.55
73.25
72.06
233115
87.09
88.11
87.60
233116
99.59
100.00
98.20
233117
93.72
93.94
90.69
233118
91,95
91.83
91.59
233119
97.47
92.66
85.53
233120
52.67
52.49
51.70
233121
98.37
98.41
96.93
233122
27.70
28.82
28.09
233123
65.29
65.13
62.11
233124
71.67
70.87
66.06
233126
94.02
88.13
93.16
233130
92.53
96.24
95.89
233131
74.26
71.86
74.63
233132
99.05
98.95
98.55
233133
69.18
68.75
69.03
233134
79.22
78.21
77.71
233135
94.05
94.43
93.95
233137
94.01
94.33
92.43
233142
97.17
97.06
96.06
233146
64.28
64.53
63.28
233143
92.55
92.13
92.59
Table 5: Top-5 baselines. The test set performance for the Regularization Cocktail against the Top-5 Most Frequent (Top-5 F) and the Top-5 Highest Ranks (Top-5 R) basclines.
17

---

Under review as a conference paper at ICLR 2021
At T we provide the results of all our experiments for the baseline, the individual regularization methods, and the regularization cocktail. All the results are calculated on the test set after retraining on the best-found hyperparameter configurations. The evaluation metric used for the performance is the balanced accuracy.
Task Ld
PN
BN
LA
SE
SWA
SC
AT
88
SD
MU
CO
CM
WD
D00
Cocktail
233090
84.13
86.78
83.99
86.48
87.
87.21
86.92
84.28
87.21
89.
60.
86.77
87 06
86,92
233091
99.70
99.85
99.70
99.70
99.55
100.00
99.85
99.85
99.69
99.59 99.85
99.55
99.85
99.85
99.85
99.85
235092
57.99
41.91
36, 14
97.31
25. 25.94
53.42
38.79
55.61
53.26
42.19
32,48
42.22
35. 76
38,70
61.46
233093
97.75
96.00
97.75
69.25
98.25
97.25
25
98.00
98.00
97.75
98.00
233088
69.40
68.69
70.83
69.76
69.40
66.43
69.29
66.43
67.14
70.00
70.36
64.29
69:29
68. 10
74.64
235094
83.77
83.17
84.36
84.39
83.36
80.82
83.17
83,20
81 598
83.77
81.47
78.
83.20
82,60
82.58
233096
70.27
66.56
71.95
71.95
65.31
72.43
67.33
72.98
235099
76,89
77.92
75.90
78.23
76.38
78. 38
76.75
75.56
78 61
78.67
82.56
82.25
76.99
78.52
82.44
233102
61.00
62.89
61.32
63.57
56.67
60.79
59.99
43.04
60.77
61.95
63.30
63.49
64.03
63.75
63.92
233103
$7.51
87.02
87.03
87.99
87.64
87.12
86.59
86.
233104
99.97
99.96
99. 566
99. 944
2.57
99.97
99.95
92.77
99.97
99 95
99 96
99.97
99.96
99.96
99.95
233106
62.83
62.46
65.70
62.16
61.85
61.89
44.63
62.05
66 29.29
65.43
64.99
66.50
67.04
68.11
233107
95.92
95.93
96.D1
96.36
95.23
95.70 76
95.77
95.37
96.22
96.52
96.
96.55
95.99
96.23
96,83
233108
$7.50
91.20
85.65
87.96
50,00
93.98
92.59
94.91
94.44
94.44
93.06
95.37
91.67
90.74
95.83
233109
67.84
73.68
66.52
68.20
66.45
65.20
66.89
66.74
67.03
68. 19.64
67.32
70.18
66.23
68.42
67.62
23311D
78.08
72.58
72.70
66.93
72.74
74.12
70.16
74.76
74 09
85.71
85.76
72.
83.14
85,99
233112
73.63
74.68
73.37
74.33
77.36
73,86
72.91
72.06
74.35
72.00
76.23
75.74
72.48
76.35
80.07
233113
87
900
99
99.91
235114
67.75
68.90
68.81
69.11
67.36
68.08
67.44
67.70
68.56
68. 59
71.93
73.13
67. 80
66.87
73.55
233115
$6.27
85.79
88.73
86.44
87.26
87.74
88.39
87.74
88.39
88.73
88.25
$8,90
87.91
86.27
$7.09
233116
97.44
100,00
96.79
97.44
87.35
99.47
97.46
99.37
97.64
99.04
99.69
233117
94.81
92.86
93.51
93.51
90.48
93.72
92.86
92.64
93.72
93.51
93.07
93.72
93.94
94.59
93.72
235118
90.46
90.86
90.73
90.75
81.72
89.91
90.69
86,
90.06
91.11
91.09
91.88
90.70
501.1
91.95
233119
97.06
97.79
$7.83
97.16
98.
97.51
97
233120
50.26
50.95
51.29
50.50
$1.63
50.92
50.17
50.23
51.00
50.72
52.35
52.10
50.41
50.30
52.67
235121
96.12
97.83
96.45
96.74
92.40
95.31
96.34
91.38
95.15
97.52
97.88
97,80
96.88
97,00
98.37
233122
16.84
20.90
24.53
16.77
18.71
24.35
23.62
23.43
17.52
27.30
233123
$1.51
51.74
50.86
53. 16
56.11
53.58
49.65
49.88
$1 144
51 51.22
66.998
61.67
51.
$5.12
65.29
235124
65.08
66.82
65.57
66.56
66.15
$7.71
65.26
64.97
58.04
67.24
70.03
68.84
66.86
67.00
71.67
233126
90.64
90.42
92.94
92.60
93.99
88.55
93.58
93.86
93.87
92.97
94,10
233130
$7.76
87.81
$8.98
88.99
70.72
$7.99
50,00
85.25
B8.35
92.43
$0.00
95.81
94.92
91.19
92.53
233131
70.94
69.28
71.59
70.94
71.31
72.14
71.59
71.59
72.32
70
72.69
72.42
70.76
70.76
74.26
233132
96.93
98.62
97.52
97.14
94.58
96.85
97.00
V7.27
96.90)
98
98.11 14
99.15
96.81
96.73
99.05
233133
63.71
65.11
65.00
66.05
64.57
66.21
62.82
64.33
65 65.98
68. 75
66.58
66.28
64.36
64.81
69.18
233134
78.05
75.87
79.05
78.22
80.38
78.38
76.88
78.38
78.38
76.88
77.38
76.54
76.88
76.21
79.12
233135
93.07
92.49
92.10
93.17
93.17
92.10
93.17
93.27
92.10
92.58
92.68
94.53
93.75
93,36
94,05
233133
91.91
93.71
92.16
92.56
90.38
91.58
91.36
$8.09
91.60
92.72
92.48
92.39
92.95
92.72
94.01
233142
92.33
35
63.59
93.60
2331443
50.00
92.30
92.76
50.00
70.81
90.28
50.00
50.31
89.26
50.00
50.00
50.00
92.26
50.00
92.53
233146
63.12
60.06
62.79
64.16 16
63.39
64,42
63.52
54.64
64.21
64.26
64.05
64.57
64.41
64.37
64,28
Table 6: Detailed Table of Results. The test set performance for the plain network, individual regularization methods and for the regularization cocktails.
Lastly, in Table 7 we present the results of Experiment 3 where we compare our method with GBDT. The results describe the balanced accuracy calculated on the test set after retraining on both methods on the best hyperparameter configuration found within the given budget.
Task ld
GBDT
Cockt.
Task ld
GBDT
Cockt.
Task ld
GBDT
Cockt.
Task ld
GBDT
Cockt.
233090
90,000
89,270
233091
99.850
99.850
233092
46.850
61,461
233093
97,500
98,000
233088
71.191
74.643
233094
80.165
82,576
233096
63.353
74.381
233099
79.830
82.443
233102
62.764
63.923
233103
88 18.341
86.619
233104
99.967
99,953
233106
68.947
68. 107
233107
97.217
96.826
233108
93.519
95,833
233109
64.985
67.617
233110
72.283
85,993
233112
72.645
80.073
233113
98.571
99.948
233114
72.926
73.546
233115
88.589
87,088
233116
100.000
99.587
233117
93.074
93.723
233118
90.457
91.950
233119
83,070
97.471
233120
52.421
52.668
233121
77.897
98.370
233122
21.144
27.70
233123
55.593
65.287
233124
63.428
71.667
233126
94.137
94.015
233130
91.797
92.535
233131
74.447
74.262
233132
98.704
99.049
233133
70.120
69.183
233134
78.878
79.217
233135
95.119
94,045
233137
74.620
94.01
233142
13.534
97.17
233143
92.514
92.531
233146
58.201
64.280
Table 7: Results of Experiment 3 The performances of the Regularization Cocktail and the GBDT algorithm over the different datasets.
18


Reviews:
[{"review_id": "p_2byJ0IY_0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper provides an empirical study of combining different regularizers. Fourteen regularizers including batch norm, weight decay, etc. are considered. The authors use BOHB (Falkner et al. 2018) to optimize for whether each regularizer is active, and additional regularizer-specific hyperparameters. Using 40 tabular datasets, they show mixtures nearly always outperform tuning a single regularizer, and that the benefits of regularization improves for smaller datasets.\n\nStrengths:\n- To my knowledge, this is the first paper that does an empirical study of combining regularizers. \n- The list of regularizers considered are extensive.\n- The paper is very well written and easy to follow. The figures are nice and easy to understand.\n\nWeaknesses:\nI think the biggest weakness of this work is that it is not very useful practically. \n- All experiments are run on tabular data. I think it’s fair to say that most practitioners who deal with regularization are interested in non tabular data, given the fact that the regularization methods were individually developed for training on non-tabular data. I find it hard to imagine extrapolating results on tabular data to images, or text.\n- The paper frames itself as a “methods” paper more than an analysis paper, where the main claims revolve around the superiority of the regularization cocktail. In fact, the take-home messages given in the conclusion explicitly recommends the use of the regularization cocktail. I find this advice not very useful because: 1) the experiments were run on tabular data, 2) regularization cocktail is too expensive to justify the improvement (if any).\n- The conclusions are trivial. It’s quite obvious that a more general method always does at least as good as the method it subsumes, as long as the tuning of the parameters can be done sufficiently. The fact that the regularization cocktail does better than individual methods, or a combination of a few, is very believable in the tabular setting, since the tuning can be done sufficiently. The fact that more regularization is needed for smaller datasets is also well-known. \n\nThe paper would be more useful with a similar experimental protocol applied to a non-tabular dataset (CIFAR-10, Fashion MNIST, SVHN, etc), but with a focus on analyzing trends (rather than highlighting the mixture method), or comparing with SOTA human designed cocktails. I recommend reject because of the lack of practicality of the paper.\n\nComments:\n- For the experiments corresponding to Figure 5, I wonder what the gap would be like for the best performing individual regularizer. I am curious because I think for a small dataset, the need for a state of the art regularizer diminishes, and just one good regularizer suffices.\n\n=== update ===\n\nI have read the revised paper, and decided to update the score (see response to authors' comment).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PBQ0PlpdH-y", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This work takes a step towards understanding the effect of automated selection of regularisation techniques and analyses the results across 42 structured datasets. It defines a search space over 13 regularisation techniques and employs one flavour of Bayesian Optimisation + Hyperband approach to find an optimal combination of regularisers. It concludes by substantiating three claims with corresponding experiments. \n\nThe work addresses an interesting problem that is relevant for full-scale automation of deep learning. The paper is easy to follow with a thorough literature survey (to my knowledge) and contains relevant experiments.\n\nI find the study can improve by addressing the following \n- Firstly, it is difficult to argue for generalisability of this work as currently this work relies on only one algorithm to find the optimal combination of regulaisers. One would need to check how much influence does BOHO have on the claims made within this work. \n- The three hypotheses are insightful but limited to small scale datasets. In particular, the third claims about gains on small datasets stands out as not-so-surprsing but challenges the effectiveness of the cocktail-system as DL models often find use in large scale problems.\n- While the study provides some descriptive insights, it falls short on prescriptive solutions for automated selection of regularisers.\n- It is interesting to note that in Exp 1 CutMix, CutOut and Mixup rank better as individual techniques and yet only CutMix crosses the threshold of 0.3 for Figure 3. Did the authors find any explanation for this? It would be interesting to see how Figure 4 changes if these are included along with BN, DO and WD. \n- Finally, I think the paper can improve with some additional information and more discussion. For instance, a brief description of BOHO, details on what the scatter plot correspond to in Figure 4, insights and discussion of results from all figures in the supplementary etc. It would also be important to establish statistical significance of the experiments with multiple re-runs. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3SUQZNxMehf", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary and contributions:*\nThis paper provides empirical evidence showing that optimizing a parametric mix of regularizers when training a model provides better generalization than using handcrafted ones.\nThe results show that the best regularizers mix is dataset dependent, and that regularization matters most when limited data is available.\n\n\n*Originality:*\nThis paper falls into the category “more parameters in the system lead to better results”.\nI am not aware of a specific paper that has explored using a parametric mix of regularizer.\nI suspect some previous papers might have explored smaller variants of this idea, but none with the systematic approach and breath of this paper.\n\n\n*Significance:*\nThis paper is part of the AutoML trend. It provides support to the idea that the regularizers should also be thrown in the optimization target box. \n(Together with network architecture, data augmentation, and the optimizer themselves).\nAs such, it provides recommendations for best practices in machine learning.\n\n\n*Strengths:* \n* The paper carefully explicits the hypotheses being tested, and design experiments to probe them.\n* The observed effect is significant to the point of justifying the additional system complexity.\n* The experiments include a reasonable amount of regularization techniques in the mix (and for comparison).\n\n\n*Weaknesses:*\n* The results are somewhat expected. More parameters make systems better, everything is task dependent, and that regularization matters more when less data is available.\n* I did not find in the paper experiments adding the validation set in the training set (aka “what if we do not split ?”). Id est, a baseline accounting for the fact that in this setup the validation set is part of the overall training set.\n* Experiments only consider tabular datasets for classification. The intuition indicates that the results would generalize to 1d audio, 2d image, or 3d point-cloud data (as well as for other tasks beyond classification); but it would be better to have some data points on this aspect. \n\n\n*Clarity:*\nThe writing is clear, and the text is overall easy to follow.\n\n\n*Correctness:*\nI did not find anything particularly wrong.\n\n\n*Relation to prior work:*\nThe related work section is satisfactory. \nI am not aware of a specific related paper should be discussed (and a quick online search did not show either).\n\n\n*Reproducibility:*\nThe general idea seems easily reproducible. The text provides enough detail to be able to reproduce the overall system.\n\n\n*Specific feedback:*\n- Section 3.1, footnote: hold-out validation versus cross-validation might not be a detail, since it affects what is considered the training set of the baseline.\nIt would be good to include results of the baselines with “default parameters” trained over train+val.\n- Section 3.2: please mention the total number of additional parameters added to the system (or at least its order of magnitude).\n- Section 4.1: please mention that the task considered is always classification, and the total number of datasets considered.\n- Section 5, experiment 1: confirm our hypothesis that -> confirm our hypothesis 1 that\n- Figure 3: what is the logic of the top row sorting ? Consider sorting by frequency of usage. Please put the numbers as percentages, to be consistent with the paper text.\n- Section 5, experiment 3: just to confirm, the validation set size was kept fix ? \n- Section 5, experiment 3: “indicate that strong regularization”; maybe I missed a nuance, but the presented results show the consequences of _tuned_ regularization. This is different from _strong_ regularization. Is there a mention of the weights used for regularization in this case tend to be higher than for other setups ? That would be expected, but I did not see it presented. \n- Section 5, experiment 3: “regularization can help...better...than generally perceived”; please cite a paper (or blog post) taking that position. I learned in ML classes that regularization matters when data is small, thus to my knowledge these results simply fit the classical ML theory (Vapnik-era Statistical learning theory).\n- Figure 4: please mention what the points represent (a dataset each ?). Also use scientific notation for the number of examples.\n\n\n*Updates after reviews and authors feedback: *\nThe updates from the author are appreciated and make the arguments of the paper clearer.\nAfter reading the other reviews and discussions, I keep my original score of \"6: Marginally above acceptance threshold\".\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0a51bbW8kl-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a study on regularization methods for the feedforward fully connected neural networks.\nThe study is formulated as hyper-parameter optimization task, heavily using Auto-Pytorch library. The paper claims as contribution (sorry for copy-pasting):\n\n1. We demonstrate the empirical accuracy gains of regularization cocktails in a systematic\nmanner via a large-scale experimental study;\n2. We challenge the status-quo practices of designing universal dataset-agnostic regularizers,\nby showing that an optimal regularization cocktail is highly dataset-dependent;\n3. We demonstrate that regularization cocktails achieve a higher gain on smaller datasets;\n4. As an overarching contribution, this paper provides previously-lacking in-depth empiri-\ncal evidence to better understand the importance of combining different mechanisms for\nregularization, one of the most fundamental concepts in machine learning.\n\n***\n\nI am highly sceptical on the paper usefulness for the community. \nIn general terms, the benchmark/empirical study type of paper typically can have one (or more) of the following contributions:\n\na) New knowledge, which was obtained as a result of a study. E.g. surprising results, practical recommendations, so on. For example, A Metric Learning Reality Check by Musgrave et. al (ECCV 2020) revealed surprising knowledge about metric learning methods. \n\nb) The methodology of such study, which was not used before. E.g. Visual Object Tracking Challenge, which become the benchmark for the tracking methods since 2013. \nc) The software and/or dataset, which were developed for the study. E.g. OpenAI Gym.\n\n\n(a) is not the case IMO, because all the recommendations are known to the practicioners, e.g. check the any Kaggle winning solution\nhttps://www.kaggle.com/sudalairajkumar/winning-solutions-of-kaggle-competitions\n\n(b) I see no novelty in using hyper-parameter optimization for the study. The paper agrees with me on this  (see Related Work, \"Positioning in the realm of AutoML\"\n\n(c) Neither software, nor dataset is proposed -- the paper uses existing ones. \n\n****\n\nNow I will go over contributions.\n\n\n1. It is well known that the regularization/augmentation/... need to be tuned to archieve the best results. One can publish a CVPR paper about such good combination, e.g. He et.al. (CVPR2019), \nBag of Tricks for Image Classification with Convolutional Neural Networks\n\n2. I don't see the support for that claim in the paper. Yes, the specific combination of regularization techniques, which performs the best on the given dataset is, perhaps, unique. But the techniques are applicable broadly, which is supported by the paper (Fig. 1), e.g. DropOut, MixUp, BatchNorm, are pretty universal.\n\n3. It is also obvious, that the less data you have, more regularization and design bias are needed for better results, see OpenAI Image GPT, or more ViT paper vs. ConvNets. \n\n4. See (1)\n\n\nOverall, if the paper spend some space on particually interesting regularization combinations/interplay of components/analisys, it might be quite useful for researchers. For now it seems as a lot of experiments were done, but no analisys is really performed.\n\nE.g. abstract says: \"there is no systematic study as to how different regularizers should be combined into the best \"cocktail\"\". \n\nBut I don't see the answer of how should they be combined.\n\n********\n\nSmall things, not contributing to the score:\n\n- AutoPyTorch is cited twice, as arXiv and as CoRR\n\n- While skip-connections and BN can be seen as \"regularization\", I would rather call them \"architecture\". Anyway, that is just matter of naming. \n\n\n### After rebuttal update.\n\nThe paper has been significantly refocused and now sells itself as a way of deep neural networks being competitive versus gradient boosting methods, which are dominating the tabular heterogeneous tasks. \n\nI was also convinced by authors responce on paper novelty, technical contribution and (after the re-focusing) potential usefulness to the community.\nThus, I am raising my rating to weak accept.\n\n### Comments on authors response (as after Nov 24 I cannot post messages, visible to authors)\n\n> 1. However, we would be thankful if you might share any prior work (paper or published practice) where the authors automatically searched for the optimal combination of regularizers for deep learning models among a large set of regularizers, as presented in this study.\n\nI am surprised with the results of the googling, but have to admit that authors are technically right and I was wrong. While it seems obvious to me, that regularization (specifically, dropout and L2 weight decay) are the hyperparameters of the deep network training, somehow papers and guides online mostly consider mostly architectural things + learning rate + (sometimes) dropout rate as a hyperparameters to optimize. \n\nhttps://arxiv.org/pdf/2006.12703.pdf\nhttps://nanonets.com/blog/hyperparameter-optimization/\n\nAnyway, I lift my objections on novelty.\n\n> 3. \"Neither software, nor dataset is proposed, the paper uses existing ones.\" : We engineered a source code that selects the application of 13 regularizers to a neural network, which required extensive programming efforts and several additions to the AutoPytorch library, as mentioned in Section 4.1.\n\nOK, I agree.\n\n\n> 4. \"It is well known that the regularization/augmentation/ methods need to be tuned ... He et.al. (CVPR2019)\" : The suggested reference is a collection of refinements (many of which are actually not regularization techniques), which have been suggested by the deep learning community for maximizing the generalization on Imagenet. That paper only summarizes a collection of some practices, however, it does not present a method that searches for the best combination of a large set of practices\n\nNo, we are discussing different things. I gave the He at.al as an example, that community is well aware about the fact, the regularization and augmentation (and other things) have to be highly tuned. I agree that He at.al and the current paper use completely different methods for solving the problem (manual tinkering vs. auto-search). What I disagree is that the community was not aware about importance of regularization tuning before this paper.\n\n>5.  E.g. DropOut was present in only 35% of the dataset cocktails, hence was not selected in the cocktails of the 65% of the datasets.\n\nAnd experiments were using fixed-size network. Of course, is the network is not wide enough for the task, the dropout might not be needed. It is also quite strong statement that \"there is no universal regularization\", given that L2 weight decay and dropout are widely used in a such different domains as image, text, speech processings, RL and so on. \n****\n\nI would like to point out the TabNet paper https://arxiv.org/pdf/1908.07442.pdf, which claimed \"beating GB methods for the tabular data\". I appreciate the fact, that unlike the TabNet, RegCocktails were using a standard deep MLP and not the attention model, yet one needs to add that reference.\n\n\n\n\n ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper in its most recent version claims that deep neural networks, when very carefully regularized, outperform methods such as Gradient Boosting Trees on tabular data. This is genuinely surprising to me (in a good way), and I suppose it is as well to the community.

The paper initially received negative reviews with two key remarks that "The results are somewhat expected." (R4, R3, R2). Indeed, the original version mainly stated that very careful regularization helps on tabular data.  Naturally, the reviewers (including myself) seen then as the second key weakness that "All experiments are run on tabular data." (R4, R3).

Based on the reviews, the Authors have clarified and changed their message. I think it is well summarized by R2 "The paper has been significantly refocused and now sells itself as a way of deep neural networks being competitive versus gradient boosting methods, which are dominating the tabular heterogeneous tasks."

As R2 said and was reflected in comments by other reviewers, "[...] convinced by authors response on paper novelty, technical contribution and (after the re-focusing) potential usefulness to the community".

Given the new message of the paper, a key new question surfaces. Is this indeed the first convincing demonstration that deep learning can outperform more standard methods on tabular data? R2 pointed out TabNet (see also Google Cloud offering) that already in 2019 claimed "beating GB methods for the tabular data". There is also NeurIPS work "Regularization Learning Networks: Deep Learning
for Tabular Datasets"; their abstract opens with "Despite their impressive performance, Deep Neural Networks (DNNs) typically underperform Gradient Boosting Trees (GBTs) on many tabular-dataset learning tasks. We propose that applying a different regularization coefficient to each weight might boost the performance of DNNs by allowing them to make more use
of the more relevant inputs". The latter work did not claim to beat GBT. Regardless, the two works should be carefully discussed and compared empirically to in the new version of the work. 

I am also not yet fully convinced by the added comparison to GDBT. Arguably, AutoML from the sklearn package is not the most popular way to use GDBT in practice. How would regularization cocktails compare to GDBT from XGBoost, optimized using either random search or bayesian optimization?

Based on the above, I have to recommend the rejection of the paper. The key reason is: *the new reframing of the paper is exciting but warrants a much more detailed and careful evaluation*.

I really appreciate the work the Authors have put in clarifying and changing the message of the paper. I understand this is disappointing that we won't be able to include the work in ICLR. Nevertheless, I hope that the Authors found the feedback useful, and wanted to thank the Authors for submitting the work for consideration in ICLR.
Decision: Reject