Venue: ICLR 2021 Conference

Reviews:

final_score: rating: null
review_content: Dear authors, 

I see that you have updated the paper ("28 Sep 2020 (modified: 11 Nov 2020)"). 
Could you please summarize the changes and answer the questions in reviews?

Best, R2
review_title: Answers/rebuttal?

final_score: rating: 6: Marginally above acceptance threshold
review_content: *Summary and contributions:*
This paper provides empirical evidence showing that optimizing a parametric mix of regularizers when training a model provides better generalization than using handcrafted ones.
The results show that the best regularizers mix is dataset dependent, and that regularization matters most when limited data is available.


*Originality:*
This paper falls into the category “more parameters in the system lead to better results”.
I am not aware of a specific paper that has explored using a parametric mix of regularizer.
I suspect some previous papers might have explored smaller variants of this idea, but none with the systematic approach and breath of this paper.


*Significance:*
This paper is part of the AutoML trend. It provides support to the idea that the regularizers should also be thrown in the optimization target box. 
(Together with network architecture, data augmentation, and the optimizer themselves).
As such, it provides recommendations for best practices in machine learning.


*Strengths:* 
* The paper carefully explicits the hypotheses being tested, and design experiments to probe them.
* The observed effect is significant to the point of justifying the additional system complexity.
* The experiments include a reasonable amount of regularization techniques in the mix (and for comparison).


*Weaknesses:*
* The results are somewhat expected. More parameters make systems better, everything is task dependent, and that regularization matters more when less data is available.
* I did not find in the paper experiments adding the validation set in the training set (aka “what if we do not split ?”). Id est, a baseline accounting for the fact that in this setup the validation set is part of the overall training set.
* Experiments only consider tabular datasets for classification. The intuition indicates that the results would generalize to 1d audio, 2d image, or 3d point-cloud data (as well as for other tasks beyond classification); but it would be better to have some data points on this aspect. 


*Clarity:*
The writing is clear, and the text is overall easy to follow.


*Correctness:*
I did not find anything particularly wrong.


*Relation to prior work:*
The related work section is satisfactory. 
I am not aware of a specific related paper should be discussed (and a quick online search did not show either).


*Reproducibility:*
The general idea seems easily reproducible. The text provides enough detail to be able to reproduce the overall system.


*Specific feedback:*
- Section 3.1, footnote: hold-out validation versus cross-validation might not be a detail, since it affects what is considered the training set of the baseline.
It would be good to include results of the baselines with “default parameters” trained over train+val.
- Section 3.2: please mention the total number of additional parameters added to the system (or at least its order of magnitude).
- Section 4.1: please mention that the task considered is always classification, and the total number of datasets considered.
- Section 5, experiment 1: confirm our hypothesis that -> confirm our hypothesis 1 that
- Figure 3: what is the logic of the top row sorting ? Consider sorting by frequency of usage. Please put the numbers as percentages, to be consistent with the paper text.
- Section 5, experiment 3: just to confirm, the validation set size was kept fix ? 
- Section 5, experiment 3: “indicate that strong regularization”; maybe I missed a nuance, but the presented results show the consequences of _tuned_ regularization. This is different from _strong_ regularization. Is there a mention of the weights used for regularization in this case tend to be higher than for other setups ? That would be expected, but I did not see it presented. 
- Section 5, experiment 3: “regularization can help...better...than generally perceived”; please cite a paper (or blog post) taking that position. I learned in ML classes that regularization matters when data is small, thus to my knowledge these results simply fit the classical ML theory (Vapnik-era Statistical learning theory).
- Figure 4: please mention what the points represent (a dataset each ?). Also use scientific notation for the number of examples.


*Updates after reviews and authors feedback: *
The updates from the author are appreciated and make the arguments of the paper clearer.
After reading the other reviews and discussions, I keep my original score of "6: Marginally above acceptance threshold".

review_title: A bag of experiments regarding parametric mixes of regularizers 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This work takes a step towards understanding the effect of automated selection of regularisation techniques and analyses the results across 42 structured datasets. It defines a search space over 13 regularisation techniques and employs one flavour of Bayesian Optimisation + Hyperband approach to find an optimal combination of regularisers. It concludes by substantiating three claims with corresponding experiments. 

The work addresses an interesting problem that is relevant for full-scale automation of deep learning. The paper is easy to follow with a thorough literature survey (to my knowledge) and contains relevant experiments.

I find the study can improve by addressing the following 
- Firstly, it is difficult to argue for generalisability of this work as currently this work relies on only one algorithm to find the optimal combination of regulaisers. One would need to check how much influence does BOHO have on the claims made within this work. 
- The three hypotheses are insightful but limited to small scale datasets. In particular, the third claims about gains on small datasets stands out as not-so-surprsing but challenges the effectiveness of the cocktail-system as DL models often find use in large scale problems.
- While the study provides some descriptive insights, it falls short on prescriptive solutions for automated selection of regularisers.
- It is interesting to note that in Exp 1 CutMix, CutOut and Mixup rank better as individual techniques and yet only CutMix crosses the threshold of 0.3 for Figure 3. Did the authors find any explanation for this? It would be interesting to see how Figure 4 changes if these are included along with BN, DO and WD. 
- Finally, I think the paper can improve with some additional information and more discussion. For instance, a brief description of BOHO, details on what the scatter plot correspond to in Figure 4, insights and discussion of results from all figures in the supplementary etc. It would also be important to establish statistical significance of the experiments with multiple re-runs. 

review_title: Interesting study, requires further analysis

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper provides an empirical study of combining different regularizers. Fourteen regularizers including batch norm, weight decay, etc. are considered. The authors use BOHB (Falkner et al. 2018) to optimize for whether each regularizer is active, and additional regularizer-specific hyperparameters. Using 40 tabular datasets, they show mixtures nearly always outperform tuning a single regularizer, and that the benefits of regularization improves for smaller datasets.

Strengths:
- To my knowledge, this is the first paper that does an empirical study of combining regularizers. 
- The list of regularizers considered are extensive.
- The paper is very well written and easy to follow. The figures are nice and easy to understand.

Weaknesses:
I think the biggest weakness of this work is that it is not very useful practically. 
- All experiments are run on tabular data. I think it’s fair to say that most practitioners who deal with regularization are interested in non tabular data, given the fact that the regularization methods were individually developed for training on non-tabular data. I find it hard to imagine extrapolating results on tabular data to images, or text.
- The paper frames itself as a “methods” paper more than an analysis paper, where the main claims revolve around the superiority of the regularization cocktail. In fact, the take-home messages given in the conclusion explicitly recommends the use of the regularization cocktail. I find this advice not very useful because: 1) the experiments were run on tabular data, 2) regularization cocktail is too expensive to justify the improvement (if any).
- The conclusions are trivial. It’s quite obvious that a more general method always does at least as good as the method it subsumes, as long as the tuning of the parameters can be done sufficiently. The fact that the regularization cocktail does better than individual methods, or a combination of a few, is very believable in the tabular setting, since the tuning can be done sufficiently. The fact that more regularization is needed for smaller datasets is also well-known. 

The paper would be more useful with a similar experimental protocol applied to a non-tabular dataset (CIFAR-10, Fashion MNIST, SVHN, etc), but with a focus on analyzing trends (rather than highlighting the mixture method), or comparing with SOTA human designed cocktails. I recommend reject because of the lack of practicality of the paper.

Comments:
- For the experiments corresponding to Figure 5, I wonder what the gap would be like for the best performing individual regularizer. I am curious because I think for a small dataset, the need for a state of the art regularizer diminishes, and just one good regularizer suffices.

=== update ===

I have read the revised paper, and decided to update the score (see response to authors' comment).

review_title: Review of "Regularization Cocktails"

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a study on regularization methods for the feedforward fully connected neural networks.
The study is formulated as hyper-parameter optimization task, heavily using Auto-Pytorch library. The paper claims as contribution (sorry for copy-pasting):

1. We demonstrate the empirical accuracy gains of regularization cocktails in a systematic
manner via a large-scale experimental study;
2. We challenge the status-quo practices of designing universal dataset-agnostic regularizers,
by showing that an optimal regularization cocktail is highly dataset-dependent;
3. We demonstrate that regularization cocktails achieve a higher gain on smaller datasets;
4. As an overarching contribution, this paper provides previously-lacking in-depth empiri-
cal evidence to better understand the importance of combining different mechanisms for
regularization, one of the most fundamental concepts in machine learning.

***

I am highly sceptical on the paper usefulness for the community. 
In general terms, the benchmark/empirical study type of paper typically can have one (or more) of the following contributions:

a) New knowledge, which was obtained as a result of a study. E.g. surprising results, practical recommendations, so on. For example, A Metric Learning Reality Check by Musgrave et. al (ECCV 2020) revealed surprising knowledge about metric learning methods. 

b) The methodology of such study, which was not used before. E.g. Visual Object Tracking Challenge, which become the benchmark for the tracking methods since 2013. 
c) The software and/or dataset, which were developed for the study. E.g. OpenAI Gym.


(a) is not the case IMO, because all the recommendations are known to the practicioners, e.g. check the any Kaggle winning solution
https://www.kaggle.com/sudalairajkumar/winning-solutions-of-kaggle-competitions

(b) I see no novelty in using hyper-parameter optimization for the study. The paper agrees with me on this  (see Related Work, "Positioning in the realm of AutoML"

(c) Neither software, nor dataset is proposed -- the paper uses existing ones. 

****

Now I will go over contributions.


1. It is well known that the regularization/augmentation/... need to be tuned to archieve the best results. One can publish a CVPR paper about such good combination, e.g. He et.al. (CVPR2019), 
Bag of Tricks for Image Classification with Convolutional Neural Networks

2. I don't see the support for that claim in the paper. Yes, the specific combination of regularization techniques, which performs the best on the given dataset is, perhaps, unique. But the techniques are applicable broadly, which is supported by the paper (Fig. 1), e.g. DropOut, MixUp, BatchNorm, are pretty universal.

3. It is also obvious, that the less data you have, more regularization and design bias are needed for better results, see OpenAI Image GPT, or more ViT paper vs. ConvNets. 

4. See (1)


Overall, if the paper spend some space on particually interesting regularization combinations/interplay of components/analisys, it might be quite useful for researchers. For now it seems as a lot of experiments were done, but no analisys is really performed.

E.g. abstract says: "there is no systematic study as to how different regularizers should be combined into the best "cocktail"". 

But I don't see the answer of how should they be combined.

********

Small things, not contributing to the score:

- AutoPyTorch is cited twice, as arXiv and as CoRR

- While skip-connections and BN can be seen as "regularization", I would rather call them "architecture". Anyway, that is just matter of naming. 


### After rebuttal update.

The paper has been significantly refocused and now sells itself as a way of deep neural networks being competitive versus gradient boosting methods, which are dominating the tabular heterogeneous tasks. 

I was also convinced by authors responce on paper novelty, technical contribution and (after the re-focusing) potential usefulness to the community.
Thus, I am raising my rating to weak accept.

### Comments on authors response (as after Nov 24 I cannot post messages, visible to authors)

> 1. However, we would be thankful if you might share any prior work (paper or published practice) where the authors automatically searched for the optimal combination of regularizers for deep learning models among a large set of regularizers, as presented in this study.

I am surprised with the results of the googling, but have to admit that authors are technically right and I was wrong. While it seems obvious to me, that regularization (specifically, dropout and L2 weight decay) are the hyperparameters of the deep network training, somehow papers and guides online mostly consider mostly architectural things + learning rate + (sometimes) dropout rate as a hyperparameters to optimize. 

https://arxiv.org/pdf/2006.12703.pdf
https://nanonets.com/blog/hyperparameter-optimization/

Anyway, I lift my objections on novelty.

> 3. "Neither software, nor dataset is proposed, the paper uses existing ones." : We engineered a source code that selects the application of 13 regularizers to a neural network, which required extensive programming efforts and several additions to the AutoPytorch library, as mentioned in Section 4.1.

OK, I agree.


> 4. "It is well known that the regularization/augmentation/ methods need to be tuned ... He et.al. (CVPR2019)" : The suggested reference is a collection of refinements (many of which are actually not regularization techniques), which have been suggested by the deep learning community for maximizing the generalization on Imagenet. That paper only summarizes a collection of some practices, however, it does not present a method that searches for the best combination of a large set of practices

No, we are discussing different things. I gave the He at.al as an example, that community is well aware about the fact, the regularization and augmentation (and other things) have to be highly tuned. I agree that He at.al and the current paper use completely different methods for solving the problem (manual tinkering vs. auto-search). What I disagree is that the community was not aware about importance of regularization tuning before this paper.

>5.  E.g. DropOut was present in only 35% of the dataset cocktails, hence was not selected in the cocktails of the 65% of the datasets.

And experiments were using fixed-size network. Of course, is the network is not wide enough for the task, the dropout might not be needed. It is also quite strong statement that "there is no universal regularization", given that L2 weight decay and dropout are widely used in a such different domains as image, text, speech processings, RL and so on. 
****

I would like to point out the TabNet paper https://arxiv.org/pdf/1908.07442.pdf, which claimed "beating GB methods for the tabular data". I appreciate the fact, that unlike the TabNet, RegCocktails were using a standard deep MLP and not the attention model, yet one needs to add that reference.




 
review_title: No contribution?

Meta-review: The paper in its most recent version claims that deep neural networks, when very carefully regularized, outperform methods such as Gradient Boosting Trees on tabular data. This is genuinely surprising to me (in a good way), and I suppose it is as well to the community.

The paper initially received negative reviews with two key remarks that "The results are somewhat expected." (R4, R3, R2). Indeed, the original version mainly stated that very careful regularization helps on tabular data.  Naturally, the reviewers (including myself) seen then as the second key weakness that "All experiments are run on tabular data." (R4, R3).

Based on the reviews, the Authors have clarified and changed their message. I think it is well summarized by R2 "The paper has been significantly refocused and now sells itself as a way of deep neural networks being competitive versus gradient boosting methods, which are dominating the tabular heterogeneous tasks."

As R2 said and was reflected in comments by other reviewers, "[...] convinced by authors response on paper novelty, technical contribution and (after the re-focusing) potential usefulness to the community".

Given the new message of the paper, a key new question surfaces. Is this indeed the first convincing demonstration that deep learning can outperform more standard methods on tabular data? R2 pointed out TabNet (see also Google Cloud offering) that already in 2019 claimed "beating GB methods for the tabular data". There is also NeurIPS work "Regularization Learning Networks: Deep Learning
for Tabular Datasets"; their abstract opens with "Despite their impressive performance, Deep Neural Networks (DNNs) typically underperform Gradient Boosting Trees (GBTs) on many tabular-dataset learning tasks. We propose that applying a different regularization coefficient to each weight might boost the performance of DNNs by allowing them to make more use
of the more relevant inputs". The latter work did not claim to beat GBT. Regardless, the two works should be carefully discussed and compared empirically to in the new version of the work. 

I am also not yet fully convinced by the added comparison to GDBT. Arguably, AutoML from the sklearn package is not the most popular way to use GDBT in practice. How would regularization cocktails compare to GDBT from XGBoost, optimized using either random search or bayesian optimization?

Based on the above, I have to recommend the rejection of the paper. The key reason is: *the new reframing of the paper is exciting but warrants a much more detailed and careful evaluation*.

I really appreciate the work the Authors have put in clarifying and changing the message of the paper. I understand this is disappointing that we won't be able to include the work in ICLR. Nevertheless, I hope that the Authors found the feedback useful, and wanted to thank the Authors for submitting the work for consideration in ICLR.
Decision: Reject