Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper supplies an empirical existence proof that an "appropriately regularized" multi-layer perceptron (MLP)-like model can outperform gradient-boosted decision trees (GBDT) on tabular data. The paper's proposed approach defines a set of parameterized techniques which includes ensembling, architectural changes, and data augmentation, and performs search over combinations from this set. The extensive experimental comparison across 40 tabular datasets shows that MLP-like models can outperform GBDTs and other neural network architectures.
main_review: I appreciated the simplicity of the proposed approach. Showing that Bayesian optimization can select the right combination from such a disparate set of techniques ("soup" came to my mind) is novel, at least to my knowledge. The paper also seems to have addressed some concerns raised in previous submissions with a thorough evaluation performed across a competitor set. However, I have some major concerns on whether the comparison makes sense. The computational cost associated with the proposed approach could be considered impractical wrt/ the standard GBDT solution, but perhaps more work could show this can be reduced. More fine-grained conclusions about why certain techniques, e.g., snapshot ensembles, are necessary and/or sufficient for identifying a high-performing MLP-like model would also have strengthened the submission.

The use of "regularization" to describe the set of techniques over which the proposed method optimizes is misleading. For example, take ensembling: just b/c ensembling has regularizing properties does not make it a regularizer in the traditional sense of a technique to mitigate against overfitting of a single model during training. Conversely, just b/c dropout has ensembling properties does not make it an ensembling technique in the traditional sense of combining the predictions of multiple trained models. Another example is what are termed structural/linearization techniques which include adding skip connects. These really fall under model selection. 

A more accurate title would be "Traditional regularization, data augmentation, model selection, model averaging, and deep learning tricks are all you need..." 
The value of the contribution comes down to how surprising the community thinks the news that some specific combination of these techniques can yield an MLP-like model or ensemble which, when searched for using 4 days of compute, outperforms GBDT on tabular datasets with <500K rows. If there was evidence that the search could be performed efficiently, that would be hugely beneficial to the message. Unfortunately, there is no finer timing analysis included in the submission.

The overloading of "regularization" in the paper isn't a pedantic issue. It makes fair comparisons wrt/ hypothesis 1 difficult. For example, AutoGluon tabular was run with only neural networks but is designed with general ML models in mind (as this paper notes). This seems artifically limiting if AutoGluon tabular natively, i.e., with AutoGluon's own model and techniques, outperforms GBDT(-like) models; since, if it does, then GBDT should no longer be considered the correct baseline of comparison. More generally, a discussion on why this paper's set of techniques (listed in Table 1) is the "right" set would help in framing which competitors are appropriate vs. unappropriate in a comparison.

* A finer analysis would be helpful to the community. Is there a method rank vs. time plot? method mean+-err balanced accuracy vs. time?

* Are there obvious efficiencies, such as training data-subsampling, in which the computational cost of the proposed approach is retained w/o compromising accuracy?

* How does the absence of one (or multiple) technique(s) from the set of techniques affect performance? E.g., if snapshot ensembles are excluded, how do the results of Table 2 change? Note, this is not a question about which techniques are selected; it is a question about which techniques are generally important/unimportant for the tabular datasets problem.

* It isn't clear how AutoGluon tabular performs natively. If it does not outperform GBDT, then this fact should be included in the paper as it will substantially increase the case for MLP-like solutions.

* Please motivate the selection for the specific set of techniques listed in Table 1.
limitations_and_societal_impact: The authors adequately addressed both of these.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes that properly optimized MLPs can perform just as well if not better on tabular data. This is an important topic as it's well know that MLPs have not achieved similar performance as GBDT on data with categorical variables. The paper is well written and reasonably comprehensive in the experiments. 
main_review: This paper proposes that properly optimized MLPs can perform just as well if not better on tabular data. This is an important topic as it's well know that MLPs have not achieved similar performance as GBDT on data with categorical variables. The paper is well written and reasonably comprehensive in the experiments. 

Positives:
- This paper is written clearly and have considered a large set of regularization techniques as well as tested on a comprehensive tabular dataset
- The results is surprising and very useful and relevant to the community
- Real world Kaggle datasets are included as well, in addition to the tiny UCI datasets.
- The discussions on limitations is also interesting and useful.

Negatives:
- the optimization being fixed is a negative. Optimizers typically have a big impact on the performance and generalization of MLPs.
- What about the different types of weight initialization for MLPs?

Given that the results are re-generated by the authors, what is not super clear is that are the numbers for non-MLP results consistent with other published papers?
limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper aims to study how regularizing basic MLP models can make them as effective as the best available models for tabular data (GBDT and more sophisticated/recent NN architectures like TabNet/NODE). The main contribution is to propose to search among 13 regularizers to find a combination that maximizes validation score, where the search is done via standard hyperparameter optimization techniques, and they find the resulting regularized MLPs outperform GBDT and more recent NN architectures across many tabular datasets.
main_review: Through a careful empirical study, this paper provides a number of important results with practical significance. Namely that MLPs can outperform XGB across a wide dataset range and that 
In particular, the conclusion that "well-regularized simple deep MLPs outperform specialized neural architectures" is an important one for the community to be informed of. However the main weakness of this paper in my eyes is it does not actually answer the question of how I should use MLPs for tabular data to get the best performance. This is because the authors did not consider how to split a limited HPO budget into tuning regularizers vs other hyperparameters like the network architecture, learning rates, etc. I also still have questions regarding the details of baselines in the empirical  comparisons.


- What makes these "regularization cocktails" different than just a hyperparameter-tuned MLP?
It seems the authors are considering implicit bias aspects of the NN as "regularization" as well, in which case wouldn't also the activation function and other hyperparameters that affect implicit bias also fit this definition of regularization?
In fact the authors say their fixed hyperparameters "have been tuned for maximizing the performance of an unregularized neural network on our dataset collection"; more details about this process would be nice to see. 
Why not tune hyperparameters like number of layers, learning rate, hidden layer size, etc? Is there some practical advantage to tuning only the regularization cocktail in MLP+C vs other hyperparameters? Given a limited HPO budget, it's still not clear to me how I should design a MLP search space and whether I should only search over regularizers or also the hyperparameters I just mentioned.


- Tuning num_round as a hyperparameter of XGBoost is a bit odd. Normally one just sets this large and  relies on early-stopping. Tuning num_round might lead to the effective number of meaningful XGB HPO trials completed within the budget much smaller than for other models. The authors state they also tried XGB + ES, but did not clarify precisely what was done there. In particular did you remove num_round from the search space and if so to what value was it set?


- The authors are missing reference to SeLU paper:

Self-Normalizing Neural Networks
Klambauer et al. (2017)
https://arxiv.org/abs/1706.02515

And an empirical comparison against this technique, which seemed to also perform well on tabular data in the original paper.


- More details of MLP+D baseline would be good to have. Eg. Was the DO-active hyperparameter fixed = True and the same number of HPO trials / total compute run for this baseline?


- Is multifidelity HPO method like BOHB able to accurately assess effect of something like Dropout? Wouldn't dropout appear to be harmful in the early stages of MLP training since it slows down the optimization (although allowing the optimization to eventually reach a better-generalizing solution)?


- Table 1 caption should include references for each regularization method. Some are not fully clear in their details such as "Multi-branch choice".


- If the authors claim "MLP with Regularization is all you Need", then why only compare against the NN model in AutoGluon Tabular, why not compare against the whole automl system which as far as I know combines NN with GBDT models?

- The authors compared against AutoGluon without HPO and with only NN models, claiming its stacking is superior to HPO. However to me, it seems stacking is not nearly as useful with only NN models and is intended for settings where many different models are trained. It would be good to see AutoGluon + HPO as well if only its NN models are being used.


- It would be good to compare plot the performance vs number of HPO trials on a dataset where MLP+C is better than XGB for both methods. Is it the case the XGB performance plateaus after certain number of HPO trials, while MLP+C performance continues to increase with additional HPO?  If so, this could indicate the XGB hyperparameter search space is too small relative to the allotted compute/runtime.
Relatedly it would be good to know how sensitive the results are to the specific compute/runtime limits chosen for on all methods here.


- Were any of the regularizers never selected in the cocktail? These could be helpful to point out.


- It would be interesting if the authors could investigate what characteristics of a dataset make it such that MLP+C >>> XGB versus the reverse. For example XGB crushes MLP on the ldpa dataset, and is crushed by MLP+C on arrhythmia. 


- It may also be interesting to produce a figure like sklearn's classifier comparison:
https://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html
showing the effects of each regularizer on the MLP decision boundary on a toy 2d dataset. 


### Update after reading author response to my original review ###
I find the paper stronger after the additional experiments the authors have run and have slightly increased my score. That said, I still find this a borderline paper. The biggest remaining concerns I have that do not seem to have been addressed in the authors' response include:

- the title needs to be changed in my opinion. It's clearly not close to "all you need" if the authors acknowledge autogluon with all of its different models will be stronger than just their regularized-MLP. 

- the authors should more clearly explain what criteria they use to decide whether a hyperparameter is regularization-related or not. Again, to me almost every hyperparameter affects inductive bias in some way. There should be a more systematic explanation for why these particular hyperparameters were considered in the search space but not other standard hyperparameters that are typically tuned in MLPs.

- the early-stopping experiment of XGBoost with num_round = 4000 could still be made more convincing. In particular, num_round could be set to a much greater value no, such that early-stopping is guaranteed to hit?  I feel at the very least num_round should be set to the maximal value such that XGBoost training runtimes do not exceed the MLP training runtimes.

- autogluon with just its MLP model should be run with HPO. I believe the autogluon recommendation of "use stacking instead of HPO" only applies when the full system with all of its models are being utilized. If you are only utilizing MLP models, then I believe HPO would outperform stacking.  To reiterate: stacking seems a bit odd to me when the ensemble only contains one type of model as in your usage.

That said, I believe the authors can easily address these in a revision, and I do think this is a high-impact paper with an important conclusion for the ML community. Papers like this are difficult to write and sometimes harshly judged since there are uncountably many comparisons/analyses that could've been run, but I think the choices made by the authors appear reasonable to me and sufficient to convincingly present some important conclusions.
limitations_and_societal_impact: I believe the authors have adequately addressed societal impact & limitations of their work.
One unaddressed limitation is that given limited training time, it's unclear whether training both MLP+D and XGBoost followed by model-selection would be competitive to MLP+C (since XGBoost is presumably much quicker to train and both MLP+D and XGBoost presumably require less HPO trials than MLP+C). Also the paper didn't consider more advanced boosting variants like CatBoost and used questionable label-encoding of categoricals for XGBoost.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: On a wide selection of tabular data datasets, this paper showed empirically that well regularized MLPs outperform recent tabular data specific architectures as well as classical ML methods on tabular data classification tasks. In particular, an array of regularizes was studied experimentally, and a principled choice of methods and hyperparameters leads to  high performing MLPs.
main_review: This paper makes a compelling case that MLPs have potential to outperform domain specific architectures. The results and figures presented show careful assessment of the experimental design. While there is not much motivation provided, the fact that MLPs  are capable, albeit with some help, to beat older methods is maybe not surprising, but the fact that a well-regularized MLP can outperform TabNet and NODE is interesting. It begs the question: Could the TabNet and NODE be regularized, or otherwise have their hyperparameters tuned and perform better yet?

Strengths: Thorough and principled experimental demonstration of the main point of the paper. Informative and legible plots, in particular, Figure 1 is easy to digest. Clear and concise writing style.

Weaknesses: There is little mentioned about why a given "regularization cocktail" might be a good combination. Also, the comparison to recent neural architectures is limited in that the recent methods are not as optimized as the MLPs. To the authors' credit, this limitation is noted in the final section of the paper and it may not harm the significance of these findings.

In summary, the main finding is important for the community to understand, even if it is not wildly surprising. Intuition and explanation behind the best performing combinations of regularizers would be nice.

___ 
Reaction to authors' response: I went back to reread existing work on NNs for tabular data. It was correctly pointed out by the authors that even works like TabNet (Arik and Pfister 2020) and TabTransformer (Huang et al. 2020) do not make an entirely compelling case that NNs outperform XGBoost. For that reason, I think additional empirical works like this paper are valuable to the community.
limitations_and_societal_impact: Yes, both the "Limitations" and "Societal Impact" are adequate.
review_title: null

Meta-review: The reviewers found this paper somewhat "unsurprising", but potentially high-impact and useful to the community. They also noted that papers like this one can be judged harshly, subjecting the authors to a potentially infinite number of experimental comparison. In this case, the consensus was that the revisions/responses from the authors did a good job addressing the concerns of the reviewers. We recommend adopting the suggestions arising from the discussion phase, particularly around the choice of title. 
Decision: Accept (Poster)