Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper aims to understand and establish, based on empirical evidence, if and why tree-based models tend to perform better than neural networks on tabular data. Existing work has indicated that this is the case, and the current paper aims to conclusively determine this over a wide range of tabular datasets. The first preliminary result, shows that the average performance of tree-based models is better than NN-based models, when controlling for hyperparameter search via the number of random searches used. The paper then aims to understand why this is the case, by considering the following 3 case feature-level hypotheses: i) neural networks perform better on smoothed-out features; ii) they are less robust to the presence of uninformative features; iii) they are rotationally invariant even when the data is not. The paper puts forth these as the main inductive biases that lead to the superior performance of tree-based models.
strengths: - The paper is clearly written and well-motivated. The problem considered in the paper is an important one to enable the adoption of NNs in domains apart from vision and NLP.
- The experiments in the paper are interesting and well-designed. The considered inductive biases make intuitive sense as the ones likely to account for the difference in performance between NNs and tree-based models.
- Some of the conclusions in the paper, such as the fact that breaking invariances can lead to better performance on tabular data, are potentially critical to the design of more performant NN architectures for tabular data
weaknesses: - One area concern for me in the experimental design was the presence of several magic numbers and vague descriptions of criteria for the choice of reference tabular datasets in Section 3.1. For example, why are high-dimensional datasets not used? What aspect of the dimensionality would invalidate the results in the paper? Why is 3000 the cutoff for 'too few samples'? Similar concerns impact the choices in Section 3.2. 
- Other types of data smoothing apart from Gaussian smoothing could be considered. 
- The paper could have included some experiments with NNs that break rotational invariances.
correctness: The experimental design is largely sound, but has some choices that need to be justified better for soundness' sake.
clarity: Apart from some clarity issues in the experimental design description, the paper is well-written.
relation to prior work: I am not well-versed enough in the prior work to judge this, but the paper seemed to contain sufficient references.
additional feedback: See weaknesses
review_title: Review for 'Why do tree-based models still outperform deep learning on tabular data?'

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: "Tabular" data comprises most of the uses of machine learning in the wild, including the large majority that don't get press; ensembles of tree-based learners, e.g. with boosting/generalized additive models, have long predominated in this area. Yet deep learning models have shown distinct advantages for more structured data. This paper establishes a benchmark of 45 varied tabular datasets to assess the situation, as the title question asks. 

The authors benchmark state-of-the-art tree-based learners against state-of-the-art deep architectures for these datasets, finding that the tree-based learners perform significantly better. Special focus is put on accounting for different hyperparameter optimization budgets. Investigating why the tree-based learners perform better by transforming training and test data in various ways, the authors highlight the importance of dealing with uninformative features, rotation non-invariance, and non-smooth prediction functions.
strengths: The authors go beyond previous benchmarks in several commendable ways.
- Clearly there is a consciousness of the importance and variation (and arbitrariness) of hyperparameter choices in models. Wonderful to see the hyperparameter grid be part of the benchmark. Hope this sets a standard; the non-smooth and small nature of tabular data problems motivates this concern very well in practice.
- "Not too easy" processing (line 115) is very sensible and I've seen it done in paper evaluations, but somehow not on a benchmark of this type - great!
- The tabular datasets themselves are larger and more varied than prior work represents.
- Confidence bars on everything, and the data behind them also provided as part of the benchmark. Given the size of the final error bars, it's clear the scrupulousness is necessary to draw conclusions from the data.

Meticulous benchmark papers for tabular data are not very common, and certainly underrepresented relative to their usefulness and need.
weaknesses: The work delivers some high-level takeaway messages that would be unsurprising to practitioners already working with tabular data:
- Tree-based learners tend to still be superior to deep learners on tabular data.
- Tree-based learners have an easier time handling nonsmooth or uninformative features.
The findings are unsurprising because they're widely understood (though this benchmark is now one of the easiest ways I know of to demonstrate these findings). 

Another area of improvement for the benchmark is in testing other uses of these models beyond 0-1 classification: 
- I'm a little nervous about preprocessing the data to make binary classification problems out of multiclass problems. The geometry of these may result in different characteristics to multiclass problems, and there are plenty of datasets for which the paper's method (using just the two most common labels) discards a lot of the interesting signal.
- It would be nice to evaluate using many different metrics instead of just accuracy. For confidence-rated prediction, class-imbalanced test data distributions, and other challenging scenarios, different loss functions suggest themselves. There's something to be learned from evaluations in the past, like [Caruana & Niculescu-Mizil '06 "An empirical comparison of supervised learning algorithms"] which justify doing this. This also provides a perspective on the different losses used to train these models. 
- The method used in Sec 3.4 still could have has issues in aggregating results, as it would inherit selection biases in overrepresenting particular datasets. Possibly relevant is a game-theoretic take on evaluating performance on datasets, e.g. from Balduzzi et al. '18 "Re-evaluating Evaluation."

The problem space benchmarked is not very large (7 models * 45 datasets) because the hyperparameter space search is meticulous. The datasets and model configurations are so extensively documented, and this is a problem of significance to many people. It would be helpful to crowdsource expansion of the grid search, and of the benchmark in general to new models/datasets. The authors have set the standard and framework for evaluation; with a small amount of continuing effort, this could be a more long-lasting resource to the community.
correctness: The method for doing evaluations seems sound. Everything in the github repo is posted as advertised.
clarity: The paper is very clear. Both results in Sec. 4.2 are practical findings that could be surprising to many tabular data learners, and perhaps could be moved up in the paper; on inspection they're currently not mentioned before/except in this section, in the middle of page 5. 

This paper's benchmarking of hyperparameter variation with respect to a collection of datasets would be of interest to explore the properties of the learned embedding. When deciding whether to explore deep learning on a particular tabular problem, a major motivator is getting a learned embedding of the data for downstream tasks. It would be interesting to see the performance of the embedding method at preserving similarities between examples, how well nonlinear classifiers do in the learned space, and how this all relates to test set accuracy.
relation to prior work: Clearly discussed. Would be good to cite Caruana & Niculescu-Mizil '06 and borrow from there.
additional feedback:
review_title: Meticulous evaluation to study highly relevant questions

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper conducts comprehensive experiments to demonstrate that tree-based model still outperforms the deep learning based model. To understand this gap, the author also conducts an empirical investigation into the differing inductive biases of tree-based models deep learning model. 
strengths: The paper defines 45 datasets from various domains with different data sizes and feature types, which is a good contribution to tabular data community. The empirical investigation for inductive biases looks interesting.
weaknesses: The paper only conducts experiments on small and medium size of datasets, people would wonder the comparison between tree-based model and deep learning based model on large scale datasets. Author may need to add more large datasets, although most large datasets are unable to satisfy the criteria proposed in the paper. Large datasets in the paper are truncated to 50,000 samples, e.g., original Covertype owns nearly 500,000 samples, but the paper only keeps one tenth of original dataset. [1] shows the superior performance of FT-transformer on large dataset, so the author may need to carefully claim "tree-based model still outperform deep learning on small scale tabular data". And large dataset with 50,000 samples defined in this paper still looks like a tiny dataset for me. 

[1]. Gorishniy, Yury, et al. "Revisiting deep learning models for tabular data." Advances in Neural Information Processing Systems 34 (2021): 18932-18943.
correctness: The paper constructs small tabular datasets and probably evaluate different methods based on these datasets. 
clarity: The paper is well-written and easy to follow. 
relation to prior work: The paper clearly discussed the difference from previous contributions. 
additional feedback: Please see the weakness.
review_title: Review for "Why do tree-based models still outperform deep learning on tabular data?"

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors propose a comparison of tree based methods with modern deep learning techniques on small-medium sized Tabular data. They perform a large-scale hyperparameter tuning sweep of numerous tree and deep learning models across a newly proposed benchmark of 45 datasets and compare results of the models. They investigate the reasons why tree models consistently outperform deep learning models including uninformative features, data orientation preservation, and irregular functions.
strengths: Given the large amount of papers claiming neural networks outperforming tree models on tabular data, and the subsequent series of refutation papers, this topic is timely and important to the community. While the Tabular Data: Deep Learning is Not All You Need paper has many similarities to this paper, this paper builds upon the prior by moving away from the limited and convoluted scope of the datasets used in the original neural network papers and towards a larger and more unified benchmark of 45 tabular datasets. The number of datasets is quite important, as to get a high confidence perspective of the state of these systems, many more datasets are required than are typically contained in papers claiming SOTA with a new method. The method to select datasets is reasonable (although I'd like future work to consider incorporating missing data into datasets as it is very common in real-world datasets).

Beyond an improved benchmark, this paper also covers reasons for the performance gap, which is of practical importance to the development of tabular NN models. In particular the authors make a very nice observation on data rotation which reverses the performance order of the models.
weaknesses: L247: Random forest feature importance is not a very accurate method for computing importance, and could induce biases when selecting features to prune according to the definition of importance by random forest (RFs may leverage features more closely to GBMs than NNs, causing important features of GBMs to be removed later than important features for NNs). Instead, iterative permutation shuffling feature importance and pruning on a per-model basis is found to be a more effective method that more closely aligns with the true importance. This would also enable a deep dive into which features are important for trees vs NN models. Overall this is a fairly minor point, but wanted to make the authors aware of alternatives. Source: https://explained.ai/rf-importance/

While the authors make the wise decision of setting the epochs to 300 and using early stopping for NN models (with 300 being a large value that often won't be reached), this is oddly not done for tree models. For example, num_estimators is used as part of the search space for XGBoost, Random Forest, and Gradient Boosting. For Random Forest, this is strange because RF will not overfit to validation data, and having fewer trees will simply make the model worse on average. It would be much more sample efficient to fix the num_trees to a single large value such as 2000, as it is redundant to search. For XGBoost and Gradient Boosting, simply use early stopping with a large patience and a very large num_estimators value, such as 10000. This will avoid needing to search for the optimal value and will greatly improve search efficiency. For the paper results, this likely won't change much due to the severity of the gap in performance between NN and trees, but it will be very important for future work if NN models become more competitive.
correctness: The paper appears to be correct in its analysis and setup.
clarity: The paper is very clear and well written.
relation to prior work: Yes, the authors cite important prior work and describe how their work extends prior work.
additional feedback: While the authors mention that the AutoMLBenchmark has issues with incorporating synthetic datasets and non-tabular datasets, I find that the codebase may be a powerful tool to re-use as it supports many datasets and has ready to use implementations of many algorithms with heavy adoption within the tabular AutoML community. Re-use and refinement of a singular tabular data benchmarking repo that is scalable and trusted for research in this vein would be greatly beneficial to the community, and it is unclear if the authors benchmark will become yet-another-tabular-benchmark or if they intend to differentiate their benchmark beyond the improvement in the choice of datasets / removal of missing values. I would be interested in clarification from the authors on this point.
review_title: Deep Learning Continues To Not Be All You Need

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors aim to compare deep learning models and tree based models on tabular data. They contributed an extensive benchmark consisting of both tree-based learners and neural networks, which is run on a set of 45 datasets from various domains. They showed that tree-based models remain state-of-the-art on medium sized tabular data and proposed explanations for this phenomenon. They also highlighted desirable biases for tabular data to guide future researchers.
strengths: The research work is novel and its potential impact is evident. It not only provides a unprecedentedly large and publicly available dataset and benchmarks, but also reveals the inductive biases of tree-based models and neural networks that could influence their performance on tabular data.

The paper is written in an organized manner, with detailed illustrations to aid the audience to understand the results. The authors made good use of figures and plots to highlight trends and point out their observations, especially in Section 5.

The authors build upon their findings and explain the limitations to their work and provides suggestions on how future research involving tabular data could be improved.

The dataset is publicly available, and this work is the first standard benchmark for tabular data that follows several clearly defined criteria. Results of their findings are presented in graphs with clear figure descriptions. The authors also mentioned limitations and future research directions.
weaknesses: The empirical evaluation (the main strength of the paper) is not comprehensive enough and may deserve more effort to find more about this difference in behavior. Additionally, although the authors indicated that they are the first to do so, they haven't argued about why it is that important and how future research might benefit from such empirical study.

There are no benchmarked tasks on the dataset. Just “Classification” and “Regression” in Figures 1 and 2 are not clear to readers.

The authors are encouraged to make a table to compare their dataset and the other related work mentioned in section 2. The authors should also consider including more details on their benchmark and dataset (such as hyperparameters, train-test splits) in the appendix section.
correctness: Everything seems correct to me
clarity: While the paper is written in an organized fashion, it has some minor language issues. I encourage the authors to make some syntactical revisions.
relation to prior work: The paper thoroughly discussed previous related work and highlighted why their research work is needed.
additional feedback: It would also be interesting to see the performance of tree-based models and NNs on large datasets. When the authors include this in the appendix, they should consider explaining whether the inductive biases that make tree-models favourable for tabular data still holds true for large datasets.

Out of curiosity, why does having images and artificial datasets in addition to tabular data prevent some benchmarks from being used in tabular deep learning? Do they only have to include tabular data so that we can use them in benchmarks extensively? It seems like you recommend so in the third paragraph of section 2 and I want to hear more about that, especially since this sentence is not cited.
review_title: Why do tree-based models still outperform deep learning on tabular data?  - Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This paper aims to answer the question of why tree-based models outperform neural networks on tabular data.
Contributions are:
* A new collection of 45 (already public available) tabular datasets (classification and regression tasks) that have been (partially) preprocessed to establish a homogeneous collection with ``clear characteristics'' of tabular data
* A benchmark study comparing tree-based models (XGBoost, random forest, gradient boosted trees) and neural networks for tabular data (MLP, Resnet, FT Transformer, SAINT) on this new benchmark; results show that tree-based models overall outperform neural networks
* An empirical investigation of the inductive bias of tree-based models, contrasted with that of neural networks for tabular data, which serves as a basis for answering the question of why tree-based models outperform neural networks
* (Raw data as obtained from performing the benchmark, including the HPO runs)
strengths: * Tabular data are still highly relevant and so is the question of which type of model to use and whether neural networks are worth it
* Datasets part of the benchmark are made accessible through re-uploads or existing uploads via OpenML; code is provided via a public GitHub repository
* The most interesting contribution of the paper is in my opinion neither the new benchmark, nor the result that tree-based methods outperform neural networks (which was already reported in some prior work - which the authors correctly cite), but rather the experimental investigation of the inductive bias of tree-based models compared to neural networks (Section 5)
  - The experimental design is convincing and the conclusions (neural networks being biased towards smooth solutions, MLP-like networks not being robust to uninformative features and rotation invariance hindering performance on tabular data) are sensible and mostly go in line with hypotheses (and findings) of prior work
  - Section 5 therefore provides a testable hypothesis to the research community regarding the design of future neural networks for tabular data (i.e., a neural network that can learn irregular representations, is robust to uninformative features and not rotation invariant should show strong performance)

weaknesses: * Overall, the paper reads much more like an empirical investigation rather than a presentation of a new tabular benchmark; in essence, the paper provides both (by performing the empirical investigation on the new tabular benchmark); but it is unclear of what additional benefit this new collection of existing datasets will be, i.e., should all new approaches for supervised learning on tabular data now be benchmarked on this new collection? If yes, one may argue that the current collection may be too homogeneous, due to the preprocessing and all the restrictions listed in 3.1
* Also, the need for coming up with a new collection of tabular datasets is not thoroughly justified, i.e., the authors mention both OpenML CC-18 (72 classification tasks) and the automl benchmark (39 classification tasks) but state that ```[they] contain tabular data, but also include images and artificial datasets, which may explain why they have not been used in tabular deep learning papers```, however, the majority of tasks in the CC-18 and automl benchmark are based on standard tabular datasets (with some exceptions like ```fashion-mnist```); also a question I had in mind when reading the paper was: Would main results and conclusions have looked different if benchmarks were performed on CC-18 or the automl benchmark instead of the new collection? Probably not, due to overlap of datasets being used in all three benchmarks (?)
* For hyperparameter tuning, a random search with a budget of around 400 iterations was used for each learner (defaults used as the first iteration). While this allows for a ``fair'' comparison, it could also be the case that the HPO problem is much more difficult for neural networks than for tree-based models (where HPO loss landscapes may be more benign). If this would be the case, part of the results could also be explained by not using a sophisticated optimizer, i.e., it could be the case that Bayesian Optimization can identify much more potent configurations of the neural networks which could allow for closing the performance gap (partially). All in all, this is only speculation, but could be investigated in a small ablation study.
* While I do find the empirical investigation regarding the inductive bias of tree-based models and neural networks interesting (Section 5), the analyses were mostly performed visually, e.g., by inspecting box plots. The experimental design, however, would, in principle, allow for more detailed statistical analyses, e.g., modeling the test score as a function of the learner and length scale (of the smoothing; in Section 5.1) and testing for significance of effects

Some additional (mostly minor) remarks:
* In the abstract and main paper it is stated that `45` datasets are part of the benchmark. However, in the Appendix, in total `55` datasets are described: numerical classification 15, numerical regression 20, categorical classification 7, categorical regression 13 (this is also in contrast to the numbers listed in the figures, i.e, Figure 1: 15, 19, 7, 14
* It may be helpful to discuss why one-hot-encoding of categorical features was employed (instead of, e.g., impact encoding or entity embedding with the latter one more frequently being used in neural networks for tabular data (Guo & Berkhahn, 2016))

All in all, my rating is mostly determined by the fact that I do not see much benefit of the new benchmark collection itself (compared to CC-18 or the automl benchmark) although I do find the empirical investigation interesting.
correctness: The benchmark collection is constructed in a sound way, although quite homogeneous (on purpose).
The experimental design is appropriate.
Evaluation is mostly performed visually - please see also my comment in the Weaknesses Section above.
clarity: The paper is well written and can be followed along easily.
relation to prior work: Prior work is discussed clearly.
The own contribution is stated clearly.
additional feedback: When trying to run ```python data/download_data.py``` (after cloning the GitHub repository) I get the following error (probably due to some directories not being created on the fly):
  ```
  Saving datasets from suite: numerical_regression
Downloading dataset cpu_act
Traceback (most recent call last):
  File "/tmp/tabular-benchmark/data/download_data.py", line 29, in <module>
    save_suite(suites_id["numerical_regression"],
  File "/tmp/tabular-benchmark/data/download_data.py", line 17, in save_suite
    with open("{}/data_{}".format(dir_name, dataset.name), "wb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'data/numerical_only/regression/data_cpu_act'
  ```
review_title: Official Review of ``Why do tree-based models still outperform deep learning on tabular data?''

Meta-review: metareview: The paper proposes a benchmark comparing modern neural network architectures with gradient boosting.

Overall the paper was reviewed quite positively with an average score of 6.33.

Most criticism by the reviewers was addressed by the authors.

Major remaining issues are:

1) While the results of BO vs random search indicated by reviewer w58A is indeed surprising and worth further investigation.

2) That the uncertainty estimates are only sampled by shuffling a single random search run is indeed not ideal. This makes the uncertainty in the later parts of the plots less reliable. 

3) The benchmarks makes no use of a standardized platform like AMLB. While I would very much prefer this, I don't think it can be a requirement for such a paper.

4) The mismatch between early-stopping and tuning the number of iterations of NNets and Boosting.


While all these points are valid, I would argue that the main findings of the benchmark are correct, the benchmark set itself is interesting and further research can look into some of the issues mentioned in 1) - 4).

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept