Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper
1) conducts a careful comparison of many deep learning methods for tabular learning
2) introduces a new baseline method for tabular learning (ResNet)
3) introduces a new fancy architecture for tabular learning (FT-Transformer)
main_review: The paper is very closely related to (and partially superceded by) [1],
but since [1] appeared on arXiv after the paper was submitted
I will try to evaluate the current paper as though [1] did not exist.

The writing is clear and the experiments seem careful and thoughtful.
Contributions 1) and 2) are important and useful.
Unfortunately, the new baseline 2) rather steals the thunder from the
new architecture 3).
The experiments don't make it clear that the new architecture is worthwhile.
The authors are very straightforward about the limitations of FT-transformer
(and its much greater cost compared to ResNet), which I appreciate.

Hyperparameter tuning is a key issue for a comparison. 
Given more time [or, more generally, computational resources] 
and more parameters (and enough data), DL models tend to improve.
Were some of these methods given more time than others?
(Including time to choose hyperparameters?)
I'd like to have seen performance as a function of time, or after a fixed amount of time.

As a more narrow question, why Optuna? Does it matter?
(Looking into the future, [1] uses BOHB [2]. Do you think swapping the hyperparameter tuner would affect your results?)

* 185 "in the ensembling mode" -> define
* 202 ALOI -> meaning of tilde is confusing, say "The ALOI dataset has nearly 1000 classes".
* I found it easy to miss the definition of "heterogeneous". Maybe use a definition environment for it?
* 240: FT-transformer is not sensitive to hyperparameters - I'd like to see more experiments demonstrating this, and looking at the quality of model as a function of training time (including hyperparameter optimization time)

[1] Kadra, Arlind, et al. "Regularization is all you Need: Simple Neural Nets can Excel on Tabular Data." arXiv preprint arXiv:2106.11189 (2021).
[2] Falkner, Stefan, Aaron Klein, and Frank Hutter. "BOHB: Robust and efficient hyperparameter optimization at scale." International Conference on Machine Learning. PMLR, 2018.

limitations_and_societal_impact: Fine.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper investigates the performance of deep learning models on tabular data and compares it with traditional models such as gradient boosted decision trees. It shows that ResNet is good baseline for deep learning models on tabular data. It also proposes a new attention based architecture which surpasses all the deep learning methods mentioned in the paper.
main_review: Pros:
- The study is quite comprehensive covering a range of datasets
- The paper analyzes multiple models from each category
- The proposed transformer based model surpasses other models in most tasks and is good baseline for future tasks
- All models are explained in detail in the supplementary materials for reproducibility
- Training runtimes are also compared

Cons:
- FT-Transformer, the proposed method, is more complex and takes longer to train on average than ResNet

Overall, the paper is very useful and provides a great overview of the different approaches for tabular data. By proposing a new method which surpasses the approaches mentioned in the paper, it even adds to the existing body of baselines for future work in the area. The code is open sourced and the models are discussed in great detail in the Appendix. 

limitations_and_societal_impact: Yes they have.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper set up a standard benchmark for recent works on NN for tabular data, and GBDT, to check whether these works indeed outperform GBDT or not. Besides, two new models, ResNet-like MLP and transformer-based models are proposed, and seem can outperform previous works.
main_review: Strengths:

- The standard benchmark is very helpful. With it, we can know which models are better. I believe this can boost the research for the tabular data learning domain. 

- The design of the FT-Transformer is simple and solid. Pre-ln is a good choice. The bias term in Tokenizer seems helpful.

Weakness & questions:
- As both xgboost and catboost are set to level-wise trees, it would be better to have the results of the leaf-wise tree, like LightGBM.

- The selected benchmark datasets are quite messy. It will be better to categorize them by applications/scenarios. Besides, as tabular data is widely used in ads click prediction and recommenders systems, it would be better if the benchmark can cover these two types of tasks. 

- For the experiment results, it seems GBDT is not outperformed by NN in tabular data. I notice there are "heterogeneous" and "non-heterogeneous" datasets in this paper. Obviously, the heterogeneous dataset is more like the tabular data in the real-world (Although multi-modal data is more common, I think we should focus on heterogeneous tabular data, as NN is already very good in other data. As for high-dimensional multi-class data, it actually is not common in real-world applications with tabular data.). From the results, GBDT is still the best one in heterogeneous data. To avoid confusion, I think the author should make the table clear (even use two tables), let readers know there are two kinds of datasets. 

- Based on 3, I think the author should make the claim more carefully.  In my opinion, the heterogeneous dataset is more like tabular data. So the sentence like "we show that the choice between GBDT and DL models highly depends on data and there is still no universally superior solution." looks ambiguous.

- Do you have a plan to host a leaderboard, like GLUE in NLP?
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: - The authors conduct extensive experiments on tabular data learning with various datasets and baselines.
- The authors discover some important takeaways for tabular learning such as when we should use shallow or deep models.
- The authors propose a novel tabular learning framework which shows the state-of-the-art performance in various datasets.
main_review: 1. Impactful paper
- This paper tackles an important problem which has significant attention from both academia and industry.
- Constructing standard experimental settings, datasets, metrics, and baselines in this field would be highly beneficial for future research in this area.
- Also, it is important for practitioners which model works well for which condition. When we should use which model is a critical question for many practitioners.
- The published codebase is clear and easy to understand and reproduce.

2. ResNet
- Have any previous works tried ResNet type of architecture for tabular data?
- If yes, it would be good to cite those papers. 
- If not, this would be another contribution of this paper.

3. Experimental details
- How to treat the categorical variables? One-hot encoding? 
- Would be good to describe the number of feature ranges in the main manuscript.
- Can we use "patience" as another hyper-parameter? How 16 is determined?

4. Sanity check
- It would be good if the authors can provide whether the implementation of each model can reproduce the results in each paper.
- For instance, the authors can utilize at least one data per each baseline and check whether the results can be reproduced.
- This would be important because it can be a sanity check whether the detailed implementation is accurate for those baselines.

5. Etc
- It would be good if the authors can clarify which dataset is regression, binary, and multi-class classification tasks in results tables.
- I would like to know whether the performance of an FT-transformer is dependent on the input order of features.
limitations_and_societal_impact: The authors properly describe the limitations of the paper and baselines.
However, it would be better if the authors can somehow tone down for the discovery and conclusions because those are only supported by the empirical results.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper provides a comparison of recently proposed deep learning (DL) models for tabular data against established gradient-boosting decision tree (GBDT) methods.
The experimental evaluation shows that there is no clear winner between DL and GDBT models.

The paper also evaluates an MLP with residual connections ('ResNet') and introduces a Transformer-based model (FT-Transformer) similar to AutoInt.
The proposed architectures (ResNet/FT-Transformer) are demonstrated to give highly competitive performance, with FT-Transformer often performing SOTA.
main_review: ## Strengths

I agree with the authors that there is a need for a thorough review and comparison of 'deep learning methods on tabular data'.
I like their overview of recent DL approaches in the introduction (and would have welcomed a more extensive overview).
The hyperparameter tuning of all involved models seems thorough at first impression.
The investigation of functional biases (S4.1) between GBDT and Neural Networks is an interesting idea.

## Weaknesses

Unfortunately, the weaknesses may outweigh the strengths for this submission.
The paper is uncomfortably split in two between (1) the comparison of established models and (2) the introduction of the FT-Transformer.
I believe that both parts would benefit from additional work.

### 1. Model Comparison

For a paper which lists the 'thorough' comparison of models on a 'wide range' of datasets as a key contribution,  the chosen selection of datasets is not adequate for a variety of reasons:
1. Only one of the datasets has categorical features. All other datasets have exclusively numerical features.
    * Categorical features are generally regarded as more challenging (especially for deep learning), so this omission may affect conclusions.
    * For this one dataset, the authors do not employ one hot encoding, which may negatively affect performance for some models.
2. 11 Datasets is definitely on the small side. (The SELU paper compares performance on 121 datasets.) 
3. All datasets are between 20k and 1M samples.
    * There are a lot of UCI datasets with <20k samples. Why were they excluded? Surely these are relevant for such a general comparison?
    * Why were datasets cut to <1M samples? E.g. the Higgs dataset (usually 11M samples) was cut in a non-standard way to <1M samples.

Further,

4. one of the conclusions of the paper is that GBDT outperforms DL models on 'heterogeneous' but not on 'homogeneous' data.
    * The separation into homogeneous and heterogeneous datasets performed by the authors *perfectly* aligns with this conclusion.
    * However,  the specific reasons for how datasets where classified as homo-/heterogeneous is not explained in detail and feels rather arbitrary to me.
        * The authors state that a dataset is heterogenous if it describes 'different physical properties' and is of 'diverse units of measurements'. As far as I can tell, the Higgs dataset should be heterogenous by this definition, but I would be happy to hear explanations from the author. (Higgs contains a bunch of particle physics properties. Why are these less homogeneous then the housing-related attributes of California housing?)
        * For epsilon, I can't find any information online about the composition of its features, so why is it homogeneous then?
    * The perfect separation between GBDT and DL performance on heterogeneous vs homogeneous datasets as well as the lack of justification makes me suspicious that it is not impossible that *the authors came up with the classification of these datasets into hetero-/homogeneous only **after** observing the results*.
     * Also, no explanation of why performance of GDBT and DL differs on hetero-/homogeneous datasets is given.

### 2. Experiments

5.) For the Adult dataset, I have compared the reported performances in this paper to publically available results at https://www.openml.org/t/7592, which shows a large 0.1 difference between the top reported AUROC values. This might be indicative of problems with the experiment routine. I have not found such irregularities for a few other datasets that I've checked.

6.) The paper makes claims of superiority of one method (e.g ResNets / FT-Transformer) over another across all observed datasets.
Why not performance in terms of average rank order across models directly, instead of eyeballing things?
Additionally, I would have preferred for the authors to also quantify variance across different splits (instead just over different model seeds), especially for datasets without established test sets.

### 3. FT-Transformer

7.) The paper is split weirdly between 'model comparison/literature review' and introducing the 'FT-Transformer' architecture. Although both parts are individually clear, the introduction of the FT-Transformer architecture comes up somewhat abruptly in the middle of the paper, after the reader has already seen results for the FT-Transformer, affecting clarity of the overall submission.

I feel that making a big deal of the FT-Transformer as a contribution generally hurts this paper, because

* It takes away space that could have been used for a more thorough comparison of architectures
* The differences to AutoInt are small, despite what the authors claim. (Adding biases to the encoding sounds like a one line code change that seems to add a 0.01 improvement. Adding the mask tokens at input is more interesting but is not ablated.)

However, both the ResNet-MLP and the FT-Transformer architecture are potentially interesting as a baselines for future work and maybe the authors could have spent more time discussing the tricks needed to make them work well. 
(Further, the conclusion that MLPs aren't horrible for tabular DL are in line with other recent work (https://arxiv.org/abs/2106.11189, previously at https://openreview.net/forum?id=2d34y5bRWxB).)

Overall, I feel like the split between 'review comparison of prior work' and 'introduction of new models'  does not work well in this submission.

### 4. Architecture Bias Experiments

The idea of randomly generating targets from GBDT/NN to study biases of the particular architectures is interesting.
However, I feel that the idea is underdeveloped in this submission:
 * What's benefit of linearly interpolating targets? (In my mind, this doesn't add anything as only $\alpha \in [0, 1]$ has a clear meaning.)
 * Why not generate data from the transformer as well? 
 * Why is XGBoost not included here? Why not extend this to all models in the spirit of the 'big comparison' paper?
 * Fig. 3 seems to suggest that CatBoost can accommodate all data while ResNets can't acccomodate CatBoost-data. How does this relate to the experiments (where ostensibly, ResNet > CatBoost for some datasets).
 * Can this be related to the statements about hetero-/homogeneous datasets?

## Conclusion

Making a good literature review and model comparison paper is hard.

This paper is split between trying to provide that comparison and introducing the FT-Transformer.
I am not convinced that the FT-Transformer contains enough information to warrant publication.
The comparison between models might be enough if appropriately extended.
However, I feel that a significant number of the above comments need to be addressed first (mainly: more datasets, compare over different CV splits, show rank order, extend 4.1 to more methods).


[edit] Score updated to a 5. See discussion below.
[edit] Score updated to a 7. See discussion below.
limitations_and_societal_impact: The paper discusses limitations of the FT-Transformer in a paragraph in S. 3.6, where they identify the quadratic scaling in the number of features as the main limitation of the architecture.
Given that a main contribution of this paper is the comparison of architectures, a discussion of the limitations here would have been nice.

I believe the authors have misunderstood the request for discussing 'potential negative societal impacts' of submissions: in the checklist, they again reference S. 3.6 where they discuss only scaling limitations and how to (potentially) alleviate them in future work. It is unclear to me how this relates to societal impact.
review_title: null

Meta-review: All reviewers have agreed on the positive aspects of the paper, and its important contributions for tabular deep learning. There were significant additions by the authors during the review process, particularly extra experimental results, method motivations, clarification of experimental setups and hyperparameter tuning. It is important to reflect these in the final version of the paper. Finally, as discussed, it would be an impactful addition to open-source the benchmarking framework. 
Decision: Accept (Poster)