Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors propose a novel architecture with sequential attention modules for tabular learning. An attention module is trained to select some elements from the (normalized) input feature, and a feature transformer takes the selected features for overall feature embedding. The model is evaluated on multiple tasks, and the proposed method outperforms prior approaches in the paper.

I am in favor of this paper as it proposes an interpretable method for feature extraction in tabular learning. The learned overall feature embedding is shown to be effective in multiple tasks. It seems to me that the proposed method may have high generalization capability because the method is trained to select sparse feature attributes for decision. The authors may study the generalization performance further.

The experiment section may not be very reliable. The numbers indicated with * are copied from other papers, but these numbers may not be compatible if there is any discrepancy in the experiment setup. It would be better if the authors can run the baselines on the datasets. 

I am a little disappointed that no study on the number of steps needed for the model. It seems non-trivial to me. In TabNet-L, the step size is 5 but in TabNet-M, the step size is 7 (even smaller?). How should we choose the right step size?
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposes a novel deep machine learning model called TabNet for learning from tabular model. The system builds on sequential feature acquisition that is aggregated to perform decisions. The performance of TabNet is evaluated in a set of experiments.

The paper proposes a network architecture which seems to achieve good performance on a set of experiments. However, I am missing a motivation for parts of the model. A broader ablation study in which certain components of the model are replaced by simpler network structures could be instructive; the current ablation study only investigates hyper-parameters and a few design choices -- but what happens for example if a feature transformer is simply replaced by fully connected layer instead of the used architecture, i.e. how general is the structure? Regarding experiments I am also missing details and a broader comparison with other methods.

Some more questions/comments:
* What is the training criterion used?
* In Table 1, what precisely is AUC? Area under "accuracy over selected features"? If so, what does the number "No sel." mean?
* Why is "No sel." performing relatively bad? Overfitting? 
* What about stronger feature selection baselines, like those based on Shapley values?
* What do the errors in table 1 mean? Which numbers are marked in bold? This seems misleading when looking at the "error bars".
* What are the std errors for all other tables?
* What is the "ratio" in Figure 6a? Why does the figure not show INVASE?
* There are other instance-wise feature selection methods, e.g. Chao Ma et al.'s "EDDI: Efficient Dynamic Discovery of High-Value Information with Partial VAE" or . How does the proposed method compare against those?
* For making a claim about tabular data in general, more datasets should be considered.

Minor comments:
* Figure 1: It would support the reader to explain the blocks in the figure caption.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new neural network for tabular data which uses sequential attention to perform instance-wise feature selection which can help learn superior decision rules, in addition to facilitating interpretability of the resulting model.  Overall, the paper is well-written and fairly easy to follow, and the idea appears conceptually well-grounded and of high practical value. However, I have concerns about the experiments.

As the proposed model is primarily empirically motivated (ie. there is no theoretical justification for why it should be better), it is imperative the authors conduct thorough experiments to convincingly demonstrate its utility.  However, the current experiments are too sparse to be fully convincing, given how easy it is nowadays to evaluate ML models across diverse collections of tabular datasets (eg. OpenML, UCI, CatBoost). While other papers on tabular-data models study many real datasets (eg. Klambauer et al), this paper only studies 4 real datasets (and does not even quantify the variability in results across multiple training/test splits). 

Also, why did the authors not compare against xgboost/lightGBM/catboost, some of the most popular tabular models, in all datasets? 

It seems different hyperparameter values were also used for each dataset, what was the search-space / hyperparameter optimization method used here? Or were hyperparameters for each dataset found through manual tuning.

Klambauer et al (2017). Self-Normalizing Neural Networks. 
https://arxiv.org/abs/1706.02515

OpenML: https://www.openml.org

UCI: https://archive.ics.uci.edu/ml/datasets.php

CatBoost: https://catboost.ai/#benchmark

Other comments:

- Why don't the authors quantitatively compare the quality of the selected features from their model vs other feature selection methods for the Syn datasets?

- What is "softmax-training" in Appendix D? If the authors just mean standard training, I recommend they use this name instead.

- Since there are many components to this model, it would help clarify their purpose to the reader if the authors could provide one concrete example of the feedforward pass of a hypothetical instantiation of their model, say for a simple datapoint with just 2 features.

- the subscript i in h_i should match the i used in M[i], P[i] to avoid confusion

- P[0] (the base case) needs to be explained.

- a[i-1] should be more clearly defined, and the base case explained in more detail (I assume it is just the input features). 
 

- I think it is a bit confusing to present the model parameters in terms of batch-size B, and think it would be better if you introduced them assuming batch-size = 1 (the generalization to batch-size B is obvious and should be familiar to readers as it's the same for all neural models).

review_title: Official Blind Review #2

Meta-review: This paper constitutes interesting progress on an important topic; the reviewers identify certain improvements and directions for future work, and I urge the authors to continue to develop refinements and extensions.
Decision: Reject