TabNet: Attentive Interpretable Tabular Learning
Authors: Sercan O. Arik
Tomas Pfister
Venue:  (2020)
Keywords: Tabular data
interpretable neural networks
attention models

Abstract: We propose a novel high-performance interpretable deep tabular data learning network, TabNet. TabNet utilizes a sequential attention mechanism that softly selects features to reason from at each decision step and then aggregates the processed information to make a final prediction decision. By explicitly selecting sparse features, TabNet learns very efficiently as the model capacity at each decision step is fully utilized for the most relevant features, resulting in a high performance model. This sparsity also enables more interpretable decision making through the visualization of feature selection masks. We demonstrate that TabNet outperforms other neural network and decision tree variants on a wide range of tabular data learning datasets and yields interpretable feature attributions and insights into the global model behavior.

Under review as a conference paper at ICLR 2020
TABNET: LEARNING
ATTENTIVE INTERPRETABLE
TABULAR
Anonymous authors Paper under double-blind review
ABSTRACT
We propose novel high-performance interpretable deep tabular data learning network, TabNet. TabNet utilizes a sequential attention mechanism that softly selects features to reason from at each decision step and then aggregates the processed information to make a final prediction decision. By explicitly selecting sparse features, TabNet learns very efficiently as the model capacity at each decision step is fully utilized for the most relevant features, resulting in a high performance model. This sparsity also enables more interpretable decision making through the visualization of feature selection masks. We demonstrate that TabNet outperforms other neural network and decision tree variants on wide range of tabular data learning datasets, especially those are not saturated in performance, and yields interpretable feature attributions and insights into the global model behavior.
INTRODUCTION
Deep neural networks have been demonstrated to be very powerful in understanding images (He et al. 2015 Simonyan & Zisserman, 2014; Zagoruyko & Komodakis 2016), text (Conneau al. [2066]
Devlin et al., 2011 8 Lai et al. 2015) and audio (van den Oord et al. [016 Amodei et al.) 2015 Chiu et al. 2018), yielding many important artificial intelligence use cases. For these data types, major enabler of the rapid research and development progress is the availability of canonical neural network architectures efficiently encode the raw data into meaningful representations. Integrated with layers, these canonical architectures yield high performance on new datasets
simple decision-makinn and related tasks with small extra tuning effort. For example, consider the image understanding task variants of convolutional layers with residual connections, e-g. the notable ResNet (He et al. 2015) architecture, can yield reasonably good performance on a new image dataset (e-g. in medical imaging) or a slightly different visual recognition problem (e.g. segmentation).
Our focus in this paper is tabular (structured) data. Tabular data indeed the most common data type in the entire addressable artificial intelligent market (Chui et al., 2018). Yet, canonical neural network architectures for tabular data understanding have been under-explored. Instead, variants of ensemble decision trees still dominate data science competitions with tabular data (Kaggle b019b). A primary reason for the popularity of tree-based approaches is their representation power for decision manifolds with approximately hyperplane boundaries that commonly observed for tabular data. In addition, in
decision tree- rree- approaches are easy to develop and fast to train. They are their basic form (e.g. by tracking decision nodes and edges) and various interpretability techniques have been shown to be effective for their ensemble form, e.g. (Lundberg et al. 20188). On the multi-
other hand, conventional neural network architectures based on stacked convolutional layer perceptrons, may not be the best fit for tabular data decision manifolds due to being vastly overparametrized the lack of appropriate inductive bias often causes them to fail to find robust solutions for tabular decision manifolds (Goodfellow et al., 2016). We argue here that neural network architectures for tabular data should be redesigned to account for 'decision-tree-
mapping.
Given the aforementioned benefits and reasonable performances of tree-based methods, why is deep learning worth exploring for tabular data? One obvious motivation is pushing the performance albeit an increased computational cost, especially with more training data. In addition, introduction of high-performann deep neural network architecture unlocks the benefits of gradient descent- based end-to-end deep learning for tabular data. For example, decision tree learning (even with gradient boosting) does not utilize back mack-propagation into their inputs to use an error signal to guide efficient learning of complex data types. On the other hand, with deep neural network architecture, complex

---

Under review as a conference paper at ICLR 2020
data types like images can be integrated into tabular data efficiently. Another well-known challenge for tree-based methods is learning from streaming data. Most algorithms for tree learning need global statistical information to select split points and straightforward modifications such (Ben-Haim & Tom-TToy 2010) typically yield lower accuracy compared to learning from full data at once, yet deep neural networks show great potential for continual learning (Parisi et al.) 2018). Lastly, deep learning models learn meaningful representations which enable new capabilities such as data-efficient domain adaptation (Goodfellow et al., 20166, generative modeling (e.g. using variational autoencoders or generative adversarial networks (Radford et al. 2015) or semi-supervised learning (Dai et al., 2017). As one example of these potential new capabilities, we demonstrate the potential of remi-supervised learning in the Appendix, showing the potential benefits of information extraction from unlabeled data which non-deep learning models are much weaker at.
workdlas
education
martall statos
occapation
rnlettorahip
capital
capttalloos houn.per week: native, country
10
Priatee
Monted dtaeppuree
Prstopecalty
Hudtond
Mal
2415
57
Indi
Profersional accupation related
Investment releted
Feecback from
previsss step
Festure séleciinn
Ieput
Fouture solection
Input procossing
Feeoback next stap
Agarepiic iriormation
Prodicted (ohether the income leanl =550k)
Figure 1: Depiction of TabNet's sparse feature selection for Adult Census Income prediction (Dua Graff 2017). TabNet employs multiple decision blocks that focus on processing subset of input features for overall decision making. Two decision blocks shown as examples process the features that are professional occupation related and investment related in order to predict the income level.
In this paper propose TabNet, a deep neural network architecture to make significant leap forward towards the optimal model design for tabular data learning. Motivated by the key problems for tabular data discussed above, the design of TabNet has two goals that are often not considered jointly: state-of-the-ar performance and interpretability. With decision tree motivations, TabNet brings
soft feature selection. but individually for each instance. Unlike other mnstance-wise feature selection methods like (Chen et al., 2018) or (Yoon et al. 2019). TabNet employs single deep learning architecture with end-to-end learning, map the raw data to the final decision with soft feature selection. The key aspects and contributions of TabNet are:
In order to learn flexible representations and enable flexible integration into end-to-end learning, unlike most tabular data methods, TabNet inputs raw tabular data without any feature preprocess- ing and is trained using conventional gradient descent -based optimization.
2. To improve performance and interpretability, TabNet utilizes sequential attention mechanism to choose which features to reason from at each decision step, as shown in Fig. I We design this features
feature selection to be instance-wise such that the model can decide which features to focus separately for each input - e.g.. for income classification capital gain may be a more important feature to focus for middle-aged individual. Explicit selection of sparse features enables interpretability as well as more efficient learning as the model parameters are fully utilized for the most salient features at the corresponding decision step.
3. Overall, our careful architecture design leads to two valuable properties for real world tabular learning problems: (1) TabNet outperforms other tabular learning models on various datasets for classification and regression problems from different domains, particularly those which are not saturated in performance; and (2) TabNet enables two kinds of interpretabilit; local interpretability that visualizes the importance of input features and how they are combined, and global interpretability that quantifics the amount of contribution of each input feature to the trained model.
2 RELATED WORK
Feature selection: Feature selection in machine learning broadly refers to the techniques for judicious selection of subset of features that are useful to build good predictor for a specified response variable. Commonly- and LASSO regularization [Guyon & Elisseeff 2003) attribute importance to the features based on the entire training data set, and are referred as global methods. On the other hand, instance-wise feature selection refers selection of the most important features, individually and differently for
feature selection techniques such as forward feature selection
2

---

Under review as a conference paper at ICLR 2020
each input. Instance- wise feature selection studied in (Chen al., 2018) by training explainer model to maximize the mutual information between the selected features and the response variable, and in (Yoon et al., 2019) by using actor-critic framework to mimic baseline model while optimizing the feature selection. (Ma et al. 2018) uses partial variational autoencoder to dynamically decide which piece of information to acquire next sequentially, that can be adapted for instance-wise feature selection. Unlike these approaches, our proposed method employs soft feature selection with controllable sparsity in end-to-end learning a single model jointly performs feature selection and output mapping, enabled by the specific design of the architecture. Thus, we demonstrate superior performance with very compact representations.
Tree-based learning: Tree- based models are the most common machine learning approaches for tabular data learning. The prominent strength of tree-based models is their efficacy in picking global features with the most statistical information gain (Grabczewski & Jankowski 2005). To improve the performance of standard tree-based models by reducing the model variance, one common approach is ensembling. Among ensembling methods, random forests (Ho 1998) use random subsets of data with randomly selected features to grow many trees. XGBoost (Chen & Guestrin (016) and LightGBM (Ke et al. 2017) are the two recent ensemble decision tree approaches that dominate most of the recent data science competitions. They are based on learning the structures of trees at first, and then updating the leaves with the streaming data.
One direction to address the limitations of decision trees is integration of neural networks. Representing decision trees with canonical neural network building blocks, as in (Humbird et al. 2018), yields redundancy in representation and inefficient learning. Soft (neural) decision trees Wann et al., 2017 Kontschieder et al. 2015) are proposed with differentiable decision functions, instead of mon-diifeeeetiaalee axis aligned splits to construct trees. Yet, abandoning axis-aligned splits loses the automatic feature selection ability, which important for learning from tabular data. In (Yang et al. 2018), a soft binning function is proposed to simulate decision trees in neural networks, which needs to enumerate all possible decisions and is inefficient. In (Ke et al., (2019. a novel neural network architecture is proposed, with the motivations of explicitly leveraging expressive feature combinations and reducing model complexity. However, learning based on transferring knowledge from a gradient boosted decision tree. Thus, it yields very limited performance improvement compared to it, and interpretability was not considered. In TTano et al. 2018), deep learning framework is proposed based on adaptively growing the architecture from primitive blocks while representation learning into edges, routing functions and leaf nodes of decision tree. Our proposed model TabNet differs from these methods as it embeds the soft feature selection ability into sequential attention- based network architecture, with controllable sparsity.
Integration of neural networks into decision trees:
Attentive table-to-text models: Table-to-ted text models extract textual information from tabular data. Recent works (Liu et al. 2017) (Bao et al., 2019) propose an architecture based on sequential mechanism for field-level attention. Despite the high-level similarities in the architecture, TabNet aims to perform the ultimate classification or regression task considering the entire input features, rather than mapping them different data type.
TABNET MODEL
3.1 PRINCIPLES
We initially consider the implementation of a decision tree-like output manifold using conventional neural network building blocks (Fig. 2). Individual feature selection is the key idea to obtain decision boundaries in hyperplane form, which can be generalized for linear combination of features where constituent coefficients determine the proportion of each feature in the decision boundary. We aim to generalize this type of tree-like functionality by:
Utilizing sparse instance-wise feature selection, learned based on the training dataset.
Constructing sequential multi-step architecture, where each decision step can contribute to portion of the decision that is based on the selected features.
Improving the model capacity by non- non-linear processing of the selected features.
Ensembling via higher feature dimension and more decision steps.
3.2 OVERALL ARCHITECTURE
Fig. depepcts the TabNet architecture. Tabular data inputs are comprised of numerical and categorical features. We use the raw numerical features and we consider mapping of categorical features with
3

---

Under review as a conference paper at ICLR 2020
Schmax
Go Go -1
W: We 0.0] box
10 b-fel, edey
Sa
7g <d
Mask MR
Masi 10.00 1)
Figure 2: Illustration of decision tree-like classification using conventional neural network blocks and the corresponding two-dimensional manifold (Iz and I2 are the input dimensions, and and d are constants). By employing multiplicative sparse masks to inputs, the relevant features are selected. The selected features are linearly transformed and after bias addition (to represent boundaries), ReLU function performs region selection by zeroing the regions that are on the negative side of the boundary. Aggregation multiple regions is based on the addition operation. As C1 and C2 get larger, the decision boundary gets sharper due the softmax.
Step
Step
Ousput
Spplt Festure transfommer
Spin
Split
Fenture transformer
Festure
tronsfomorr
Attenbive transformer
Mack
transfrmer
Fewtures
Feature attributta
(a) TabNet architecture
Feattune trensformer
Attentive
transformer
across dedision steps
Decision step dependent
Prior ocales
Sparseraa
VOLS
(b) Feature transformer
(c) Attentive transformer
Figure 3: (a) TabNet architecture, composed of feature transformer, attentive transformer and feature masking at each decision step. Split block divides the processed representation into two, to be used at the attentive transformer of the subsequent step and to be used towards construction of the overall output. At each decision step. its feature selection mask can provide insights about its functionality, and the masks can be aggregated (using the Agg. block) ultimately to obtain global feature important attribution behavior. (b) A feature transformer block example 4-layer network is shown, where 2 of them are shared across all decision steps and of them are decision step-dependent. Each layer composed An attentive transformer block example single layer mapping is modulated with prior scale information, which aggregates how much each feature has been used before the current decision step. Normalization of the coefficients is employed using sparsemax (Martins & Astudillo, 2016) for sparse selection of the most salient features at each decision step.
fully-connected layer, batch normalization and GLU nonlinearity. (c)
4

---

Under review as a conference paper at ICLR 2020
trainable embeddings We do not consider any global normalization of features, but merely apply batch normalization. We pass the same D-dimensional features f E RB to each decision step, where B is the batch size. TabNet is based on sequential multi-step processing, with Nateps decision steps. The 2th step inputs the processed information from the (i 1)th step decide which features to use and outputs the processed feature representation to be aggregated into the overall decision. The idea of top-down attention in sequential form is inspired from its applications in processing visual and language data such as for visual question answering (Hudson & Manning 2018 or in reinforcement learning (Mott al. 2019) while searching for a small subset of relevant information in high dimensional input. Ablation studies in Appendix focus on the impact of various TabNet design choices, explained next. Overall, the performance is not too sensetitive to most hyperparameters. and guidelines on selection of the important hyperparameters are also provided in Appendix.
Feature selection: We employ a learnable sparse mask M[i] E RB for soft selection of the salient features. Through sparse selection of the most salient features, the learning capacity of a decision step is not wasted irrelevant features, and thus the model becomes more parameter efficient. The masking is in multiplicative form, M[i] f. We use an attentive transformer (sce Fig, 3 to obtain the masks using the processed features from the preceding step, a 1]:
M[i] sparsemax 1] h, (a 1)))-
Sparsemax normalization (aartins & Astudillo 2016) encourages sparsity by mapping the Euclidean projection onto the probabilistic simplex, which is observed to be superior in performance and aligned with the goal of sparse feature selection for most real-world datasets. Note that Eq. 1 has the normalization property, Σn=1 Mi 1. h is a trainable function, shown in Fig. 3 using fully-connected layer, followed by batch normalization. P(i) is the prior scale term, denoting how much a particular feature has been used previously:
Pli II (7j))
where y is a relaxation parameter- when 1, feature is enforced to be used only at one decision step and as y increases, more flexibility is provided to use feature at multiple decision steps. P[O] is initialized all ones. To further control the sparsity of the selected features, we propose sparsity regularization in the form of entropy (Granvvalet & Bengio 2004):
Σ 1
Natera
D 1== Σ Mbj(i] (Mg( + c),
B
Isparse
(3)
Nsteps
where is a small number for numerical stability. We add the sparsity regularization the overall loss, with a coefficient Asparse- Sparsity may provide a favorable inductive bias for convergence to higher accuracy for some datasets where most of the input features are redundant.
Feature processing: We process the filtered features using a feature transformer (see Fig. 3 to obtain the features that are split for the decision step output and information for the subsequent step:
[di). a [i]] = (M(M f),
where d[i] E Bx and a[i] E Bx,, For parameter-effie efficcien learning and efficient convergence with the high model capacity, a feature transformer should comprise layers that are shared across all decision steps (as the same features are input across different decision steps), as well as decision step-dependent layers. In Fig. 3 we show the implementation of block concatenation of two shared layers and two decision step-dependent layers. Each fully-connected laver is followed by batch normalization and gated linear unit (GLU) nonlinearity (Dauphin al. 2016 eventually connected a normalized residual connection with normalization. Normalization with v 0.5 helps to stabilize learning by ensuring that the variance throughout the network does not change dramatically, as discussed in (Gehring et al. 2017).. For faster training, we aim for very large batch sizes in practice. To improve performance with large batch sizes, all batch normalization operations, except the one applied to the input features, are implemented in ghost batch normalization (Hoffer et al. 2017) form, with virtual batch size By and momentum m.. For the input features, we observe the benefit of
For example, if there are three possible categories A, B and C for particular features, they can be learned to be mapped to scalars 0.4, 0.1, and -0.2 respectively, along with the training of the model. In GL.U, first linear mapping is applied to the intermediate representation and the dimensionality is doubled, and then second half of the output is used to determine nonlinear processing on the first half.
5

---

Under review as a conference paper at ICLR 2020
low-variance averaging and hence avoid ghost batch normalization. After the last layer, split the processed representation into dii and afi]. Inspired by decision-tree like aggregation as in Fig. 2 we construct the overall decision embedding as:
Nateps
dout
ReL
(5)
==1
Finally, we apply a linear mapping W Wont for the final decision. When discrete outputs are required, we additionally employ a softmax function during training (and argmax during inference).
4 EXPERIMENTS
We evaluate the performance of TabNet in wide range of problems, that contain regression or classification tasks. We specifically focus on tabular datasets with published benchmarks, based on notable tree-based and neural network-based approaches.
For all datasets, categorical inputs are mapped to a angle-dimensional trainable scalar with a learnable embedding" and numerical columns input without and preprocessing! 44 We use standard classification (softmax cross entropy) and regression (mean squared error) loss functions and we train until convergence. Hyperparameters of the TabNet models are optimized listed in Appendix. TabNet performance is not very sensitive to most hyperparameters as shown with ablation studies in Appendix. In all of the experiments where we cite results from other papers, we use the same training. validation and testing data split with the original work. Adam optimization algorithm (Kigmma & Ba 2014) and Glorot uniform initialization are used for training of all models.
validation and
4.1 PERFORMANCE
Comparison to methods that integrate explicit feature selection. For this comparison, we con- sider the 6 synthetic tabular datasets from (Chen et al. (0018) As the datasets small (10k training samples), efficient feature selection is crucial for high performance in this task. The synthetic datasets are constructed in such a way that only subset of the features determine the output. For Syn1, Syn2 and Syn3 datasets, the 'salient' features are the same for all instances, so that an accurate global feature selection mechanism should be optimal. E.g.. the ground truth output of the Syn2 dataset only depends on features X3-Xg. For Syn4, Syn5 and Syn6 datasets, the salient features are instance dependent. E.g., for Syn4 dataset, X₁I is the indicator, and the ground truth output depends on either X1-X₂ or X3-X6 depending the value of X11- This instance dependence makes global feature selection suboptimal, as the globally-salient features would be redundant for some instances.
Table 1: TabNet achieves high performance with small number of network parameters. Mean and std. of test area under the receiving operating characteristic curve (AUC) on synthetic datascts from (Chen al. 2018), for TabNet vs. other feature selection selection-based neural network models: No sel.: using all input features without any feature selection, Global: using only globally-salient features, Tree refers to Tree Ensembles (Geurts et 2006), LASSO: LASSO -eso-rrgullarized model, L.2X (Chen)
ct al.
2018) and INVASE (Yoon et al.) 2019) are instance-wise feature selection frameworks. Bold
numbers the best method for each dataset.
Model
Synl
Syn2
Syn3 .854 + .044
Syn4
Syn5
Syn6
No sel.
.578 .004
.789 1 .003
.558 .021
.662 4 .013
.692 1 .015
Tree
.574 .101
.872 + .003
.899 + .001
.684 .017
.741 .004
.771 + .031
Lasso
.498 + .006
.555 + .061
.866 + .003
.512 + .031
.691 + .024
.727 + .025
L2X
.498 .005
.823 + .029
.862 + .009
.678 .024
.709 .008
.827 = .017
INVASE
.690 + .006
.877 + .003
.902 + .003
.787 1 .004
.784 - .005
.877 1 .003
Global
.686 .005
.83 1 .003
.900 1 .003
.774 1 .006
.784 .005
.858 1 .004
TabNet
.682 + .005
.892 + .004
.897 + .003
.776 + .017
.789 .009
.878 + .004
Table shows the performance TabNet vs. other techniques. including no selection, using only globally-salient features, Tree Ensembles (Geurts et al. 2006), LASSO regularization, L2X (Chen
In some problems, higher dimensional embedding mapping may slightly improve the performance, but interpretation of individual embedding dimensions may become challenging.
SSpecially-designn feature engineering, e.g. logarithmic transformation of variables highly-skewed distribu- tions, may further improve the results but we leave out of the scope of this paper.

---

Under review as a conference paper at ICLR 2020
et al. 2018 and INVASE (Yoon al. 2019 We observe that TabNet outperforms all other methods and on par with INVES.. For Synl. Syn2 and Syn3 datasets, we observe that the TabNet performance is very close global feature selection. For Syn4, Syn5 and Syn6 datasets, observe that TabNet improves global feature selection, which would contain redundant features. (Feature selection is visualized in Sec. 4.2 All other methods utilize predictive model with 43k parameters, and the total number of trainable parameters is 101k for INVASE due to the two other networks in the actor-critic framework. On the other hand, TabNet is single neural network architecture, and the total number of parameters is 26k for Synl-Sym I-yn3 datasets and 3 31k for Syn4-Synd Synó datasets. This compact end-to-end representation is one of TabNet's valuable properties.
Comparison to models that do not employ explicit feature selection. We compare TabNet nigh-performance tabular data learning models that are demonstrated on the following problems:
Forest cover type (Dua Grafi 2017): Classification forest cover type from cartographic variables.
Poker hand (Dua & Graff 2017p: Classification of the poker hand from the raw input features of suit and rank attributes of the cards.
Sarcos robotics arm inverse dynamics (Vijayakumar & Schaal 2000): Regression for inverse dynamics of seven degrees-of freedom
offrreeoof-freddm of an anthropomorphic robot arm.
Higgs boson (Baldi et al. 2014): Distinguishing between a signal process which produces Higgs bosons and background process which does not.
Table 2: Performance for forest cover type dataset. The performance of the comparison models' from (Mitchell et al., 2018). Autolnt models pairwise feature interactions with an attention-bass deep neural network (Song al. 2018). AutoMI. Tables (denoted as **) is automated machine learning development tool based on ensemble of models including linear feed-forward deep neural network, gradient boosted decision tree, AdaNet (Cortes al., 2016) and ensembles (AutoML, 2019). For AutoML Tables (**), the amount of node hours reflects the measure of the count of searched models for the ensemble and their complexity[ [ single TabNet model, without Ane-ggained hyperparameter search, can outperform the accuracy of ensemble models with very thorough hyperparameter search.
Model
Test accuracy
XGBoost
89.34*
LightGBM
89.28*
CatBoost
$5.14*
AutoInt
90.24*
AutoMI. Tables (2 node hours)"*
94.56**
AutoML Tables (5 node hours)**
94.95**
AutoML Tables (IO node hours)"*
96.67**
AutoML Tables (30 node hours)"*
96.93**
TabNet
96.99
Table 3; Performance for poker hand induction dataset. The input-output relationship is deterministic and hand-crafted rules implemented with several lines of code can get 100% accuracy. Yet, neural networks and decision tree models severely suffer from the imbalanced data and cannot learn the required sorting and ranking operations with the raw input features. The results for comparison models* are from (Yang et al., 2018).
Model
Test accuracy (%)
Decision tree*
50.0*
Multi layer perceptron"
50.0"
Deep neural decision tree"
65.1*
XGBoost
71.1
LightGBBM
70.0
CatBoost
66.6
TabNer
99.3
Rule-based
100.0
Tables 2-5 show the performance comparisons. We observe that TabNet outperforms multi-layer perceptrons and the variants of ensemble decision trees on all four datasets. TabNet allocates the learning capacity to salient features, and it yields more compact model terms of the number of parameters. When the model size is constrained, we observe the superior performance of TabNet even
7

---

Under review as a conference paper at ICLR 2020
Table 4: Performance for Sarcos robotics arm inverse dynamics dataset. Three TabNet models of different sizes are considered (denoted with -S, -M and -L). The performance of the comparison
models* are from (Tanno et al. 2018
Model
Test MSE
Number of parameters
Random forest"
2.39*
16.7K
Stochastic decision tree*
2
28K
Multi layer perceptron"
2.13'
0.14M
Adaptive neural tree ensemble"
1.23
0.60M
Gradient boosted tree"
1.44
0.99M
TabNet-S
1.25
6.3K
TabNet-M
0.28
0.59M
TabNet-L
0.14
1.75M
Table 5: Performance Higgs boson dataset. Two TabNet models of different sizes considered (denoted with -S and -M). The performance of the comparison models' are from (Mocanu et al. 2018). Sparse evolutionary training applies non-structured sparsity integrated into training, yielding low number of parameters. With its compact representation, TabNet, (without any further pruning or extra non-structured sparsity). yields almost similar performance with sparse evolutionary training for the same number of parameters. Gradient boosted tree models are implemented using (Tnssrfloww 2019) see Appendix for details.
Model
Test accuracy (%)
Number of parameters
Sparse evolutionary trained multi layer pereeptron"
78.47"
81K
Gradient boosted tree-S
74.22
0.12M
Gradient boosted tree-M
75,97
0.69M
Malti-layer perceptron"
78.44
2.04M
Gradient boosted tree-L
76.98
6.96M
TabNet-S TabNet-M
78.25
81K
78.84
0.66M
compared to the decision tree variants. The performance is only slightly worse than the evolutionary sparsification algorithms (Mocanu et al.] 2018). Yet, the sparsity learned in TabNet is structured unlike the alternative approaches - i.e. does not degrade the operational intensity of the model (Wen et al. (016)) and can efficiently utilize modern multi-core processors. Also note that we do not consider any matrix sparsification techniques such as adaptive pruning (Narang et al. 2017) which could further improve the parameter-efffciencyy
4.2 INTERPRETABILITY
The feature selection masks in TabNet can be used to build insights selected features each step. Such capability would not be possible for conventional neural networks with fully- fully-connected layers, each subsequent layer hidden units would jointly process all features without sparsity-controle selection mechanism. For feature selection masks, Mb, = 0. then j feature of the 6th sample should have 0 contribution to the overall decision. If f, were linear function, the coefficient Mb would correspond to the feature importance of fb.j. Although each decision step employs non-linear processing. their outputs are combined later linear way. Our goal quantify an aggregate feature importance beyond analysis of each step as well. Combination of the masks at different decision steps require coefficient that can weigh the relative importance of each step in the decision.
We use 76 [i] = Na ReLU(db.e [i)) to denote the aggregate decision contribution at 7th decision step for the 5th sample. Intuitively, [b,c[i] < O, then all features at ith decision step should have contribution the overall decision and its value increases, it plays higher role in the overall linear combination given in Eq. 5 Scaling the decision mask at each decision step with mh [i]. we propose the aggregate feature importance mask as:
Natepo
N Natepe
Magg-bJ -bJ
mb [i]
Σ
Th [i]
(6)
1=1
Normalization is used to ensure D M Magg b.j 1 for each sample.
8

---

Under review as a conference paper at ICLR 2020
Syn2 dataset
Syn3 datanet
Figure 4: Feature importance masks Mi (that indicate which features selected at 1th step) and the aggregate feature importance mask M agg showing the global instance-wise feature selection for Syn2 and Syn3 datasets from (Chen 2018). Brighter colors show higher value. E.g. for Syn2 dataset, only four features (X) are used.
M
XIX
MI2)
Synó detxeet
Man
XBI
M121
MLII
Figure 5: Feature importance masks M (that indicate which features are selected at 1th step) and the aggregate feature importance mask Magg showing the global instance-wiss feature selection for Syn4 and Syn6 datasets from (Chen et al. 2018). Brighter colors show a higher value. E.g. for Syn4 dataset, the chosen features depend on the value X11-

---

Under review as a conference paper at ICLR 2020
show the aggregate feature importance masks for the synthetic datasets discussed in Sec. for better illustration here, unlike Sec. 4.1 the models are trained with 10M training samples rather than 10K
Figs. and 5
obtain sharper feature selection masks). The ground truth output of the Syn2 dataset only depends on features X3-X6, and the ground truth output of the Syn3 dataset only depends on features X7-X10- We observe that the aggregate masks are almost all-zero for irrelevant features and they merely focus on relevant ones. For Syn4 dataset, X11 is the indicator, and the ground truth output depends on either X or X3-Xe depending on the value of X11. For Syn6 dataset, X11 is the indicator, and the ground truth output depends on either X3-X6 or X7-X10 depending on the value of X11. For both, TabNet yields accurate instance-wise feature selection uses majority of the weights in two masks to focus on XI1, and assigns almost all-zero weights to irrelevant features (the ones other than one of the two feature groups based on the value of X11).
Footure iepartrnce the fostare "Odor"
Importance ranking
Fasture
Skater
Nobossl
Paen
12
Pelayceahip
TOUNET DOcBUT
Plakk clew
10
(a)
(b)
(c)
Figure 6: (a) Comparison to previous work for the ratio of the feature importance of "Odor" feature to all the features of the top feature for the mushroom edibility prediction (Dua & Graff] 2017) (task: classify whether mushroom is edible poisonous). With "Odor" feature only, 98.5% test accuracy can be obtained, so a high feature importance is expected to be assigned to it, as observed with TabNet. (b) Comparison previous work for importance ranking of features in the Adult Census Income dataset (Dua & Graff 2017) (task: distinguish whether a person's 's income is above $50,000). (c) Impact of the most important feature on the decision manifold. T-SNE of the decision manifold for Adult Census Income test samples and the impact of the most dominant feature 'Age'.
Fig. ((a) shows the feature importance score of the top feature obtained with TabNet vs. other explamability techniques from (Ibrahim et al. 2019) for mushroom edibility prediction. Mushroom edibility is a simple pattern recognition problem TabNet achieves 100% test accuracy. It is indeed known (Dua & Graff (017)) that with "Odor" feature only, model can get > 98.5% test accuracy (Dua & Graff] 2017). so a high feature importance is expected for it, as observed with TabNet. Fig. b(b) shows the importance ranking of features for TabNet vs. other explainability techniques from (Lnddberg et al. 2018) (Nbviewer) 2019) for Adult Census Income prediction. TabNet achieves 85.7% test accuracy for this problem. We observe the commonality of the most important features ("Age", "Capital gain/loss". "Education number", "Relationship") and the least important features ("Native country", "Race", "Gender", "Work class"). ). For the same problem, Fig. (6(c shows the impact of the most important feature on the output decision by visualizing the T-SNE of the decision manifold. A clear separation between age groups is observed, underlining the importance of the "Age" feature, as suggested by its high value in the aggregate feature importance mask of TabNet.
5 CONCLUSIONS
We propose TabNet, novel deep learning architecture for tabular learning. TabNet utilizes sequential attention mechanism to choose a subset of cemantically-meaninee each decision step. The selected features are processed the representation, that contributes to the overall decision output and sends information to the next decision step. Instance-wise feature selection enables efficient learning as the model capacity is fully used for the most salient features, and also yiclds more interpretable decision making via visualization of selection masks. We demonstrate that TabNet outperforms previous work across tabular datasets from different domains.
features to process
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, et al. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv:1512.025555.000 2015.
AutoMI.. AutoMI. Tables - Google Cloud, 2019. URL
com/
automlttables
Baldi, P. Sadowski, and D. Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature Commun., Jul 2014.
Pierre Baldi. Autoencoders, unsupervised learning, and deep architectures. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning, volume 27 of Proceedings of Machine Learning Research, pp. 37-49, Bellevue, Washington, USA, 02 Jul 2012. PMLR.
J. Bao, D. Tang, N. Duan, Z. Yan, M. Zhou, and T. Zhao. Text generation from tables. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(2) Feb 2019.
Yael Ben-Haim and Elad Tow-Tov. A streaming parallel decision tree algorithm. J. Mach. Learn Res., 11:849-872 872, March 2010.
Catboost. Benchmarks. httt::///ghhub...cc///tttttto//s 2019-11-10.
2019. Accessed:
Jianbo Chen, Le Song, Martin Wainwright, and Michael I. Jordan. Learning to explain: An information-thhorret perspective on model interpretation. arXiv:1802.07814, 2018.
Tianqi Chen and Carlos Guestrin. XGboost: A scalable tree boosting system. In KDD, 2016.
C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, et al. State-of-the-ar speech recognition with sequence-to-sequeecc models. In ICASSP, April 2018.
Michael Chui, James Manyika, Mehdi Miremadi, Nicolaus Henke, Rita Chung, et al. Notes from the ai frontier. McKinsey Global Institute, 2018.
Alexis Conneau, Holger Schwenk, Loic Barrault, and Yann LeCun. Very deep convolutional networks for natural language processing. arXiv: PrrXiv:666...0 /0////666607777 2016.
Corinna Cortes, Xavi Gonzalvo, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang. Adanet: Adaptive structural learning of artificial neural networks. arXiv:1607.01097, 2016.
Zihang Dai, Zhilin Yang, Fan Yang, William W. Cohen, and Ruslan Salakhutdinov. Good semi- supervised learning that requires a bad GAN. arxiv 2017.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks, arXiv:1612.0880 08083, 2016.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. arxiv.1810.04805. 2018.
Dheeru Dua and Casey Graff. UCI Machine Learning Repository, 2017. URL http:://archh
ics. uci. www.ummm.mmmmmmmmmmmmmmm
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence sequence learning. arXiv: 7775000112 2017.
Pierre Geurts, Damien Emstt and Louis Wehenkel. Extremely randomized trees. Machine Learning, 63(1):3 42, Apr 2006. ISSN 1573-0565. doi: 10.1007//10994-000 120066226-1.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
K. Grabezewski and Jankowski. Feature selection with decision tree criterion. In HIS), Nov 2005.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NIPS 2004, 2004.
11

---

Under review as a conference paper at ICLR 2020
Isabelle Guyon and André Elisseeff. An introduction to variable and feature selection. J. Mach. Learn. Res., 3:1157-1182, March 2003.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv:1512.03385 2015.
Tin Kam Ho. The random subspace method for constructing decision forests. IEEE Trans. on PAMI, 20(8):832 844, Aug 1998.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. arXiv: 2017.
Drew A. Hudson and Christopher D. Manning. Compositional attention networks for machine reasoning. arXiv: /.03:/8803.00000 2018.
K. D. Humbird, J. L. Peterson, and R. G. McClarren. Deep neural network initialization with decision trees. IEEE Transactions on Neural Networks and Learning Systems, 2018.
Mark Ibrahim, Melissa Louie, Ceena Modarres, and John W. Paisley. Global explanations of neural networks: Mapping the landscape of predictions. arXivv1902.022384, 2019.
Kaggle. Rossmann store sales, httpe//ttttess.........oo/eeooot store- sales
2019a. Accessed: 2019-11-10.
Kaggle. Historical data science trends on Kaggle. https://vww
wwtp://wwwww.....e////.. shi vamb/ dssa-sccience trends on oo-ooo----n -anggel 2019b. Accessed: 2019-04-20.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, et al. Lightgbm: A highly efficient gradient boosting decision tree. NIPS. 2017.
Guolin Ke, Jia Zhang, Zhenhui Xu, Jiang Bian, and Tie- Liu. TabNN: A universal neural net forum/f
network solution for tabular data, 2019. URL
rleJssCqY7
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
Kontschieder, M. Fiterau, A. Criminisi, and S. R. Bulò. Deep neural decision forests. In ICCV, 2015.
Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. Recurrent convolutional neural networks for text classification. In AAAI, 2015.
Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. Table-to-text generation by structure-aware seq2seq learning. arXiv: 2017.
Scott M. Lundberg, Gabriel G. Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv: arXiv:1802.088 03888, 2018.
Chao Ma, Sebastian Tschiatschek, Konstantina Palla, José Miguel Hernández-Lobate Sebastian Nowozin, and Cheng Zhang. EDDI: efficient dynamic discovery of high-value information with partial VAE, arXiv: 1809, 1424 2018.
Freddie Mac. Loan level dataset. http:/ / /www. freddiemac. hataaacc.ccneeeeeeac///ttttsssse sf www.oonneeeeaateeeeeeeeeeeeeeeee
page 2019. Accessed: 2019-3-10.
André T. Martins and Ramón Fernández Astudillo, From softmax to sparsemax: sparse model of attention and multi-label classification. UrXiv:1602.02066 2016.
Rory Mitchell, Andrey Adinets, Thejaswi Rao, and Eibe Frank. Xgboost: Scalable GPU accelerated learning. arXiv:1806, 118880111446 1248, 2018.
Decebal Mocanu, Elena Mocanu, Peter Stone, Phuong Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 12 2018.
12

---

Under review as a conference paper at ICLR 2020
Alex Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo J. Rezende. S3TA: A soft, spatial, sequential, top-down attention model, 2019. URL
Eorum?2@++BlgooRRc10)
Sharan Narang, Gregory Diamos, Shubho Sengupta, and Erich Elsen. Exploring sparsity in recurrent neural networks, arXiv: 19, 2017.
Nbviewer. Notebook on Nbviewer, 2019. URL https://novvexeer.pppptee
sccpsttta__seeeeeeeeeeeeeee for httpa//aww.a....aaaaaaa/aa/aaaaa ottpop/....aoa..a.............ar ipynbt
N. C. Oza. Online bagging and boosting. In 2005 IEEE International Conference Systems, Man
and Cybernetics, volume 3, pp. 2340-2345 Vol. 3, Oct 2005.
German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: review. (rXXiv:802.075660 2018.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In NIPS, pp. 6638-6648. 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv:1511.06444 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv: 1409. 1556, 2014.
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Au- toint: Automatic feature interaction learning via self-attentive neural networks. arxiv: 1810. 1921, 2018.
Ryutaro Tanno, Kai Arulkumaran, Daniel C. Alexander, Antonio Criminisi, and Aditya V. Nori. Adaptive neural trees. 1807 2018.
Tensorflow. Classifying higgs boson processes in the higgs data set, 2019. URL httpt:tt///// htttps://githuss com/ http://ensorroooocoeeeooo http:///rooooetlooaa/eeeeee/eeea
trees
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, et al. Wavenet: generative model for raw audio. 1609. 03499, 2016.
Sethu Vijayakumar and Stefan Schaal. Locally weighted projection regression: An o(n) algorithm for incremental real time learning in high dimensional space. In ICML, 2000.
Suhang Wang, Charu Aggarwal, and Huan Liu. Using random forest to inspire neural network and improving on it. In SDM, 2017.
Wei Wen, Chumpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. arXiv: 1608. 03665, 2016.
Yongxin Yang, Irene Garcia Morillo, and Timothy M. Hospedales. Deep neural decision trees. arXiv:1806.06988, 2018
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. INVASE: Instance-wise variable selection using neural networks. In ICLR, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. urXiv:1605.07146 2016.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv: 12,0.0942, 2017.
13

---

Under review as a conference paper at ICLR 2020
A
SIMPLIFIED DIAGRAM FOR TABNET FEEDFORWARD PASS
Step
FC
Output
ReLu
ReLU
Split
0
Split
1
a[1]
Split
d
Festure transformer
Feature transformer fd
Feature
transformer 64.
|,,0001
|0,0. Rsl
Attentive transformar hy{
Mask M110-11,0,01
Attentive transformar hg/
Mask
1120-1000111
[8g. Rg.
BIN
Features Rgy
Figure 7: Simplified diagram for TabNet feedforward pass for input with 3 features, assuming
Nsteps = 2. At the first step, the model selects only the first feature, and applies feature processing on it. At the second step. the model selects the last feature, and applies the feature processing Lastly, the two outputs are combined for the final decision.
B ADDITIONAL RESULTS
B. RETAIL DATASET WITH TIME COMPONENT
In this section, show additional results real-world tabular data learning problem Rossmann store sales forecasting (Kaggle, 2019a). This dataset has time-dependent features. Time information input day, month, and year columns. We observe that TabNet outperforms alternative methods that are commonly used for such problems.
Table 6: Performance for Rossmann store sales dataset (Kaggle 2019a). We use the exactly same preprocessing and data split with (Catboost 2019) data from 2014 is used for training and validation. whereas 2015 is used for testing. The performance of the comparison models* are from (Catboost) 2019).
Model
Test MSE
XGBoost
490.83"
LightGBM
504.76*
CatBoost*
489.75*
TabNet
485.12
B.2 KDD DATASETS
Table 7: Performance on three KDD datasets Customer Relationship Management: Appetency, Churn and Upselling. We apply the similar preprocessing and data partitioning as (Prokhorenkova
et al. 2018) The performance of the comparison models" are from (Prokhorenkova al. 2018).
Model
Appetency test
Churn test
Upselling test accuracy (%)
accuracy (%)
accuracy (%)
XGBoost
98.2*
92.7*
95.1"
CatBoost"
98.2*
92.8
95.1*
TabNet
98.2
92.7
95.0
We experiment TabNet on four KDD datasets: the three Customer Relationship Management and Census Income. These datasets show saturated behavior in achievable performance (even simple
14

---

Under review as a conference paper at ICLR 2020
Table 8: Performance for KDD Census Income (Dua & Graff 2017). The task is income prediction
from demographic and employment related variables. The performance of the comparison models" from Oza 2005).
Model
Test accuracy (%)
XGBoost
95.76
CatBoost
95.72
Multi-layes perceptron"
95.19
Boosting, Multi-layer perceptron"
94.86
Bagging, Multi-layer perceptron"
95.33
TabNet
95.49
models yield similar results). For these cases, TabNet shows very similar (or slightly worse) perfor- mance than XGBoost and CatBoost, that are known to be very robust as they contain high amount of ensembles.
B.3 LOAN DELINQUENCY PREDICTION
Table 9: Performance for loan delinquency prediction on proprictary dataset, constructed from (Mac 2019). The task is to classify loan delinquency status (among four categories), from many input features including personal information and financial status. The training dataset consists of 93k samples. The dataset is highly imbalanced as the delinguency situation is observed rarely.
Model
Test mean per class accuracy
XGBoost
0.55
H2OAutoML (with 15 models)
0.60
Multi-layer perceptron
0.46
TabNet
0.86
We consider TabNet for a real-world problem in financial services industry: loan delinquency prediction. On a proprictary dataset, we demonstrate strong outperformance of TabNet, especially finding rare delinquency cases without any special techniques on anomaly detection.
C EXPERIMENT HYPERPARAMETERS
For all datasets, start hyperparameter tuning with pre-defined value space. Na and Na are chosen from {8, 16, 24, 32, 64, 128). Nateps chosen from {3, 43,4,5 5, 6, 7, 8, 9, 10). y is chosen from {1.0, 1.2, 1.5, 2.0). Asparse is chosen from {0. 0.000001, 0.0001, 0.001, 0.01, 0.13. B is chosen from {256, 512, 1024, 2048, 4096, 8192, 16384, 32768), By is chosen from (256, 512, 1024, 2048, 4096) and my is chosen from {0.6, 0.7, 0.8, 0.9, 0.95, 0.98). If the model size is not under the desired cutoff (e.g. for Table comparisons), we decrease the value to satisfy the model size constraint.
C.I SYNTHETIC DATASETS
All TabNet models use N₂ = N. = 16, B = 3000, By 100, mg 0.7. For Synl we use Aeporse 0.02. Nateps and y 2.0; for Syn2 and Syn3 we use Asparse 0.01, N Nateps and 7 = 2.0; and for Syn4, Syn5 and Syn6 we use Asparse = 0.005, Nsteps = 5 and 7 = 1.5. Each feature transformer block uses two shared and two decision step-dependent fully-connected layer, ghost batch normalization and GLU blocks. All models use Adam optimization lcarning rate of 0.02 (decayed 0.7 every 200 iterations with an exponential decay) for 4k iterations.
For visualizations in Section 4.2 also train TabNet models with datasets of size 10M samples. For this case, we choose N₂ = N₂ = 32, Aeporse = 0.001, B = 10000, By = 100, mg = 0.9. Adam optimization is used with learning rate of 0.02 (decayed 0.9 every 2k iterations with an exponential decay) for 15k iterations. For Syn2 and Syn3, Nsteps = 4 and 7 = 2. For Syn4 and Syn6, Nsteps and 1.5.
15

---

Under review as a conference paper at ICLR 2020
C.2 FOREST COVER TYPE DATASET
We use the exact same partitioning of the train, evaluation and test datasets with (Mitchell et al. 2018) for a fair comparison.
TabNet model uses Na = N₂ = 64, Asparse = 0.0001, = 16384, BV = 512, m. = 0.7, Nateps 5 and y 1.5. Each feature transformer block uses two shared and two decision step- dependent fully-connected layer, ghost batch normalization and GLU blocks. Adam optimization is used with learning rate of 0.02 (decayed 0.95 every 0.5k iterations with an exponential decay) for 130k iterations.
C.3 POKER HANDS DATASET
TabNet uses N₂ = 24, N₂ 8, Aeparee = 0.001, B 4096, By = 256, mB = 0.8, Netepe 4 and x 1.5. Each feature transformer block uses two shared and two decision step-dependent fully-connected layer, ghost batch normalization and GLU blocks. Adam optimization is used with a learning rate of 0.02 (decayed 0.9 every 10k iterations with exponential decay) for 71k iterations.
C.4 SARCOS DATASET
TabNet-S model uses Na = N = Asparse 0.0001, B = 4096, By 256, mB = 0.9, Nsteps = and 7 1.2. Each feature transformer block uses one shared and two decision step- dependent fully-connected layer, ghost batch normalization and GLU blocks. Adam optimization is used with a learning rate of 0.01 (decayed 0.95 every 8k iterations with an exponential decay) for 600k iterations.
TabNet-M model uses Na = N₂ = 64, Asparse = 0.0001, B = 4096, Bv = 128, mg = 0.8. Nateps 7 and y 1.5. Each feature transformer block uses two shared and two decision step- dependent collececndeeed layer, ghost batch normalization and GLU blocks. Adam optimization is used with learning rate of 0.01 (decayed 0.95 every 8k iterations with exponential decay) for 600k iterations.
The T model uses Na Na 128, Apparse 0.0001, B 4096, By 128, ms 0.8, Nsteps 5 and 7 1.5. Each feature transformer block uses two shared and two decision step- dependent fully-connected layer, ghost batch normalization and GLU blocks. Adam optimization used with a learning rate of 0.02 (decayed 0.9 every 8k iterations with an exponential decay) for 600k iterations.
C.5 HIGGS DATASET
S model uses Na- 24, Na = 26, Asparse 0.000001, B = 16384, By = 512, mB 0.6. Nsteps 5 and 7 = 1.5. Each feature transformer block uses two shared and two decision step- dependent fully-connected layer, ghost batch normalization and GLU blocks. Adam optimization is used with a learning rate of 0.02 (decayed 0.9 every 20k iterations with an exponential decay) for 870k iterations.
TabNet-M model uses N₂ = 96, N₂ = 32, Asparse = 0.000001, B = 8192, By = 256, ma = 0.9, Nsteps 8 and 7 2.0. Each feature transformer block uses shared and two decision step- dependent fully-connected layer, ghost batch normalization and GLU blocks. Adam optimization is used with learning rate of 0.025 (decayed 0.9 every 10k iterations with an exponential decay) for 370k iterations.
For gradient boosted trees, we use the implementation [Tensorllow 2019). We choose the learning rate of 0.1 and optimize the maximum depth to 8, based on the performance. The Gradient boosted tree-S model uses 50 trees, the Gradient boosted tree-M model uses 300 trees and the Gradient boosted tree-I. model uses 3000 trees.
C.6 MUSHROOM EDIBILITY DATASET
TabNet model uses N₂ = N₂ 8, Asparse 0.001, B = 2048, By = 128. mB = 0.9, Nateps and 7 1.5. Each feature transformer block uses shared and two decision step-dependent
16

---

Under review as a conference paper at ICLR 2020
fully-com connectd layer, ghost batch normalization and GLU blocks. Adam optimization is used with learning rate of 0.01 (decayed 0.8 every 400 iterations with an exponential decay) for 10k iterations.
C.7 ADULT CENSUS INCOME DATASET
TabNet model uses No No 16, Aeparee 0.0001, B 4096, By 128, mB 0.98, Nsteps = 5 and y = 1.5. Each feature transformer block uses two shared and two decision step- dependent layer, ghost batch normalization and GLU blocks. Adam optimization is used with learning rate 0.02 (decayed 0.4 every 2.5k iterations with an exponential decay) for 7.7k iterations.
When only 40 labeled examples are used instead of the full dataset, based re-optimization of hyperparameters on the validation set, we modify B 128, Asparse = 0.01 and the learning rate of 0.005 (decayed 0.95 every 10 iterations with an exponential decay) for 100 iterations.
C.8 ROSSMANN DATASET
TabNet model uses N₂ Na = 32, Aeparse = 0.001, 4096, By = 512, mB = 0.8. Nateps 5 and 7 1.2. Each feature transformer block uses shared and two decision step- dependent fully-connected layer, ghost batch normalization and GLU blocks. Adam optimization is used with learning rate of 0.002 (decayed 0.95 every 2000 iterations with an exponential decay) for 15k iterations.
D GUIDELINES FOR HYPERPARAMETER SELECTION
We consider datasets ranging from ~ 10K to ~ 10M training points, with varying degrees of fitting difficulty. TabNet obtains high performance for all with few general principles hyperparameter selection:
Most datasets yield the best results for Nsteps E [3, 10). Typically, we observe that when iiformation-beariinn features, the optimal value of Nsteps tends be higher. On the other hand, increasing it beyond some value may adversely affect training dynamics some paths in the network becomes deeper and there ill-conditioned matrices. A very high value of Noteps typically suffers from overfitting and yields poor generalization.
there are
potentially-probleemt
Adjustment of the values of N₂ and N,, is the most efficient way of obtaining a trade-off between performance and complexity. Na Na is reasonable choice for most datasets. A very high value of Na and N. may suffer from overfitting and yield poor generalization.
An optimal choice of can have major role on the overall performance. Typically larger
Netepe value favors for a larger y.
A large batch size is beneficial for performance if the memory constraints permit, as large 1-10 % of the total training dataset size suggested. The virtual batch size is typically much smaller than the batch size.
Initially large learning rate is important, which should be gradually decayed until conver- gence.
When the model size is constrained, the hyperparameter search becomes more complicated. Because the optimal ways to increase the representation capacity may be chosen among different options, such as increasing the number of steps or the unit size. For example, increasing the number of units while slightly decreasing the step size can be better way of optimal utilization of the limited capacity constrained by the size on the number of parameters.
ABLATION STUDIES
In Table I0, we show the impact various design and hyperparameter choices. For all cases, the number of iterations optimized the validation
17

---

Under review as a conference paper at ICLR 2020
Table 10: Ablation studies for the TabNet model for the forest cover type dataset.
Test accuracy % (difference)
Number of parameters
Ablation cases
Base (Na = N. 64, y 1.5, Nsteps 5, Asparse 0.0001, feature transformer block composed of two shared and two decision step-dependent layers,
96.99 (0)
470k
B 16384)
Decreasing capacity via number of units (with
94.99 (-2.00)
129k
N₂ = N₂ = 32)
Decreasing capacity via number of decision steps (with 96.22 (-0.77)
328k
N Nateps 3)
Increasing capacity via number of decision steps (with
95.48 (-1.51)
755k
N Nsteps 9)
Decreasing capacity via all-shared feature transformer
96.74 (-0.25)
143k
blocks
Increasing capacity decision step-dependent feature
96.76 (-0.23)
703k
transformer blocks
Replacing feature transformer block with single shared
95.32 (-1.67)
35k
layer
Replacing feature transformer block with a single shared
93.92 (-3.07)
27k
layer, with ReLU instead of GLU
Replacing feature transformer block with two shared
96.34 (-0.66)
71k
layers
Replacing feature transformer block with two shared
96.54 (-0.45)
27 lk
layers and decision step-dependent layer Replacing feature transformer block with a single decision-step dependent layer Replacing feature transformer block with single
94.71 (-0.28)
105k
96.24 (-0.75)
208k
decision-step dependent layer, with N₂ = N₂ 128 Replacing feature transformer block with single
decision-step dependent layer, with N₂ = N₂ = 128 and replacing GLU with ReLU Replacing feature transformer block with a single
95.67 (-1.32)
139k
decision-step dependent layer, with N₂ N, 256 and
96.41 (-0.58)
278k
replacing GLU with ReLU Reducing the impact of prior scale (with 7 3.0)
96.49 (-0.50)
470k
Increasing the impact of prior scale (with 1.0)
96.67 (-0.32)
470k
No sparsity larizaaiittionn (with Xeporee = 0)
96.50 (-0.49)
470k
High sparsity regularization (with Asparse 0.01)
93.87 (-3.12)
470k
Small batch size (B 4096)
96.42 (-0.57)
470k
18

---

Under review as a conference paper at ICLR 2020
Obtaining high performance necessitates appropriately-adieee acteristics of the dataset. Decreasing the number of units Na, Na or the number of decision steps Nateps are efficient ways of gradually decreasing the capacity without significant degradation in performance. On the other hand, increasing these parameters beyond some value causes optimization issues and do yield performance benefits.
model capacity based on the char-
Replacing the feature transformer block with a very simpler alternative, such as a single shared layer, can still give strong performance while yielding very compact model architecture. This shows the importance of the inductive bias introduced with feature selection and sequential attention.
To push for the performance further, increasing the depth of the feature transformer is the effective approach. While increasing the depth, parameter sharing between feature transformer blocks across different decisions an efficient way decrease model size without significant degradation from performance. We indeed observe the benefit of partial parameter sharing, compared to fully decision step-dependent blocks or fully shared blocks. We observe the empirical benefit of GLU, compared to conventional nonlinearities like ReLU.
The strength of sparse feature selection depends on the two parameters we introduce: 7 and Asparse- We show that optimal choice of these two is important for performance. A y close to 1, or a high Asporse may yield too tight constraints on strength sparsity and may hurt performance. On the other hand, there is still the benefit of sufficient low y and sufficiently high Asporse to aid learning of the model via a favorable inductive bias.
Lastly, given the fixed model architecture, we show the benefit of large-batch training, enabled by ghost batch normalization (Hoffer et al. (2017). The optimal batch size for TabNet seems considerably higher than the conventional batch sizes used for other data types, such as images speech.
F MIXUP TRAINING
In (Zhang et al. 2017), mixup training was shown beneficial for tabular data learning, on small-scale datasets with simple neural network models comprising fully-connected layers. We experiment mixup training with TabNet and did not observe superior performance compared to standard softmax training. For Covertype dataset, the best mixup model (for mixup parameter a=3.3) yields a test accuracy of 96.28% roughly 0.7% lower than softmax training. For Higgs dataset, for the best TabNet-S model, the best mixup model (for mixup parameter a=0.1) yields test accuracy of 78.1 %,, roughly 0.1 % lower than softmax training. We hypothesize that linearization of the inputs may cause significant shifts in the input distribution and thus adversely affect the feature selection blocks of TabNet.
G SEMI-SUPERVISED LEARNING
Reconstructed features
Feature transformer
Feature transformer
Stop
Stop
Encoded representation
Figure 8: Decoder architecture to transform the encoded representation into reconstructed tabular data features. Each decision step composed of feature transformer block (see Fig. 3). and fully-connected layer.
We explore the capability of TabNet in learning semantically representations by integrat- ing it into an autoencoder framework (Baldi, 2012). For this purpose, we propose a simple decoder architecture, shown in Fig. 8 The decoder composed of feature transformer block (as given in Fig. 3). followed by a fully fully-connected layer at each decision step. Different decision steps are summed output the reconstructed features.
19

---

Under review as a conference paper at ICLR 2020
We propose additive reconstruction loss (with coefficient Aunsup) between the input features X and the reconstructed features X. The reconstruction loss (computed over unlabeled data batch of size By) is in the form of L2 loss, normalized with the population standard deviation of the ground truth data, and scaled by S:
Bu D
(Xb.j Xb.j) Sb. Bu b= XbJ
Lunsup (X By
b=1j=1
Bu E By
Xb,)²
Normalization with the input value is observed to be erucial, the tabular data features may have very different ranges. A straightforward approach in conventional reconstruction loss is scaling with uniform mask, Sb.= = 1/D, As more promising alternative, we propose that scaling should be based on feature importance values, such that the autoencoder should prioritize learning the representation for features that are the most important for decision making. We use the feature important mask
SbJ -gggbb Σラ= M.g to promote learning for the most salient features. M is inferred from the TabNet for the batch of unlabeled training samples, and fixed the computation of loss avoid the trivial solutions of fitting the easiest features.
For semi-supervised learning experiments, we consider the Adult Census Income dataset. We randomly choose 50 samples the labeled set. We fix the TabNet model with the aferomentioned hyperparameters. As the original learning hyperparameters overfit very quickly for 50 samples, we reoptimize the learning rate 0.01 (decayed 0.9 every 100 iterations with exponential decay) and trained for 800 iterations. For the autoencoder, we also fix the TabNet architecture, and optimize the decoder and learning hyperparameters. We use a decoder architecture with N₂ .N. 16, B = 128, and m 0.98. We use an unlabeled batch size of By 2048. The model with uniform masking uses Aunsup 0.2, X Asporse 0.005, the number of decoder steps of N Nsteps learning rate of 0.005 (decayed 0.95 every 4k iterations with an exponential decay) and trained for 20.6k iterations. The model with feature importance mask uses Aunsup 0.1, Asparse 0.005, the number of decoder steps of Nsteps 4 and learning rate of 0.01 (decayed 0.9 every 10k iterations with an exponential decay) and trained for 63k iterations. Since feature importance masking focuses on reconstructing the most salient features, the learning capacity of the optimal decoder is lower.
and
Table 11: Results for semi-supervised learning for Adult Census Income, along with the supervised
learning benchmarks. 50 samples with labels are randomly chosen from the training dataset. We reoptimize the learning hyperparameters on separate validation set for a fair comparison.
Dataset
Learning setting
Test accuracy (%)
50 labeled
Fully-supervised
76.8
50 labeled 26015 unlabeled
Semi-supervised (autoencoder with uniform mask)
78.9
50 labeled 26015 unlabeled
Semi-supervised (autoencoder with feature importance mask)
80.6
26065 labeled
Fully-supervised
85.7
Table II shows the semi-supervised learning performance, along with the two supervised learning benchmarks: when trained without additional unlabeled data and when trained after labeling the entire dataset. We observe significant boost in performance with the contributions from the unlabeled data, closing the gap towards the supervised learning baseline of the entire dataset. Focusing on the most important features in autoencoding helps improving the semi-supervised learning performance.
20


Reviews:
[{"review_id": "rJlzbfWGcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a novel architecture with sequential attention modules for tabular learning. An attention module is trained to select some elements from the (normalized) input feature, and a feature transformer takes the selected features for overall feature embedding. The model is evaluated on multiple tasks, and the proposed method outperforms prior approaches in the paper.\n\nI am in favor of this paper as it proposes an interpretable method for feature extraction in tabular learning. The learned overall feature embedding is shown to be effective in multiple tasks. It seems to me that the proposed method may have high generalization capability because the method is trained to select sparse feature attributes for decision. The authors may study the generalization performance further.\n\nThe experiment section may not be very reliable. The numbers indicated with * are copied from other papers, but these numbers may not be compatible if there is any discrepancy in the experiment setup. It would be better if the authors can run the baselines on the datasets. \n\nI am a little disappointed that no study on the number of steps needed for the model. It seems non-trivial to me. In TabNet-L, the step size is 5 but in TabNet-M, the step size is 7 (even smaller?). How should we choose the right step size?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1l40wry5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel deep machine learning model called TabNet for learning from tabular model. The system builds on sequential feature acquisition that is aggregated to perform decisions. The performance of TabNet is evaluated in a set of experiments.\n\nThe paper proposes a network architecture which seems to achieve good performance on a set of experiments. However, I am missing a motivation for parts of the model. A broader ablation study in which certain components of the model are replaced by simpler network structures could be instructive; the current ablation study only investigates hyper-parameters and a few design choices -- but what happens for example if a feature transformer is simply replaced by fully connected layer instead of the used architecture, i.e. how general is the structure? Regarding experiments I am also missing details and a broader comparison with other methods.\n\nSome more questions/comments:\n* What is the training criterion used?\n* In Table 1, what precisely is AUC? Area under \"accuracy over selected features\"? If so, what does the number \"No sel.\" mean?\n* Why is \"No sel.\" performing relatively bad? Overfitting? \n* What about stronger feature selection baselines, like those based on Shapley values?\n* What do the errors in table 1 mean? Which numbers are marked in bold? This seems misleading when looking at the \"error bars\".\n* What are the std errors for all other tables?\n* What is the \"ratio\" in Figure 6a? Why does the figure not show INVASE?\n* There are other instance-wise feature selection methods, e.g. Chao Ma et al.'s \"EDDI: Efficient Dynamic Discovery of High-Value Information with Partial VAE\" or . How does the proposed method compare against those?\n* For making a claim about tabular data in general, more datasets should be considered.\n\nMinor comments:\n* Figure 1: It would support the reader to explain the blocks in the figure caption.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJll0MB6YH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new neural network for tabular data which uses sequential attention to perform instance-wise feature selection which can help learn superior decision rules, in addition to facilitating interpretability of the resulting model.  Overall, the paper is well-written and fairly easy to follow, and the idea appears conceptually well-grounded and of high practical value. However, I have concerns about the experiments.\n\nAs the proposed model is primarily empirically motivated (ie. there is no theoretical justification for why it should be better), it is imperative the authors conduct thorough experiments to convincingly demonstrate its utility.  However, the current experiments are too sparse to be fully convincing, given how easy it is nowadays to evaluate ML models across diverse collections of tabular datasets (eg. OpenML, UCI, CatBoost). While other papers on tabular-data models study many real datasets (eg. Klambauer et al), this paper only studies 4 real datasets (and does not even quantify the variability in results across multiple training/test splits). \n\nAlso, why did the authors not compare against xgboost/lightGBM/catboost, some of the most popular tabular models, in all datasets? \n\nIt seems different hyperparameter values were also used for each dataset, what was the search-space / hyperparameter optimization method used here? Or were hyperparameters for each dataset found through manual tuning.\n\nKlambauer et al (2017). Self-Normalizing Neural Networks. \nhttps://arxiv.org/abs/1706.02515\n\nOpenML: https://www.openml.org\n\nUCI: https://archive.ics.uci.edu/ml/datasets.php\n\nCatBoost: https://catboost.ai/#benchmark\n\nOther comments:\n\n- Why don't the authors quantitatively compare the quality of the selected features from their model vs other feature selection methods for the Syn datasets?\n\n- What is \"softmax-training\" in Appendix D? If the authors just mean standard training, I recommend they use this name instead.\n\n- Since there are many components to this model, it would help clarify their purpose to the reader if the authors could provide one concrete example of the feedforward pass of a hypothetical instantiation of their model, say for a simple datapoint with just 2 features.\n\n- the subscript i in h_i should match the i used in M[i], P[i] to avoid confusion\n\n- P[0] (the base case) needs to be explained.\n\n- a[i-1] should be more clearly defined, and the base case explained in more detail (I assume it is just the input features). \n \n\n- I think it is a bit confusing to present the model parameters in terms of batch-size B, and think it would be better if you introduced them assuming batch-size = 1 (the generalization to batch-size B is obvious and should be familiar to readers as it's the same for all neural models).\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper constitutes interesting progress on an important topic; the reviewers identify certain improvements and directions for future work, and I urge the authors to continue to develop refinements and extensions.
Decision: Reject