Revisiting Deep Learning Models for Tabular Data

1 Introduction . Due to the tremendous success of deep learning on such data domains as images , audio and texts ( Goodfellow et al. , 2016 ) , there has been a lot of research interest to extend this success to problems with data stored in tabular format . In these problems , data points are represented as vectors of heterogeneous features , which is typical for industrial applications and ML competitions , where neural networks have a strong non-deep competitor in the form of GBDT ( Chen and Guestrin , 2016 ; Ke et al. , 2017 ; Prokhorenkova et al. , 2018 ) . Along with potentially higher performance , using deep learning for tabular data is appealing as it would allow constructing multi-modal pipelines for problems , where only one part of the input is tabular , and other parts include images , audio and other DL-friendly data . Such pipelines can then be trained end-to-end by gradient optimization for all modalities . For these reasons , a large number of DL solutions were recently proposed , and new models continue to emerge ( Arik and Pfister , 2020 ; Badirli et al. , 2020 ; Hazimeh et al. , 2020 ; Huang et al. , 2020 ; Klambauer et al. , 2017 ; Popov et al. , 2020 ; Song et al. , 2019 ; Wang et al. , 2017 , 2020 ) . Unfortunately , due to the lack of established benchmarks ( such as ImageNet ( Deng et al. , 2009 ) for computer vision or GLUE ( Wang et al. , 2019a ) for NLP ) , existing papers use different datasets for evaluation and proposed DL models are often not adequately compared to each other . Therefore , from the current literature , it is unclear what DL model generally performs better than others and whether GBDT is surpassed by DL models . Additionally , despite the large number of novel architectures , the field still lacks simple and reliable solutions that allow achieving competitive performance with moderate effort and provide stable performance across many tasks . In that regard , Multilayer ∗Correspondence to : yura.gorishniy @ phystech.edu 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Perceptron ( MLP ) remains the main simple baseline for the field , however , it does not always represent a significant challenge for other competitors . The described problems impede the research process and make the observations from the papers not conclusive enough . Therefore , we believe it is timely to review the recent developments from the field and raise the bar of baselines in tabular DL . We start with a hypothesis that well-studied DL architecture blocks may be underexplored in the context of tabular data and may be used to design better baselines . Thus , we take inspiration from well-known battle-tested architectures from other fields and obtain two simple models for tabular data . The first one is a ResNet-like architecture ( He et al. , 2015 ) and the second one is FT-Transformer — our simple adaptation of the Transformer architecture ( Vaswani et al. , 2017 ) for tabular data . Then , we compare these models with many existing solutions on a diverse set of tasks under the same protocols of training and hyperparameters tuning . First , we reveal that none of the considered DL models can consistently outperform the ResNet-like model . Given its simplicity , it can serve as a strong baseline for future work . Second , FT-Transformer demonstrates the best performance on most tasks and becomes a new powerful solution for the field . Interestingly , FT-Transformer turns out to be a more universal architecture for tabular data : it performs well on a wider range of tasks than the more “ conventional ” ResNet and other DL models . Finally , we compare the best DL models to GBDT and conclude that there is still no universally superior solution . We summarize the contributions of our paper as follows : 1 . We thoroughly evaluate the main models for tabular DL on a diverse set of tasks to investigate their relative performance . 2 . We demonstrate that a simple ResNet-like architecture is an effective baseline for tabular DL , which was overlooked by existing literature . Given its simplicity , we recommend this baseline for comparison in future tabular DL works . 3 . We introduce FT-Transformer — a simple adaptation of the Transformer architecture for tabular data that becomes a new powerful solution for the field . We observe that it is a more universal architecture : it performs well on a wider range of tasks than other DL models . 4 . We reveal that there is still no universally superior solution among GBDT and deep models . 2 Related work . The “ shallow ” state-of-the-art for problems with tabular data is currently ensembles of decision trees , such as GBDT ( Gradient Boosting Decision Tree ) ( Friedman , 2001 ) , which are typically the top-choice in various ML competitions . At the moment , there are several established GBDT libraries , such as XGBoost ( Chen and Guestrin , 2016 ) , LightGBM ( Ke et al. , 2017 ) , CatBoost ( Prokhorenkova et al. , 2018 ) , which are widely used by both ML researchers and practitioners . While these implementations vary in detail , on most of the tasks , their performances do not differ much ( Prokhorenkova et al. , 2018 ) . During several recent years , a large number of deep learning models for tabular data have been developed ( Arik and Pfister , 2020 ; Badirli et al. , 2020 ; Hazimeh et al. , 2020 ; Huang et al. , 2020 ; Klambauer et al. , 2017 ; Popov et al. , 2020 ; Song et al. , 2019 ; Wang et al. , 2017 ) . Most of these models can be roughly categorized into three groups , which we briefly describe below . Differentiable trees . The first group of models is motivated by the strong performance of decision tree ensembles for tabular data . Since decision trees are not differentiable and do not allow gradient optimization , they can not be used as a component for pipelines trained in the end-to-end fashion . To address this issue , several works ( Hazimeh et al. , 2020 ; Kontschieder et al. , 2015 ; Popov et al. , 2020 ; Yang et al. , 2018 ) propose to “ smooth ” decision functions in the internal tree nodes to make the overall tree function and tree routing differentiable . While the methods of this family can outperform GBDT on some tasks ( Popov et al. , 2020 ) , in our experiments , they do not consistently outperform ResNet . Attention-based models . Due to the ubiquitous success of attention-based architectures for different domains ( Dosovitskiy et al. , 2021 ; Vaswani et al. , 2017 ) , several authors propose to employ attentionlike modules for tabular DL as well ( Arik and Pfister , 2020 ; Huang et al. , 2020 ; Song et al. , 2019 ) . In our experiments , we show that the properly tuned ResNet outperforms the existing attention-based models . Nevertheless , we identify an effective way to apply the Transformer architecture ( Vaswani et al. , 2017 ) to tabular data : the resulting architecture outperforms ResNet on most of the tasks . Explicit modeling of multiplicative interactions . In the literature on recommender systems and click-through-rate prediction , several works criticize MLP since it is unsuitable for modeling multiplicative interactions between features ( Beutel et al. , 2018 ; Qin et al. , 2021 ; Wang et al. , 2017 ) . Inspired by this motivation , some works ( Beutel et al. , 2018 ; Wang et al. , 2017 , 2020 ) have proposed different ways to incorporate feature products into MLP . In our experiments , however , we do not find such methods to be superior to properly tuned baselines . The literature also proposes some other architectural designs ( Badirli et al. , 2020 ; Klambauer et al. , 2017 ) that can not be explicitly assigned to any of the groups above . Overall , the community has developed a variety of models that are evaluated on different benchmarks and are rarely compared to each other . Our work aims to establish a fair comparison of them and identify the solutions that consistently provide high performance . 3 Models for tabular data problems . In this section , we describe the main deep architectures that we highlight in our work , as well as the existing solutions included in the comparison . Since we argue that the field needs strong easy-to-use baselines , we try to reuse well-established DL building blocks as much as possible when designing ResNet ( section 3.2 ) and FT-Transformer ( section 3.3 ) . We hope this approach will result in conceptually familiar models that require less effort to achieve good performance . Additional discussion and technical details for all the models are provided in supplementary . Notation . In this work , we consider supervised learning problems . D= { ( xi , yi ) } ni=1 denotes a dataset , where xi= ( x ( num ) i , x ( cat ) i ) ∈ X represents numerical x ( num ) ij and categorical x ( cat ) ij features of an object and yi ∈ Y denotes the corresponding object label . The total number of features is denoted as k. The dataset is split into three disjoint subsets : D = Dtrain ∪ Dval ∪ Dtest , where Dtrain is used for training , Dval is used for early stopping and hyperparameter tuning , and Dtest is used for the final evaluation . We consider three types of tasks : binary classification Y = { 0 , 1 } , multiclass classification Y = { 1 , . . . , C } and regression Y = R . 3.1 MLP . We formalize the “ MLP ” architecture in Equation 1 . MLP ( x ) = Linear ( MLPBlock ( . . . ( MLPBlock ( x ) ) ) ) MLPBlock ( x ) = Dropout ( ReLU ( Linear ( x ) ) ) ( 1 ) 3.2 ResNet . We are aware of one attempt to design a ResNet-like baseline ( Klambauer et al. , 2017 ) where the reported results were not competitive . However , given ResNet ’ s success story in computer vision ( He et al. , 2015 ) and its recent achievements on NLP tasks ( Sun and Iyyer , 2021 ) , we give it a second try and construct a simple variation of ResNet as described in Equation 2 . The main building block is simplified compared to the original architecture , and there is an almost clear path from the input to output which we find to be beneficial for the optimization . Overall , we expect this architecture to outperform MLP on tasks where deeper representations can be helpful . ResNet ( x ) = Prediction ( ResNetBlock ( . . . ( ResNetBlock ( Linear ( x ) ) ) ) ) ResNetBlock ( x ) = x+ Dropout ( Linear ( Dropout ( ReLU ( Linear ( BatchNorm ( x ) ) ) ) ) ) Prediction ( x ) = Linear ( ReLU ( BatchNorm ( x ) ) ) ( 2 ) 3.3 FT-Transformer . In this section , we introduce FT-Transformer ( Feature Tokenizer + Transformer ) — a simple adaptation of the Transformer architecture ( Vaswani et al. , 2017 ) for the tabular domain . Figure 1 demonstrates the main parts of FT-Transformer . In a nutshell , our model transforms all features ( categorical and numerical ) to embeddings and applies a stack of Transformer layers to the embeddings . Thus , every Transformer layer operates on the feature level of one object . We compare FT-Transformer to conceptually similar AutoInt in section 5.2 . Feature Tokenizer . The Feature Tokenizer module ( see Figure 2 ) transforms the input features x to embeddings T ∈ Rk×d . The embedding for a given feature xj is computed as follows : Tj = bj + fj ( xj ) ∈ Rd fj : Xj → Rd . where bj is the j-th feature bias , f ( num ) j is implemented as the element-wise multiplication with the vector W ( num ) j ∈ Rd and f ( cat ) j is implemented as the lookup table W ( cat ) j ∈ RSj×d for categorical features . Overall : T ( num ) j = b ( num ) j + x ( num ) j ·W ( num ) j ∈ Rd , T ( cat ) j = b ( cat ) j + e T j W ( cat ) j ∈ Rd , T = stack [ T ( num ) 1 , . . . , T ( num ) k ( num ) , T ( cat ) 1 , . . . , T ( cat ) k ( cat ) ] ∈ Rk×d . where eTj is a one-hot vector for the corresponding categorical feature . Transformer . At this stage , the embedding of the [ CLS ] token ( or “ classification token ” , or “ output token ” , see Devlin et al . ( 2019 ) ) is appended to T and L Transformer layers F1 , . . . , FL are applied : T0 = stack [ [ CLS ] , T ] Ti = Fi ( Ti−1 ) . We use the PreNorm variant for easier optimization ( Wang et al. , 2019b ) , see Figure 2 . In the PreNorm setting , we also found it to be necessary to remove the first normalization from the first Transformer layer to achieve good performance . See the original paper ( Vaswani et al. , 2017 ) for the background on Multi-Head Self-Attention ( MHSA ) and the Feed Forward module . See supplementary for details such as activations , placement of normalizations and dropout modules ( Srivastava et al. , 2014 ) . Prediction . The final representation of the [ CLS ] token is used for prediction : ŷ = Linear ( ReLU ( LayerNorm ( T [ CLS ] L ) ) ) . Limitations . FT-Transformer requires more resources ( both hardware and time ) for training than simple models such as ResNet and may not be easily scaled to datasets when the number of features is “ too large ” ( it is determined by the available hardware and time budget ) . Consequently , widespread usage of FT-Transformer for solving tabular data problems can lead to greater CO2 emissions produced by ML pipelines , since tabular data problems are ubiquitous . The main cause of the described problem lies in the quadratic complexity of the vanilla MHSA with respect to the number of features . However , the issue can be alleviated by using efficient approximations of MHSA ( Tay et al. , 2020 ) . Additionally , it is still possible to distill FT-Transformer into simpler architectures for better inference performance . We report training times and the used hardware in supplementary .

Summary: This paper investigates the performance of deep learning models on tabular data and compares it with traditional models such as gradient boosted decision trees. It shows that ResNet is good baseline for deep learning models on tabular data. It also proposes a new attention based architecture which surpasses all the deep learning methods mentioned in the paper.