Revisiting Deep Learning Models for Tabular Data

1 Introduction . Due to the tremendous success of deep learning on such data domains as images , audio and texts ( Goodfellow et al. , 2016 ) , there has been a lot of research interest to extend this success to problems with data stored in tabular format . In these problems , data points are represented as vectors of heterogeneous features , which is typical for industrial applications and ML competitions , where neural networks have a strong non-deep competitor in the form of GBDT ( Chen and Guestrin , 2016 ; Ke et al. , 2017 ; Prokhorenkova et al. , 2018 ) . Along with potentially higher performance , using deep learning for tabular data is appealing as it would allow constructing multi-modal pipelines for problems , where only one part of the input is tabular , and other parts include images , audio and other DL-friendly data . Such pipelines can then be trained end-to-end by gradient optimization for all modalities . For these reasons , a large number of DL solutions were recently proposed , and new models continue to emerge ( Arik and Pfister , 2020 ; Badirli et al. , 2020 ; Hazimeh et al. , 2020 ; Huang et al. , 2020 ; Klambauer et al. , 2017 ; Popov et al. , 2020 ; Song et al. , 2019 ; Wang et al. , 2017 , 2020 ) . Unfortunately , due to the lack of established benchmarks ( such as ImageNet ( Deng et al. , 2009 ) for computer vision or GLUE ( Wang et al. , 2019a ) for NLP ) , existing papers use different datasets for evaluation and proposed DL models are often not adequately compared to each other . Therefore , from the current literature , it is unclear what DL model generally performs better than others and whether GBDT is surpassed by DL models . Additionally , despite the large number of novel architectures , the field still lacks simple and reliable solutions that allow achieving competitive performance with moderate effort and provide stable performance across many tasks . In that regard , Multilayer ∗Correspondence to : yura.gorishniy @ phystech.edu 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Perceptron ( MLP ) remains the main simple baseline for the field , however , it does not always represent a significant challenge for other competitors . The described problems impede the research process and make the observations from the papers not conclusive enough . Therefore , we believe it is timely to review the recent developments from the field and raise the bar of baselines in tabular DL . We start with a hypothesis that well-studied DL architecture blocks may be underexplored in the context of tabular data and may be used to design better baselines . Thus , we take inspiration from well-known battle-tested architectures from other fields and obtain two simple models for tabular data . The first one is a ResNet-like architecture ( He et al. , 2015 ) and the second one is FT-Transformer — our simple adaptation of the Transformer architecture ( Vaswani et al. , 2017 ) for tabular data . Then , we compare these models with many existing solutions on a diverse set of tasks under the same protocols of training and hyperparameters tuning . First , we reveal that none of the considered DL models can consistently outperform the ResNet-like model . Given its simplicity , it can serve as a strong baseline for future work . Second , FT-Transformer demonstrates the best performance on most tasks and becomes a new powerful solution for the field . Interestingly , FT-Transformer turns out to be a more universal architecture for tabular data : it performs well on a wider range of tasks than the more “ conventional ” ResNet and other DL models . Finally , we compare the best DL models to GBDT and conclude that there is still no universally superior solution . We summarize the contributions of our paper as follows : 1 . We thoroughly evaluate the main models for tabular DL on a diverse set of tasks to investigate their relative performance . 2 . We demonstrate that a simple ResNet-like architecture is an effective baseline for tabular DL , which was overlooked by existing literature . Given its simplicity , we recommend this baseline for comparison in future tabular DL works . 3 . We introduce FT-Transformer — a simple adaptation of the Transformer architecture for tabular data that becomes a new powerful solution for the field . We observe that it is a more universal architecture : it performs well on a wider range of tasks than other DL models . 4 . We reveal that there is still no universally superior solution among GBDT and deep models . 2 Related work . The “ shallow ” state-of-the-art for problems with tabular data is currently ensembles of decision trees , such as GBDT ( Gradient Boosting Decision Tree ) ( Friedman , 2001 ) , which are typically the top-choice in various ML competitions . At the moment , there are several established GBDT libraries , such as XGBoost ( Chen and Guestrin , 2016 ) , LightGBM ( Ke et al. , 2017 ) , CatBoost ( Prokhorenkova et al. , 2018 ) , which are widely used by both ML researchers and practitioners . While these implementations vary in detail , on most of the tasks , their performances do not differ much ( Prokhorenkova et al. , 2018 ) . During several recent years , a large number of deep learning models for tabular data have been developed ( Arik and Pfister , 2020 ; Badirli et al. , 2020 ; Hazimeh et al. , 2020 ; Huang et al. , 2020 ; Klambauer et al. , 2017 ; Popov et al. , 2020 ; Song et al. , 2019 ; Wang et al. , 2017 ) . Most of these models can be roughly categorized into three groups , which we briefly describe below . Differentiable trees . The first group of models is motivated by the strong performance of decision tree ensembles for tabular data . Since decision trees are not differentiable and do not allow gradient optimization , they can not be used as a component for pipelines trained in the end-to-end fashion . To address this issue , several works ( Hazimeh et al. , 2020 ; Kontschieder et al. , 2015 ; Popov et al. , 2020 ; Yang et al. , 2018 ) propose to “ smooth ” decision functions in the internal tree nodes to make the overall tree function and tree routing differentiable . While the methods of this family can outperform GBDT on some tasks ( Popov et al. , 2020 ) , in our experiments , they do not consistently outperform ResNet . Attention-based models . Due to the ubiquitous success of attention-based architectures for different domains ( Dosovitskiy et al. , 2021 ; Vaswani et al. , 2017 ) , several authors propose to employ attentionlike modules for tabular DL as well ( Arik and Pfister , 2020 ; Huang et al. , 2020 ; Song et al. , 2019 ) . In our experiments , we show that the properly tuned ResNet outperforms the existing attention-based models . Nevertheless , we identify an effective way to apply the Transformer architecture ( Vaswani et al. , 2017 ) to tabular data : the resulting architecture outperforms ResNet on most of the tasks . Explicit modeling of multiplicative interactions . In the literature on recommender systems and click-through-rate prediction , several works criticize MLP since it is unsuitable for modeling multiplicative interactions between features ( Beutel et al. , 2018 ; Qin et al. , 2021 ; Wang et al. , 2017 ) . Inspired by this motivation , some works ( Beutel et al. , 2018 ; Wang et al. , 2017 , 2020 ) have proposed different ways to incorporate feature products into MLP . In our experiments , however , we do not find such methods to be superior to properly tuned baselines . The literature also proposes some other architectural designs ( Badirli et al. , 2020 ; Klambauer et al. , 2017 ) that can not be explicitly assigned to any of the groups above . Overall , the community has developed a variety of models that are evaluated on different benchmarks and are rarely compared to each other . Our work aims to establish a fair comparison of them and identify the solutions that consistently provide high performance . 3 Models for tabular data problems . In this section , we describe the main deep architectures that we highlight in our work , as well as the existing solutions included in the comparison . Since we argue that the field needs strong easy-to-use baselines , we try to reuse well-established DL building blocks as much as possible when designing ResNet ( section 3.2 ) and FT-Transformer ( section 3.3 ) . We hope this approach will result in conceptually familiar models that require less effort to achieve good performance . Additional discussion and technical details for all the models are provided in supplementary . Notation . In this work , we consider supervised learning problems . D= { ( xi , yi ) } ni=1 denotes a dataset , where xi= ( x ( num ) i , x ( cat ) i ) ∈ X represents numerical x ( num ) ij and categorical x ( cat ) ij features of an object and yi ∈ Y denotes the corresponding object label . The total number of features is denoted as k. The dataset is split into three disjoint subsets : D = Dtrain ∪ Dval ∪ Dtest , where Dtrain is used for training , Dval is used for early stopping and hyperparameter tuning , and Dtest is used for the final evaluation . We consider three types of tasks : binary classification Y = { 0 , 1 } , multiclass classification Y = { 1 , . . . , C } and regression Y = R . 3.1 MLP . We formalize the “ MLP ” architecture in Equation 1 . MLP ( x ) = Linear ( MLPBlock ( . . . ( MLPBlock ( x ) ) ) ) MLPBlock ( x ) = Dropout ( ReLU ( Linear ( x ) ) ) ( 1 ) 3.2 ResNet . We are aware of one attempt to design a ResNet-like baseline ( Klambauer et al. , 2017 ) where the reported results were not competitive . However , given ResNet ’ s success story in computer vision ( He et al. , 2015 ) and its recent achievements on NLP tasks ( Sun and Iyyer , 2021 ) , we give it a second try and construct a simple variation of ResNet as described in Equation 2 . The main building block is simplified compared to the original architecture , and there is an almost clear path from the input to output which we find to be beneficial for the optimization . Overall , we expect this architecture to outperform MLP on tasks where deeper representations can be helpful . ResNet ( x ) = Prediction ( ResNetBlock ( . . . ( ResNetBlock ( Linear ( x ) ) ) ) ) ResNetBlock ( x ) = x+ Dropout ( Linear ( Dropout ( ReLU ( Linear ( BatchNorm ( x ) ) ) ) ) ) Prediction ( x ) = Linear ( ReLU ( BatchNorm ( x ) ) ) ( 2 ) 3.3 FT-Transformer . In this section , we introduce FT-Transformer ( Feature Tokenizer + Transformer ) — a simple adaptation of the Transformer architecture ( Vaswani et al. , 2017 ) for the tabular domain . Figure 1 demonstrates the main parts of FT-Transformer . In a nutshell , our model transforms all features ( categorical and numerical ) to embeddings and applies a stack of Transformer layers to the embeddings . Thus , every Transformer layer operates on the feature level of one object . We compare FT-Transformer to conceptually similar AutoInt in section 5.2 . Feature Tokenizer . The Feature Tokenizer module ( see Figure 2 ) transforms the input features x to embeddings T ∈ Rk×d . The embedding for a given feature xj is computed as follows : Tj = bj + fj ( xj ) ∈ Rd fj : Xj → Rd . where bj is the j-th feature bias , f ( num ) j is implemented as the element-wise multiplication with the vector W ( num ) j ∈ Rd and f ( cat ) j is implemented as the lookup table W ( cat ) j ∈ RSj×d for categorical features . Overall : T ( num ) j = b ( num ) j + x ( num ) j ·W ( num ) j ∈ Rd , T ( cat ) j = b ( cat ) j + e T j W ( cat ) j ∈ Rd , T = stack [ T ( num ) 1 , . . . , T ( num ) k ( num ) , T ( cat ) 1 , . . . , T ( cat ) k ( cat ) ] ∈ Rk×d . where eTj is a one-hot vector for the corresponding categorical feature . Transformer . At this stage , the embedding of the [ CLS ] token ( or “ classification token ” , or “ output token ” , see Devlin et al . ( 2019 ) ) is appended to T and L Transformer layers F1 , . . . , FL are applied : T0 = stack [ [ CLS ] , T ] Ti = Fi ( Ti−1 ) . We use the PreNorm variant for easier optimization ( Wang et al. , 2019b ) , see Figure 2 . In the PreNorm setting , we also found it to be necessary to remove the first normalization from the first Transformer layer to achieve good performance . See the original paper ( Vaswani et al. , 2017 ) for the background on Multi-Head Self-Attention ( MHSA ) and the Feed Forward module . See supplementary for details such as activations , placement of normalizations and dropout modules ( Srivastava et al. , 2014 ) . Prediction . The final representation of the [ CLS ] token is used for prediction : ŷ = Linear ( ReLU ( LayerNorm ( T [ CLS ] L ) ) ) . Limitations . FT-Transformer requires more resources ( both hardware and time ) for training than simple models such as ResNet and may not be easily scaled to datasets when the number of features is “ too large ” ( it is determined by the available hardware and time budget ) . Consequently , widespread usage of FT-Transformer for solving tabular data problems can lead to greater CO2 emissions produced by ML pipelines , since tabular data problems are ubiquitous . The main cause of the described problem lies in the quadratic complexity of the vanilla MHSA with respect to the number of features . However , the issue can be alleviated by using efficient approximations of MHSA ( Tay et al. , 2020 ) . Additionally , it is still possible to distill FT-Transformer into simpler architectures for better inference performance . We report training times and the used hardware in supplementary .

Summary: - The authors conduct extensive experiments on tabular data learning with various datasets and baselines. - The authors discover some important takeaways for tabular learning such as when we should use shallow or deep models. - The authors propose a novel tabular learning framework which shows the state-of-the-art performance in various datasets.