Robust Deep Neural Networks for Heterogeneous Tabular Data 

1 INTRODUCTION . Tabular data is the most commonly used form of data , and it is ubiquitous in various applications , such as medical diagnosis based on patient history ( Fatima et al. , 2017 ) , predictive analytics for financial applications ( Dastile et al. , 2020 ) , cybersecurity ( Buczak & Guven , 2015 ) , and so forth . Although DNNs perform outstandingly well on homogeneous data , e.g. , visual , audio , and textual data ( Goodfellow et al. , 2016 ) , heterogeneous , tabular data still pose a challenge to these models ( Shwartz-Ziv & Armon , 2021 ) . We hypothesize that the moderate performance of DNNs on tabular data comes from two major factors . The first is the inductive bias ( es ) ( Katzir et al. , 2020 ) ; for example , DNNs assume that certain spatial structures are present in the data ( Mitchell et al. , 2017 ) , whereas tabular data do not have any spatial connections . The second reason is the high information loss during the data preprocessing step since tabular input data need to undergo cleansing ( dealing with missing , noisy , and inconsistent values ) , uniform discretized representation ( handling categorical and continuous values together ) , and scaling ( standardized representation of features ) steps . Along with these feature-processing steps , important information contained in the data may get lost , and , hence , the preprocessed feature vectors ( especially when one-hot encoded ) may negatively impact training and learning effectiveness ( García et al. , 2015 ) . As reported by Hancock & Khoshgoftaar ( 2020 ) , an efficient transformation of heterogeneous data for training DNNs is still a significant challenge . Typically , when heterogeneous tabular data are involved , the first choice across all machine learning ( ML ) algorithms are ensemble models based on decision trees ( Nielsen , 2016 ) , such as random forests ( RF ) ( Breiman , 2001 ) or gradient-boosted decision trees ( GBDT ) ( Friedman , 2002 ) . Since the inductive bias ( es ) of the methods based on decision trees are well suited to non-spatial heterogeneous data , the data preprocessing step is reduced to a minimum . In particular , the most common implementations of the GBDT algorithm—XGBoost ( Chen & Guestrin , 2016 ) , LightGBM ( Ke et al. , 2017 ) , and CatBoost ( Prokhorenkova et al. , 2018 ) — handle the missing values internally by searching for the best approximation of missing data points . However , the most significant computational disadvantage of the decision tree–based methods is while training the need to store ( almost ) the entire dataset in memory ( Katzir et al. , 2020 ) . Furthermore , in the multimodal datasets in which different data types are involved ( e.g. , visual and tabular data ) , decision tree–based models are not able to provide state-of-the-art results , whereas DNN models allow for batch-learning ( no need to store the whole dataset ) , and for those multimodal data tasks , DNNs demonstrate state-of-the-art performance ( Gu & Budhkar , 2021 ) . Towards the goal of significantly boosting DNNs on tabular data , we propose DeepTLF , a novel deep tabular learning framework that exploits the advantages of the GBDT algorithm as well as the flexibility of DNNs . The key element of the framework is a novel encoding algorithm , TreeDrivenEncoder , which transforms the heterogeneous tabular data into homogeneous data by distilling knowledge from nodes of trained decision trees . Thus , DeepTLF can preserve most of the information that is contained in the original data and encoded in the structure of the decision trees and benefit from preprocessing power of decision tree-based algorithms . Through experiments on various freely available real-world datasets , we demonstrate the advantages of such a composite learning approach for different prediction tasks . We argue that by transforming heterogeneous tabular data into homogeneous vectors , we can drastically improve the performance of DNNs on tabular data . The main contributions of this work are : ( I ) We propose a deep tabular learning framework - DeepTLF that combines the preprocessing strengths of GBDTs with the learning flexibility of DNNs . ( II ) The proposed framework builds on a generic approach for transforming heterogeneous tabular data into homogeneous vectors using the structure of decision trees from a gradient boosting model using a novel encoding function – TreeDrivenEncoder . Hence , the transformation approach can also be used independently from the presented deep learning framework . ( III ) In extensive experiments on various datasets and compared with state-of-the-art ML approaches , we show that the proposed framework mitigates well-known data-processing challenges and leads to unprecedented predictive performance , outperforming all the competitors . ( IV ) We provide an open-source implementation of DeepTLF https : //github.com/xxxx/DeepTLF 2 RELATED WORK . In this section , we briefly review the main ideas from prior work that are relevant to our framework and data encoding step . We provide more information on the state-of-the-art DL approaches for tabular data in the Appendix B . Independent works Moosmann et al . ( 2007 ) ; Geurts et al . ( 2006 ) demonstrate that data can be encoded using the RF algorithm by accessing leaf indices in the decision trees . The idea was also utilized by ( He et al. , 2014 ) , where instead of the RF model , trees from a GBDT model are used for the categorical data encoding . These works demonstrates that the decision trees are a powerful and convenient way to implement non-linear and categorical feature transformations for heterogeneous data . The DeepGBM framework Ke et al . ( 2019 ) further evolved the idea of distilling knowledge from decision trees leaf index by encoding them using a neural network for online learning tasks . This approach is quite popular , but the leaf indices ’ from a decision tree embedding do not fully represent the whole decision tree structure . Thus , each boosted tree is treated as a new meta categorical feature , which might be an issue for the DNNs ( Hancock & Khoshgoftaar , 2020 ) . In contrast to related methods , our aim is to holistically distill the information from decision trees , by utilizes the whole decision tree , not only the output leafs . The DeepTLF combines the advantages of GBDT ( such as handling missing values , categorical variables ) with the learning flexibility of DNNs to achieve superior and robust prediction performance . Also , Medvedev & D ’ yakonov ( 2020 ) demonstrated that a DNN which was trained using distilled data can outperform models trained on the whole original data . Other approaches such as NODE ( Popov et al. , 2019 ) , Net-DNF ( Katzir et al. , 2020 ) try to mimic the decision trees using DNNs . A difference to our framework is that DeepTLF is more robust to the data inconsistencies and does not require new DNN architectures , hence , it is straightforward to use . 3 DEEPTLF : DEEP TABULAR LEARNING FRAMEWORK . In this section , we present the main components of our DeepTLF framework . As shown in Fig . 1 , DeepTLF consists of three major components : ( 1 ) an ensemble of decision trees ( in this work , we utilize the GBDT algorithm ) , ( 2 ) a TreeDrivenEncoder that performs the transformation of the original data into homogeneous , binary feature vectors by distilling the information contained in the structures of the decision trees through the TreeDrivenEncoder algorithm , and ( 3 ) a deep neural network model trained on the binary feature vectors obtained from the TreeDrivenEncoder algorithm . We will describe the details of each component in the following subsections . 3.1 GRADIENT BOOSTED DECISION TREE . For the data encoding step , we selected one of the most powerful algorithms on tabular data , namely the gradient boosted Decision Trees ( GBDT ) algorithm ( Friedman , 2002 ) . GBDT is a well-known and widely used ensemble algorithm for tabular data both in research and industrial applications ( Chen & Guestrin , 2016 ) and is particularly successful for tasks containing heterogeneous features , small dataset sizes , and `` noisy '' data ( Nielsen , 2016 ) . Especially when it comes to handling variance and bias , gradient boosting ensembles show highly competitive performance in comparison with state-ofthe-art learning approaches ( Friedman , 2002 ; Nielsen , 2016 ) . In addition , multiple evaluations have empirically demonstrated that the decision trees of a GBDT ensemble preserve the information from the original data and can be used for further data processing ( He et al. , 2014 ; Ke et al. , 2019 ) . The key idea of the GBDT algorithm is to construct a strong model by iterative addition of weak learners . Formally , at each iteration k of the gradient boosting algorithm , the GBDT model ϕ can be defined as : ϕk ( x ) = ϕk−1 ( x ) + λhk ( x ) , ( 1 ) where x is an input feature vector , ϕk−1 is the strong model constructed at the previous iteration , h is a weak learner from a family of functionsH , and λ is the learning rate . hk = argmin h∈H ∑ i ( −∂L ( ϕ k−1 ( xi ) , yi ) ∂ϕk−1 ( xi ) − h ( xi ) ) 2 . ( 2 ) More specifically , a pseudo-residual −∂L ( ϕ k−1 ( xi ) , yi ) ∂ϕk−1 ( xi ) should be approximated as well as possible by the current weak model h ( xi ) . The gradient w.r.t . the current predictions indicate how these predictions should be changed in order to minimize the loss function . Informally , gradient boosting can be thought of as performing gradient descent in the functional space . The set of weak learnersH is usually formed by shallow decision trees , which are directly trained on the original data . Consequently , almost no data preparation is needed , and the information loss is minimized . We denote a GBDT model as a set of decision trees : GB ( X , y ) = { T1 , T2 , ... , Tk } , where X is a matrix of observations , y is a vector or matrix of labels . 3.2 KNOWLEDGE DISTILLATION FROM GRADIENT BOOSTED DECISION TREES USING TREEDRIVENENCODER . The trained GBDT model provides structural data information , which also encodes dependencies between the input features with respect to the prediction task . In order to distill the knowledge from a tree-based model , we propose a novel data transformation algorithm – TreeDrivenEncoder . For every input vector from the original data , the proposed encoding method maps all features occurring in the decision trees of the GBDT ensemble to a binary feature vector xb . This has the advantage that the neural network in the final component can form its own feature representations from homogeneous data . In Fig . 2 , we illustrate the transformation obtained by applying the TreeDrivenEncoder algorithm on a toy example . There we have two input feature vectors x1 and x2 with categorical and numerical values are encoded into corresponding homogeneous binary feature vectors xb1 and x b 2 . To formally describe the TreeDrivenEncoder algorithm , we first need a definition of the decision trees : Definition 1 ( Decision Tree ) Let T be a structure T = ( V , E , µ ) , where V is a set of nodes , E ⊆ V × V is a set of edges and µ = ( µv ) v∈V is a sequence of mapping functions µv : Rd → V ∪ { ∅ } that map input vectors to ( child ) nodes . We call T a ( binary ) decision tree if it satisfies the following properties : 1 . ( V , E ) is a directed acyclic graph 2 . There is exactly one designated node vr ∈ V , called the root , which has no entering edges , i.e . for a node v ∈ V : v = vr ⇔ ∀w ∈ V : ( w , v ) 6∈ E. 3 . Every node v ∈ V \ { vr } has exactly one entering edge with the parent node at its other end : w ∈ V : ( w , v ) ∈ E ⇔ w = parent ( v ) . 4 . Each node has either two or zero outgoing edges . We call the nodes with two outgoing edges inner nodes and all others nodes leaves . We denote the sets of inner nodes and leaves with VI and VL , respectively . 5. µv maps feature vectors from inner nodes to their child nodes and from leaves to ∅ . v ∈ VI ⇒ ∀x ∈ Rd : ( v , µv ( x ) ) ∈ E , ( 3 ) v ∈ VL ⇒ ∀x ∈ Rd : µv ( x ) = ∅ . ( 4 ) In the following , we denote the number of inner nodes as |T | = |VI | . Furthermore , we assume that the child nodes can be identified as left or right child . For each inner node v ∈ VI , we use a modified mapping function µ̃v : Rd → { 0 , 1 } ( i.e. , a Boolean function ) where 0 encodes the left child and 1 encodes the right child . For an input vector x ∈ Rd , we exploit the structure of T to derive a binary vector of length |T | . To this end , as shown in Alg . 1 , we employ a breadth-first-search approach on the nodes of T . More specifically , for every feature that is evaluated at an inner node v of T , we retrieve the corresponding value from x and evaluate that value at v based on the associated Boolean function . Note that other node visiting strategies ( e.g. , depth-first search ) can be used as well . It is only important that the same strategy is used across all decision trees and vectors . Finally , we concatenate all the vectors generated from the single decision trees of the ensemble T on the input vector x , which gives us the final binary representation xb of x . We kindly ask the Reader to refer to the supplementary materials - we summarize the full algorithm in Alg . 1 , For mathematical completeness , the mapping obtained by applying TreeDrivenEncoder is formalized as follows . Given the feature vector x that represents an instance from the training dataset D and a trained decision tree ensemble T ( i.e. , a collection of decision trees ) on the same dataset , we exploit the structure of each tree T ∈ T to produce a binary feature vector for the original feature vector x = ( x1 , ... , xd ) and employ a transformation function : mapT : Rd → { 0 , 1 } |T | , ( 5 ) mapT : x 7→ ( µ̃v ( x ) ) v∈VI , ( 6 ) where VI again represents the inner nodes in a well-defined order and |T | their number . The mapping is performed such that at an inner node v of T , the corresponding component xj of x is mapped to 1 if the Boolean function at v evaluates to true for xj and 0 otherwise . Note that we apply the transformation function to each node in the decision tree T , even if a node does not belong to the decision path of x , hence , it holds that mapT ( x ) ∈ { 0 , 1 } |T | . For the multiple decision trees T1 , ... , Tk we construct a function TreeDrivenEncoder : Rd → { 0 , 1 } ∑k i=1 |Ti| ( 7 ) with TreeDrivenEncoder ( x ) = ( mapT1 ( x ) ... mapTk ( x ) ) ( 8 )

Summary: For the problem of learning (supervised classification and regression) on tabular data, the authors propose to used the decision functions of tree based ensemble methods as input features for a deep neural network (DNN). Tabular data is typically heterogenous, that is the data come from different modalities (continuous, categorical, missing values). As decision trees are very well tailored to such data the authors show that with their method (DeepTLF) no further preprocessing or data cleansing is necessary (apart from what's being performed in the decision tree). On numerous experiments on 7 public tabular datasets, they show that their method outperforms different baseline and competitive approaches, thereby supporting the claim that the tree based input features preserve sufficiently rich information of the input data.