Year: 2021

Abstract:
Although deep neural networks (DNNs) constitute the state-of-the-art in many tasks based on image, audio, or text data, their performance on heterogeneous, tabular data is typically inferior to that of decision tree ensembles. To bridge the gap between the difficulty of DNNs to handle tabular data and leverage the flexibility of deep learning under input heterogeneity, we propose DeepTLF, a framework for deep tabular learning. The core idea of our method is to transform the heterogeneous input data into homogeneous data to boost the performance of DNNs considerably. For the transformation step, we develop a novel knowledge distillations approach, TreeDrivenEncoder, which exploits the structure of decision trees trained on the available heterogeneous data to map the original input vectors onto homogeneous vectors that a DNN can use to improve the predictive performance. Through extensive and challenging experiments on various real-world datasets, we demonstrate that the DeepTLF pipeline leads to higher predictive performance. On average, our framework shows 19.6% performance improvement in comparison to DNNs. The DeepTLF code is publicly available.

heading: 1 INTRODUCTION
text: Tabular data is the most commonly used form of data, and it is ubiquitous in various applications, such as medical diagnosis based on patient history (Fatima et al., 2017), predictive analytics for financial applications (Dastile et al., 2020), cybersecurity (Buczak & Guven, 2015), and so forth. Although DNNs perform outstandingly well on homogeneous data, e.g., visual, audio, and textual data (Goodfellow et al., 2016), heterogeneous, tabular data still pose a challenge to these models (Shwartz-Ziv & Armon, 2021).
We hypothesize that the moderate performance of DNNs on tabular data comes from two major factors. The first is the inductive bias(es) (Katzir et al., 2020); for example, DNNs assume that certain spatial structures are present in the data (Mitchell et al., 2017), whereas tabular data do not have any spatial connections. The second reason is the high information loss during the data preprocessing step since tabular input data need to undergo cleansing (dealing with missing, noisy, and inconsistent values), uniform discretized representation (handling categorical and continuous values together), and scaling (standardized representation of features) steps. Along with these feature-processing steps, important information contained in the data may get lost, and, hence, the preprocessed feature vectors (especially when one-hot encoded) may negatively impact training and learning effectiveness (García et al., 2015). As reported by Hancock & Khoshgoftaar (2020), an efficient transformation of heterogeneous data for training DNNs is still a significant challenge.
Typically, when heterogeneous tabular data are involved, the first choice across all machine learning (ML) algorithms are ensemble models based on decision trees (Nielsen, 2016), such as random forests (RF) (Breiman, 2001) or gradient-boosted decision trees (GBDT) (Friedman, 2002). Since the inductive bias(es) of the methods based on decision trees are well suited to non-spatial heterogeneous data, the data preprocessing step is reduced to a minimum. In particular, the most common implementations of the GBDT algorithm—XGBoost (Chen & Guestrin, 2016), LightGBM (Ke et al., 2017), and CatBoost (Prokhorenkova et al., 2018) — handle the missing values internally by searching for the best approximation of missing data points.
However, the most significant computational disadvantage of the decision tree–based methods is while training the need to store (almost) the entire dataset in memory (Katzir et al., 2020). Furthermore,
in the multimodal datasets in which different data types are involved (e.g., visual and tabular data), decision tree–based models are not able to provide state-of-the-art results, whereas DNN models allow for batch-learning (no need to store the whole dataset), and for those multimodal data tasks, DNNs demonstrate state-of-the-art performance (Gu & Budhkar, 2021).
Towards the goal of significantly boosting DNNs on tabular data, we propose DeepTLF, a novel deep tabular learning framework that exploits the advantages of the GBDT algorithm as well as the flexibility of DNNs. The key element of the framework is a novel encoding algorithm, TreeDrivenEncoder, which transforms the heterogeneous tabular data into homogeneous data by distilling knowledge from nodes of trained decision trees. Thus, DeepTLF can preserve most of the information that is contained in the original data and encoded in the structure of the decision trees and benefit from preprocessing power of decision tree-based algorithms.
Through experiments on various freely available real-world datasets, we demonstrate the advantages of such a composite learning approach for different prediction tasks. We argue that by transforming heterogeneous tabular data into homogeneous vectors, we can drastically improve the performance of DNNs on tabular data.
The main contributions of this work are: (I) We propose a deep tabular learning framework - DeepTLF that combines the preprocessing strengths of GBDTs with the learning flexibility of DNNs. (II) The proposed framework builds on a generic approach for transforming heterogeneous tabular data into homogeneous vectors using the structure of decision trees from a gradient boosting model using a novel encoding function – TreeDrivenEncoder. Hence, the transformation approach can also be used independently from the presented deep learning framework. (III) In extensive experiments on various datasets and compared with state-of-the-art ML approaches, we show that the proposed framework mitigates well-known data-processing challenges and leads to unprecedented predictive performance, outperforming all the competitors. (IV) We provide an open-source implementation of DeepTLF https://github.com/xxxx/DeepTLF

heading: 2 RELATED WORK
text: In this section, we briefly review the main ideas from prior work that are relevant to our framework and data encoding step. We provide more information on the state-of-the-art DL approaches for tabular data in the Appendix B.
Independent works Moosmann et al. (2007); Geurts et al. (2006) demonstrate that data can be encoded using the RF algorithm by accessing leaf indices in the decision trees. The idea was also utilized by (He et al., 2014), where instead of the RF model, trees from a GBDT model are used for the categorical data encoding. These works demonstrates that the decision trees are a powerful and convenient way to implement non-linear and categorical feature transformations for heterogeneous data. The DeepGBM framework Ke et al. (2019) further evolved the idea of distilling knowledge from decision trees leaf index by encoding them using a neural network for online learning tasks. This approach is quite popular, but the leaf indices’ from a decision tree embedding do not fully represent the whole decision tree structure. Thus, each boosted tree is treated as a new meta categorical feature, which might be an issue for the DNNs (Hancock & Khoshgoftaar, 2020).
In contrast to related methods, our aim is to holistically distill the information from decision trees, by utilizes the whole decision tree, not only the output leafs. The DeepTLF combines the advantages of GBDT (such as handling missing values, categorical variables) with the learning flexibility of DNNs to achieve superior and robust prediction performance. Also, Medvedev & D’yakonov (2020) demonstrated that a DNN which was trained using distilled data can outperform models trained on the whole original data.
Other approaches such as NODE (Popov et al., 2019), Net-DNF (Katzir et al., 2020) try to mimic the decision trees using DNNs. A difference to our framework is that DeepTLF is more robust to the data inconsistencies and does not require new DNN architectures, hence, it is straightforward to use.

heading: 3 DEEPTLF: DEEP TABULAR LEARNING FRAMEWORK
text: In this section, we present the main components of our DeepTLF framework. As shown in Fig. 1, DeepTLF consists of three major components: (1) an ensemble of decision trees (in this work, we utilize the GBDT algorithm), (2) a TreeDrivenEncoder that performs the transformation of the original data into homogeneous, binary feature vectors by distilling the information contained in the structures of the decision trees through the TreeDrivenEncoder algorithm, and (3) a deep neural network model trained on the binary feature vectors obtained from the TreeDrivenEncoder algorithm. We will describe the details of each component in the following subsections.

heading: 3.1 GRADIENT BOOSTED DECISION TREE
text: For the data encoding step, we selected one of the most powerful algorithms on tabular data, namely the gradient boosted Decision Trees (GBDT) algorithm (Friedman, 2002). GBDT is a well-known and widely used ensemble algorithm for tabular data both in research and industrial applications (Chen & Guestrin, 2016) and is particularly successful for tasks containing heterogeneous features, small dataset sizes, and "noisy" data (Nielsen, 2016). Especially when it comes to handling variance and bias, gradient boosting ensembles show highly competitive performance in comparison with state-ofthe-art learning approaches (Friedman, 2002; Nielsen, 2016). In addition, multiple evaluations have empirically demonstrated that the decision trees of a GBDT ensemble preserve the information from the original data and can be used for further data processing (He et al., 2014; Ke et al., 2019).
The key idea of the GBDT algorithm is to construct a strong model by iterative addition of weak learners. Formally, at each iteration k of the gradient boosting algorithm, the GBDT model ϕ can be defined as:
ϕk(x) = ϕk−1(x) + λhk(x), (1)
where x is an input feature vector, ϕk−1 is the strong model constructed at the previous iteration, h is a weak learner from a family of functionsH, and λ is the learning rate.
hk = argmin h∈H ∑ i ( −∂L(ϕ k−1(xi), yi) ∂ϕk−1(xi) − h(xi) )2 . (2)
More specifically, a pseudo-residual −∂L(ϕ k−1(xi),yi)
∂ϕk−1(xi) should be approximated as well as possible
by the current weak model h(xi). The gradient w.r.t. the current predictions indicate how these predictions should be changed in order to minimize the loss function. Informally, gradient boosting can be thought of as performing gradient descent in the functional space.
The set of weak learnersH is usually formed by shallow decision trees, which are directly trained on the original data. Consequently, almost no data preparation is needed, and the information loss is minimized. We denote a GBDT model as a set of decision trees:
GB(X,y) = {T1, T2, ..., Tk},
where X is a matrix of observations, y is a vector or matrix of labels.

heading: 3.2 KNOWLEDGE DISTILLATION FROM GRADIENT BOOSTED DECISION TREES USING TREEDRIVENENCODER
text: The trained GBDT model provides structural data information, which also encodes dependencies between the input features with respect to the prediction task. In order to distill the knowledge from a tree-based model, we propose a novel data transformation algorithm – TreeDrivenEncoder. For every input vector from the original data, the proposed encoding method maps all features occurring in the decision trees of the GBDT ensemble to a binary feature vector xb. This has the advantage that the neural network in the final component can form its own feature representations from homogeneous data. In Fig. 2, we illustrate the transformation obtained by applying the TreeDrivenEncoder algorithm on a toy example. There we have two input feature vectors x1 and x2 with categorical and numerical values are encoded into corresponding homogeneous binary feature vectors xb1 and x b 2.
To formally describe the TreeDrivenEncoder algorithm, we first need a definition of the decision trees:
Definition 1 (Decision Tree) Let T be a structure T = (V,E, µ), where V is a set of nodes, E ⊆ V × V is a set of edges and µ = (µv)v∈V is a sequence of mapping functions µv : Rd → V ∪ {∅} that map input vectors to (child) nodes. We call T a (binary) decision tree if it satisfies the following properties:
1. (V,E) is a directed acyclic graph
2. There is exactly one designated node vr ∈ V , called the root, which has no entering edges, i.e. for a node v ∈ V :
v = vr ⇔ ∀w ∈ V : (w, v) 6∈ E.
3. Every node v ∈ V \{vr} has exactly one entering edge with the parent node at its other end:
w ∈ V : (w, v) ∈ E ⇔ w = parent(v) .
4. Each node has either two or zero outgoing edges. We call the nodes with two outgoing edges inner nodes and all others nodes leaves. We denote the sets of inner nodes and leaves with VI and VL, respectively.
5. µv maps feature vectors from inner nodes to their child nodes and from leaves to ∅. v ∈ VI ⇒ ∀x ∈ Rd : (v, µv(x)) ∈ E, (3) v ∈ VL ⇒ ∀x ∈ Rd : µv(x) = ∅. (4)
In the following, we denote the number of inner nodes as |T | = |VI |. Furthermore, we assume that the child nodes can be identified as left or right child. For each inner node v ∈ VI , we use a modified mapping function µ̃v : Rd → {0, 1} (i.e., a Boolean function) where 0 encodes the left child and 1 encodes the right child.
For an input vector x ∈ Rd, we exploit the structure of T to derive a binary vector of length |T |. To this end, as shown in Alg. 1, we employ a breadth-first-search approach on the nodes of T . More specifically, for every feature that is evaluated at an inner node v of T , we retrieve the corresponding value from x and evaluate that value at v based on the associated Boolean function. Note that other node visiting strategies (e.g., depth-first search) can be used as well. It is only important that the same strategy is used across all decision trees and vectors.
Finally, we concatenate all the vectors generated from the single decision trees of the ensemble T on the input vector x, which gives us the final binary representation xb of x. We kindly ask the Reader to refer to the supplementary materials - we summarize the full algorithm in Alg. 1,
For mathematical completeness, the mapping obtained by applying TreeDrivenEncoder is formalized as follows. Given the feature vector x that represents an instance from the training dataset D and a trained decision tree ensemble T (i.e., a collection of decision trees) on the same dataset, we exploit the structure of each tree T ∈ T to produce a binary feature vector for the original feature vector x = (x1, ..., xd) and employ a transformation function:
mapT : Rd → {0, 1}|T |, (5) mapT : x 7→ (µ̃v(x))v∈VI , (6)
where VI again represents the inner nodes in a well-defined order and |T | their number. The mapping is performed such that at an inner node v of T , the corresponding component xj of x is mapped to 1 if the Boolean function at v evaluates to true for xj and 0 otherwise. Note that we apply the transformation function to each node in the decision tree T , even if a node does not belong to the decision path of x, hence, it holds that mapT (x) ∈ {0, 1}|T |. For the multiple decision trees T1, ..., Tk we construct a function
TreeDrivenEncoder : Rd → {0, 1} ∑k i=1 |Ti| (7)
with TreeDrivenEncoder(x) = (mapT1(x)...mapTk(x)) (8)

heading: 3.3 DEEP LEARNING MODELS FOR ENCODED HOMOGENEOUS DATA
text: After the data distillation by the TreeDrivenEncoder algorithm, the new binary representations of the feature vectors are used to train and validate a chosen neural network model.

heading: 4 EXPERIMENTS
text: To evaluate the performance of DeepTLF against state-of-the-art models, we employ several realworld heterogeneous datasets of varying sizes from different application domains. In the following,
we first provide details about our experimental setup, including a description of the datasets. We then compare the performance between different ML models and DeepTLF. After, we present a training, inference, and data preprocessing runtime analysis. For the purpose of reproducibility we provide technical details of the experiments. Furthermore, in the Appendix A, we provide more empirical results to support our work.

heading: 4.1 EXPERIMENTAL SETTINGS

heading: 4.1.1 DATASETS
text: For the evaluation of DeepTLF, we used six heterogeneous and one multimodal dataset from different domains as described in Table 4, each dataset was previously featured in multiple published studies. The web access points and disruptions of each dataset are in the Appendix D.1. The data is pre-processed in the same way for each experiment; we do normalization and missing values subsection steps, except for GBDT and DeepTLF; since these approaches can handle missing values independently.

heading: 4.1.2 BASELINE MODELS
text: For the baseline models, we select the following algorithms: LR, linear or logistic regression models; Random Forest (RF) (Breiman, 2001); for GBDT (Friedman, 2002), we utilize the XGBoost implementation (Chen & Guestrin, 2016); DNN, A deep neural network with three fully-connected layers and two DropOut layers (Srivastava et al., 2014); Leafs+LR, A hybrid model, combining leaf
index from a trained GBDT model and generalized linear models proposed by He et al. (2014); RLNs (Shavitt & Segal, 2018), Regularization Learning Networks (RLNs) is a dedicated to tabular learning DNN, which uses the counterfactual loss to tune its regularization hyperparameters efficiently. TabNet (Arik & Pfister, 2019) is a deep tabular data learning architecture, which uses sequential attention to choose which features to reason from at each decision step; Neural Oblivious Decision Ensembles (NODE) (Popov et al., 2019) is a deep tabular data learning architecture, which generalizes ensembles of oblivious decision trees, but benefits from both end-to-end gradient-based optimization and the power of multilayer hierarchical representation learning; DeepGBM (Ke et al., 2019), a deep learning framework distilled by the GBDT algorithm; Net-DNF (Katzir et al., 2020); VIME (Yoon et al., 2020), a self-supervised learning framework for tabular data; TabTransformer (Huang et al., 2020), a framework built using self-attention transformers; Lastly, DeepTLF (the proposed algorithm), consisting of a four fully-connected layers with the two DropOut layers to lower the overfitting effect. We deliberately select a relatively simple neural network model without advanced layers such as batch normalization or attention (transformer) to demonstrate the power of our approach. By applying more sophisticated DL techniques, the model performance can be further improved.

heading: 4.2 PERFORMANCE EVALUATION
text: Main benchmark. In our performance evaluation, we partitioned each of the datasets using (stratified) 5-fold cross-validation. We use the following quality measures cross-entropy loss for classification tasks and mean-squared error (MSE) for regression tasks. Results are reported in terms of mean and standard deviation values in Table 2.
Corrupted data. We also compare the performance of DeepTLF with a plain DNN and GBDT under corrupted data to verify how the proposed model performs in scenarios of noisy labels and noisy data in the training dataset (Fig. 4).
Noisy training data and labels. We use two different setups: noisy training labels and noisy training data. We artificially corrupted the customer churn dataset (Sec. 4.1.1) by introducing random noise either to the training labels (labels were shuffled) and the training dataset. Note that for validation purposes, the test dataset was not corrupted. A distinguishing strength of the DeepTLF framework compared to other state-of-the-art approaches in the field is that it can handle missing values internally through the proposed gradient-boosting embeddings.
Missing values experiment. Figures 9 in the Appendix show the performance of DNN, GBDT, and DeepTLF models with different proportions of missing in the training dataset. As we can see, the performance of the DNNs drops drastically, while DeepTLF shows stable performance.
Sensitivity to hyperparameters. This experiment demonstrates how the GBDT hyperparameters contribute to the final performance of the DeepTLF such as the number of decision trees (Fig. 5) and learning rate (Fig. 10). For comparison purposes, we also add the GBDT baseline to the figures. It can be seen that DeepTLF does not require extensive hyperparameter tuning, since it reaches the saturation level.
t-SNE visualizations. We also compare t-SNE visualizations (Van der Maaten & Hinton, 2008) of the default of the clients dataset and a TreeDrivenEncoder encoded version of the same dataset,
the results are shown in Fig. 3. It can be seen that TreeDrivenEncoder indeed preserves valuable information from the trained decision trees.
Multimodal data. In this experiment, we demonstrate how our framework performs on multimodal data. We select the e-commerce clothing reviews dataset (Agarap, 2018), which has two data modalities textual and tabular data, and compare to DNN and DeepTLF models on unseen validation data (Fig. 6). The only difference between the DNN and DeepTLF models in this experiment is the tabular data representation, for DNN it is the original heterogeneous dataset, where the proposed framework utilizes TreeDrivenEncoder for the data transformation step. The results demonstrate the efficiency of our framework in the multimodal setting.
Training/Inference Runtime Comparison. Finally, we compare the runtime performance between several DL-based algorithms with GBDT (XGBoost (Chen & Guestrin, 2016)). Table 3 summarizes our results. To make a fair comparison, we used the latest available versions of the corresponding implementations. Also, we utilize the same DL framework, PyTorch (Paszke et al., 2017), and the same number of epochs for each DL-based baseline. One of the possible reasons for the gap between the proposed method and other DL-based approaches is that DeepTLF utilizes a simple deep neural network, whereas other approaches apply transformer networks or specialized decision tree-like layers. We also report the data preprocessing time for each baseline.
5 DISCUSSION
Empirical evaluations. We can derive the following observations from all experiments: Our framework, DeepTFL, combines the preprocessing strengths of gradient-enhanced decision trees with the learning flexibility of deep neural networks. It can handle heterogeneity in the data very well and hence shows to be highly efficient. Also, it shows a stable performance independently of data size. Moreover, the proposed encoding algorithm can be employed for multimodal learning problems Ngiam et al. (2011); Gu & Budhkar (2021), where multimodal data involve both tabular and other data sources (text, image, sound) in an integrated manner (i.e., end-to-end deep learning) while achieving a robust performance. Finally, with regard to data quality issues (noisy data and labels, misssing values), our approach clearly outperforms the DNN and GBDT models, thus showing to be applicable to many realworld applications where data loss occurs frequently.
Decision tree model choice. Noteworthy, the proposed prediction approach can use any decision tree ensemble as a basic algorithm; in this work, we adopted GBDT
because of its well-known superior performance and its robust feature handling capacities. In addition,
the GBDT algorithm sequentially constructs the trees; at each step, the next tree maximally reduces the loss given the current loss. Thus, there are conditional dependencies between the trees in the GBDT ensemble, and as a consequence, they provide adequate coverage of the data distribution.
Hyperparameter selection for DeepTLF. In our experiments, we demonstrate that the DeepTLF framework does not required extensive tinning for the of the decision tree ensemble part (Fig. 5 and Fig. 8), after reaching the saturation level, the number of trees does not have significant effect.
Tabular data encoding. Besides constructing a new homogeneous representation for the heterogeneous, tabular data, TreeDrivenEncoder encodes information about the whole dataset, as represented by the structures of the decision trees, which can be seen as a local feature selection (and feature engineering). Furthermore, in terms of efficient representation, the encoded binary data has a drastically smaller size than the original heterogeneous data, since real-valued features are typically represented as 32-bit float types. In contrast, a binary vector can be efficiently represented by a sequence of Boolean values (i.e., 1 bit per value). This allows for efficient training in the final component for the DeepTLF model.
Future work and limitations. Further analysis is needed to investigate the performance of DeepTLF in online learning scenarios. We also see further potential in improving the efficiency of DeepTLF by replacing the decision trees with an efficient neural net transformation layer, thus achieving an end-to-end deep learning mechanism for heterogeneous and multimodal data. Further improvements of our approach could be the usage of more advanced deep learning architectures such as convolution or attention-based (Transformers) neural networks.

heading: 6 CONCLUSION
text: In this work, we discussed the challenge of learning from heterogeneous tabular data with deep neural networks. The challenge stems from the concurrent existence of numerical and categorical feature types, complex, irregular dependencies between the features, and other data-related issues such as scales, outliers, and missing values. To address the challenge, we proposed DeepTLF, a framework that exploits the decision trees’ structures from an ensemble model to map the original data into a homogeneous feature space where deep neural networks can be effectively and robustly trained. This allows DeepTLF to distill and conserve relevant information in the original data and utilize it in the deep-learning process. Furthermore, the distillation step reduces the required preprocessing to a minimum and can mitigate the mentioned data-related issues by exploiting decision trees’ dataprocessing advantages (internal handling, missing values, and data scaling). Our extensive empirical evaluation on real-world datasets of different sizes and modalities convincingly showed that DeepTLF consistently outperforms the evaluated competitors, which are state-of-the-art approaches in this field. Also, the proposed framework showed robust performance on corrupted data (noisy labels, noisy data, and missing values). Compared to most approaches in this field, DeepTLF is easy to use and does not require changes to existing ML pipelines, which is essential for many practical applications. Moreover, we provide an open-source implementation of DeepTLF which can be used researchers and practitioners for various learning tasks on heterogeneous or multimodal tabular data.

heading: 7 ETHICS STATEMENT
text: The proposed deep tabular framework and data encoding algorithm closes the performance gap between deep learning and tree-based ensemble approaches, such as gradient boosting decision trees, on heterogeneous tabular data. Thus, with DeepTLF, well-studied deep learning and decision-treebased approaches, along with privacy-preserving mechanisms, can be also applied to challenging
settings related to heterogeneous tabular data. Moreover, the homogeneous decision tree encoded vectors from the DeepTLF algorithm can be used for further interpretation tasks by implementing importance analysis using SHAP (Lundberg & Lee, 2017) or LIME (Ribeiro et al., 2016) algorithms.

heading: 8 REPRODUCIBILITY DETAILS
text: For the purpose of reproducibility, we provide details of experiment settings, including hyperparameter ranges for tuning, and access points for datasets of the study are available in Appendix D. Besides, we also make publicly available our implementation of the DeepTLF framework with the scoring function which we used in this study.

heading: A ADDITIONAL EXPERIMENTS
text: In this Section, we provide more experimental results.
Validation loss curves. We examine DNNs and our DeepTLF model separately using only the validation (unseen) data. To enable a fair comparison, the deep learning part in DeepTLF is identical to the DNN we used. The results are presented in Fig. 7.
Is there a correlation between GBDT’s performance and the final performance of the DeepTLF? In this experiment, we examine the correlation between the performance of GBDT (which is used for data encoding) and DeepTLF. In other words, we want to demonstrate that if the performance of the GBDT improves, the performance of the DeepTLF rises. Fig. 11 presents the results of the experiments; as it can be observed, there is indeed a high positive correlation between the performance of the GBDT and DeepTLF. It is also noticeable that the DeepTLF performance in many cased the GBDT results.
A "sanity check" experiment. In this simple experiment, we want to verify that our heterogeneous encoding function distills knowledge better than a random heterogeneous "encoding function". We design the random function in the what it extracts random features f with random splitting value. The experiment confirms that indeed the proposed encoder performs better than a random set of Boolean rules for a given dataset (Fig. 12).

heading: B EXTENDED RELATED WORK
text: In recent years, deep learning on tabular data has received much attention from the machine learning and data science communities (Arik & Pfister, 2019; Popov et al., 2019; Yin et al., 2020; Huang et al., 2020; Guo et al., 2017; Ke et al., 2019; He et al., 2014; Shavitt & Segal, 2018; Katzir et al., 2020; Yoon et al., 2020; Padhi et al., 2020; Levy et al., 2020; Ballet et al., 2019; Akrami et al., 2020; Gupta
et al., 2021). The existing approaches can be grouped into two broad categories - architecture-based and data transformation-based models.
Architecture-based models. This group aims at developing new deep learning architectures for heterogeneous data (Arik & Pfister, 2019; Popov et al., 2019; Guo et al., 2017; Ke et al., 2019; Shavitt & Segal, 2018). For example, the authors of (Guo et al., 2017) proposed distinct neural network architecture for reducing the preprocessing and feature engineering effort by introducing a data sharing strategy between a deep and a wide network so that low- and high-level interactions between the inputs can be learned simultaneously, based on the ideas of Factorization Machines (FM) proposed in (Rendle, 2010). The work (Lian et al., 2018) extended the sharing strategy using the FM for structured data further. In (Ke et al., 2019), the authors propose an integrated solution by introducing two special neural networks, one for handling categorical features and another for numerical data. However, for mentioned approaches ((Guo et al., 2017; Lian et al., 2018; Ke et al., 2019)), it is not clear how other data-related issues, such as missing values, different scaling of numeric features, and noise, influence the predictions produced by the models.
Another line of research in this group tries to combine the advantages of decision trees and neural networks. For example, the authors of (Rota Bulo & Kontschieder, 2014) introduced Neural Decision Forests, an ensemble of neural decision trees, where split functions in each tree node are randomized Multi-Layer Perceptrons (MLPs). Another approach (Denoyer & Gallinari, 2014) presented a strategy for selecting paths in a neural directed acyclic graph to produce the prediction for a given input. Hence, the selected neural paths are specialized to specific inputs. In (Wang et al., 2017), the authors empirically showed that neural networks with random forest structure could have better generalization ability across various input domains.
A fully differentiable architecture for deep learning, which generalizes ensembles of oblivious decision trees on tabular, is also introduced in (Popov et al., 2019). Their architecture (coined NODE) employs the entmax transformation (Peters et al., 2019) and thus maps a vector of real-valued scores to a discrete probability distribution.
Other approaches focus on architectures that build on attention-based (Transformers) mechanisms (Vaswani et al., 2017). For example, the authors of (Arik & Pfister, 2019) and (Huang et al., 2020) propose an attentive transformer architecture for deep tabular learning. Their architecture also offers the possibility to interpret the input features; however, for reliable performance, a large amount of training data is needed. Another drawback is that the attention mechanism is only applied to categorical data. Hence, the continuous data does not throw the self-attention block, meaning that correlations between categorical and continuous features are dropped. The work (Yoon et al., 2020) proposes a variation of a transformer and offers semi-supervised learning.
However, no clear statements can be drawn for all methods described so far regarding the relationship between data heterogeneity and prediction quality (especially robustness under noisy data or labels).
Moreover, many of the solutions in this line of research are quite challenging from a practical perspective since it is often unclear which architectural choices should be employed in realistic scenarios. These architecture-based approaches generally rely on novel neural network architectures, which are difficult to (re-)implement and optimize for specific real-world use cases. Also, the work (?) promotes localized decisions that are taken over small subsets of the features. Especially for critical, data-intensive applications, e.g., data streaming, large-scale recommendation systems (Baylor et al., 2017), and many more, it is not clear what additional adjustments to the working pipeline are needed.
Data transformation-based models. Another way for improving the predictive quality in the presence of tabular data is to transform heterogeneous data into homogeneous feature vectors. The transformation can range from simple data preprocessing, such as the normalization of numerical variables or binary encoding of categorical variables, to linear or non-linear embedding schemes (e.g., generated by advanced autoencoders) (García et al., 2015; Hancock & Khoshgoftaar, 2020). The advantage of such data transformation approaches is that they do not require adapting the deep learning architecture. However, they may reduce the information content by smoothing critical values that might have been highly relevant for the final prediction.
The method proposed by Moosmann et al. (2007) showed how the data could be encoded using the Random Forests (RF) algorithm by accessing leaf indices in the decision trees. The same idea was presented in (He et al., 2014), where instead of the random forest algorithm, trees from the Gradient Boosted Decision Tree (GBDT) are used for the categorical data encoding. This work empirically demonstrates that the boosted decision trees are a powerful and convenient way to implement non-linear and categorical feature transformations for heterogeneous data. The DeepGBM framework (Ke et al., 2019) further evolved the idea of distilling knowledge from decision trees leaf index by encoding them using a neural network for online learning tasks.
This leaf embedding approach received much attention. However, the leaf indices’ from decision tree embeddings do not fully represent the structure of the decision tree. Hence, each boosted tree is treated as a new meta categorical feature. Bruch et al. (2020) proposed a gradient-descent-based strategy that exploits the decision tree structure to propagate gradients in the learning process. Instead, our approach has a strong focus on the exploitation of the tree structures for the data transformation process, that is, the transformation of the heterogeneous tabular data into homogeneous vectors that are especially suited for deep learning but also other machine learning techniques. The observation that local Boolean features can be quite informative for global modeling is also reported in (Pedapati et al., 2020), where the authors exploit sparse local contrastive explanations of a black-box model to obtain custom Boolean features. A globally transparent model is then trained on the Boolean features only; empirically, the global model shows a predictive performance that is only slightly worse than that of state-of-the-art approaches.
In summary, in contrast to state-of-the-art methods that exploit decision tree structures and mainly focus on leaf indices, DeepTLF utilizes the whole decision tree structure from a GBDT model, and it furthermore considers the representation of each feature independently in the information distillation process. The framework proposed in this work combines the advantages of gradient boosted trees (such as handling different scales, different attribute types, missing values, outliers, and many more) with the learning flexibility of neural networks to achieve excellent predictive performance.

heading: C TREEDRIVENENCODER ALGORITHM
text: In this section, we present the psedo code for our TreeDrivenEncoder algorithm (Alg. 1).

heading: D REPRODUCIBILITY DETAILS
text: We use the following implementation of baseline ML models: kNN, RF, LR, algorithms are from the widely used open-source machine learning python library Scikit-Learn Pedregosa et al. (2011), for GBDT we select the python version of distributed gradient boosting library XGBoost Chen & Guestrin (2016), RLN, we use the official TensorFlow implementation from the GitHub repository1, we use well-tested PyTorch implementation of TabNet2. We use the official implementation of
1https://github.com/irashavitt/regularization_learning_networks 2https://github.com/dreamquark-ai/tabnet
Algorithm 1 For a GBDT model T and an instance x from the underlying dataset, the TreeDrivenEncoder procedure visits the inner nodes of each T ∈ T (in a breadth-first search manner) and exploits their Boolean functions to construct a binary vector according to the feature values of x.
procedure TREEDRIVENENCODER(x, T )) xb vector of length 0 for tree T ∈ T do
u vector of length |T | . binary vector we aim to construct i := 0 . position index in the binary vector Q := ∅ an empty queue Q.enqueue(T.root) while Q.notEmpty do
v := Q.dequeue() x := getFeatureValue(x, v) . get from x the value of the feature that is evaluated at v if v.evaluate(x) == true then . evaluate x at v
add u(i) = 1 else
add u(i) = 0 end if i++ for all children v′ of v do
if v′ is an inner node (v′ ∈ VI ) then Q.enqueue(v′)
end if end for
end while xb = concat(xb,u)
end for return xb
end procedure
Net-DNF3. We use the official PyTorch implementation of NODE from the GitHub repository4, we adapt the official implementation of DeepGBM5 to our need using the PyTorch framework. VIME6 and TabTansformer7 implementations are from official GitHub repositories. DeepTLF, for the encoding part, we employ the XGBoost8 implementation of the GBDT algorithm; for the deep learning part, PyTorch (Paszke et al., 2017) is used. Additionally, we will provide a TensorFlow implementation. Note, other implementations of GBDT can be used as well. We select the AdaBelief Optimizer Zhuang et al. (2020) for proposed framework.
For the hyper-parameter selection task for all baseline, we apply the tree-structured parzen estimator (TPE) optimization algorithmBergstra et al. (2011) using the HyperOpt library Bergstra et al. (2013).
t-SNE Experiment For the t-SNE experiments, we normalized the original datasets. We did not pre-procces the encoded homogeneous dataset after the TreeDrivenEncoder transformation.
Computing infrastructure. Out experimental setup for all experiments has two RTX2080Ti GPUs and a single CPU AMD 3960X 24-Core.
D.1 DATASETS DESCRIPTION
Among these, the HIGGS dataset, which stems from experimental physics, is the largest dataset in our evaluation. As an exemplary dataset from the financial industry, we include the dataset defaults of clients, which contains information on default payments, demographic factors, credit data, history
3https://github.com/amramabutbul/DisjunctiveNormalFormNet 4https://github.com/Qwicen/node 5https://github.com/motefly/DeepGBM 6https://github.com/jsyoon0823/VIME 7https://github.com/lucidrains/tab-transformer-pytorch 8https://xgboost.readthedocs.io/en/latest/
of payment, and bill Statements of credit card clients in Taiwan from April 2005 to September 2005. In addition, the Zillow dataset represents typical heterogeneous data from the real estate sector. It is important to emphasize that in this dataset around 47 % of the data inputs are missing values. The avocado dataset is another representative of tabular datasets, which provides historical data on avocado prices. The telecom churn dataset presents customer data of different feature types with the goal to estimate the behavior of a customer. Lastly, we employ the California housing dataset which contains information about house pricing in 1990. E-commerce clothing reviews dataset (Agarap, 2018) has multiple data modalities - text and tabular data.
All these datasets are collected from real-world problems and contain numerical as well as categorical data. Moreover, these datasets are freely available online and common in tabular data processing: each dataset was previously featured in multiple published studies. We deliberately chose these six datasets to cover different domain areas (web, natural sciences, etc.), tasks (classification and regression), and different dataset sizes.
We prepossessed the data in the same way for every machine learning model by applying standard normalization. For the linear regression, logistic regression, and models based on neural networks, the missing values were substituted with zeros since these methods cannot handle them otherwise.

Review summaries:
This paper proposes DeepTLF, a new framework for prediction tasks using tabular data. DeepTLF first trains gradient boosted decision trees (GBDT) using the entire tabular training samples. Then it uses node values as the input the neural network predictor (e.g. classifier or regressor) for the actual prediction phase. In essence, the authors rely on GBDT's capability to handle heterogeneous tabular data with potentially many missing values to derive a binary representation of the given sample, and feed that representation to a downstream neural network. DeepTLF was able to demonstrate superior performance in real-world tabular datasets.

For the problem of learning (supervised classification and regression) on tabular data, the authors propose to used the decision functions of tree based ensemble methods as input features for a deep neural network (DNN). Tabular data is typically heterogenous, that is the data come from different modalities (continuous, categorical, missing values). As decision trees are very well tailored to such data the authors show that with their method (DeepTLF) no further preprocessing or data cleansing is necessary (apart from what's being performed in the decision tree). On numerous experiments on 7 public tabular datasets, they show that their method outperforms different baseline and competitive approaches, thereby supporting the claim that the tree based input features preserve sufficiently rich information of the input data.