Robust Deep Neural Networks for Heterogeneous Tabular Data 
Authors: Vadim Borisov
Klaus Broelemann
Enkelejda Kasneci
Gjergji Kasneci
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep Learning
Deep Neural Networks
Tabular Data
Gradient Boosted Decision Trees

Abstract: Although deep neural networks (DNNs) constitute the state-of-the-art in many tasks based on image, audio, or text data, their performance on heterogeneous, tabular data is typically inferior to that of decision tree ensembles. To bridge the gap between the difficulty of DNNs to handle tabular data and leverage the flexibility of deep learning under input heterogeneity, we propose DeepTLF, a framework for deep tabular learning. The core idea of our method is to transform the heterogeneous input data into homogeneous data to boost the performance of DNNs considerably. For the transformation step, we develop a novel knowledge distillations approach, TreeDrivenEncoder, which exploits the structure of decision trees trained on the available heterogeneous data to map the original input vectors onto homogeneous vectors that a DNN can use to improve the predictive performance. Through extensive and challenging experiments on various real-world datasets, we demonstrate that the DeepTLF pipeline leads to higher predictive performance. On average, our framework shows 19.6\% performance improvement in comparison to DNNs. The DeepTLF code is publicly available.

Under review as a conference paper at ICLR 2022
ROBUST DEEP NEURAL NETWORKS FOR HETEROGENEOUS TABULAR DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Although deep neural networks (DNNs) constitute the state-of-the-art in many tasks based on image, audio, or text data, their performance on heterogeneous, tabular data is typically inferior to that of decision tree ensembles. To bridge the gap between the difficulty of DNNs to handle tabular data and leverage the flexibility of deep learning under input heterogeneity, we propose DeepTLF. a framework for deep tabular learning. The core idea of our method is to transform the heterogeneous input data into homogeneous data to boost the performance of DNNs considerably. For the transformation step, we develop a novel knowledge distillations approach, Tree.DriveEEncodrr, which exploits the structure of decision trees trained on the available heterogeneous data to map the original input vectors onto homogeneous vectors that DNN can use to improve the predictive perfor- mance. Through extensive and challenging experiments on various real-world datasets, we demonstrate that the DeepTLF pipeline leads to higher predictive performance. On average, our framework shows 19.6% performance improvement in comparison to DNNs. The DeepTLF code is publicly available.
INTRODUCTION
Tabular data the most commonly used form of data, and is ubiquitous in various applications, such as medical diagnosis based on patient history (Fatima et al., 2017), predictive analytics for financial applications (Dastile al., 2020), cybersecurity (Buczak & Guven, 2015), and forth. Although DNNs perform outstandingly well on homogeneous data, e.g., visual, audio, and textual data (Good- fellow et al., 2016), heterogeneous, tabular data still pose a challenge to these models (Shwartz-Ziv Armon, 2021).
We hypothesize that the moderate performance of DNNs on tabular data comes from two major factors. The first is the inductive bias(es) (Katzir al., 2020); for example, DNNs assume that certain spatial structures are present in the data (Mitchell et al., 2017), whereas tabular data do not have any spatial connections. The second reason is the high information loss during the data preprocessing step since tabular input data need to undergo cleansing (dealing with missing, noisy, and inconsistent values), uniform discretized representation (handling categorical and continuous values together), and scaling (standardized representation of features) steps. Along with these feature-processing processing steps, important information contained in the data may get lost, and, hence, the preprocessed feature vectors (especially when one-hot encoded) may negatively impact training and learning effectiveness (Gareía et al., 2015). As reported by Hancock Khoshgoftaar (2020), an efficient transformation of heterogeneous data for training DNNs is still significant challenge.
Typically, when heterogeneous tabular data are involved, the first choice across all machine learning (MI.) algorithms are ensemble models based on decision trees (Nielsen, 2016), such as random forests (RF) (Breiman, 2001) or gradient-boosted decision trees (GBDT) (Friedman, 2002). Since the inductive bias(es) of the methods based decision trees are well suited non-spatial heterogencous data, the data preprocessing step is reduced to minimum. In particular, the most common implemen- tations of the GBDT algorithm- Sarrrtthmm-GBBosttttt XGBoost (Chen & Guestrin, 2016), LightGBM (Ke al., 2017), and CatBoost (Prokhorenkova et al., 2018) - handle the missing values internally by searching for the best approximation of missing data points.
However, the most significant computational disadvantage of the decision tree rree-based methods is while training the need store (almost) the entire dataset in memory (Katzir et al., 2020). Furthermore,

---

Under review as a conference paper at ICLR 2022
in the multimodal datasets in which different data types involved (e.g., visual and tabular data), decision tree-based models are not able to provide state-of-the-ee art results, whereas DNN models allow for batch-learning (no need to store the whole dataset), and for those multimodal data tasks, DNNs demonstrate state-of-the-art performance (Gu & Budhkar, 2021).
Towards the goal of significantly boosting DNNs on tabular data, propose Deep'TLF, novel deep tabular learning framework that exploits the advantages of the GBDT algorithm as well as the flexibil- ity of DNNs. The key element of the framework is a novel encoding algorithm, Enenoer which transforms the heterogeneous tabular data into homogeneous data by distilling knowledge from nodes of trained decision trees. Thus, DeepTLF can preserve most of the information that contained in the original data and encoded in the structure of the decision trees and benefit from preprocessing power of decision tree-based algorithms.
Through experiments on various freely available real-world datasets, we demonstrate the advantages of such composite learning approach for different prediction tasks. We argue that by transforming heterogeneous tabular data into homogeneous vectors, we can drastically improve the performance of DNNs on tabular data.
The main contributions of this work are: (I) We propose deep tabular learning framework - Dee that combines the preprocessing strengths GBDT's with the learning flexibility of DNNs. (II) The proposed framework builds on generic approach for transforming heterogencous tabular data into homogeneous vectors using the structure of decision trees from a gradient boosting model using a novel encoding function TreeDDivenEncoderr Hence, the transformation approach can also be used independently from the presented deep learning framework. (III) In extensive experiments on various datasets and compared with state-of-the-an ML approaches, we show that the proposed framework mitigates well-known data-processing challenges and leads to unprecedented predictive performance, outperforming all the competitors. (IV) We provide an open-source implementation of DeepTLF https://gihbb..om//xx///
RELATED WORK
In this section, we briefly review the main ideas from prior work that are relevant to framework and data encoding step. We provide more information on the state-of-the-ar DL approaches for tabular data in the Appendix B.
Independent works Moosmann et al. (2007); Geurts et al. (2006) demonstrate that data can be encoded using the RF algorithm by accessing leaf indices in the decision trees. The idea was also utilized by (He et al., 2014), where instead of the RF model, trees from a GBDT model are used for the categorical data encoding. These works demonstrates that the decision trees a powerful and convenient way to implement non-linear and categorical feature transformations for heterogeneous data. The DeepGBM framework Ke et al. (2019) further evolved the idea of distilling knowledge from decision trees leaf index by encoding them using a neural network for online learning tasks. This approach is quite popular, but the leaf indices' from a decision tree embedding do not fully represent the whole decision tree structure. Thus, each boosted tree is treated as a new meta categorical feature, which might be an issue for the DNNs (Hancock & Khoshgoftaar, 2020).
In contrast to related methods, our aim is to holistically distill the information from decision trees, by utilizes the whole decision tree, not only the output leafs. The Deep'TI. combines the advantages of GBDT (such handling missing values, categorical variables) with the lcarning flexibility of DNNs to achieve superior and robust prediction performance. Also, Medvedev & D' D'yakonov (2020) demonstrated that DNN which was trained using distilled data outperform models trained on the whole original data.
Other approaches such NODE (Popov et al., 2019), Net- (Katzir al., 2020) try to mimic the decision trees using DNNs. A difference to our framework is that DeepTLF is more robust to the data inconsistencies and does not require new DNN architectures, hence, it is straightforward to use.
2

---

Under review as a conference paper at ICLR 2022
Hetarogeneous Tabular Data
Data Encoding
Homogeneous Data
Neural Network
TmeeDriveEncooder
Gradient Boosting Deciaion Treee
DeepTLF
Figure I: A data pipeline for the Deep framework. First, the original tabular data is used to train gradient boosted decision trees (GBDT) model. The heterogeneous data (i.e., training well the test data) are transformed by exploiting the structures of the decision trees in the ensemble. More specifically, the TreeDrivenEnncder algorithm distills information from trained decision trees of the GBDT model to produce homogeneous binary vectors. These vectors are then used to train DNN. Note that TLe does require data preprocessing (such as normalization handling missing or categorical values); therefore, in total, dramatically speeds up the data preprocessing time.
DEEPTLF: DEEP TABULAR LEARNING FRAMEWORK
In this section, we present the main components of our DeepTLF framework. As shown in Fig. DeepTLLF consists of three major components: (1) ensemble of decision trees (in this work, we utilize the GBDT algorithm),, (2) TreeDrivenEncoder that performs the transformation of the original data into homogeneous, binary feature vectors by distilling the information contained in the structures of the decision trees through the TreeDrivenEneoder algorithm, and (3) a deep neural network model trained the binary feature vectors obtained from the Enedoer algorithm. We will describe the details of each component in the following subsections.
3.1 GRADIENT BOOSTED DECISION TREE
For the data encoding step, we selected one of the most powerful algorithms on tabular data, namely the gradient boosted Decision Trees (GBDT) algorithm (Friedman, 2002). GBDT is well-known and widely used ensemble algorithm for tabular data both research and industrial applications (Chen & Guestrin, 2016) and is particularly successful for tasks containing heterogeneous features, small dataset sizes, and "noisy" data (Nielsen, 2016). Especially when comes to handling variance and bias, gradient boosting ensembles show highly competitive performance in comparison with state-of- the-art learning approaches (Friedman, 2002; Nielsen, 2016). In addition, multiple evaluations have empirically demonstrated that the decision trees of GBD)T ensemble preserve the information from the original data and can be used for further data processing (He et al., 2014; Ke et al., 2019).
The key idea of the GBDT algorithm construct strong model by iterative addition of weak learners. Formally, at each iteration of the gradient boosting algorithm, the GBDT model can be defined
* = (xx) + (x)(x),
where x an input feature vector, - is the strong model constructed the previous iteration, is a weak learner from a family of functions H. and À is the learning rate.
a (xx). vi) (xp)-
h
arg min
Σ
h(xx;)
More specifically, a pseudo-residual aL(Y - (x,,..)) should be approximated as well as possible (x.)
by the current weak model h(xx). The gradient w.r.t. the current predictions indicate how these predictions should be changed order to minimize the loss function. Informally, gradient boosting can be thought of as performing gradient descent in the functional space.
3

---

Under review as a conference paper at ICLR 2022
1
X2
f1
fi
X
X2
f12 11
0
f1
1
14
f2 20
f2
19
23
fe
20
f is red
fa 2 2
0
f3 is
f3
red
f12
R2
f1 1
f4
0.3
2.3
fa 2 2
0
Heterogeneous Tabular
Homogeneous Tabular
Trained Decision Trees
Feature Vectors
Feature Vectors
Figure 2: A toy example of the proposed data transformation performed by the TreeDrivenEncoder algorithm. On the left, we see two heterogeneous input feature vectors Xı and X2 from the original dataset D, where X, RR., f1+ f2. are numerical features, and fa is categorical feature in tabular dataset. Note there is also missing value for the feature fa in X2. To encode the input data, we use two trained decision trees on the dataset 2, with inner nodes in total. By evaluating the Boolean function in each inner node for given input vector, we construct two homogeneous feature vectors x and x2, where a component of these vectors is set to if the corresponding Boolean function evaluates to true and otherwise.
The set of weak learners H is usually formed by shallow decision trees, which are directly trained on the original data. Consequently, almost no data preparation needed, and the information loss is minimized. We denote GBDT model set of decision trees:
GB(X, = {Ti, T2, Tk)
where X matrix of observations, y is vector matrix of labels.
3.2 KNOWLEDGE DISTILLATION FROM GRADIENT BOOSTED DECISION TREES USING TREEDRIVENENCUDER
The trained GBDT model provides structural data information, which also encodes dependencies between the input features with respect to the prediction task. In order to distill the knowledge from tree-based model, we propose a novel data transformation algorithm - Drieenrivenn....rr For every input vector from the original data, the proposed encoding method maps all features occurring in the decision trees of the GBDT ensemble to a binary feature vector x". This has the advantage that the neural network in the final component can form its feature representations from homogeneous data. In Fig. 2, we illustrate the transformation obtained by applying the TreeDDrivenEncode algorithm toy example. There have two input feature vectors X₁ and x2 with categorical and numerical values are encoded into corresponding homogeneous binary feature vectors x and x
To formally describe the TreeodivenEnnoddr algorithm, we first need definition of the decision trees:
Definition (Decision Tree) Let T be structure T TV, where V is a set of nodes, E C V x V a set of edges and μ (yu)vey is sequence of mapping functions pa : Rd that map input vectors to (child) nodes. We call T a (binary) decision tree if it satisfies the following properties:
(6)
1. (V, is a directed acyclic graph
2. There is exactly one designated node U, V, called the root, which has no entering edges, i.e. for a node v E V:
== = Vw E V (w, u) $ E.
3. Every node v E (Vr has exactly one entering edge with the parent node at its other end:
w E (w, E ==prentt()))
E
4. Each node has either inner nodes and all others nodes leaves. We denote the sets of inner nodes and leaves with V, and Vi. respectively.
outgoing edges. We call the nodes with two outgoing edges
4

---

Under review as a conference paper at ICLR 2022
5. Me maps feature vectors from inner nodes their child nodes and from leaves 0.
V1 Vx R (v. Mu (x(x)) E,
(3)
VE VL V pe(x) 0.
(4)
In the following, denote the number of inner nodes [T] |Vil Furthermore, assume that the child nodes can be identified as left or right child. For each inner node u E V1, we use a modified mapping function Pe IR + {0,1} (i.e., Boolean function) where O encodes the left child and 1
encodes the right child.
For imput vector Rd, exploit the structure of T to derive a binary vector of length [T]. To this end, as shown in Alg. 1, we employ a breadth-first-eenn
approach the nodes of T. More specifically, for every feature that is evaluated at an inner node T, we retrieve the corresponding value from x and evaluate that value at based on the associated Boolean function. Note that other node visiting strategies (e.g., depth-first search) can be used as well. It is only important that the same strategy is used across all decision trees and vectors.
Tapoy
Finally, we concatenate all the vectors generated from the single decision trees of the ensemble T on the input vector x, which gives us the final binary representation x" of x. We kindly ask the Reader to refer to the supplementary materials we summarize the full algorithm in Alg. 1,
5NE 1
Debact
Rapey
For mathematical completeness, the mapping obtained by applying TreeDrivenEncoder is formalized follows. Given the feature vector x that represents an instance from the training dataset ID and trained decision tree ensemble T (i.e., a collection of decision trees) on the same dataset, we exploit the structure of each tree T E T to produce binary feature vector for the original feature vector x = (f1....,;a Fa) and employ transformation function:
SHE
mapr Id
{0, 10.1)
(5)
5NE
mapr x + (x))x))
(6)
Figure 3: t-SNE visualizations of original heterogeneous tab- ular default of clients dataset (top). and the same dataset after the TreendivenEneoder transformation (bottom).
where V, again represents the inner nodes in well-defined order and [T]] their number. The mapping is performed such that at an inner node of T, the corresponding component x, of x mapped to 1 if the Boolean function at e evaluates to true for æj and 0 otherwise. Note that we apply the transformation function each node in the decision tree T, even if node does not belong to the decision path of x, hence, it holds that mapr (x) E {0, 131"1
For the multiple decision trees Ti, ..., Ta construct function
TreeDrivenEhcoder Rd + {0, 10,1
(7)
with
TreeDrivenEncoderr (mapr, (x)... (x))
(8)
3.3
DEEP LEARNING MODELS FOR ENCODED HOMOGENEOU'S DATA
After the data distillation by the TreeDrivenEnnoder algorithm, the new binary representations the feature vectors are used to train and validate chosen neural network model.
4
EXPERIMENTS
To evaluate the performance of DeepTLF against state-of-the-ee models, we employ several real- world heterogeneous datasets of varying sizes from different application domains. In the following,
5

---

Under review as a conference paper at ICLR 2022
Table Details of the datasets used in the experimental evaluations. #Sample is the number of data points, #Num is the number of numerical variables, and #Cat is the number of categorical variables dataset.
Dataset
#Samples
#Num
#Cat
Task
DI
HIGGS
11,000,000
28
0
Binary classification
D2
Default of clients
30,000
14
9
Binary classification
D3
Telecom churn
51,047
38
18
Binary classification
D4
Zillow
167 (67,888
31
27
Regression
D5
Avocado prices
18,249
8
3
Regression
D6
California housing
20,640
8
0
Regression
D)7
E-commerce clothing reviews
23,486
6
4
Binary classification
Table 2: Experimental results based on (stratified) 5-fold cross-validation. We use same fold splitting strategy for every dataset. The cross-entropy measure (lower is better) is selected for classification tasks and MSE measure (lower is better) is selected for regression problems respectively. The top results for each dataset marked in bold.
Classification Datasets
Regression Datasets
D1
D2
D3
D4
DS
D6
LR
0.637=0.001
0.470:10.001
0.58410.001
0.028=0.001
0.96610.001
0.552-0.063
RF
0.502:10.001
0.444:1.007
0.56410.003
0.028=0.001
0.02510.001
0.254=0.008
GBDT
0.498110000
0.429:10.006
0.55910.000
0.027=0.003
0.02611.003
0.217=0.021
Leafs+LR
0.65910.001
0.453:10.002
0.58010.002
0.029-0.001
0.10510.0036
0.358_0.032
Deep Neural Network Models
DNN
0.511110.001
0,437:10.005
0.57911.002
0.02810.000
0.06911.002
0.339-0.122
RLN
0.507110.02
0.433:10.051
0,59911.0001
0.39910.042
0.275:1.224
0,947-0.228
DeepGBM
0.487+0.001
0.457+00.023
0.589+0.001
0.001
0.038+0.045
0.299
TabNet
0.503++0.001
0.447++0.001
0.591+0.005
0.049+0.001
0.073+0.002
0.455 106
VIME
0.514:10.001
0.453:1-00006
0.593+100002
0.030=0.003
0.120
0.684-0.023
TabTrrnsformer
0.5002
0.515=+0.003
0.650=0.021
0.029=0.001
0.073+0.002
0.994-0.501
TabNNet
0.503=+0.001
0.447++0.001
0.591+0.005
0.049=0.001
0.073+0.002
0.455-0.106
Net-DNF
0.561110.001
0,512-10.001
0.59410.000
0.027=0.001
0.321+0.093
2.491-0.051
NODE
0.489=10.006
0.458:10.006
0.598110.001
0.028=0.001
0. 0.104
0.722-0.052
DeepTLF (ours)
0.001
0.427:10.006
0.007
0.026:1.0011
0.021
0.215
we first provide details about our experimental setup, including a description the datasets. We then compare the performance between different ML models and DeepTLF. After, present training, inference, and data preprocessing runtime analysis. For the purpose of reproducibility we provide technical details of the experiments. Furthermore, in the Appendix A, we provide more empirical results to support our work.
4.1
EXPERIMENTAL SETTINGS
4.1.1 DATASETS
For the evaluation of DeepTLF, we used six heterogeneous and one multimodal dataset from dif- ferent domains as described in Table 4, each dataset was previously featured in multiple published studies. The web access points and disruptions of each dataset are the Appendix D. The data is pre-processed in the same way for each experiment; we do normalization and missing values subsection steps, except for GBDT and DeepTLF; since these approaches can handle missing values independently.
4.1.2 BASELINE MODELS
For the baseline models, we select the following algorithms: LR, linear or logistic regression models; Random Forest (RF) (Breiman, 2001); for GBDT (Friedman, 2002), utilize the XGBoost implementation (Chen Guestrin, 2016); DNN, A deep neural network with three fully-connected layers and two DropOut layers (Srivastava al., 2014); Leafs+LR, A hybrid model, combining leaf

---

Under review as a conference paper at ICLR 2022
8.63
DeepTLF
Test ROC AUC score
0.55
GBDT
DNN
ROC AUC
0.60
0.500
0.55
0:33
Deep TLF
GBDT
1.50
DNN
20
30
55
65
10 20
15
80
90
% noisy labels train
%
noisy data samples
data
Figure 4: Left: Noisy labels experiment. Right: Noisy data experiment. The DNN here is identical to the DL part in the Deep'LL.. Note the test data not corrupted. We report the ROC AUC value (higher is better). Results are averages over five trials for the telecom churn (D3) dataset.
index from trained GBDT model and generalized linear models proposed by He et al. (2014); RLNs (Shavitt & Segal, 2018), Regularization Learning Networks (RLNs) is dedicated to tabular learning DNN, which uses the counterfactual loss to tune its regularization hyperparameters efficiently. TabNet (Arik & Pfister, 2019) is a deep tabular data learning architecture, which uses sequential attention to choose which features reason from each decision step: Neural Oblivious Decision Ensembles (NODE) (Popov et al., 2019) is a deep tabular data learning architecture, which generalizes ensembles of oblivious decision trees, but benefits from both end-to-end gradient-based optimization and the power of multilayer hierarchical representation learning: DeepGBM (Ke et al., 2019), a deep learning framework distilled by the GBDT algorithm: Net-DNF (Katzir et al., 2020); VIME (Yoon et al., 2020), self- supervisee learning framework for tabular data; TabTransformer (Huang et al., 2020), a framework built using self-atte celf-attention transformers: Lastly, DeepTLF (the proposed algorithm), consisting of a four fully-connected layers with the two DropOut layers to lower the overfitting effect. We deliberately select a relatively simple neural network model without advanced layers such as batch normalization attention (transformer) to demonstrate the power of our approach. By applying more sophisticated DL techniques, the model performance can be further improved.
4.2 PERFORMANCE EVALCATION
Main benchmark. our performance evaluation, we partitioned each of the datasets using (strati- fied) 5-fold cross-valiaation. We use the following quality measures cross-entropy loss for classifica- tion tasks and mean-squared error (MSE) for regression tasks. Results are reported in terms of mean and standard deviation values in Table 2.
Corrupted data. We also compare the performance of DeepTLF with plain DNN and GBDT under corrupted data verify how the proposed model performs scenarios of noisy labels and noisy data in the training dataset (Fig. 4).
Noisy training data and labels. We two different setups: noisy training labels and noisy training data. We artificially corrupted the customer churn dataset (Sec. 1) by introducing random noise either to the training labels (labels were shuffled) and the training dataset. Note that for validation purposes, the test dataset was not corrupted. A distinguishing strength of the Deep'TLF framework compared to other state-of-thh trr approaches in the field that can handle missing values internally through the proposed gradient-boosting embeddings.
Missing values experiment. Figures 9 in the Appendix show the performance of DNN, GBDT, and DeepTLF models with different proportions of missing in the training dataset. As we can see, the performance of the DNNs drops drastically, while DeepTLF shows stable performance.
Sensitivity to hyperparameters. This experiment demonstrates how the GBDT hyperparameters contribute to the final performance of the DeepTLF such as the number of decision trees (Fig. 5) and learning rate (Fig. 10). For comparison purposes, also add the GBDT baseline to the figures. It can be seen that DeepTLF does not require extensive hyperparameter tuning, since it reaches the saturation level.
t-SNE visualizations. We also compare t-SNE visualizations (Van der Maaten & Hinton, 2008) of the default of the clients dataset and TreeDrivenEnnoder encoded version of the same dataset,
7

---

Under review as a conference paper at ICLR 2022
Table 3: A comparison of the training and inference runtime for selected models from different categories on the whole Zillow dataset (167.888 samples). The results related to the training, inference and preprocessing time are averages over five runs over the whole dataset for training and inference tests. The data preprocessing step includes: data scalling, handling missing values.
Model
Training time (8)
Inference time (s)
Data preprocessing
#Learning
time (s)
parameters
GBDT (CPU)
13.5
0.5
0
200 trees, depth 4
GBDT (GPU)
3.1
0.3
0
200 trees, depth 4
DeePGBM (GPU)
23.9
5.2
0.3
222,548 weights
TabNet (GPU)
79.1
2,2
0.3
584,832 weights
NODE (GPU)
310.2
15.5
0.3
27,105,922 weights
Dee (GPL)
15.1
3.2
0
80,351 weights
the results are shown in Fig. 3. It can be seen that TreeDDivvenFnoodrr indeed preserves valuable information from the trained decision trees.
Multimodal data. In this experiment, we demonstrate how framework performs on multimodal data. We select the e-commerce clothing reviews dataset (Agarap, 2018), which has two data modalities textual and tabular data, and compare DNN and Deep models on unseen validation data (Fig. 6). The only difference between the DNN and DeepTLF models in this experiment is the tabular data representation, for DNN it is the original heterogeneous dataset, where the proposed framework utilizes TreeDrivenEncode for the data transformation step. The results demonstrate the efficiency of our framework the multimodal setting.
Training/Inference Runtime Comparison. Finally, we compare the runtime performance between several base algorithms with GBDT (XGBoost (Chen & Guestrin, 2016)). Table summarizes our results. To make a fair comparison, we used the latest available versions of the corresponding implementations. Also, we utilize the same DL framework, PyTorch (Paszke et al., 2017), and the same number of epochs for each DL-based baseline. One of the possible reasons for the gap between the proposed method and other DL-based approaches is that DLe utilizes a simple deep neural network, whereas other approaches apply transformer networks specialized decision tree-like layers. We also report the data preprocessing time for each baseline.
5 DISCUSSION
Empirical evaluations. We can derive the following observations from all experiments: Our framework, DeepTFL, combines the preprocessing strengths of gradient-enhanced decision trees with the learning flex- ibility of deep neural networks. It can handle hetero- geneity in the data very well and hence shows to be highly efficient. Also, it shows stable performance independently of data size. Moreover, the proposed encoding algorithm can be employed for mudtimodal learning problems Ngiam et al. (2011), I); Gu & Budhkar (2021), where multimodal data involve both tabular and other data sources (text, image, sound) in an integrated manner (i.e., end-to-end deep learning) while achiev- ing robust performance. Finally, with regard to data quality issues (noisy data and labels, misssing values), approach clearly outperforms the DNN and GBDT models, thus showing to be applicable to many real- world applications where data loss occurs frequently.
0.61
DeepTLF
GBDT
Test entropy
0.59
0.58
0.57
0.56
30
100
of decision trees
Figure 5: A relationship between a num- ber of trees in the GBDT and the final per- formance of the proposed DeepTLLF frame- work. The precise same GBDT model is used for the data encoding in the DeepTTF. Results are averaged over five trials for the D3 dataset.
Decision tree model choice. Noteworthy, the proposed any decision tree ensemble as a basic algorithm; in this work, we adopted GBDT because of its well-known superior performance and its robust feature handling capacities. In addition,
prediction approach
8

---

Under review as a conference paper at ICLR 2022
the GBDT algorithm sequentially constructs the trees; at each step. the next tree maximally reduces the loss given the current loss. Thus, there are conditional dependencies between the trees in the GBDT ensemble, and consequence, they provide adequate coverage of the data distribution.
Hyperparameter selection for Deep'TLE. In our ex- periments, demonstrate that the Deep'TLF frame- work does not required extensive tinning for the of the decision tree ensemble part (Fig. 5 and Fig. 8), after reaching the saturation level, the number of trees does have significant effect.
0.75
DeF
DNN
0.70
Cross-entropy
0.65
0.60
Tabular data encoding, Besides constructing new homogeneous representation for the heterogeneous, tabular data, DreennevveEneoder encodes information about the whole dataset, as represented by the structures of the decision trees, which can be seen as a local fea- ture selection (and feature engineering). Furthermore, in terms of efficient representation, the encoded binary data has a drastically smaller size than the original het- erogeneous data, since real-valued features typically represented as 32-bit float types. In contrast, binary vector be efficiently represented by sequence of Boolean values (i.e., bit per value). This allows for ef- ficient training in the final component for the DeepTLF model.
0.55
0.50
0:45
0
10
10
20
25
Epoch
Figure 6: A multimodal data experiment on the textual and tabular from the D7 dataset, the DNN and DeepTLF have identical DL architectures and training setups. Results are averaged over five trials.
Future work and limitations. Further analysis is needed investigate the performance of Deep in online learning scenarios. We also see further potential in improving the efficiency of Deep'TLe TLF by replacing the decision trees with an efficient neural net transformation layer, thus achieving an end-to-end deep learning mechanism for heterogeneous and multimodal data. Further improvements of our approach could be the usage of more advanced deep learning architectures such as convolution or attention-based (Transformers) neural networks.
CONCLUSION
In this work, we discussed the challenge of learning from heterogeneous tabular data with deep neural networks. The challenge stems from the concurrent existence of numerical and categorical feature types, complex. irregular dependencies between the features, and other data-related issues such as scales, outliers, and missing values. To address the challenge, we proposed DeepTLF, a framework that exploits the decision trees' structures from ensemble model to map the original data into a homogeneous feature space where deep neural networks can be effectively and robustly trained. This allows DeepTTLF to distill and conserve relevant information in the original data and utilize it in the deep-learning process. Furthermore, the distillation step reduces the required preprocessing minimum and can mitigate the mentioned data-related issues by exploiting decision trees' data- processing advantages (internal handling. missing values, and data scaling). Our extensive empirical evaluation on real-world datasets of different sizes and modalities convincingly showed that Deep consistently outperforms the evaluated competitors, which are state-of ofttte-rttttttt approaches in this field. Also, the proposed framework showed robust performance on corrupted data (noisy labels, noisy data, and missing values). Compared to most approaches in this field, Deep'TIF is easy to use and does not require changes to existing ML pipelines, which is essential for many practical applications. Moreover, provide open-source implementation of DeepTLF which can be used researchers and practitioners for various learning tasks on heterogeneous or multimodal tabular data.
7 ETHICS STATEMENT
The proposed deep tabular framework and data encoding algorithm closes the performance gap between deep learning and tree- rrsed ensemble approaches, such gradient boosting decision trees, on heterogeneous tabular data. Thus, with DeepTLF, well-studied deep learning and decision tree- based approaches, along with privacy- mechanisms, can be also applied to challenging

---

Under review as a conference paper at ICLR 2022
settings related to heterogeneous tabular data. Moreover, the homogeneous decision tree encoded vectors from the DeepTLF algorithm can be used for further interpretation tasks by implementing importance analysis using SHAP (Lundberg & Lee, 2017) LIME (Ribeiro et al., 2016) algorithms.
REPRODUCIBILITY DETAILS
For the purpose of reproducibility, provide details of experiment settings, including hyper- parameter ranges for tuning. and access points for datasets the study are available in Appendix D. Besides, also make publicly available our implementation of the Deep'TLF framework with the scoring function which we used in this study.
REFERENCES
Abien Fred Agarap. Statistical analysis on e-commerce reviews, with sentiment classification using bidirectional recurrent neural network (mnn) arXiv preprint wrXiv:1805.03687, 2018.
Haleh Akrami, Sergul Aydore, Richard M Leahy, and Anand A Joshi. Robust variational autoencoder for tabular data with beta divergence. arXiv preprint arXiv:2006.000 08204, 2020.
Sercan O Arik and Tomas Pfister. TabNet: Attentive interpretable tabular learning. arXiv preprint /908:009000444 2019.
Vincent Ballet, Xavier Renard, Jonathan Aigrain, Thibault Laugel. Pascal Frossard, and Marcin Detyniecki. Imperceptible adversarial attacks on tabular data. arXiv preprint 2019.
Denis Baylor, Eric Breck, Heng-Tze Cheng, Noah Fiedel, Chuan Yu Foo, Zakaria Haque, Salem Haykal, Mustafa Ispir, Vihan Jain, Levent Koc, et al. Tfx: tensorflow-based production-seale machine learning platform. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1387-1395, 2017.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for optimization. In 25th annual conference neural information processing systems (NIPS 201 I), volume 24. Neural Information Processing Systems Foundation, 2011.
James Bergstra, Dan Yamins, David D Cox, et al. Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms. In Proceedings of the 12th Python science conference, volume 13, pp. 20. Citeseer, 2013.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Sebastian Bruch, Jan Pfeifer, and Mathieu Guillame-hert. Learning representations for axis-aligned decision forests through input perturbation. arXiv preprint vxXiv:200..1476/ 2020.
Anna L Buczak and Erhan Guven. A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Communications surveys & tutorials, 1000:1153-176 2015.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Xolani Dastile, Turgay Celik, and Moshe Potsane. Statistical and machine learning models in credit scoring: A systematic literature survey. Applied Soft Computing, 91: 91:106263, 2020.
Ludovic Denoyer and Patrick Gallinari. Deep sequential neural network. arXiv preprint arXiv:1410.05 2014.
Meherwar Fatima, Maruf Pasha, et al. Survey of machine learning algorithms for disease diagnostic. Journal of Intelligent Learning Systems and Applications, 9(01):1, 2017.
Jerome H Friedman. Stochastic gradient boosting. Computational statistics & data analysis, 38(4): 367-378, 2002.
10

---

Under review as a conference paper at ICLR 2022
Salvador García, Julián Luengo, and Francisco Herrera. Data preprocessing in data mining, vol- ume 72. Springer, 2015.
Pierre Geurts, Damien Ernst, and Louis Wehenkel. Extremely randomized trees. Machine learning, 63(1):3-42, 2006.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Ken Gu and Akshay Budhkar. A package for learning on tabular and text data with transformers. In Proceedings of the Third Workshop on Multimodal Artificial Intelligence, pp. 69-73, 2021.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. DeepFM: a factorization- machine based neural network for ctr prediction. arXiv preprint arXiv: 04247, 2017.
Kavya Gupta, Beatrice Pesquet-Popeseu. Fatch Kaakai, and Jean- Christophe Pesquet. A quantitative analysis of the robustness of neural networks for tabular data. ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 8057-8061. IEEE, 2021.
John T Hancock and Taghi Khoshgoftaar. Survey on categorical data for neural networks. Journal of Big Data, 7:1-41, 2020.
Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et al. Practical lessons from predicting clicks on ads at facebook. In Proceedings of the Eighth International Workshop Data Mining for Online Advertising, pp. 1-9, 2014.
Xin Huang. Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.0668 2020.
Liran Katzir, Gal Elidan, and Ran El-Yaniv. Net-DNF: Effective deep modeling of tabular data. In International Conference on Learning Representations, 2020.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in neural information processing systems, pp. 3146-3154, 2017.
Guolin Ke, Zhenhui Xu, Jia Zhang, Jiang Bian, and Tie- Yan Liu. DeepGBM: A deep learning framework distilled by ghdt for online prediction tasks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 384-394 2019.
Eden Levy, Yael Mathov, Ziv Katzir, Asaf Shabtai, and Yuval Elovici. Not all datasets are born equal: On heterogeneous data and adversarial examples. arXiv preprint arXiv:2010.03180 2020.
Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. XDeepFM: Combining explicit and implicit feature interactions for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1754-1763 2018.
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predic- tions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 4765-4774. Curran Associates, Inc., 2017. URL http: http:://pppprsss/ nips.cc.//.cc.......o..oo cappr/ 7062-a-uni fieiffied--eee -owp----pppaacc------t--eeeeeeee
modelllooooooooooooooo -rred ctions. .ddf.
Dmitry Medvedev and Alexander D' D'yakonov. New properties of the data distillation method when working with tabular data. arXiv preprint PrXii:20000098:33 2020.
Benjamin R Mitchell et al. The Spatial Inductive Bias of Deep Learning PhD thesis, Johns Hopkins University, 2017.
Frank Moosmann, Bill Triggs, and Frederic Jurie. Fast discriminative visual codebooks using randomized clustering forests. In Advances in neural information processing systems, pp. 985-992, 2007.
11

---

Under review as a conference paper at ICLR 2022
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multi- modal deep learning. In JCML, 2011.
Didrik Nielsen. Tree boosting with xgboost-why does xgboost win" every" machine learning competition? Master's thesis, NTNU, 2016.
Inkit Padhi, Yair Schiff, Igor Meinyk, Mattia Rigotti, Youssef Mrouch, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman. Tabular transformers for modeling multivariate time series. arXiv preprint ar arxiv:201 1.000...01433 2020.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, W, 2017.
Tejaswini Pedapati, Avinash Balakrishnan, Karthikeyan Shanmugam, and Amit Dhurandhar. Learning global transparent models consistent with local contrastive explanations. Advances in Neural Information Processing Systems, 33, 2020.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn Machine learning in python. The Journal of Machine Learning Research, 12:2825-2830, 2.
Ben Peters, Vlad Niculae, and André FT Martins. Sparse sequence-to-ssqqueee models. arXiv preprint 1005:v:9005.070000 2019.
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep 2019.
learning tabular data. arXiv preprint
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: unbiased boosting with categorical features. In Advances in neural information processing systems, pp. 6638-6648, 2018.
Steffen Rendle.
Factorization machines. In 2010 IEEE International Conference Data Mining, pp- 995-1000. IEEE, 2010.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1 1134-1144, 2016.
Samuel Rota Bulo and Peter Kontschieder. Neural decision forests for semantic image labelling. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognition, pp. 81-88, 2014.
Ira Shavitt and Eran Segal. Regularization learning networks: deep learning for tabular datasets. In Advances in Neural Information Processing Systems, pp. 1379-1389, 2018.
Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need, 2021.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15296): 2014. URL http://jmm org/ www.:orggg//p///aeoooooo http//ppprssss/// /v15 srivastaval html.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint 170.:v:700.03777 2017.
Suhang Wang, Charu Aggarwal, and Huan Liu. Using a random forest to inspire a neural network and improving on it. In Proceedings of the 2017 SIAM international conference on data mining, pp. 1-9. SIAM, 2017.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint understanding of textual and tabular data, arXiv preprint 4rX11:2005.08314, 2020.
12

---

Under review as a conference paper at ICLR 2022
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervisee learning to tabular domaindim. Advances in Neural Information Processing Systems, 33, 2020.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar Tatikonda, Nicha Dvornek, Xenophon Pa- pademetris, and James S Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. arXiv preprint arXiv:2010.07668,000 2020.
13

---

Under review as a conference paper at ICLR 2022
DXN
DON DepTTLP
DMN Dee
DayTTLF
0.50
M55
[U0
0.90
16.5
0.56
Epoch
(a) DI HIGGS dataset
(b) 1D3-Telecom churn dataset (c) D6 California housing dataset
Figure The comparison of the DeepTLF (a green line) and the deep neural model (DNN) (a red line) models on validation (unseen) data. DeepTLF and DNN models have the exact same architecture. The results are computed over ten runs with different random seeds.
0.259
0.01
DeepTIF GBDT
cooo-entrooyyy.ooooo loss
Tex RDC AUC ScORe
0,7255
0.01
7200
0006
0.0
02129
0.57
DapTLF
DeedTLF
02125
GBOT
GID1
50
100
200
330
50
100
2011
302
decison frees
of decition trees
of decisinn troes
(a) Accuracy score
(b) ROC AUC score
(c) Cross-entt cross-entopy loss
Figure 8: A relationship between number of decision trees and the Deep'TLF performance. The accuracy score (higher better), ROC AUC score (higher is better), cross-entopy loss (lower better) metrics for the same experiment. The exact same GBDT model is used for the data encoding in the DeepTLF. The results averages over five trials for the telecom churn (D3) dataset.
A ADDITIONAL EXPERIMENTS
In this Section, provide experimental results.
Validation loss curves. We examine DNNs and our DeepTLF model separately using only the validation (unseen) data. To enable fair comparison, the deep learning part in DeepTLF is identical to the DNN we used. The results are presented in Fig. 7.
Is there correlation between GBDT's performance and the final performance of the DeepTLF? In this experiment, we examine the correlation between the performance of GBDT (which is used for data encoding) and DeepTLF. In other words, want demonstrate that if the performance of the GBDT improves, the performance of the DeepTLF rises. Fig. 11 presents the results of the experiments; the performance of the GBDT and DeepTLF. It is also noticeable that the DeepTLF performance in many cased the GBDT results.
can be observed, there is indeed a high positive correlation between
A "sanity check" experiment. In this simple experiment, we want to verify that our heterogeneous encoding function distills knowledge better than random heterogeneous "encoding function". We design the random function in the what it extracts random features with random splitting value. The experiment confirms that indeed proposed encoder performs better than a random set of Boolean rules for given dataset (Fig. 12).
B EXTENDED RELATED WORK
In recent years, deep learning on tabular data has received much attention from the machine learning and data science communities (Arik & Pfister, 2019; Popov al., 2019; Yin et al., 2020; Huang et al., 2020; Guo et al., 2017; Ke et al., 2019; He et al., 2014; Shavitt & Segal, 2018: Katzir et al., 2020; Yoon et al., 2020; Padhi al., 2020; Levy al., 2020; Ballet et al., 2019; Akrami et al., 2020; Gupta
14

---

Under review as a conference paper at ICLR 2022
2.72M
Test ROC AUC score
6.81
Accrrayy
DMN
DNN
Conn-entrepp lee
0.00
8.50
0.08
Dee
0.57
GBDT
0.50
30
65
35
20
10
55 6G
miss
soing
missing
(a) Accuracy score
(b) ROC AUC score
(c) Cross-entopy loss
Figure 9; The missing data experiment. The accuracy score (higher is better), ROC AUC score (higher is better), cross-entopy loss (lower better) metrics for the same experiment. The exact same GBDT model is used for the data encoding in the DeepTLF. The DNN model is identical in training and architecture to the DeepTLF's DNN part. The results are averages over five trials for the telecom churn (D3) dataset.
SET
633
6.0
678
2.A
631
641
8.30
(a) Accuracy score
(b) ROC AUC score
(c) Cross-entopy loss
Figure 10: Learning rate. The results are averages over five trials for the telecom churn (D3) dataset.
Test creas entrapy loss GBDT
A 0.55
RDC AUC, GBDT
8.700
0.00
acoursy
0.700
0.58
0/20
1012
0.335
3.500 0.005
0.370 0.255 0.30
socuracy,
Text ROC AUC, DeepTLF
Test cros-enttogy loss, DoepTFF
(a) Accuracy score
(b) ROC AUC score
(c) Cross-entopy loss
Figure Correlation plots for different quality measurements. The exact same GBDT model is used for the data encoding in the DeepTLF. The results demonstrate that there is indeed a high positive relationship between the performance GBDT and DeepTLF: Thus, the proposed data distillation algorithm can successfully distill the knowledge from trees. The results are averaged over five trials for the telecom churn (D3) dataset.
15

---

Under review as a conference paper at ICLR 2022
0.52
Cross-ent ees-entropp loss
0..00
DLF
0.48
Random
0.16
0.41
5
10
15
20
25
Epoch
Figure 12: A "sanity check" experiment. A comparison of the TreeDrivenEncoder and random encoding functions. The random encoding mimics the TreeDrivenEncoder, but it selects a random feature and splitting value. The experiment verifies that the TreeoDrivenEnnoder is able to distill the knowledge using trained decision trees in a GBDT algorithm. The results are averaged over five trials.
et al., 2021). The existing approaches can be grouped into two broad categories architecture-based and data transformation-base models.
Architecture-base based models. This group aims at developing new deep learning architectures for heterogeneous data (Arik & Pfister, 2019; Popov et al., 2019; Guo et al., 2017; Ke et al., 2019; Shavitt & Segal, 2018). For example, the authors of (Guo et al., 2017) proposed distinct neural network architecture for reducing the preprocessing and feature engineering effort by introducing a data sharing strategy between a deep and wide network so that low- and high-level interactions between the inputs can be learned simultaneously, based on the ideas of Factorization Machines (FM) proposed in (Rendle, 2010). The work (Lian et al., 2018) extended the sharing strategy using the FM for structured data further. In (Ke et al., 2019), the authors propose an integrated solution by introducing two special neural networks, one for handling categorical features and another for numerical data. However, for mentioned approaches ((Guo et al., 2017: Lian et al., 2018 8;; Ke et al., 2019)), it is not clear how other data-related issues, such as missing values, different scaling of numeric features, and noise, influence the predictions produced by the models.
Another line of research in this group tries to combine the advantages of decision trees and neural networks. For example, the authors of (Rota Bulo & Kontschicder, 2014) introduced Neural Decision Forests, ensemble of neural decision trees, where split functions in each tree node randomized Multi-Layer Perceptrons (MLPs). Another approach (Denoyer & Gallinari, 2014) presented a strategy for selecting paths in neural directed acyclic graph to produce the prediction for given input. Hence, the selected neural paths are specialized to specific inputs. In (Wang et al., 2017) 7), the authors empirically showed that neural networks with random forest structure could have better generalization ability across various input domains.
fully differentiable architecture for deep learning, which generalizes ensembles of oblivious decision trees on tabular, is also introduced in (Popov et al., 2019). Their architecture (coined NODE) employs the entmax transformation (Peters et al., 2019) and thus maps a vector of real-valued scores to discrete probability distribution.
Other approaches focus on architectures that build on iitention-based (Transformers) mecha- nisms (Vaswani et al., 2017). For example, the authors of (Arik & Pfister, 2019) and (Huang al., 2020) propose an attentive transformer architectre for deep tabular learning. Their architecture also offers the possibility to interpret the input features; however, for reliable performance, large amount of training data is needed. Another drawback is that the attention mechanism is only applied to categorical data. Hence, the continuous data does not throw the self-attention block, meaning that correlations between categorical and continuous features are dropped. The work (Yoon et al., 2020) proposes variation of transformer and offers semi-supervised learning.
However, no clear statements can be drawn for all methods described so far regarding the relationship between data heterogeneity and prediction quality (especially robustness under noisy data labels).
16

---

Under review as a conference paper at ICLR 2022
Moreover, many of the solutions in this line of research quite challenging from practical perspective since it is often unclear which architectural choices should be employed in realistic scenarios. These architecture-baase which are difficult to (re-)implement and optimize for specific real-world use cases. Also, the work (?) promotes localized decisions that are taken over small subsets of the features. Especially for critical, data-intensive applications, e.g., data streaming, large-scale recommendation systems (Baylor et al., 2017), and many more, is not clear what additional adjustments the working pipeline are needed.
approaches generally rely novel neural network architectures,
Data transformation-based models. Another way for improving the predictive quality in the presence of tabular data is to transform heterogeneous data into homogeneous feature vectors. The transformation can range from simple data preprocessing, such as the normalization of numerical variables or binary encoding of categorical variables, to linear or non-linear embedding schemes (e.g., generated by advanced autoencoders) (Garefa et al., 2015; Hancock & Khoshgoftaar, 2020). The advantage of such data transformation approaches is that they do not require adapting the deep learning architecture. However, they may reduce the information content by smoothing critical values that might have been highly relevant for the final prediction.
The method proposed by Moosmann et al. (2007) showed how the data could be encoded using the Random Forests (RF) algorithm by accessing Icaf indices in the decision trees. The same idea was presented in (He et al., 2014), where instead of the random forest algorithm, trees from the Gradient Boosted Decision Tree (GBDT) are used for the categorical data encoding. This work empirically demonstrates that the boosted decision trees are a powerful and convenient way to implement non-linear and categorical feature transformations for heterogeneous data. The DeepGBM framework (Ke et al., 2019) further evolved the idea of distilling knowledge from decision trees leaf index by encoding them using a neural network for online learning tasks.
This leaf embedding upproach received much attention. However, the leaf indices' from decision tree embeddings do not fully represent the structure of the decision tree. Hence, each boosted tree treated as new meta categorical feature. Bruch et al. (2020) proposed a gradient-deeccet--avee strategy that exploits the decision tree structure to propagate gradients in the learning process. Instead, our approach has a strong focus on the exploitation of the tree structures for the data transformation process, that is, the transformation of the heterogeneous tabular data into homogeneous vectors that are especially suited for deep learning but also other machine learning techniques. The observation that local Boolean features can be quite informative for global modeling is also reported in (Pedapati et al., 2020). where the authors exploit sparse local contrastive explanations of a black- box model to obtain custom Boolean features. A globally transparent model is then trained on the Boolean features only; empirically, the global model shows a predictive performance that is only slightly worse than that of state-of-the-ar approaches.
In summary, in contrast to state-of-the-art methods that exploit decision tree structures and mainly focus on leaf indices, DeepTLF utilizes the whole decision tree structure from a GBDT model, and furthermore considers the representation of each feature independently in the information distillation process. The framework proposed in this work combines the advantages of gradient boosted trees (such as handling different scales, different attribute types, missing values, outliers, and many more) with the learning flexibility of neural networks to achieve excellent predictive performance.
C TREEDRIVENENCODER ALGORITHM
In this section, we present the psedo code for our TreeDDivenEncoddr algorithm (Alg, 1).
D REPRODUCIBILITY DETAILS
We use the following implementation of baseline ML models: KNN, RF, LR, algorithms are from the widely used open-source machine learning python library Seikit-Learn Pedregosa al. (2011). for GBDT we select the python version of distributed gradient boosting library XGBoost Chen & Guestrin (2016), RLN, we use the official TensorFlow implementation from the GitHub repository'. we use well-tested PyTorch implementation of TabNet'. We use the official implementation of
Phttps: thub. http:////...tttccoooo/e/oaaa///a
17

---

Under review as a conference paper at ICLR 2022
Algorithm 1 For GBDT model T and an instance from the underlying dataset, the TreeDrivenEn- coder procedure visits the inner nodes of each T (in a breadth-first search manner) and exploits their Boolean functions to construct binary vector according to the feature values of x.
procedure TREED REEEDRIVENENCODRRC.. T))
x" vector of length
for tree T E T do
vector of length [T]
binary vector we aim construct position index in the binary vector
i ==
0 an empty queue
Q cennueue(T.rrof
while Q.notEmpty do
t Q.dequeue()
x getFeature Value(x, v) > get from x the value of the feature that is evaluated evaluate at
if v.evaluate(z)) == true then
add u(i)
else
add u(i) = 0
end if
i++
for all children v' of v do
if o' is an inner node (v' E V/) then Q.enqueue(r')
end if end for end while x" =concat(x", u)) end for return x end procedure
N We use the official PyTorch implementation of NODE from the GitHub repository', we adapt the official implementation of DeepGBM' to our need using the PyTorch framework. VIME and TabTansformer' implementations are from official GitHub repositories. DeepTLF, for the encoding part, we employ the XGBoost implementation of the GBDT algorithm; for the deep learning part, PyTorch (Paszke et al., 2017) is used. Additionally, implementation. Note, other implementations of GBDT can be used as well. We select the AdaBelief Optimizer Zhuang et al. (2020) for proposed framework.
will provide TensorFlow
For the (TPE) optimization algorithmBergstra et al. (2011) using the HyperOpt library Bergstra et al. (2013).
selection task for all baseline, we apply the tree-structured parzen estimator
t-SNE Experiment For the t-SNE experiments, we normalized the original datasets. We did not pre-procces the encoded homogeneous dataset after the TreendeieenEnooer transformation.
Computing infrastructure. Out experimental setup for all experiments has two RTX2080Ti GPUs and single CPU AMD 3960X 24-Core.
D.I DATASETS DESCRIPTION
Among these, the HIGGS dataset, which stems from experimental physics, is the largest dataset in our evaluation. As an exemplary dataset from the financial industry, we include the dataset defaults of clients, which contains information on default payments, demographic factors, credit data, history
18

---

Under review as a conference paper at ICLR 2022
Table 4: URLs for datasets of the study,
Dataset
URL
DI
HIGGS
Sttpp: //wrchive.....oo lcs uci. acc..dd//nl /tt.//aattttttt/oooooae /HI02S
D2
Default of clients
bttpp://wwx. http://wwwwwwgg..oo//otoooo www.ga..gooooooooooooo.... con/ucini/ seraalt-ar-coede card- elioneo-datasee
D3
Telecom churn
stspp://wwk. www.kk.ggcc.oooooo of c/ ox-prize-1
D4
Zillow
butlga http:////wwwww//// kayg Iu. www//wus foess wwwp:////wuuua////a///aa/oo /ovouaddo pricos
D5
Avocado prices
but httw:///wwww./. kagg wttp://ww....aaoa////aaaao http://aaaattaaaaaaaoaaaaaaa
castomer castomer-churee
D6
California housing
attpe / wwww kaggle. le com/ http:////o.////o///////////aaaaa
D7
E-commerce clothing reviews
attpes /wwww//www.....// leggle, www.gggpl.....o..o.ooooo wwwp:///wwpoeeeeeeeooooooeeeoooo
of payment, and bill Statements of credit card clients in Taiwan from April 2005 September 2005. In addition, the Zillow dataset represents typical heterogeneous data from the real estate sector. It important emphasize that in this dataset around 47 % of the data inputs are missing values. The avocado dataset is another representative of tabular datasets, which provides historical data on avocado prices. The telecom churn dataset presents customer data of different feature types with the goal to estimate the behavior of customer. Lastly, we employ the California housing dataset which contains information about house pricing 1990. E-commerce clothing reviews dataset (Agarap, 2018) has multiple data modalities text and tabular data.
All these datasets are collected from real- world problems and contain numerical as well as categorical data. Moreover, these datasets are freely available online and common in tabular data processing: each dataset was previously featured multiple published studies. We deliberately chose these six datasets cover different domain areas (web, natural sciences, etc.), tasks (classification and regression), and different dataset sizes.
We prepossessed the data in the same way for every machine learning model applying standard normalization. For the linear regression, logistic regression, and models based on neural networks, the missing values were substituted with zeros since these methods cannot handle them otherwise.
19


Reviews:
[{"review_id": "LXfygnMAj1Y", "reviewer": "Reviewer_gWeP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strength:\n1. the paper is clearly written, and easy to follow.\n2. the proposed idea is simple and easy to understand.\n3. the experiment part results look promising.\n\nweakness & question:\n1. some details in experiments are not clear. For example, the used hyper-parameters, especially for GBDT. And is the results run by one hyper-parameters, and by hyper-parameter search tools? To get a fair comparison, a throughout hyper-parameter search is needed.\n2. Can you also provide the AUC metric for the binary classification tasks?\n3. Are there any multiclass tasks?\n4. Do you think it is possible to get rid of GBDT (as a feature extractor), and independently design a feature extractor, and perform a similar performance?\n\nI will boost my score if the experimental details are provided and reasonable.", "summary_of_the_paper": "This paper proposes an encoding method (to encode structural information in GBDT) for DNN with tabular data, to handle the heterogeneous features. The method is simple, and the results look very promising. \n", "summary_of_the_review": "This paper is clearly written, and the proposed method is simple, while seems to work. But some details of the experiment, especially the hyper-parameters, are missed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DRA9hbpHS8", "reviewer": "Reviewer_vaip", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is nice and quite elegant, leveraging the flexibility of tree based methods for use in deep learning. To my knowledge, this is a novel idea, although it may not be super obscure. At least I did not came across any literature where this approach had been followed.\n\nThe paper is well written and easy to follow. If we deduct the recap of gradient boosting and the formal definition of a decision tree, the actual description of the approach is just about one page. Large part of the paper and the appendix is dedicated to extensive experiments, comparing the approach to simple baselines and dedicated tabular data methods.\n\nThe major claims made in the paper are supported\n- *DeepTLF can preserve most of the information that is contained in the original data.* I think this is shown with the experiments (Table 2) in so far as DeepTLF yield state of the art performance on all datasets  \n - *it dramatically speeds up the data preprocessing time (Fig 1, caption)* Well, yes, once the tree is trained no further explicit preprocessing (data scaling and handling missing values) is required as this is moved into the decision functions. \n\nI have three bigger concerns:\n1. The comparison with DNN in Table 2 may not be fair because DNN has 3 fully connected layers while DeepTLF has 4 fully connected layers. Depending on the number of decision functions, the first layer in DeepTLF may also have more parameters than DNN. The architecture of DNN is not further described. Are there embedding layers for categorical columns  or are they treated numerically? How are missing values treated? That can make a big difference to the performance of DNN.\n2. Table 3 should include all(!) competitors, in particular DNN. For the training time of DeepTLF it should be made explicit how much time is spend on training the tree ensemble (is it the same as GBDT, second line) and how much for training the network. For data pre-processing time it should be made clear what time is required to prepare the dataset for training (e.g. computing averages and variances or inferring candidates for missing values) and how much preprocessing time is required to do inference on a new and unseen sample (e.g. scale fields appropriately, fill in missing values). Both should be set into relation to training time and inference time respectively.\n3. I'm wondering what I could do if there are no classification or regression targets to train a tree ensemble in the first place (e.g. for  unsupervised or self-supervised learning approaches) Have the authors done any experiments in this regard? At least this questions should be addressed in the discussion. \n\n\n#### Minor issues\n- Fig 2, caption: $\\mathbf x_i \\in \\mathbb R\\times\\mathbb R\\times\\\\{red,-\\\\}\\times\\mathbb R\\$\nsimilar in Definition 1\n- p5,l15: *\"It is only important that the same strategy is used across all decision trees and vectors.”* why is that? Isn’t it just important that the order of $\\tilde\\mu_v$ is the same for all $\\mathbf x$?\n- p5,l28: $\\mathbf x=(x_1,…,x_d)^\\top$, Euklidean vectors are column vectors by convention, similar in (8)\n- how to deal with duplicate $\\tilde\\mu_v$ ?\n\n\n\n", "summary_of_the_paper": "For the problem of learning (supervised classification and regression) on tabular data, the authors propose to used the decision functions of tree based ensemble methods as input features for a deep neural network (DNN). Tabular data is typically heterogenous, that is the data come from different modalities (continuous, categorical, missing values). As decision trees are very well tailored to such data the authors show that with their method (DeepTLF) no further preprocessing or data cleansing is necessary (apart from what's being performed in the decision tree). On numerous experiments on 7 public tabular datasets, they show that their method outperforms different baseline and competitive approaches, thereby supporting the claim that the tree based input features preserve sufficiently rich information of the input data. ", "summary_of_the_review": "Interesting and to my knowledge novel idea to use decision functions of tree ensembles as input features to neural networks. The experiments show that this method works, but are not 100% conclusive if it is superior in all aspects, in particular comparing to DNN (see my concerns above).\nI rate the paper *\"6: marginally above the acceptance threshold\"* and am willing to increase my rating if my concerns are fully addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CQWXTxaTI8", "reviewer": "Reviewer_Q6we", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper proposes a very simple idea (i.e. Use a two-stage feature derivation process for tabular data), which outperforms many contemporary large-scale neural network models for tabular data for several real-world datasets.\n- Comparison with contemporary tabular neural nets are appreciated.\n\nWeaknesses:\n- The technical novelty is quite limited, since all the authors do is train GBDT first on the entire training set, extracts the internal node values as a feature representation, use those feature representations as new training samples to train a downstream neural network. As discussed in the paper, if the whole process was done end-to-end, it would have been more interesting.\n- Experiment details are missing/incomplete. D1, D2, D3 are used for binary classification, and the authors do not discuss the ratio between positive and negative samples (class imbalance). Also, for binary classification, researchers typically use AUROC or AUPRC due to class imbalance, but the authors use cross entropy loss instead.\n- The proposed method barely outperforms GBDT in all datasets except for D1. The authors claim that they used a small DNN model for DeepTLF to demonstrate the power of their approach, but since that power is not observed in Table 2, it would have been more convincing if the authors had used several variations of DeepTLF (e.g. deeper DNN, Transformer) and show the power of DeepTLF.\n- In Table 3, it is not clear whether the training time measure a single minibatch update, or a single epoch, or the entire training process.\n- Is there any reason D7 was not included in Table 2?\n- The authors state that D7 consists of textual and tabular data. Does this mean that the cell values of the table can either be a continuous value, categorical value, discrete value, and word tokens? It is not clear how GDBT would be able to process such table. If not, then what does D7 look like as a multimodal tabular data?", "summary_of_the_paper": "This paper proposes DeepTLF, a new framework for prediction tasks using tabular data. DeepTLF first trains gradient boosted decision trees (GBDT) using the entire tabular training samples. Then it uses node values as the input the neural network predictor (e.g. classifier or regressor) for the actual prediction phase. In essence, the authors rely on GBDT's capability to handle heterogeneous tabular data with potentially many missing values to derive a binary representation of the given sample, and feed that representation to a downstream neural network. DeepTLF was able to demonstrate superior performance in real-world tabular datasets.", "summary_of_the_review": "The proposed method, DeepTLF, is a very straightforward combination of GDBT and DNN, which seems to outperform all modern tabular neural nets, which I find very interesting. However, the technical novelty is quite limited, and considerable amount of experiment details are missing, therefore making it difficult to accurately evaluate the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a method called DeepTLF that handles heterogeneous tabular data by using GBDT as an encoder for a DNN.

The paper is clearly written and the method works as intended.

There is however the issue of novelty (raised by Q6we). The method indeed relies of the capacity of GBDT to represent the data, the internal node values are used as features to train a downstream neural network. This process is straightforward, which is good from an application perspective, though the paper offers limited insights to the community from a scientific perspective.

Another reviewer concern was that of incompleteness of experiments and lack of certain details (reviewers vaip and gWeP). This was answered in the rebuttal, which the reviewers acknowledged, however, the authors did not provide a revised version of the manuscript, when ICLR in fact allowed (and actually encouraged) revised versions to be submitted by Nov 22. Without a revised version, it is difficult for the reviewers to assess whether the text in the final manuscript will actually accurately reflect the changes they suggested. This justifiably caused two of the reviewers to keep their original scores (they explicitly stated the lack of an updated manuscript as the reason).

Given lack of an update, coupled with the issue of novelty, I conclude the paper is not ready to be accepted in its current form.
Decision: Reject