Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data
Authors: Sergei Popov
Stanislav Morozov
Artem Babenko
Venue:  (2020)
Keywords: tabular data
architectures
DNN

Abstract: Nowadays, deep neural networks (DNNs) have become the main instrument for machine learning tasks within a wide range of domains, including vision, NLP, and speech. Meanwhile, in an important case of heterogenous tabular data, the advantage of DNNs over shallow counterparts remains questionable. In particular, there is no sufficient evidence that deep learning machinery allows constructing methods that outperform gradient boosting decision trees (GBDT), which are often the top choice for tabular problems. In this paper, we introduce Neural Oblivious Decision Ensembles (NODE), a new deep learning architecture, designed to work with any tabular data. In a nutshell, the proposed NODE architecture generalizes ensembles of oblivious decision trees, but benefits from both end-to-end gradient-based optimization and the power of multi-layer hierarchical representation learning. With an extensive experimental comparison to the leading GBDT packages on a large number of tabular datasets, we demonstrate the advantage of the proposed NODE architecture, which outperforms the competitors on most of the tasks. We open-source the PyTorch implementation of NODE and believe that it will become a universal framework for machine learning on tabular data.

Published as a conference paper at ICLR 2020
NEURAL OBLIVIOUS DECISION ENSEMBLES FOR DEEP LEARNING ON TABULAR DATA
Sergei Popoy
Stanislav Morozov
Yandex
Yandex Lomonosov Moscow State University
sapopovéyander-team.
Stanis--morozov@anddee. ru
Artem Babenko
Yandex
National Research University
Higher School of Economics
artem. www.enko@phyteecc.cc... edu
ABSTRACT
Nowadays, deep neural networks (DNNs) have become the main instrument for machine learning tasks within a wide range of domains, including vision, NLP, and speech. Meanwhile, in an important case of heterogenous tabular data, the advantage of DNNs over shallow counterparts remains questionable. In particular, there no sufficient evidence that deep learning machinery allows constructing methods that outperform gradient boosting decision trees (GBDT), which are of- ten the top choice for tabular problems. In this paper, we introduce Neural Obliv- ious Decision Ensembles (NODE), new deep learning architecture, designed to work with any tabular data. nutshell, the proposed NODE architecture gen- eralizes ensembles of oblivious decision trees, but benefits from both end-to-end gradient- based optimization and the power of multi-layer hierarchical representa- tion learning. With extensive experimental comparison the leading GBDT packages on large number of tabular datasets, we demonstrate the advantage most of
the proposed NODE architecture, which outperforms the competitors the tasks. We open-source the PyTTrch implementation of NODE and believe that it will become a universal framework for machine learning on tabular data.
INTRODUCTION
The recent rise of deep neural networks (DNN) resulted in substantial breakthrough for large number machine learning tasks in computer vision, natural language processing, speech recogni- tion, reinforcement learning (Goodfellow et al. 2016). Both gradient-based optimization via back- propagation (Rumelhart et al. 1985) and hierarchical representation learning appear to be crucial in increasing the performance of machine learning for these problems by large margin.
While the superiority of deep architectures in these domains is undoubtful, machine learning for tabular data still did fully benefit from the DNN power. Namely, the state-of-the-art perfor- mance in problems with tabular heterogeneous data is often achieved by "shallow" models, such as gradient boosted decision trees (GBDT) (Friedman) 2000]; Chen Guestrin 2016 Ke al. 2017 Prokhorenkova et al. 2018). While the importance of deep learning on tabular data is recognized by the ML community, and many works address this problem (Zhou & Feng 2017 Yang et al., 2018 Miller et al.] 2017 Lay et al.) 2018) Feng et al. 2018 Ke et al. 2018). the proposed DNN approaches do not consistently outperform the state-of-the-arr shallow models by a notable margin. In particular, to the best of our knowledge, there is still universal DNN approach that was shown to systematically outperform the leading GBDT packages (e.g., XGBoost (Chen Guestrin 2016)). As additional evidence, large number of Kaggle ML competitions with tabular data still by the shallow GBDT methods (Harasymiv) 2013). Overall, at the moment, there is no dominant deep learning solution for tabular data problems, and we aim to reduce this gap by our paper.

---

Published as a conference paper at ICLR 2020
We introduce Neural Oblivious Decision Ensembles (NODE), new DNN architecture, designed to work with tabular problems. The NODE architecture is partially inspired by the recent CatBoost package (Prokhorenkova al. 2018). which was shown to provide state-of-the-ang performance on a large number of tabular datasets. In a nutshell, CatBoost performs gradient boosting on oblivi- ous decision trees (decision tables) (Kohavi 1994 Lou & Obukhov 2017). which makes inference very efficient, and the method is quite resistant to overfitting. In its essence, the proposed NODE architecture generalizes CatBoost, making the splitting feature choice and decision tree routing dif- ferentiable. As result, the NODE architecture is fully differentiable and could be incorporated in any computational graph of existing DL packages, such as TensorFlow or PyTorch. Further- more, NODE allows constructing multi-layer architectures, which resembles "deep" GBDT that is trained end-to-end, which was never proposed before. Besides the usage of oblivious decision ta- bles, another important design choice is the recent enimax transformation (Peters et al. 2019), which effectively performs a "soft" splitting feature choice in decision trees inside the NODE architecture. As discussed in the following sections, these design choices are critical to obtain state-of- off-the-ar art performance. In a large number of experiments, we compare the proposed approach with the lead- ing GBDT implementations with tuned hyperparameters and demonstrate that NODE outperforms competitors consistently on most of the datasets.
Overall, the main contributions of our paper can be summarized as follows:
1. We introduce
DNN architecture for machine learning tabular data. To the best of our knowledge, our method is the first successful example of deep architectures that substantially outperforms leading GBDT packages
tabular data.
2. Via an extensive experimental evaluation on a large number of datasets, we show that the proposed NODE architecture outperforms existing GBDT implementations.
3. The PyTorch implementation of NODE is available online
The rest of the paper is organized follows. In Section method. The proposed Neural Oblivious Decision Ensembles architecture is described in Section 3 and experimentally evaluated in Section Section 5 concludes the paper.
review prior work relevant to
RELATED WORK
In this section, we briefly review the main ideas from prior work that are relevant to our method.
The state-of-the-art for tabular data. Ensembles of decision trees, such as GBDT (Friedman 2001] or random forests (Barandiaran 1998), are currently the top choice for tabular data problems. Currently, there are several leading GBDT packages, such as XGBoost (Chen & Guestrin 2016), LightGBM (Ke et al. 2017). CatBoost (Prokhorenkova et al. 2018). which are widely used by both academicians and ML practitioners. While these implementations vary in details, tasks their performances do not differ much (Prokhorenkova et al. 2018 Anghel et al.). The most important distinction of CatBoost is that it uses oblivious decision trees (ODTss) as weak learners. As ODTs are also an important ingredient of our NODE architecture, we discuss them below.
most of the
Oblivious Decision Trees. An oblivious decision tree a regular tree of depth that is constrained to use the same splitting feature and splitting threshold in all internal nodes of the same depth. This constraint essentially allows representing an ODT as table with 2d entries, corresponding to all possible combinations of d splits (Lou & Obukhov 2017). Of course, due to the constraints above, ODT's are significantly weaker learners compared to unconstrained decision trees. However, when used in an ensemble, such trees are less prone to overfitting, which was shown to synergize well with gradient boosting (Prokhorenkova et al., 2018). Furthermore, the inference in ODTs very efficient: one can compute d independent binary splits in parallel and return the appropriate table entry. In contrast, non-oblivious decision trees require evaluating d splits sequentially.
Differentiable trees. The significant drawback of tree-based approaches is that they usually do not allow end-to-end optimization and employ greedy, local optimization procedures for tree construc- tion. Thus, they cannot be used as component for pipelines, trained in end-to-end fashion. To address this issue, several works (Kontschieder et al. 2015 Yang et al. 2018 Lay et al. 2018)
https://gi thwb. cwmob..com//// wwt../cmm/i.i.e... httm///wicee//n//////ee
2

---

Published as a conference paper at ICLR 2020
propose to "soften" decision functions in the internal tree nodes to make the overall tree function and tree routing differentiable. In our work, we advocate the usage of the recent entmax transforma- tion (Peters al. 2019) to "soften" decision trees. We confirm its advantages over the previously proposed approaches in the experimental section.
Entmax. The key building block of our model is the entmax transformation (Peters et al.) 2019) which maps a vector of real-valued scores to a discrete probability distribution. This transforma- tion generalizes the traditional softmax and its sparsity-enforcing alternative sparsemax (Martins & Astudillo 2016), which has already received significant attention in a wide range of applications: probabilistic inference, topic modeling, neural attention (Niculae Blondel 2017 Niculae al.) 2018 Lin et al., (2019). The entmax is capable to produce sparse probability distributions, where majority of probabilities are exactly equal to O. In this work, we argue that entmax is also an appropriate inductive bias in our model, which allows differentiable split decision construction in the internal tree nodes. Intuitively, entmax can learn splitting decisions based on a small subset of data features (up to one, as in classical decision trees), avoiding undesired influence from others. As an additional advantage, using entmax for feature selection allows for computationally efficient inference using the sparse pre-computed choice vectors as described below in Section
Multi-layer non-differentiabbl architectures. Another line of work (iilerr et al., 2017 Zhou & Feng 2017: Feng al.) 2018) promotes the construction of multi- mulli--lyyer architectures from non- differentiable blocks, such as random forests or GBDT ensembles. For instance, (Zhou & Feng. 2017 Miller al. (017) propose use stacking of several random forests, which are trained sepa- rately. In recent work, (Feng et al. 2018) introduces the multi-layer GBDTs and proposes a training procedure that does not require each layer component be differentiable. While these works report marginal improvements over shallow counterparts, they lack the capability for end-to-end training, which could result in inferior performance. contrast, we argue that end-to-end training is crucial and confirm this claim in the experimental section.
Specific DNN for tabular data. While a number of prior works propose architectures designed for tabular data (Ke et al., 2018 [Shavitt& & Segal 2018), they mostly do not compare with the properly tuned GBDT implementations, which are the most appropriate baselines. The recent preprint (Ke) al. 2018) reports the marginal improvement over GBDT with default parameters, but in experiments, the baseline performance is much higher. To the best our knowledge, our approach is the first to consistently outperform the tuned GBDTs over large number of datasets.
NEURAL OBLIVIOUS DECISION ENSEMBLES
We introduce the Neural Oblivious Decision Ensemble (NODE) architecture with a layer-wise struc- ture similar to existing deep learning models. In nutshell, our architecture consists of differentiable oblivious decision trees (ODT) that are trained end-to-end by backpropagation. We deseribe our im- plementation of the differentiable NODE layer in Section 3.1 the full model architecture in Section and the training and inference procedures in section
3.2
3.3
3.1 DIFFERENTIABLE OBLIVIOUS DECISION TREES
The core building block of our model is a Neural Oblivious Decision Ensemble (NODE) layer. The layer is composed of m differentiable oblivious decision trees (ODTs) of equal depth d. As an input, all m trees get common vector R", containing numeric features. Below we describe design of a single differentiable ODT.
In its essence, ODT is a decision table that splits the data along d splitting features and compares each feature to a learned threshold. Then, the tree returns one of the 2d possible responses, corre- sponding to the comparisons result. Therefore, each ODT is completely determined by its splitting
2x2x2
features f E IRd, splitting thresholds E Rd and a d-dimensional tensor of responses It E In this notation, the tree output is defined as:
hxx)= R bı), ..., 1ff((x) ba)],
where 1 (-) denotes the Heaviside function.
3

---

Published as a conference paper at ICLR 2020
input 000000000
tree root
entmax choice
C (x] (x)- poj-bb.
F (g(F,(x)-bb)
cr,
F,,FF (x) b
F
cr (F,lx)
Ryn
Rin
Rin
RT.
output
Figure 1: The single ODT inside the NODE layer. The splitting features and the splitting thresholds shared across all the internal nodes of the same depth. The output is a sum of leaf responses scaled by the choice weights.
To make the tree output differentiable, we replace the splitting feature choice and the compar- ison operator (f)(r) b2) by their continuous counterparts. There are several existing approaches that can be used for modelling differentiable choice functions decision trees (Yang et al. 2018). for instance, REINFORCE (Williams) 1992) Gumbel-softmax (Jang et al. (016)). However, these approaches typically require long training time, which can be crucial in practice.
Instead, we propose to use the d-entmax function (Peters et al., 2019) as it is able to learn sparse choices, depending only on a few features, via standard gradient descent. This function is general- ization of softmax in its variational form: sof (mfa(aaax argmoxpealll (p, x)) + [p)) where (p) is Shannon entropy. We can define o-entmax by replacing (p) with Tsallis a-entropy
The choice function is hence replaced by a weighted sum of features, with weights computed as a-entmax (a=1.5) over the learnable feature selection matrix F E Rdxn;
f((x) = Σ 2j entmar, (Fir) 1=1
(2)
Similarly, relax the Heaviside function ((f((z) by) two-class entmax, which denote as a, (e)==)t ([z, 0)). As different features can have different characteristic scales, we use the scaled version e, (x) = 0& [f(())bb bb: where b, and T3 are learnable parameters for thresholds and
scales respectively.
2x2x2
Based the e; (z) values, define a "choice" tensor C tensor R by computing the outer product of all c::
of the same size the response
ca (x) (
c2(2) 02
ca(x))
C(x) =
x
(z)
2 @
1 ca(x))
(3)
The final prediction then computed as weighted linear combination of response tensor entries with weights from the entries of choice tensor C:
h(x))
Σ
R Ch (I)
(4)
(0.1))
Note, that this relaxation equals to the classic non-differentiable ODT ((x)(T)) iff both feature selec- tion and threshold functions reach one-hot state, i.c. entmax always returns non-zero weights for single feature and always return exactly zeros or ones.
IIf is unfamiliar with this definition, highly recommend reading Peters al. 2019
4

---

Published as a conference paper at ICLR 2020
Finally, the output of the NODE layer is composed concatenation of the outputs of m individual trees (1) Am(())
Multidimensional tree outputs. In the description above, we assumed that tree outputs are one- dimensional h(x)) E R. For classification problems, where NODE predicts probabilities of each class, we use multidimensional tree outputs h(z)) E RR where |C] is number of classes.
3.2 GOING DEEPER WITH THE NODE ARCHITECTURE
The NODE layer, described above, can be trained alone or within complex structure, like fully- connected layers that can be organized into the multi-layer architectures. In this work, we introduce a new architecture, following the popular DenseNet (Huang et al. 2017) model and train it end-to- end via backpropagation.
input
trea
9
layer
layer
layer 3
Concatenation: cat (a, b)
Σ
prediction
Figure 2:
The NODE architecture, consisting of densely connected NODE layers. Each layer con- tains several trees whose outputs are concatenated and serve as input for the subsequent layer. The final prediction is obtained by averaging the outputs of all trees from all the layers.
Similar to DenseNet, architecture is sequence of & NODE layers (see Section 3.1 where each layer uses a concatenation of all previous layers as its input. The input layer 0 of this architecture corresponds to the input features I, accessible by all successor layers. Due to such a design, our architecture capable to learn both shallow and deep decision rules. A single tree on i-th layer can rely chains of up to layer outputs features, allowing it to capture complex dependencies. The resulting prediction is a simple average of all decision trees from all layers.
Note, in the multi- architecture described above, tree outputs h(x) from early layers are used as inputs for subsequent layers. Therefore, we do not restrict the dimensionality of h((x) to be equal to the number of classes, and allow to have an arbitrary dimensionality which correspond to
2x2x2xl
the (d + 1) dimmensiiaa response tensor R E IR
When averaging the predictions from all layers, only first |C| coordinates of 1((x)) are used for classification problems and the first for regression problems. Overall, / is an additional hyperparameter with typical values in [1,3].
3.3 TRAINING
Here we summarize the details of our training protocol.
Data preprocessing. First, we transform each data feature follow normal distribution via quan- tile transform]] In experiments, we observed that this step was important for stable training and faster convergence.
Initialization. Before training, we perform the data-aware initialization Mishhin & Matas 2016) to obtain a good initial parameter values. In particular, we initialize the feature selection matrix
Akkeeamm.reprocesssigg hwwp:pprrrroocciinnuuuaittttaaaa QuantileTrnssfommm
5

---

Published as a conference paper at ICLR 2020
uniformly Fiy (0, 1), while the thresholds b are initialized with random feature values f(()) observed in the first data batch. The scales T; are initialized such a way that all the samples in the first batch belong the linear region of aa and hence receive nonzero gradients. Finally, the response tensor entries are initialized with the standard normal distribution R(i,,..., va] ~ N(0, 1).
Training. As for existing DNN architectures, NODE is trained end-to-end via mini-batch SGD. We jointly optimize all model parameters: F, R. In this work, we experimented with traditional objective functions (cross-entropy for classification and mean squared error for regression), but any differentiable objective can be used as well. As an optimization method, we use the recent Quasi- Hyperbolic Adam with parameters recommended in the original paper (Ma & Yarats 2018). We also average the model parameters over = 5 consecutive checkpoints (Izmailov et al. 2018 and pick the optimal stopping point on the hold-out validation dataset.
Inference. During training, significant fraction of time is spent computing the entmax function and multiplying
the choice tensor. Once the model trained, one can pre-compute entmax feature selectors and store them sparse vector (e.g., in coordinate (coo) format), making inference more efficient.
EXPERIMENTS
In this section, we report the results of a comparison between our approach and the leading GBDT packages. We also provide several ablation studies that demonstrate the influence of each design choice in the proposed NODE architecture.
4.1 COMPARISON TO THE STATE-OF SEATE-O--THE-AAEE ARE-ART.
As main experiments, compare the proposed NODE architecture with two state-of-the-are GBDT implementations on a large number of datasets. In all the experiments we set parameter in the entmax transformation 1.5. All other details of the comparison protocol are described below.
Datasets. We perform most of the experiments on six open-source tabular datasets from different domains: Epsilon, YearPrediction, Higgs, Microsoft, Yahoo, Click. The detailed description of the datasets is available in appendix. All the datasets provide train/test splits, and we used 20% samples from the train set as a validation set to tune the hyperparameters. For each dataset, we fix the train/val/test splits for a fair comparison. For the classification datasets (Epsilon, Higgs, Click), we minimize cross-entropy loss and report the classification error. For the regression and ranking datasets (YearPrediction, Microsoft, Yahoo), we minimize and report mean squared error (which corresponds to the pointwise approach to learning-to-
Methods. We compare the proposed NODE architecture the following baselines:
Catboost. The recent GBDT implementation (Prokhorenkova et al. 2018) that uses oblivious decision trees as weak learners. We use the open-souree implementation, provided by the authors.
XGBoost. The most popular GBDT implementation widely used in machine learning competi- tions (Chen & Guestrin 2016). We use the open-source implementation, provided by the authors. FCNN. Deep neural network, consisting of several fully-connected layers with ReLU non- linearity layers (Nair& & Hinton 2010).
Regimes, We perform comparison in two following regimes that are the most important in practice:
this regime, we compare the methods as easy-to-tune toolkits that could be used by a non-professional audience. Namely, here we do not tune hyperparameters and use the default ones provided by the GBDT packages. The only tunable parameter here is number of trees (up to 2048) in CatBoost/XGBoost, which is set based on the validation set. We do not compare with FCNN in this regime, as it typically requires much tuning, and did not find the set of parameters, appropriate for all datasets. The default architecture in our model contains only a single layer with 2048 decision trees of depth six. Both of these hyperparameters were inherited from the CatBoost package settings for oblivious decision trees. With these parameters, the NODE architecture shallow, but it still benefits from end-to-end training via back-propagation.
Default hyperparameters.

---

Published as a conference paper at ICLR 2020
Epsilon
YearPrediction
Higgs
Microsoft
Yahoo
Click
Default hyperparameters
CatBoust
0.111942e-4
80.6810.04
0.243412c-4
0.5587-2c-4
0.578113e-4
0.3438
XGBoost
0.1144
81.11
0.2000
0.5637
0.5756
0.3461
NODE
0.1043::4e-4 77.43
0.2412 4.55544 00..33e4406666--66 0.5666115e-40.333 0:33090-----33
Table 1: The comparison of NODE with the shallow state-of-ther aftt---------tttt- counterparts with default hy- perparameters. The results are computed over ten runs with different random seeds.
Epsilon
YearPrediction
Higgs
Microsoft
Yahoo
Click
Tuned hyperparameters
CatBoost
0.1113+44e-
09.67+00112
0.2378-1e:
0.5565+22-4
0.5632+3e-4
0.3401 +
XGBoost
0.1112:::c 4
78.53:10.09
0.2328-3c 4 0.554411c-
0.5420:
0.33341:22 3
FCNN
0.1058:1c-
81.17::0.81
0.2139
0.5595::2c 4
0.569117e
0.3324
NODE
0.1034
06.21=0011
0.2101
0.5570+226
0.5692+22
0.3312:t2--3
mGBDT
0OM
80.67
OOM
OOM
OOM
OOM
DeepForest
0.1179
0.2391
0.3333
Table 2: The comparison of NODE with both shallow and deep counterparts with hyperparameters tuned for optimal performance. The results computed over ten with different random seeds.
Tuned hyperparameters. this regime, we tune the hyperparameters for both NODE and the competitors the validation subsets. The optimal configuration for NODE contains between two and eight NODE layers, while the total number trees across all the layers does not exceed 2048. The details of hyperparameter optimization are provided in appendix.
The results of the comparison are summarized in Table | and Table 2 For all methods, we report mean performance and standard deviations computed over ten runs with different random seeds. Several key observations are highlighted below:
1. With default hyperparameters, the proposed NODE architecture consistently outperforms both CatBoost and XGBoost on all datasets. The results advocate the usage of NODE as a handy tool for machine learning tabular problems.
2. With tuned hyperparameters, NODE also outperforms the competitors most of the tasks. Two exceptions are the Yahoo and Microsoft datasets, where tuned XGBoost provides the highest performance. Given the large advantage of XGBoost over CatBoost on Yahoo, we speculate that the usage of oblivious decision trees is an inappropriate inductive bias for this dataset. This implies that NODE should be extended to non-oblivious trees, which we leave for future work. In the regime with tuned hyperparameters on some datasets FCNN outperforms GBDT, while on others GBDT is superior. Meanwhile, the proposed NODE architecture appears to be universal instrument, providing the highest performance on most the tasks.
For completeness we also aimed to compare to previously proposed architectures for deep learning on tabular data. Unfortunately, many works did not publish the source code. We were only able to perform a partial comparison with mGBDT (Feng et al.) 2018) and DeepForest (Zhou & Feng 2017). which source code is available. For both baselines, use the implementations, provided by the authors, and tune the parameters on the validation set. Note, that the DeepForest implementation available only for classification problems. Moreover, both implementations do not scale well, and for many datasets, we obtained Out-Or-Memory error (OOM). On datasets in our experiments it turns out that properly tuned GBDTs outperform both (Feng et al. 2018) and (Zhou & Feng 2017).
4.2 ABLATIVE ANALYSIS
In this section, we analyze the key architecture components that define our model.
Choice functions. Constructing differentiable decision trees requires function that selects items from set. Such function is required for both splitting feature selection and decision tree routing. We experimented with four possible options, each having different implications:
7

---

Published as a conference paper at ICLR 2020
Dataset
YearPrediction
Epsilon
Function
softmax
Gumbel
sparsemax
entmax
softmax
Gumbel
sparsemax
entmax
layer 78.41
79.39
78.13
77.43
0.1045
0.1979
0.1083
0.1043
layers
77.61
79.31
76.81
77.05
0.1041
0.2884
0.1052
0.1031
4 layers 77.58
79.69
76.60
76.21
0.1034
0.2908
0.1058
0,1033
layers 77.47
80.49
76.31
76.17
0.1036
0.3081
0.1058
0.1036
Table 3: The experimental comparison of various choice functions and architecture depths. The values represent mean squared error for YearPrediction and classification error rate for Epsilon.
kytr3
23%
Rys1
limes bype
1ye13
WWrS
byere
2.9
Figure 3: NODE on UCI Higgs dataset: Left-Top: individual feature importance distributions for both original and learned features. Left-Bottom: mean absolute contribution of individual trees to the final response. Right: responses dependence feature importances. See details in the text.
Softmax learns dense decision rules where all items have nonzero weights;
Gumbel-Softmax (Jang et al., 2016) learns to stochastically sample single element from set;
Sparsemax (Martins & Astudillo 2016 learns sparse decision rules, where only few items have nonzero weights;
Entmax (Peters et al. 2019) generalizes both sparsemax and softmax; it is able to learn sparse decision rules, but smoother than sparsemax, being more appropriate for gradient-based opti- mization. In comparison a parameter was set to 1.5.
We experimentally compare the four options above with both shallow and deep architectures in Ta- ble 3 We use the same choice function for both feature selection and tree routing across all experi- ments. In Gumbel-Softmax, we replaced it with hard argmax one-hot during inference. The results clearly show that Entmax with 1=1.5 outperforms the competitors across all experiments. First, Ta- ble demonsttaaes that sparsemax and softmax are not universal choice functions. For instance, on the YearPrediction dataset, sparsemax outperforms softmax, while on the Epsilon dataset softmax superior. In turn, entmax provides great empirical performance across all datasets. Another obser- vation is that Gumbel-Softmax is unable to learn deep architectures with both constant and annealed temperature schedules. This behavior is probably caused by the stochasticity of Gumbel-Softmax and the responses on the former layers are too noisy to produce useful features for the latter layers.
Feature importance. In this series of experiments, by the NODE architecture. We begin by estimating the feature importances from different layers of multi-layer ensemble via permutation feature importance, initially introduced in (Breiman) 2001). Namely, for 10.000 objects from the Higgs dataset we randomly shuffle the values of each feature (original or learnt on some NODE layer) and compute the increase in the classification error. Then for each layer, we split feature importance values into seven equal bins and calculate the total feature importance of each bin, shown Figure 3 (left-top). We discovered that the features from the first layer are used the most, with feature importances decreasing with depth. This figure shows that deep layers are able to produce important features, even though earlier layers have an advantage because of the DenseNet architecture. Next, we estimated the mean absolute contribution of individual trees to the final response, reported on Figure 3 (left-bottom). One can see the reverse trend, deep trees tend to contribute more to the final response. Figure (right) clearly shows that there is anticorrelation of feature importances and contributions in the final response, which implics that the main role of ealier layers is to produce informative features, while the latter layers mostly use them for accurate prediction.
analyze the internal representations, learned
Training/ Innfrenn/I runtime. Finally, compare the NODE runtime to the timings of the state- of-the--rt GBDT implementations. In Table 4 we report the training and inference time for million of objects from the YearPrediction dataset. In this experiment, evaluate ensembles of 1024 trees
8

---

Published as a conference paper at ICLR 2020
of depth six with all other parameters set to their default values. Our GPU setup has single 1080Ti GPU and CPU cores. In turn, our CPU setup has a 28-core Xeon E5-2660 v4 processor (which costs almost twice much as the GPU). We CatBoost v0.15 and XGBoost v0.90 baselines, while NODE inference runs on PyTorch v1.1.0. Overall, NODE inference time is on par with heavily optimized GBDT libraries despite being implemented in pure PyTorch (i.e. custom kernels).
Method Training Inference
NODE 8 layers 1080Ti
XGBoost Xeon XGBoost 1080Ti CatBoost Xeon
7min 42s
5min 39s
Imin 13s
4ls 4.62s
8.56s
5.94s
4.45s
Table 4: Training and inference runtime for models with 1024 trees of depth six on the YearPredic-- tion dataset, averaged over five runs. Both training and inference eight-layer NODE architecture on GPU on par with shallow counterparts of the same total number of trees in ensemble.
5 CONCLUSION
In this paper, we introduce a new DNN architecture for deep learning on heterogeneous tabular data. The architecture is differentiable deep GBDTs, trained end-to-end via backpropagation. In extensive experiments, we demonstrate the advantages of our architecture over existing competitors with the default and tuned hyperparameters. layer into complex pipelines trained via back-propagation. For instance, in multi-modal problems, the NODE layer could be employed as a way to incorporate the tabular data, as CNNs are currently used for images, RNNs used for sequences.
promising research direction is incorporating the NODE
REFERENCES
Andreea Anghel, Nikolaos Papandreou, Thomas Parnell, Alessandro de Palma, and Haralampos Pozidis. Benchmarking and optimization of gradient boosting decision tree algorithms.
lñigo Barandiaran. The random subspace method for constructing decision forests. IEEE transac- fions pattern analysis and machine intelligence, 1998.
Leo Breiman. Random forests. Machine Learning, 45(1):5-32, 2001. doi: 10.1023/A: 1010933404324. URL /attpa///bbbb.ccrg/000022///2110
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Feng, Yang Yu, and Zhi-Hua Zhou, Multi-layered gradient boosting decision trees. In Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Pro- cessing Systems 2018, NeurIPS, 2018
Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pp. 1189-1232, 2001.
Ian Goodfellow. Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Vasyl Harasymiv. Lessons from 2 million machine learning models on kaggle, 2015.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv: 1803.05000 2018.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. CoRR, abs/1611..0144 2016. URL attp://abll uni -trier de/ de/db/ db/ httr.dddddd//oaaa.../ journals/ corr//orri 611 html #JangGP16

---

Published as a conference paper at ICLR 2020
Guolin Ke, Qi Meng. Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, pp. 1146-3154, 2017.
Guolin Ke, Jia Zhang, Zhenhui Xu, Jiang Bian, and Tie- Yan Liu. Tabnn: A universal neural network
solution for tabular data. 2018.
Ron Kohavi. Bottom-up induction of oblivious read-once decision graphs: strengths and limitations. In AAAI, 1994.
Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Bulo. Deep neural decision forests. In Proceedings of the IEEE international conference on computer vision, 2015.
Nathan Lay, Adam Harrison, Sharon Schreiber, Gitesh Dawer, and Adrian Barbu. Random hinge forest for differentiable learning. arXiv preprint arXiv: /802.03882, 2018.
Tianyi Lin, Zhiyue Hu, and Xin Guo. Sparsemax and relaxed wasserstein for topic sparsity. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, 2019.
Yin Lou and Mikhail Obukhov. Bdt: Gradient boosted decision tables for high accuracy and scoring efficiency. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017.
Jerry Ma and Denis Yarats. Quasi-hyperbolic momentum and adam for deep learning. arXiv preprint
2018
Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning, 2016.
Kevin Miller, Chris Hettinger, Jeffrey Humpherys, Tyler Jarvis, and David Kartchner. Forward thinking: building deep random forests. arXiv preprint arXiv:1705. 000000076666.0 2017.
Dmytro Mishkin and Jiri Matas. All you need is good init. In 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-I0), 2010.
Vlad Niculae and Mathieu Blondel. A regularized framework for sparse and structured neural atten- tion. In Advances in Neural Information Processing Systems, 2017.
Vlad Niculae, André FT Martins, Mathieu Blondel, and Claire Cardie. Sparsemap: Differentiable sparse structured inference. arXiv preprint arXiv: 880.04223, 2018.
Ben Peters, Vlad Niculae, and André T. Martins. Sparse sequence-to-sqquene models. In ACL, 20/9, pp. 1504-1519, 2019.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In Advances in Neural Information Processing Systems, pp. 6638-6643 2018
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Ira Shavitt and Eran Segal. Regularization learning networks: Deep learning for tabular datasets. In Advances in Neural Information Processing Systems, pp. 1389, 2018
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229-256, 1992.
Yongxin Yang, Irene Garcia Morillo, and Timothy M Hospedales. Deep neural decision trees. arXiv preprint arXiv: 06988, 2018
Zhi-Hua Zhou and Ji Feng. Deep forest: Towards an alternative to deep neural networks. In Proceed- ings of the Sixth International Joint Conference Artificial Intelligence, IJCAI, 2017.
10

---

Published as a conference paper at ICLR 2020
Train
Test
Features
Task
Metric
Description
Epsilor
400K
100K
2000
Classification
Error
PASCAL Challenge 2008
YearPredictior 6
463K
51.6K
90
Regression
MSE
Million Song Dataset
Higg
10.5M
500K
28
Classification
Error
UCI ML Higgs
Microsof
723K
241K
136
Regression
MSE
MSLR-WEBB10 WEB10K
Yahoo
544K
165K
699
Regression
MSE
Yahoo LETOR dataset
Click
800K
200K
11
Classification
Error
2012 KDD Cup
Table 5: The datasets used in our experiments.
A APPENDIX
A.1 DESCRIPTION OF THE DATASETS
In our experiments, we used six tabular datasets, described in Table 5 (1) Epsilon is high dimen- sional dataset from the PASCAL Large Scale Learning Challenge 2008, The problem a binary classification. (2) YcarPrediction is subset of Million Song Dataset. It is regression dataset, and the task is to predict the release year of the song by using the audio features. It contains tracks from 1922 to 2011. (3) Higgs is dataset from the UCI ML Repository. The problem is predict whether the given event produces Higgs bosons or not. (4) Microsoft a Learning to Rank Dataset. consists of 136-dimensional feature vectors extracted from query-url pairs. Each pair has rele- vance judgment labels, which take values from (irrelevant) to 4 (perfectly relevant) (5) Yahoo is very similar ranking dataset with query-url pairs labeled from to 4. We treat both ranking problems as regression (which corresponds to the pointwise approach to learning-to- (oriing)t-raak)) (6) Click is a subset of data from the 2012 KDD Cup. For the subset construction, we randomly sample 500.000 objects of positive class and 500.000 objects of negative class. The categorical features were converted to numerical ones via Leave-One-Out encoder from category package of the scikit-learn library.
A.2 OPTIMIZATION OF IYPERPPARAMETER
In order tune the hyperparameters, performed random stratified split of full training data into train set (80%) and validation set (20%) for the Epsilon, YcarPrediction, Higgs, Microsoft, and Click datasets. For Yahoo, we use train/val/test split provided by the dataset authors. We use the Hyperop library to optimize Catboost, XGBoost, and FCNN hyperparameters. For each method, we perform 50 steps of Tree-structured Parzen Estimator (TPE) optimization algorithm. we choose the set of hyperparameters, corresponding to the smallest loss the validation set.
final configuration,
A.2.1 CATBOOST AND XGBOOST
On each iteration Hyperopt, the number of trees was set based on the validation set, with maximal trees count set to 2048. Below is the list of hyperparameters and their search spaces for Catboost.
learning rate: Log-Uniforn distribution [e-5,
random :trengtht Discrete uniform distribution [1, 20]
one_hot.max_ste Discrete uniform distribution 25]
12 reg:: Log- Uniform distribution [1, 10]
bagging temperature: Uniform distribution [0,
leaf estimation iterations: Discrete uniform distribution [1, 10]

---

Published as a conference paper at ICLR 2020
XGBoost tuned parameters and their search spaces:
eta: Log-Unifoma distribution [e-?,1]
depthh Discrete uniform distribution [2, 10]
subsample: Uniform distribution [0.5,
colsample bytree: Uniform distribution [0.5, 1]
bylevel: Uniform distribution [0.5, 1]
hinn.chlld.wiightt Cog-Uniform distribution [e [e-16, ee]
alpha: Uniform choice {0. Log- Uniformf distribution (e 16 21]]
lambda: Uniform choice (0, Log- Log-Uniform distribution
16
E22 {²]
gamma: Uniform choice {0, Log-Uniform distribution
A.2.2 FCNN
Fully connected neural networks tuned using Hyperas library, which is Keras wrapper for Hyperopt. We consider FCNN constructed from the following blocks: Rense-Rel.U-Drroouut The number of units in each layer is independent of each other, and dropout value is the same for the whole network. The networks are trained with the Adam optimizer with averaging the model pa- rameters over c=5 consecutive checkpoints (Izmailov al. 2018) and early stopping on validation. Batch size is fixed to 1024 for all datasets. Below is the list of tuned hyperparameters.
Architecture: either Sequential (each layer is receives previous layer activations) or DenseNet (each layer receives activations from all previous layers
Number of layers: Discrete uniform distribution [2.7]
Number of units: Dicrete uniform distribution over set {128, 256, 512, 1024]
Learning rate: Uniform distribution [le 4, le 2]
Dropout: Uniform distribution [0, 0.5]
A.2.3 NODE
Neural Oblivious Decision Ensembles were tuned by grid search over the following hyperparameter values. In the multi-layer NODE, we use the same architecture for all layers, i.c., the same number of trees of the same depth. total total_tree_cour here denotes the total number of trees all layers. For each dataset, we use the maximal batch size, which fits in the GPU memory. We always use learning rate 10-3
nayJay:ss: {2, 4, 8}
total_tree trt.teeeeuut: { {1024, 2048]
tree depth: (6. 8}
tree tree_output dim: (2, 3}
http:://www.ccientt..dee. http::/www...ttt.........oooonna chin:/lawwmtoollsaaaattsshhaaam
"hups///rccivv..ice wwwp:iiiiiiiii..iiiiieeeeiee
10
"htt::/wwww..ddd.org/kdddd
hhups://github. https://githhh.cmmmaaapnmppppppo com/maxppmmerttaayyyrr...a
12


Reviews:
[{"review_id": "rkeiA3I19H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new method to make ensembles of decision trees differentiable, and trainable with (stochastic) gradient descent. The proposed technique relies on the concept of \"oblivious decision trees\", which are a kind of decision trees that use the same classifier (i.e. a feature and threshold) for all the nodes that have the same depth. This means that for an oblivious decision tree of depth d, only d classifiers are learned. Said otherwise, an oblivious decision tree is a classifier that split the data using d splitting features, giving a decision table of size 2^d. To make oblivious decision trees differentiable, the authors propose to learn linear classifiers using all the features, but add a sparsity inducing operator on the weights of the classifiers (the entmax transformation). Similarly, the step function used to split the data is replaced by a continuous version (here a binary entmax transformation). Finally, the decision function is obtained by taking the outer product of all the scores of the classifiers: [c_1(x), 1-c_1(x)] o [c_2(x), 1-c_2(x)] ... This \"choice\" operator transforms the d dimensional vectors of the classifier scores to a 2^d dimensional vector. Another interpretation of the proposed \"differentiable oblivious decision trees\" is a two layer neural network, with sparsity on the weights of the first layer,\nand an activation function combining the entmax transformation and the outer product operator. The authors then propose to combine multiple differentiable decision trees in one layer, giving the neural decision oblivious ensemble (NODE). Finally, several NODE layers can be combined in a dense net fashion, to obtain a deep decision tree model. The proposed method is evaluated on 6 datasets (half classification, half regression), and compared to existing decision tree methods such as XGBoost or CatBoost, as well as feed forward neural networks.\n\nThe paper is clearly written, ideas are well presented, and it is easy to follow the derivation of the method. As a minor comment, I would suggest to the authors to give more details on the EntMax method, as it is quite important for the method, but not really introduced in the paper. The proposed algorithm is sound, and a nice way to make decision trees differentiable. One concern that I have though, is that it seems that NODE are close to fully connected neural networks, with sparsity on the weights. Indeed, I think that there are two ingredients in the paper to derive the method: adding sparsity to the weights and the outer product operator (as described in the previous paragraph). In particular, the improvement over vanilla feed forward neural networks seem small in the experimental section. I thus believe that it would be interesting to study if both two differences with feed forward networks are important, or if only is enough to get better results.\n\nTo conclude, I believe that this is a well written paper, proposing a differentiable version of decision trees which is interesting. However, the proposed method relies on existing techniques, such as EntMax, and I wonder if the (relatively small) improvement compared to feed forward network comes from these. I believe that it would thus be interesting to compare the method with feed forward network with sparsity on the weights. For now, I am putting a weak reject decision, but I am willing to reconsider my rating based on the author response.\n\nQuestions to the authors:\n(1) do you use the same data preprocessing for all methods (quantile transform)?\n(2) would it make sense to evaluate the effects of each the entmax and the outer product operator separately in the context of fully connected networks?\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJepxs90tS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary:\n\nThe paper considers training oblivious trees ensemble with gradient descent by introducing a relaxation for feature selection and node thresholding. The relaxation is based on the recently introduced EntMax. The approach is compared with standard gradient boosting tree learning on benchmark datasets.\n\nReview Summary:\n\nThe paper reads well, is technically sound. The approach is novel and relevant to ICLR. Reference to related work are appropriate. Experimental comparison with CatBoost, neural nets could be more rigorous, more ablations could give a complete picture. Overall this is a good paper that gives an extra tool applicable to many practical settings.\n\nDetailed Review:\n\nThe introduction needs to define \"tabular data\". In your case, it seems that you mean mostly numerical heterogeneous features. Could you comment on using categorical features as well? \n\nThe method is clearly explained and references are appropriate, so most of my questions relate to the empirical setup and results.\n\nFirst, it seems to me that the paper would be much stronger if you were to reproduce the results from an established paper. If you take the catboost paper (arXiv:1706.09516v5 [cs.LG] 20 Jan 2019), the error on epsilon dataset is 10.9 which is better than the number your report, similarly click reports 15.6 error rate. To me, the paper would be much better if you simply added an FCNN and a NODE column to Table 2 and 3 of the catboost paper. It does not mean that your approach has to be better in all cases, but it will give a clear picture of when it is useful and it would clear any doubt on the tuning of the catboost baseline.\n\nSecond, the model you propose builds upon the densenet idea while the FCNN you compare with has no densenet connections. It would be fairer to consider neural net with this kind of residual.\n\nThird, I feel you need to report results over CPU as well. Boosted trees primary advantage is their low cost on regular CPU, the entmax formulation requires integrating over more leaves \nthan typical thresholded trees and it would be interesting to compare the effect on CPU. Reporting timing with batch and individual sample evaluation would make sense as well.\n\n As a side note, I would advise to define entmax with its equation. It is too recent to consider it should be known by the reader.\n\nOverall, this is a good paper than reads well. The method is novel, interesting and practical. With the extra experiments, it would make an excellent ICLR paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgbSpbaKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tries to ask if there is a good neural net architecture that works as effectively as gradient boosting decision trees on tabular data. The authors propose an architecture (NODE) that satisfies this conditions. NODE is an architecture consisting of differentiable oblivious decision trees that can be trained end to end via back propagation. The paper is readable and the experiments are well presented. They make use of an alpha-entmax transformation to obtain a differentiable architecture. The approach seems well motivated in the literature. It is unclear how novel the contribution is. It is unclear if in the experimental section the datasets used are standard for this classes of tasks. Would be good to mention if it is the case. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Neural Oblivious Decision Ensembles, a formulation of ensembles of decision trees that is end-to-end differentiable and can use multi-layer representation learning. The reviewers are in agreement that this is a novel and useful tool, although there was some mild concern about the extent of the improvement over other methods. Post-discussion, I am recommending the paper be accepted.
Decision: Accept (Poster)