Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper introduces a new method to make ensembles of decision trees differentiable, and trainable with (stochastic) gradient descent. The proposed technique relies on the concept of "oblivious decision trees", which are a kind of decision trees that use the same classifier (i.e. a feature and threshold) for all the nodes that have the same depth. This means that for an oblivious decision tree of depth d, only d classifiers are learned. Said otherwise, an oblivious decision tree is a classifier that split the data using d splitting features, giving a decision table of size 2^d. To make oblivious decision trees differentiable, the authors propose to learn linear classifiers using all the features, but add a sparsity inducing operator on the weights of the classifiers (the entmax transformation). Similarly, the step function used to split the data is replaced by a continuous version (here a binary entmax transformation). Finally, the decision function is obtained by taking the outer product of all the scores of the classifiers: [c_1(x), 1-c_1(x)] o [c_2(x), 1-c_2(x)] ... This "choice" operator transforms the d dimensional vectors of the classifier scores to a 2^d dimensional vector. Another interpretation of the proposed "differentiable oblivious decision trees" is a two layer neural network, with sparsity on the weights of the first layer,
and an activation function combining the entmax transformation and the outer product operator. The authors then propose to combine multiple differentiable decision trees in one layer, giving the neural decision oblivious ensemble (NODE). Finally, several NODE layers can be combined in a dense net fashion, to obtain a deep decision tree model. The proposed method is evaluated on 6 datasets (half classification, half regression), and compared to existing decision tree methods such as XGBoost or CatBoost, as well as feed forward neural networks.

The paper is clearly written, ideas are well presented, and it is easy to follow the derivation of the method. As a minor comment, I would suggest to the authors to give more details on the EntMax method, as it is quite important for the method, but not really introduced in the paper. The proposed algorithm is sound, and a nice way to make decision trees differentiable. One concern that I have though, is that it seems that NODE are close to fully connected neural networks, with sparsity on the weights. Indeed, I think that there are two ingredients in the paper to derive the method: adding sparsity to the weights and the outer product operator (as described in the previous paragraph). In particular, the improvement over vanilla feed forward neural networks seem small in the experimental section. I thus believe that it would be interesting to study if both two differences with feed forward networks are important, or if only is enough to get better results.

To conclude, I believe that this is a well written paper, proposing a differentiable version of decision trees which is interesting. However, the proposed method relies on existing techniques, such as EntMax, and I wonder if the (relatively small) improvement compared to feed forward network comes from these. I believe that it would thus be interesting to compare the method with feed forward network with sparsity on the weights. For now, I am putting a weak reject decision, but I am willing to reconsider my rating based on the author response.

Questions to the authors:
(1) do you use the same data preprocessing for all methods (quantile transform)?
(2) would it make sense to evaluate the effects of each the entmax and the outer product operator separately in the context of fully connected networks?

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: Paper Summary:

The paper considers training oblivious trees ensemble with gradient descent by introducing a relaxation for feature selection and node thresholding. The relaxation is based on the recently introduced EntMax. The approach is compared with standard gradient boosting tree learning on benchmark datasets.

Review Summary:

The paper reads well, is technically sound. The approach is novel and relevant to ICLR. Reference to related work are appropriate. Experimental comparison with CatBoost, neural nets could be more rigorous, more ablations could give a complete picture. Overall this is a good paper that gives an extra tool applicable to many practical settings.

Detailed Review:

The introduction needs to define "tabular data". In your case, it seems that you mean mostly numerical heterogeneous features. Could you comment on using categorical features as well? 

The method is clearly explained and references are appropriate, so most of my questions relate to the empirical setup and results.

First, it seems to me that the paper would be much stronger if you were to reproduce the results from an established paper. If you take the catboost paper (arXiv:1706.09516v5 [cs.LG] 20 Jan 2019), the error on epsilon dataset is 10.9 which is better than the number your report, similarly click reports 15.6 error rate. To me, the paper would be much better if you simply added an FCNN and a NODE column to Table 2 and 3 of the catboost paper. It does not mean that your approach has to be better in all cases, but it will give a clear picture of when it is useful and it would clear any doubt on the tuning of the catboost baseline.

Second, the model you propose builds upon the densenet idea while the FCNN you compare with has no densenet connections. It would be fairer to consider neural net with this kind of residual.

Third, I feel you need to report results over CPU as well. Boosted trees primary advantage is their low cost on regular CPU, the entmax formulation requires integrating over more leaves 
than typical thresholded trees and it would be interesting to compare the effect on CPU. Reporting timing with batch and individual sample evaluation would make sense as well.

 As a side note, I would advise to define entmax with its equation. It is too recent to consider it should be known by the reader.

Overall, this is a good paper than reads well. The method is novel, interesting and practical. With the extra experiments, it would make an excellent ICLR paper.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper tries to ask if there is a good neural net architecture that works as effectively as gradient boosting decision trees on tabular data. The authors propose an architecture (NODE) that satisfies this conditions. NODE is an architecture consisting of differentiable oblivious decision trees that can be trained end to end via back propagation. The paper is readable and the experiments are well presented. They make use of an alpha-entmax transformation to obtain a differentiable architecture. The approach seems well motivated in the literature. It is unclear how novel the contribution is. It is unclear if in the experimental section the datasets used are standard for this classes of tasks. Would be good to mention if it is the case. 
review_title: Official Blind Review #1

Meta-review: This paper proposes Neural Oblivious Decision Ensembles, a formulation of ensembles of decision trees that is end-to-end differentiable and can use multi-layer representation learning. The reviewers are in agreement that this is a novel and useful tool, although there was some mild concern about the extent of the improvement over other methods. Post-discussion, I am recommending the paper be accepted.
Decision: Accept (Poster)