Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Aim to improve the interpretability and the accuracy of the neural network, this paper takes a step further on the integration of NN with a decision tree. It will replace the final linear layer of the NN with a decision tree induced by pre-trained model weights. It takes advantage of both hard and soft decision trees and designs suitable tree supervision loss thereon. Extensive experiments verify the design choice of the proposed components. On both small-scale and large-scale datasets, it beats the decision tree counterparts. Also, on the aspects of generalization and interpretability, it shows the strength compared to NN.

This work is a good try to combine the two techniques NN and decision tree. It finally makes the combination to achieve comparable accuracy with the NN and also enjoy the benefit in the aspects of generalization and interpretability. Recent SOTA of capsule networks which are based on the NN backbone and this work are both achieved comparable performance with NN. They show a promising direction for studying representation learning. Researchers can delve deeper based on this work to further exploit how to integrate decision tree into NN and the characteristics of the combination (e.g. adversarial examples). 

With the decision tree, we can visualize the decision process the bring the benefits of interpretability. The paper proposes to label the decision nodes with WordNet and show the applications of zero-shot generalization, high-level concepts, dataset debugging, and improved human trust. There are lots to do on the aspects. Also, the zero-shot and high-level concept experiments are really intriguing. Using the pre-trained model weights to construct the tree and the proposed tree losses to train can help the generalization in such a significant way, though the performance would depend on the accuracy of the superclasses labeling and the agglomerative clustering. Where the benefits come from? The method is only used the same information as the NN and the tree is also constructed based on the pre-trained weights. Does the way of making hierarchy decisions help here? If you do not enforce the second term of the equation (3), will the phenomenon be the same?

Overall, the paper is very easy to follow and the figures really help understanding. Extensive experiments help to know the performance, effectiveness of the proposed components, and also its unusual applications.

----------------------------
Some concerns and comments are listed below:

Will you update the weights of the intermediate nodes? 

On large-scale datasets, the paper currently only tests using the EfficientNet. The reviewer wonders if the author can use more advanced backbones to see the performance changes.

The reviewer is unsure of the specific way to label the decision nodes. Will you use the wordvec provided in the wordnet and compare it with the decision nodes' feature? Since your structure is different from the WordNet, how do you match the classes with the nodes?

-----------------after rebuttal------------

I would like to keep my origin score due to the pros listed above.
review_title: Take a step further on integrating the NN and Decision Tree

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a neural-backed decision tree that aims to improve both the accuracy and the interpretability of deep learning models. Training under a newly introduced tree supervision loss, the authors show that NBDTs can outperform and out-generalize some modern architectures on several image datasets.

Overall this paper is well written and established. The idea of using a differentiable oblique decision tree to replace the final linear layer is interesting. The authors provide clear illustration of the procedure and promising experimental results.

Questions:

1. What is the main intuition that NBDTs can outperform the original network?
2. Given the classes are in the leaves, does the ordering of classes in the leaf layer matter? How should one determine which two classes are in the same bottom subtrees?

Minor comments:

1. Figure 2 Step A: y_d -> y_k or y_4.
2. Why are there many n/a results in Table 1?
3. Section 3.1 in the Compute node probabilities paragraph, the definition of r_i seems confusing.
4. What does NN mean in Table 1? How different is it from CNN-RNN?
review_title: A Nice Paper

final_score: rating: 7: Good paper, accept
review_content: The authors did a fantastic job of answering questions, revising their manuscript in accord with reviewer feedback (Sec 3.4 title), and even adding new experimental results based on reviewer suggestions (mid-training hierarchy) and reflecting best practices in interpretability research. I was really impressed by their nimbleness and responsiveness. I will raise my score to a 7: I think this is a very solid paper and excellent research effort around a nascent idea. In particular, I think its impact is limited by

- its close coupling to naturally hierarchical problems, e.g., multi-class classification with a taxonomy
- its close coupling to image data and tasks
- its heuristic nature: fully train neural net, infer hierarchy via clustering, retrain neural net, then map a priori labels onto inferred hierarchy

The "10" version of this paper (maybe future work?) would propose a way to infer the hierarchy on the fly and show how to apply it onto other kinds of data and problems with different structures.

-----

This submission proposes a modification of neural networks that replaces the "final linear layer with a decision tree." The term "decision tree" is applied somewhat loosely to a hierarchical neural architecture akin to a hierarchical softmax. In the current work (as I understand it), this hierarchy is induced from a pre-trained multi-output, e.g., multiclass, neural network via a hierarchical clustering and subsequent averaging of the output weights. At inference time, path probabilties can be computed based on the chain rule. Predictions can be made based on either a greedy traversal of the tree (choosing the most likely child at each step, a la hierarchical softmax) or by choosing the most probable leaf, which requires computing all path probabilities. Empirical results across three standard image datasets are suggestive, if not conclusive, and the paper concludes with some interesting, albeit cursory, examples of potential "interpretability" applications.

The submission summarizes its contributions at the end of Section 1 as follows:
1. It proposes a tree-structured loss to augment supervised neural network training (predominantly for multiclass classification problems).
2. It describes a heuristic to induce a hierarchy in the output weights of a pre-trained multi-output neural network, enabling decision tree-like inference and provide evidence it is more effective than other approaches for inducing hierarchies.
3. It presents simple case studies of how the induced hierarchy can be used for traditional "interpretability" tasks, like debugging and generating explanations.

I appreciate the idea at the center of this paper -- adding simple hierarchical structure to a multi-output neural network, with the aim of increased interpretability -- but I feel the work as it is presented is nascent and the manuscript itself is flawed. I lean toward rejection at the moment, but I could be persuaded to change my mind by some combination of solid revisions, convincing author response, or vociferous advocacy from other reviewers.

I will briefly extol the paper's strengths before providing a longer discussion of what I consider to be its key weakness. First, I really like the last sentence in the paper:

"This challenges the conventional supposition of a dichotomy between accuracy and interpretability, paving the way for jointly accurate and interpretable models in real-world deployments."

Weaknesses in the evaluation of its interpretability claims aside, I agree with this statement. I think the case studies presented do provide evidence of improved interpretability alongside small accuracy improvements. I think this paper does succeed in demonstrating that accuracy and interpretability are not necessarily competing objectives, at least for certain tasks (multiclass classification of images).

A laundry list of other strengths:

- The motivation is strong (modulo weakness discussed below): there is a growing need to provide human-understandable insights into decisions made by complex machine learning models.
- The proposed approach is simple and elegant, easy to implement, and empirically effective. I'm quite impressed that the proposed tree loss appears to improve accuracy (!) on multiple tasks.
- I also think this paper lays groundwork for a direction of research that the community could continue to build on.

I think that the manuscript's largest flaw, ironically, regards interpretability, its primary motivation. The work's central claim is that the tree-structured decision layer delivers improved interpretability with comparable or slightly improved accuracy. In its discussion of this claim, the manuscript provides no precise definition of "interpretable," making it difficult to verify the claim qualitatively or quantitatively. Section 5 presents a vignette of case studies, but the discussion of each is quite limited. In particular, none of the use cases is fully motivated or placed in the context of previous research on interpretability definitions [1][2]. The cursory presentation of results for each do the results a disservice by making it difficult for the reader to recognize and assess their significance.

To quote the introduction from Lipton's _The Mythos of Model Interpretability_ [1],

"Despite the absence of a definition, papers frequently make claims about the interpretability of various models. From this, we might conclude that either: (i) the definition of interpretability is universally agreed upon, but no one has managed to set it in writing, or (ii) the term interpretability is ill-defined, and thus claims regarding interpretability of various models may exhibit a quasi-scientific character."

I believe the paper would be strengthened by focusing on one use case, e.g., debugging or human trust, using the ~1 page dedicated to Section 5 to motivate it more fully and to present the results in detail. If the primary use case is generalization or debugging, then I suggest designing a quantitative analysis so defend against claims of cherry picking the best results (a common problem in presenting "example" interpretability results).

Section 5.4 includes a quantitative evaluation, but I question whether mere human preference is evidence of "human trust." More recent research on trust appear to use more elaborate studies in which trust is measured by subjects' rate of success in performing a particular task aided by the machine learning model [3].

I want to caveat the above: I really appreciate this line of work and think it has value. There is an ongoing discussion in our community about rewarding good ideas, rather than punishing imperfect or incomplete execution. I also acknowledge that I am far from an expert in the latest interpretability research. Nonetheless, my understanding is that interpretability researchers have grown more skeptical of interpretability claims about new methods absent a rigorous framework (definitional and/or experimental) for evaluating those claims.

When I read this paper, I find it hard to escape the conclusion that its interpretability claims rest on the presupposition that trees are naturally more interpretable (and further that readers will accept this dogma). I disagree with this assertion (see below), but even if it were generally true, I still think the paper would be strengthened by adding a more rigorous discussion and analysis of its claims. Propose a definition or criterion (see [1][2] for ideas), ideally one that could be assessed qualitatively and evaluated empirically, then apply it.

Regarding the claim about trees in Section 5: "The interpretability of a decision tree is well-established when input features are easily understood (e.g. tabular data in medicine or finance)."

I would dispute that this is "well-established" for anything but the simplest decision tree models, with a single tree consisting of a small number of splits using a handful of features, which are rare in realistic settings. The most commonly used tree-structured models (gradient boosted decision trees and random forests) are not readily interpretable, even for tabular data and especially for high dimensional inputs. This has made research like SHAP [4] of great interest to practitioners.

What is more, even for tabular data, the neural decision trees described in this paper are (to my understanding) basically a cascade of linear classifiers, with split each having access to all features at once. This does do not lend itself to the same kind of "interpretation" one gets for classic decision trees that use one feature per split. With even modestly deep hierarchies, the resulting "explanations" would rapidly become quite complex.

I see one other weakness in the proposed method itself: as I understand things, it requires access to a pretrained neural network. At the very least, one needs pre-existing output weights to cluster in order to induce a hierarchy -- and the induced hierarchy is a necessary component in the presented results. This isn't a fatal flaw -- learning a hierarchy on the fly could be left for future work. Nonetheless, it limits the work's usefulness and potential impact.

What is more, I don't think the manuscript is sufficiently clear about this requirement: on my first pass through the paper, I came away with the impression that there was a way to learn the hierarchy while training the neural net -- the inclusion of a section entitled "Training with Tree Supervision Loss" seems to imply this. I suggest revising the text to make it crystal clear that it is not possible to use the tree loss to train a neural net from scratch -- at least, not without a predefined hierachy (perhaps from a previous training run or prior knowledge).

I will now summarize the improvements I suggest for strengthening the manuscript:
1. Focus on one definition of interpretability and then analyze central claims through that lens. Introduce it early in the paper (introduction) and then dedicate Section 5 to it, rather than trying to cover lots of use cases superficially.
2. Make the limitations of the proposed approach VERY clear. In particular, you need a predefined hierarchy to train with the tree loss, and you need pretrained neural net (or pre-existing weights, at least) to induce a hierarchy based on clustering.
3. If the intention is for this approach to be used exclusively for finetuning or adapting an existing neural network, then this should be made clear in the text. Consider renaming Section 3.3.
4. Justify (or reword) statements like "the interpretability of a decision tree is well-established" or "neural features are visually interpretable" (a single reference does not suffice...the Olah distill survey draws no such definitive conclusions).

I have a few questions:
- One thing that is not clear: when training with tree loss, are weights shared across nodes? In particular, the weight vector for an inner node is the average of its descendent leaf node weight vectors. When training with tree loss, do we then treat that inner node weight vector as a set of independent parameters with separate updates? Or do we continue to treat it as a sum of leaf parameters, so that leaf and inner node updates affect the same parameters, as in an RNN or recursive network.
- Is there possibly a heuristic that could approximate "learning the hierarchy?" For example, train with a basic loss for enough iterations until the output weights start to converge, then pause training to induce the hierarchy and then resume training with the tree loss. Part of the heuristic could be guidance about how to detect when the output weights have sufficiently converged.
- What are the key differences between this approach and a hierarchical softmax? My understanding is that they're basically equivalent at inference time (except maybe traditional hierarchical softmax uses hard decisions?). What about during training? Is it maybe the use of negative sampling for hierarchical softmax?
- How would this approach perform for extremely high dimensional output spaces -- one of the primary motivations for hierarchical softmax? I imagine that for some output cardinality, "soft" inference becomes computationally infeasible.

[1] Lipton. The Mythos of Model Interpretability. https://arxiv.org/abs/1606.03490.
[2] Doshi-Velez and Kim. Towards A Rigorous Science of Interpretable Machine Learning. https://arxiv.org/abs/1702.08608.
[3] Poursabzi-Sangdeh, et al. Manipulating and Measuring Model Interpretability. https://arxiv.org/abs/1802.07810.
[4] Lundberg, et al. From local explanations to global understanding with explainable AI for trees. https://www.nature.com/articles/s42256-019-0138-9.
review_title: Compelling interpretability methodological work but minor flaws in motivation, lack of discussion about practical limitations

final_score: rating: 6: Marginally above acceptance threshold
review_content: Significance:
This article seems to be a useful contribution to the literature on interpretable deep networks. However, the paper could be strengthened by demonstrating and analyzing the interpretability of approaches to other types of data, such as sequential data.

Novelty:
This paper's main contribution is to offer a new hybrid model that combines a deep neural network with a tree. The authors used the weights of the last layer of a DNN to build a tree from bottom to top, where each inner node in the tree is the average weight of each child. Then, the authors used softmax to compute the probability of  routing for each child. 

Potential Impact:
The approach presented in this paper is well-evaluated in computer vision but potentially useful in many other settings.

Technical Quality:
+ The technical content of the paper appears to be correct, albeit there is some room for improvement. 
+ Page 2, the authors said, "These models likewise limit interpretability by supporting no more than depth-2 trees." Having the depth-2 tree actually improves the interpretability since it is easier to follow the model prediction. For example, a tree of depth-2 considers more interpretable than a tree of depth-4. The authors should rephrase this sentence.
+ While the authors claim that linearly increasing the weight in NBDT is the superior method, why the NBDT with the constant rate overperforms on CIFAR-10 as shown in Table 3?
+ Based on the NBDT's explanation, all leaf nodes should have the same depth, but in the example shown in Figure 6 and supplement, the leaf nodes are in different depths. The authors need to explain why the final tree has leaf nodes with different depths.
+ The authors did not compare their method's interpretability with a similar methods, such as Adaptive Neural Trees (Tanno et al. 2019).  I suggest running this experiment since Adaptive Neural Trees has a different interface than NBDT. . While the number of participants is noticeable in the interpretability study, it seems that participants only answer one question. Adding more questions could strengthen the paper. Further, the authors should provide a summary of participants (e.g., age, education, and gender).
+ The paper will be strengthened if the authors run an experiment without using a pre-trained neural network on a small dataset like MNIST to demonstrate their algorithm's effectiveness. 

Presentation/Clarity
+ While the paper is fairly readable, there is room for improvement in the clarity.
+ Page 3, the last paragraph forgot a period after the parentheses. "path (Figure 1 C, Appendix Table 6) This ..."
+ Page 7, figure 4, I believe that the authors mean "without NBDT" instead of "without ResNet."
+ While the authors explained how they label the tree nodes with WordNet in the supplement, there is no explanation in the main paper. Labeling the tree's nodes is an important part of the algorithm and should be included in the main paper.
+ Page 5, the authors used "Hierarchy Ablation" and "Loss Ablation" subtitle in the paper. The word "Ablation" seems inappropriate in this context.

Reproducibility
The paper describes all the algorithms in full detail and provides enough information for an expert reader to reproduce its results. However, the authors did not discuss when they start to increase the W in equation 3, how to determine to stop the W from increasing, and with what rate the W should be increased.
review_title: The paper presents an algorithm (NBDT) for interpretable image classification, demonstrating a higher preference rate (65.9%) when compared to a saliency map for the model explanation among 374 participants.  

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a method to make neural networks more accurate and interpretable by replacing their final layers with a probabilistic decision tree. As a result, the network can produce a sequence of decisions that leads to the final classification result, given an input image. The method is trained with soft decisions by assigning probabilities to each leaf, which are associated with a single class. The tree decision hyperplanes are constructed automatically from the backbone networks final dense layer and finetuned. The fact that decisions are soft solves the differentiablility problem of decisions as in various other similar papers, cited or uncited (more below).

The paper is not written very clearly, so it would be hard to reproduce. It's not clear in places if indices correspond to nodes or classes, as it is used interchangeably. The text misses a proper mathematical formulation of the operations done in inner nodes, and this all makes it difficult to understand what the loss is and how it can update the decisions in the tree. Perhaps it's possible to understand all the details by re-reading the text several times, but the paper definitely lacks clarity. A nice comparison would be with the Deep Neural Decision Forest (DNDF) paper. I'd expect that level of clarity from an ICLR paper.

The justification behind using the hyperplanes from the last layer of the CNN for constructing inner nodes is not explained. The visualization in Figure 2 indicates that the averages used for the parent nodes act like clusters - and somehow averaging them forms bigger clusters, but what really happens is that the rows of the final layer, which are unnormalized hyperplanes, are averaged to form new hyperplanes that are assumed to cover both classes, which would not be the case most of the time. I'm not sure if this has a reasonable geometric meaning, but the visualization gives the wrong idea.

The paper makes the claim: "Unlike previous decision trees or hierarchical classifiers, NBDTs use path probabilities for inference".
There are certainly many papers that use path probabilities for inference. In fact it's been the norm for discrete decisions, since hard decisions are much harder to formulate in a differentiable manner. DFDN uses path probabilities, as do older papers like "Decision Forests, Convolutional Networks and the Models in-Between" by Ioannou et al. (uncited). I don't understand this claim.
The biggest difference here is that each class gets exactly one dedicated leaf, instead of each leaf storing a distribution. It is not clear to me why this is a good idea though - it's clearly not preferred in the decision tree literature.

Another interesting point - most papers I know on the subject actually try to enable sparse activation at test time for efficiency, which is the harder problem to solve. See CondConv (Neurips 2019 - uncited), SplineNets (Neurips 2018 - uncited), Outrageously Large Neural Networks by Shazeer et al. (uncited), Conditional Information Gain by Bicici et al., etc. Activating all the branches of the tree or graph becomes prohibitively costly for deeper trees.

Interpretability is another major claim. The way it works is by using WordNet to assign higher level compound classes to images, such as Animal -> Turtle. This way by grouping similar leaves together, the inner nodes are assigned a meaning. This needs to be done when constructing the hierarchy, so the tree structure is manually given from what I understand. But then how are the pairs of nodes selected? It's not clear from the text, at least it is not explained clearly. In the Figure 5, a decision tree is given, which could only have been constructed by hand - i.e. with someone knowing that cats and dogs are the closest pairs. Where does this information come from for ImageNet when there are 1000 classes? WordNet?

Another concern about interpretability is that it is claimed that other good performing methods like DFDN are not interpretable. I don't see why that's the case if we use WordNet to assign meaning for the inner nodes of such solutions. I'd agree that it is not straightforward to do so, but with WordNet I can imagine how it could be done. So I'm not convinced about the claim that this solution is the most interpretable one when the paper does not explain why other solutions aren't in a persuasive manner.

One final concern is with the comparisons with similar models. The paper says these were reimplemented and tested with different backbones. Where are these implementations taken from, for instance for DFDN, which is quite complex? How can the reader trust these numbers?

I find the claims unconvincing and the results unpersuasive. I think the paper needs i) a better mathematical formulation to clarify the method, ii) better explanation of how the trees are constructed (e.g. if WordNet is used), iii) better understanding of differences from similar work. Currently I don't think it meets the bar for ICLR.
 

TreeSup result is likely wrong in Table 3.

After Rebuttal:

The authors did a great job in addressing most of the issues I have, and made many changes that helped with the clarity of the paper. There are still some remaining issues like Figure 2 assigning a wrong geometric meaning to the clusters formed by taking means of hyperplanes, and the uncited references, which are simply added to the references section (which should be fixed). But I think the added survey results are a great addition and a persuasive proof about the increased interpretabililty of these models. Therefore I'll increase my score to 6.

review_title: Interesting results but unconvincing claims.

Meta-review: This paper initially received mixed ratings but after the rebuttal, all reviewers recommended acceptance. Reviewers appreciate the novel technical ideas and extensive experimental results. 
Decision: Accept (Poster)