NBDT: Neural-Backed Decision Tree
Authors: Alvin Wan
Lisa Dunlap
Daniel Ho
Jihan Yin
Scott Lee
Suzanne Petryk
Sarah Adel Bargal
Joseph E. Gonzalez
Venue: ICLR 2021 Poster (2021)
Keywords: explainability
computer vision
interpretability

Abstract: Machine learning applications such as finance and medicine demand accurate and justifiable predictions, barring most deep learning methods from use. In response, previous work combines decision trees with deep learning, yielding models that (1) sacrifice interpretability for accuracy or (2) sacrifice accuracy for interpretability. We forgo this dilemma by jointly improving accuracy and interpretability using Neural-Backed Decision Trees (NBDTs). NBDTs replace a neural network's final linear layer with a differentiable sequence of decisions and a surrogate loss. This forces the model to learn high-level concepts and lessens reliance on highly-uncertain decisions, yielding (1) accuracy: NBDTs match or outperform modern neural networks on CIFAR, ImageNet and better generalize to unseen classes by up to 16%. Furthermore, our surrogate loss improves the original model's accuracy by up to 2%. NBDTs also afford (2) interpretability: improving human trustby clearly identifying model mistakes and assisting in dataset debugging. Code and pretrained NBDTs are at https://github.com/alvinwan/neural-backed-decision-trees.

Published as a conference paper at ICLR 2021
NBDT: NEURAL-BACKED DECISION TREE
Alvin Wan,, Lisa Dunlapi, Daniel Hoj, Jihan Yin,, Scott Lee Suzanne Petryki, Sarah Adel Bargalz, Joseph F. Gonzalez UC Berkeley1, Boston University2 {alvinwan, ddunlap, daninlhe, jihan.yin, scoct leo. operty,,,,,, web.ggllbuedd..
[egonzal] | @barkaley.
ABSTRACT
Machine learning applications such as finance and medicine demand accurate and justifiable predictions, barring most deep learning methods from use. response. previous work combines decision trees with deep learning, yielding models that (1) sacrifice interpretability for accuracy (2) sacrifice accuracy for interpretabil- ity. We forgo this dilemma by jointly improving accuracy and interpretability us- ing Neural-Backed Decision Trees (NBDTs). NBDTs replace neural network's s final linear layer with differentiable sequence of decisions and surrogate loss. This forces the model to learn high-level concepts and lessens reliance on highly- uncertain decisions, yielding (1) accuracy: NBDTs match outperform modern neural networks on CIFAR, ImageNet and better generalize to unseen classes by up 16%. Furthermore, our surrogate loss improves the original model's accu- racy by up to 2%. NBDT's also afford (2) interpretability: improving human trust by clearly identifying model mistakes and assisting in dataset debugging. Code and pretrained NBDTs are at github.com/alviiie www.aaaawwaaaaaaaaaaaaeeeeeeeeee
INTRODUCTION
Many computer vision applications (e.g. medical imaging and autonomous driving) require insight into the model's 's decision process, complicating applications of deep learning which are tradition- ally black box. Recent efforts in explainable computer vision attempt to address this need and can be grouped into one of two categories: (1) saliency maps and (2) sequential decision processes. Saliency maps retroactively explain model predictions by identifying which pixels most affected the prediction. However, by focusing on the input, saliency maps fail to capture the model's decision making process. For example, saliency offers no insight for misclassification when the model is "looking" at the right object for the wrong reasons. Alternatively, we can gain insight into the model's s decision process by breaking up predictions into a sequence of smaller semantically mean- ingful decisions as in rule-based models like decision trees. However, existing efforts to fuse deep learning and decision trees suffer from (1) significant accuracy loss, relative to contemporary mod- els (e.g., residual networks), (2) reduced interpretability due accuracy optimizations (e.g.. impure leaves and ensembles), and (3) tree structures that offer limited insight into the model's 's credibility.
To address these, propose Neural-Backed Decision Trees (NBDTs) to jointly improve both (1) accuracy and (2) interpretability of modern neural networks, utilizing decision rules that pre- serve (3) properties like sequential, discrete decisions; pure leaves; and non-ensembled predictions. These properties in unison enable unique insights, as we show. We acknowledge that there is no universally-acceptedd definition of interpretability (Lundberg al. 2020 Doshi- Velez & Kim 2017 Lipton 2016). so to show interpretability, we adopt a definition offered by (Poursabzi-Sangeel et al. (2018: A model is interpretable if a human can validate its prediction, determining when the model has made a sizable mistake. We picked this definition for its importance to downstream benefits we can evaluate, specifically (1) model or dataset debugging and (2) improving human trust. To ac- complish this, NBDTs replace the final linear layer of neural network with a differentiable oblique decision tree and, unlike its predecessors (i.e. decision trees, hierarchical classifiers), uses a hierar- chy derived from model parameters, does not employ hierarchical softmax, and can be created from any existing classification neural network without architectural modifications. These improvements
denotes equal contribution

---

Published as a conference paper at ICLR 2021
tailor the hierarchy to the network rather than overfit to the feature space, lessens the decision tree's 's reliance on highly uncertain decisions, and encourages accurate recogmition of high-level concepts. These benefits culminate in joint improvement of accuracy and interpretability. Our contributions:
1. We propose a tree supervision loss, yielding NBDTs that match/outperform and out- generalize modern neural networks (WideResNet, EfficientNet) on ImageNet, Tinylma- geNet200, and CIFARI00. Our loss also improves the original model by up to 2%.
2. We propose alternative hierarchies for oblique decision trees induced hierarchies built using pre-trained neural network weights that outperform both data-based hierarchies (e.g. built with information gain) and existing hierarchies (e.g. WordNet), in accuracy.
3. We show NBDT explanations
more helpful the user when identifying model mis- takes, preferred when using the model to assist in challenging classífication tasks, and can be used to identify ambiguous ImageNet labels.
RELATED WORKS
Saliency Maps. Numerous efforts (Springenberg et al. 2014 Zeiler & Fergus 2014 Simonyan
iet al., 2013| Zhang et al.
2016; Selvaraju et al.] 2017; Ribeiro et al.] [016 Petsiuk et al. 2018
Sundararajan et al. 2017) have explored the design of saliency maps identifying pixels that most in-
fluenced the model prediction. White-box techniques (Springenberg et al. 2014 Zeiler & Fergus
2014 Simonyan et al. 2013 Selvaraju et al. 2017 Sundararajan et al.) 201 use the network's 's pa-
rameters to determine salient image regions, and black-box techniques (Ribeiro et al. 2016 Petsiuk
et al. 2018) determine pixel importance by measuring the prediction's 's response to perturbed inputs. However, saliency does not explain the model's 's decision process (e.g. Was the model confused early on, distinguishing between Animal and Vehicle? Or is it only confused between dog breeds?).
Transfer to Explainable Models. Prior to the recent success of deep learning, decision trees were state-of-the-art on a wide variety of learning tasks and the gold standard for interpretability. Despite this recency, study the intersection of neural network and decision tree dates back three decades, where neural networks were seeded with decision tree weights (Banerjee] 1990, 1994; Ivanova & Kubat 1995a b), and decision trees were created from neural network queries (Krishnan al. [999)
Boz 2000 Dancey et al., 2004 Craven & Shavlik 1996: 1994), like distillation (Hinton et al. 2015)
The modern analog of both sets of work (Humburd al. 2018 88; Siu 2019 Frosst & Hinton 2017)
evaluate on feature-sparse, sample-sparse regimes such as the UCI datasets (Dua & Grafi (017) or
MNIST (eCun et al. 2010) and perform poorly on standard image classification tasks.
Hybrid Models. Recent work produces hybrid decision tree and neural network models to scale up to datasets like CIFARIO (Krizhevsky, 2009). CIFAR100 (Krizhevsky) 2009), TinyImageNet (Le)
& Yang 2015). and ImageNet (Deng et al. 2009p. One category of models organizes the neural network into hierarchy, dynamically selecting branches to run inference (Veit Belongie 2018
McGill & Peronal 2017: Teja Mullapudi et al. 2018 Redmon & Farhadi 2017] Murdock et al.
2016). However, these models use impure leaves resulting in uninterpretatble, stochastic paths.
Other approaches fuse deep learning into each decision tree node: an entire neural network (Murthy
et al.
2016), several layers (Murdock et al., 2016 Roy & Todorovic 2016), a linear layer (Ahmed
al.
2016). or some other parameterization of neural network output (Kontschieder et al.] 2015).
These models see reduced interpretability by using k-way decisions with large k (via depth-2 trees)
Ahmed et al. 2016 Guo et al. 2018) or employing an ensemble (Kontschieder al.
2015 Ahmed
et al. 2016), which is often referred to as a "black box" (Carvalho et al. [2199] Rudin
2018
Hierarchical Classification Siila & Freitas 2011). One set of approaches directly uses a pre- existing hierarchy over classes, such as WordNet (Redmon & Farhadi) 2017 Brust & Denzler 2019 Deng et af.). However conceptual similarity is not indicative of visual similarity. Other models build hierarchy using the training set directly, via a classic data-dependent metric like Gini impurity (Alaniz & Akata 2019) or information gain (Rota Bulo & Kontschieder 2014 Biçici et all 2018). These models are instead prone to overfitting, per (Tanno | 2019). Finally, several works introduce hierarchical surrogate losses (Wu et al. 2017 Deng et al. 2012). such as hierarchical softmax (Mohammed & Umaashankar 2018), but as the authors note, these methods quickly suffer images (e.g. beyond CIFARIO).
from major accuracy loss with more classes or
We demonstrate hierarchical classifiers attain higher accuracy without hierarchical softmax.
2

---

Published as a conference paper at ICLR 2021
My
(2t, my)
(xt, (gg)
(4s os)
0.8
= 0.7
0.6
(at max)
(e, us? 0.8
(e, mg.
0.4
0.6
0.4
0.1
0.9 = 0.5
0.5
uv
15%
10%
25%
12%
4%
90%
30%
30%
A. Hard
B.
C.
Soft
Figure I: Hard and Soft Decision Trees. A. Hard: is the classic "hard" oblique decision trec. Each node picks the child node with largest inner product, and visits that node next. Continue until leaf is reached. B. Soft: is the "soft" variant, where each node simply returns prohabilities, as normalized inner products, of each child. For each leaf, compute the probability of its path to the root. Pick leaf with the highest probability. C. Hard vs. Soft: Assume is the correct class. With hard inference, the mistake at the root (red) is irreoveerbbl.. However, with soft inference, the highly-uncertain decisions the root and wa superseded by the highly certain decision at wy (green). This means the model can still correctly pick wa despite a mistake at the root. In short, soft inference can tolerate mistakes in highly uncertain decisions.
METHOD
Neural-Backed Decision Trees (NBDTs) replace a network's final linear layer with decision tree. Unlike classical decision trees or many hierarchical classifiers, NBDTs use path probabilities for inference (Sec 3.1 tolerate highly-uncertain intermediate decisions, build hierarchy from pre- trained model weights (Sec 3.2 & 3.3 to lessen overfitting, and train with a hierarchical loss (Sec significantly better learn high-level decisions (e.g., Animal Vehicle).
3.1 INFERENCE
Our NBDT first featurizes each sample using the neural network backbone; the backbone consists of all neural network layers before the final linear layer. Second, run the final fully-connected layer as oblique decision tree. However, (a) classic decision tree cannot recover from a mistake early in the hierarchy and (b) just running classic decision tree on neural features drops accuracy significantly, by up to 11% (Table D.. Thus, present modificd decision rules (Figure I B):
1. Seed oblique decision rule weights with neural network weights. An oblique decision tree supports only binary decisions, using hyperplane for each decision. Instead, associate weight vector n, with each node. For leaf nodes, where the fully-com mollecconnected layer's weights W R PxK K For all inner nodes, where i [K + NJ, find all leaves k E L(i) in node i's subtree and average their weights: n, = ΣkEL()) (x)l
E [1, K]. each n, wy is a row vector from
Compute node probabilities. Child probabilities are given by softmax inner products. For each sample and node compute the probability of each child j E C(i) using p(jli) SOFTMAX((71; [i), where i7; ((n)=)))EE(())
3. Pick leaf using path probabilities. Inspired by Deng al. (012)). consider leaf, its class and its path from the root Pa. The probability of each node E Pe traversing the next node in the path (i) P& C(i) denoted (i)li. Then, the probability of leaf and its class k is
p(k) IIE.R,
In soft inference, the final class prediction i defined over these class probabilities,
h argmax, argmax,Ilee,
Our inference strategy has two benefits: (a) Since the architecture is unchanged, the fully-connected layer can be run regularly (Table as decision rules (Table D. and (b) unlike decision trees and
3

---

Published as a conference paper at ICLR 2021
Step A. Load Weights
Step B. Set Leaf Voctors
Step C. Set Parcnt Veetors
Step D.
Sct Anocator Vectors
A. Load the weights of a pre-trained model's final fully- connected layer, with weight matrix W E Px Step B. Take rows Wg W and normalize for each leaf node's weight. For example, the wn A is assigned the nodes for the parents' weight. For example, wh and wa (red and purple) in B are averaged to make wi (blue) in C. Step D. For each ancestor, average all leaf node weights in its subtree. That average is the ancestor's weight. Here, the ancestor is the root, its weight is the average of all Icaf weights w1, W22 Uss W/4.
Figure 2: Building Induced Hierarchies.
leaf
Step C. Average each pair of leaf
other conditionally- exxeutted models Tanno et al, 2019 Veit & Belongie 2018 our method can recover from mistake early in the hierarchy with sufficient uncertainty in the incorrect path (Figure CC, Appendix Table This inference mode bests classic tree inference (Appendix
3.2 BUILDING INDUCED HIERARCHIES
Existing cecision-trre-baseee methods use (a) hierarchies built with data-dependent heuristics like information gain or (b) existing hierarchies like WordNet. However, the former overfits to the data, and the latter focuses on conceptual rather than visual similarity: For example, by virtue of being an animal, Bird is closer to Cat than to Plane, according to WordNet. However, the opposite is true for visual similarity: by virtue of being the sky, Bird more visually similar to Plane than to Cat. Thus, to prevent overfitting and reflect visual similarity, we build hierarchy using model weights.
Our hierarchy requires pre-trained model weights. Take row vectors wt : E [1, K]. each repre- senting class, from the fully-connected layer weights W. Then, run hierarchical agglomerative clustering on the normalized class representatives wt/||wwl] which nodes and groups of nodes iteratively paired. As described in Sec 3.1 each leaf node's weight is a row vector W/ E W (Figure 2 Step B) and each inner node's weight n, the average of its leaf node's weights (Figure Step C). This hierarchy is the induced hierarchy (Figure 2
Agglomerative clustering decides
3.3 LABELING DECISION NODES WITH WORDNET
WordNet is a hierarchy of nouns. To assign WordNet meaning nodes, compute the earliest common ancestor for all leaves in a subtree: For example, say Dog and Cat are two leaves that share parent. To find WordNet meaning for the parent, find all ancestor concepts that Dog and Cat share, like Mammal, Animal, and Living Thing. The earliest shared ancestor is Mammal, so we assign Manmal to the parent of Dog and Cat. We repeat for all inner nodes.
However, the WordNet corpus is lacking in concepts that are not themselves objects, like object attributes (e.g., Pencil and Wire both cylindrical) and (b) abstract visual ideas like context (e.g., fish and boat are both aquatic). Many of these which are littered across our induced hierarchies (Appendix Figure 14 Despite this limitation, we use WordNet assign meaning to intermediate decision nodes, with more sophisticated methods left to future work.
3.4 FINE-TUNING WITH TREE SUPERVISION LOss
Even though standard cross entropy loss separates representatives for each leaf, it is not trained to separate representatives for each inner node (Table "None"). To amend this, we add tree super-
4

---

Published as a conference paper at ICLR 2021
Table Results. NBDT outperforms competing decision-tree treeioo--teeebeee methods by up 18% and also outperform the original neural network by 1%. "Expl?" indicates the method retains interpretable properties:
pure leaves, sequential decisions, non-ensemble. Methods without this check see reduced interpretability. We hold the highest decision--ree-bascel accuracy. These results are taken directly from the original papers (n/a
denotes results missing from original papers); XOC (Alaniz & Akata 2019). DCDJ (Back et al. 2017), NofE
Ahmed al. 2016), DDN (Murthy al. 2016), ANT (Tanno al. 2019), CNN-RNN (Guo
2018). We
train DNDF Kontschieder el al.j 2015)
with an updated RI8 backbone, as they did not report CIFAR accuracy.
Method
Backbone
Expl?
CIFARIO
CIFARI00
TinyImageNet
WideResNer28x10
97.62%
82.09%
67.65%
ANT-A*
sla
V
93.28%
air
DDN
NiN
X
90.32%
68.35%
DCDJ
NiN
69.0%
NofE
ResNe156-4x
76.24%
CNN-RNN
WideResNNe22xx1
76.23%
m/
NBDT-S (Ours)
WideResNet28x10
V
97.55 %
82.97 %
67.72% %
NN
ResNet18
X
94.97%
75.92%
64. 13%
DNDF
ResNet18
X
94.32%
67.18%
44.56%
XOC
ResNet18
93.12%
ResNet18
93.97%
64.45%
52.09%
NBDT-S (Ours)
ResNet18
94.82%
77.09% %
64.23%
vision loss, cross entropy loss over the class distribution of path probabilities Dabai = {p(m)) (Eq. I from Sec 3.1 with viry-ng weights Wt, Be where t is the epoch count:
== CROSSOENTROP D tw2 CROSSENTROPY ( Diabel)
(3)
Longiaal
Cson
Our tree supervision loss Luot requires pre-defined hierarchy. We find that (a) tree supervision loss damages learning speed early in training, when leaf weights are nonsensical. Thus, tree supervision weight Wt grows linearly from Wo 0 to wT 0.5 for CIFARIO, CIFAR100, and to wr = for TinylmageNet, ImageNet; 3g [0, 1] decays linearly where possible, fine-tuning with Looft only when the original model accuracy is not reproducible. (c) Unlike hierarchical softmax, path-probas proo-prrbability cross entropy loss Loof disproportionately up- weights decisions earlier in the hierarchy, encouraging accurate high-level decisions; this reflected out-generalization of the baseline neural network by up 16% unseen classes (Table 6)
time. (b) We re-train
4 EXPERIMENTS
NBDTs obtain state-of-the-art results for interpretable models and match outperform modern neural networks on image classification. We report results on different models (ResNet, WideRes- Net, EfficientNet) and datasets (CIFARIO, CIFARI00, TinylmageNet, ImageNet). We additionally conduct ablation studies to verify the hierarchy and loss designs, find that our training procedure improves the original neural network's accuracy up to 2%, and show that NBDTs improve gen- eralization to unseen classes by up to 16%. All reported improvements are absolute.
4.1 RESULTS
Small-scale Datasets. Our method (Table matches or outperforms recently state-of-the of-the--rtt-e neural networks. On CIFARIO and TinyImageNet, NBDT accuracy falls within 0.15% of the baseline neural network. On CIFAR100, NBDT accuracy outperforms the baseline by 11%
Large-scale Dataset. On ImageNet (Table 3 NBDTs obtain 76.60% top-1 accuracy, outperform- ing the strongest competitor NofE by 15%. Note that take the best competing results for any decision-tree-bate method, but the strongest competitors hinder interpretability by using ensembles of models like decision forest (DNDF, DCDJ) or feature shallow trees with only depth 2 (NofF)).
5

---

Published as a conference paper at ICLR 2021
Figure 3: ImageNet Results. NBDT outperforms all competing decision-frre-haseed methods by at least 14%, staying within 0.6% of EfficientNet accuracy.
"EfficientNet" is FageieenNNet-FdgeTTUU-------
Method
NBDT (ours)
NBDT (ours)
XOC
NofE
Backbone
EfficientNet
ResNet18
R5sNet152
AlexNet
Original Acc
77.23%
60.76%
78.31%
56.55%
Delta Ace
-0.63%
+0.50%
-17.5%
+4.7%
Explainable Ace
76.60%
61.26%
60.77%
61.29%
Table 2: Comparisons of Hierarchies. We demonstrate that our weight-space hierarchy bests taxonomy and data-dependent hierarchies. In particular, the induced hierarchy achieves better performance than (a) the WordNet hierarchy, (b) classic decision tree's information gain hicrarchy, built over neural features ("Info Gain") "). and (c) an oblique decision tree built over neural features ("OC1").
Dataset
Backbone
Original
Induced
Info Gain
WordNet
OC1
CIFAR
ResNet18
94.97%
94.82%
93.97%
94.37%
94.33%
CTFAR 100
ResNet18
75.92%
77.09 %
64.45%
74.08%
38.67%
TinYImageNe1200
ResNet18
64.13%
64.23%
52.09%
60.26%
15.63%
4.2 ANALYSIS
Analyses show that NBDT improvements are dominated by significantly improved ability to distinguish higher-level concepts (c.g., Animal vs. Vehicle).
Comparison of Hierarchies. Table 2 shows that our induced hierarchies outperform alternatives. In particular, data- deta-dependent hierarchies overfit, and the existing WordNet hierarchy focuses on conceptual rather than visual similarity.
Comparisons of Losses. Previous work suggests hierarchical softmax (Appendix C. for hierarchical classifiers. However, our results suggest otherwise: NBDTss trained with hierarchical softmax see ~3% less accuracy than with tree supervision loss on TinylmageNet (Table
is necessary
Original Neural Network. Per Sec 3.1
the original neural network's fully-connected layer normally, after training with tree supervision loss. Using this, we find that the original neural network's 's accuracy improves by up to 2% on CIFAR100, TinylmageNet (Table 5
Zero-Shot Superclass Generalization. We define a "superclass" to be the hypernym of several
classes. (e.g. Animal is superclass of Cat and Dog). Using WordNet (per See (.22). we (1) identify which superclasses each NBDT inner node is deciding between (c.g. Animal vs. Vehicle). (2) We find unseen classes that belong the same superclass, from different dataset. (e.g. Pull Turtle images from ImageNet). (3) Evaluate the model to ensure the unseen class is classified into the correct superclass (e.g. ensure Turtle is classified as Animal). For an NBDT, this is straightforward: one of the inner nodes classifies Animal vs. Vehicle (Sec 3.3) For standard neural network, consider the superclass that the final prediction belongs to. (i.e. When evaluating Animal vs. Vehicle on Turtle image, the CIFAR-trainer model may predict any CIFAR Animal class). See Appendix B.2 for details. Our NBDT consistently bests the original neural network by 8%+ (Table 6). When discerning Carnivore Ungulate, NBDT outperforms the original neural network by 16%.
Mid-Training Hierarchy: We test NBDTs without using pre- tre-trained weights, instead constructing hierarchies during training from the curtially-trained network's 's weights. Tree supervision loss with mid-training hierarchies reliably improve the original neural network's 's accuracy, up to ~0.6%% and the NBDT itself can match the original neural network's 's accuracy (Table However, this underper- forms NBDT (Table D. showing fully-trained weights are still preferred for hierarchy construction.
5 INTERPRETABILITY
By breaking complex decisions into smaller intermediate decisions, decision trees provide insight into the decision process. However, when the intermediate decisions are themselves neural network

---

Published as a conference paper at ICLR 2021
Table 3: Comparisons of Losses. Training the NBDT using tree supervision loss with linearly increasing weight ("TreeSup(O"") is superior to training (a) with -weight tree supervision loss ("TreeSup"), (b) with a hierarchical softmax ("HrchSmax") and (c) without extra loss terms. ("None"). A the accuracy difference between our soft loss and hierarchical softmax.
Dataset
Backbone
Original
TreeSup(t)
TreeSup
None
HrchSmax
CIFARIO
ResNet18
94.97%
94.82% %
94.76%
94.38%
93.97%
CIFARIOU 100
ResNet18
75.92%
77.09
74.92%
61.93%
74.09%
TinylmageNe1200
ResNet18
64. 1.3%
64.23% %
62.74%
45.51 1%
61.12%
Table 4: Mid- Mid-Training Hierarchy. Constructing and using hierarchies early and often in training yields the highest performing models. All experiments use ResNet18 backhones. Per See B.4 [8, we are the loss term coefficients. Hierarchies reconstructed every "Period" epochs, starting at "Start" and ending at "End".
Hierarchy Updates
CIFARIO
CIFARI00
Start
End
Period
NBDT
NN+TSL
NN
NBDT
NN+TSL
NN
67
120
10
94.88%
94.97%
94.97%
76.04%
76.56%
75.92%
90
140
10
94.29%
94.84%
94.97%
75.44%
76.29%
75.92%
90
140
94.52%
94.89%
94.97%
75.08%
76.11%
75.92%
120
121
10
94.52%
94.92%
94.97%
74.97%
75.88%
75.92%
predictions, extracting insight becomes more challenging. To address this, we adopt benchmarks and an interpretability definition offered by [Poursazzi-Sandell a human can validate its prediction, determining when the model has made a sizable mistake. To assess this, we adapt Poousaaoi- Sangdeh et al. (2018)ss benchmarks computer vision and show (a) humans can identify misclassifications with NEDT explanations more accurately than with saliency explanations (See [.T]. (b) a way to utilize NBDT's entropy to identify ambiguous labels (Sec. 5.4) and (c) that humans prefer to agree with NBDT predictions when given a challenging image classification task (Sec. 5.2 & 5.3). Note that these analyses depend on three model properties that NBDT preserves: (I) discrete, sequential decisions, that one path is selected; (2) pure leaves, so that one path picks one class; and (3) non-ensembled predictions, so that path to prediction attribution is discrete. In all surveys, use CIFAR 1--trained models with ResNet18 backbones.
al. (2018): A model is interpretable
SURVEY: IDENTIFYING FAULTY MODEL PREDICTIONS
In this section we aim answer a question posed in (Poursabzi-Sangdel) al, 2018) "How well can someone detect when the model has made a sizable mistake?" In this survey, each user is given images, of which are correctly classified and is mis-classified. Users must predict which image was incorrectly classified given the model explanations and b) without the final prediction. For saliency maps, this near-impossible task as saliency usually highlights the main object in the image, regardless of wrong right. However, hierarchical methods provide sensible sequence of
Table 5: Original Neural Network. We compare the model's accuracy before and af- ter the tree supervision loss, using ResNet18. WideResNet on CIFAR100. TayyImageNet. Our loss increases the original network accu- racy consistently by ~ 8 2.4%. NN-S S is the network trained with the tree supervision loss.
Table 6: Zero-Shot Superclass Generalization. We eval- uate 11-tRio-d NBDT (ResNet18 backbone) in- node's ability generalize beyond label TinyImageNer with superclass labels (e.g. label Dog with Animal) and evaluate nodes distinguishing between said superclasses. We compare to the baseline ResNet18: check if the prediction is within the right superclass.
classes. We
Dataset Backbone
NN
NN-S
Phctas Superclasses
R18
NBDT-S
C100
R18
75.92%
76.96%
71
Animal vs. Vehicle
66.08%
74.79%
T200
R18
64.13%
66.55%
36
Placental vs. Vertebrate
45.50%
54.89 %
C100
WRN28
82.09%
82.87 %
19
Carnivore vs. Ungulate
51.37%
67.78% %
T200
WRN28
67.65%
68.51%
9
Motor Vehicle vs. Craft
69.33%
77.78 %

---

Published as a conference paper at ICLR 2021
Figure 4: CIFARIO Blurry Images. To make the classification task difficult for humans, the CIFARIO images are downsampled by 4x. This forces least partial reliance model predictions. allowing evaluate which explanations are convincing enough carn the user's agreement.
intermediate decisions that can be checked. This is reflected in the results: For each explainability technique, we collected 600 survey responses. When given saliency maps and class probabilities, only 87 predictions were correctly identified as wrong. In comparison, when given the NBDT series of predicted classes and child probabilities (e.g., "Animal (90%) + Mammal (95%)", without the final leaf prediction) 237 images were correctly identified as wrong. Thus, respondents can better recognize mistakes in NBDT explanations nearly times better.
Although NBDT provides more information than saliency maps about misclassification, a majority the remaining 363 NBDT predictions were not correctly identified. To explain this, we note that 37% of all NBDT errors occur at the final binary decision, between two leaves; since we provide all decisions except the final one, these leaf errors would be impossible to distinguish.
5.2 SURVEY: EXPLANATION-GUIDER IMAGE CLASSIFICATION
In this section we aim answer a question posed in (Poursabzi-Sangde et al. 2018 "To what extent do people follow model's 's predictions when it is beneficial to do so?". In this first survey, each user is asked classify severely blurred image (Fig D. This survey affirms the problem's 'ss difficulty. decimating human performance to not much more than guessing: 163 of 600 responses are correct (27.2% accuracy).
In the next survey, we offer the blurred image and two sets of predictions: (1) the original neural network's predicted class and its saliency map, and (2) the NBDT predicted class and the sequence of decisions that led up to it ("Animal, Mammal, Cat"). For all examples, the two models predict different classes. In 30% of the examples, NBDT is right and the original model is wrong. In another 30%, the opposite is true. In the last 40%, both models are wrong. As shown in Fig. 4 the image extremely blurry, so the user must rely on the models to inform their prediction. When offered model predictions, in this survey, 255 of 600 responses improvement over no model guidance. We observe that humans trust NBDT-explained prediction more often than the saliency-explaieed predictions. Out of 600 responses, 312 responses agreed with the NBDT's prediction, 167 responses agreed with the base model's prediction, and 119 responses disagreed with both model's predictions. Note that majority of user decisions (~ 80%) agreed with either model prediction, even though neither model prediction was correct in 40% of examples, showing our images were sufficiently blurred to force reliance on the models. Furthermore, 52% of responses agreed with NBDT (against saliency's 's 28%), even though only 30% of NBDT predictions were correct, showing improvement in model trust.
correct (42.5% accuracy), 15.3 point
5.3 SURVEY: HUMAN-DIAGNOSHE LEVEL OF TRUST
The explanation of an NBDT prediction is the visualization of the path traversed. We then compare these NBDT explanations to other explainability methods in human studies. Specifically, we ask participants to pick an expert trust (Appendix, Figure 13]. based on the expert's explanation a saliency map Nets GradCAM), a decision tree (NBDT), or neither. We only use samples where ResNet18 and NBDT predictions agree. Of 374 respondents that picked one method over the other, 65.9% prefer NBDT explanations; for misclassified samples, 73.5% % prefer NBDT. This supports the previous survey's results, showing humans trust NBDT' more than current saliency techniques when explicitly asked.
5.4 ANALYSIS: IDENTIFYING FAULTY DATASET LABELS
There are several types of ambiguous labels (Figure 5). any of which could hurt model performance for image classification dataset like ImageNet. To find these images, entropy in NBDT
8

---

Published as a conference paper at ICLR 2021
deerr
NBDT EXPLANATION
98.5%
ankmall
96.4%
dog
dog
34.5%
98.9%
96.0%
wensbratte
96.6%
bArd
96.6%
vertebeste
sird
trusk
trakk
sirplere
NNN
Bird (96%), Dog (0.8%) Cat (0.4%)
Cat (80%), Dog (18%), Automobile (0.3%)
Figure 5: Types of Ambiguous Labels. All these examples have ambiguous labels. With NBDT (top), the decision rule deciding between equally-plausible classes has low certainty (red, 30-50%). All other decision rules have high certainty (blue, 96%+). The juxtaposition of high and low certainty decision rules makes ambiguous labels casy to distinguish. By contrast, ResNet18 (bottom) still picks one class with high probabilityy (Left) An extreme example of a "spug" that may plausibly belong to two classes. (Right) Image containing two animals of different classes. Photo ownership: "SSugg" by Ame Fredriksen gyyporama.com. Used with permission. Second image is CC-0 licensed at pexels.com.
RESNET-18 ENTROPY (BASELINE)
NBDT PATH ENTROPY (OURS)
Figure 6: ImageNet Ambiguous Labels. These images suggest that NBDT path entropy uniquely identifies ambiguous labels in Imagenet, without object detection labels. We plot ImageNet validation samples that in- duce the most 2-class confusion, using TinyImagene2200 trained models. Note that ImageNet classes not include people. (Left) Run ResNet18 and find samples that (a) maximize entropy between the top 2 classes and (b) minimize entropy across all classes, where the top classes are averaged. Despite high model uncertainty, half the classes from the training set hee, orange, bridge, hanana, remote control and do not show visual ambiguity. (Right) For NBDT, compute entropy for each node's predicted distribution; take the difference be- tween the largest and smallest values. Now, half of the images contain truly ambiguous content for a classifier; we draw green boxes around pairs of objects that could each plausibly be used for the image class.
decisions, which find is a much stronger indicator of ambiguity than entropy in the original neural network prediction. The intuition is as follows: If all intermediate decisions have high certainty except for few decisions, those decisions are deciding between multiple equally plausible cases. Using this intuition, we can identify ambiguous labels by finding samples with high "path entropy" or highly disparate entropies for intermediate decisions on the NBDT prediction path.
Per Figure 6 the highest "path entropy" samples in ImageNet contain multiple objects, where each object could plausibly be used for the image class. In contrast, samples that induce the highest entropy in the baseline neural network do not suggest ambiguous labels. This suggests NBDT entropy is more informative compared to that of a standard neural network.
CONCLUSION
In this work, we propose Neural-Backer Decision Trees that see (1) improved accuracy: NBDTs out-generalize (16%+), improve (2%+), and match (0. (0.15%) or outperform (1%+) state-of-the-arr neural networks on CIFARIO, CIFAR 100, TinyImageNet and ImageNet. We also show (2) im- proved interpretability by drawing unique insights from our hierarchy, confirming that humans trust NBDT's over saliency and illustrate how path entropy can be used to identify ambiguous labels. This challenges the conventional supposition of dichotomy between accuracy and interpretability, paving the way for jointly accurate and interpretable models in real-world deployments.

---

Published as a conference paper at ICLR 2021
REFERENCES
Karim Ahmed, Mohammadharis Baig, and Lorenzo Torresani. Network of experts for large-scale image categorization. volume 9911, April 2016.
Stephan Alaniz and Zeynep Akata. XOC: explainable observer-classifier for explainable binary decisions. CoRR, abs/1902.01780, 2019.
Seungryul Baek, Kwang In Kim, and Tae- Tae-Kyun Kim. Deep convolutional decision jungle for image classification. CoRR, abs/1700.02003, 2017.
Arunava Banerjee. Initializing neural networks using decision trees. 1990.
Arunava Banerjee. Initializing neural networks using decision trees. In Proceedings of the Inter- national Workshop on Computational Learning and Natural Learning Systems, pp. 3-15. MIT Press, 1994.
Ufuk Can Biçici, Cem Keskin, and Lale Akarun. Conditional information gain networks. In 2018 24th International Conference on Pattern Recognition (ICPR), pp. 1990-1395, IEEE, 2018.
Olcay Boz. Converting a trained neural network to a decision tree dectext decision tree extractor. In ICMLA, 2000.
-lexane-r Brust and Joachim Denzler. Integrating domain knowledge: using hierarchies to improve deep classifiers. In Asian Conference on Pattern Recognition, pp. 3-16. Springer, 2019.
Diogo V Carvalho, Eduardo M Percira, and Jaime S Cardoso. Machine learning interpretability: A survey on methods and metrics. Electronics, 8(8):832, 2019.
Mark Craven and Jude W Shavlik. Extracting troe-structured representations of trained networks.
In Advances in neural information processing systems, pp. 24-30, 1996.
Mark W Craven and Jude Shavlik. Using sampling and queries extract rules from trained neural networks. In Machine learning proceedings 1994, pp. 37-45. Elsevier, 1994.
Darren Dancey, David McLean, and Zuhair Bandar. Decision tree extraction from trained neural networks. January 2004.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR09, 2009.
Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classification using label relation graphs.
Jia Deng. Jonathan Krause, Alexander Berg, and Li Fei-Fei. Hedging your bets: Optimizing accuracy-speciffctt trade-offs in large scale visual recognition. In 2012 IEEE Conference Computer Vision and Pattern Recognition, pp. 3450-3457. IEEE, 2012.
Finale Doshi-Veles and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint /tt.://00.0000 2017.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL httt:://tcchhhhi ics. uci edu.dduuni ml
Nicholas Frosst and Geoffrey E. Hinton. Distilling a neural network into a soft decision tree. CoRR, abs/1711.09784, 2017.
Yanming Guo, Yu Liu, Erwin M Bakker, Yuanhao Guo, and Michael Lew. Cun-rmn: a large-scale hierarchical image classification framework. Multimedia Tools and Applications, 77(8):10251- 10271, 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv: 000.:-0033.33333 2015.
10

---

Published as a conference paper at ICLR 2021
Kelli Humbird, Lue Peterson, and Ryan McClarren. Deep neural network initialization with decision trees. IEEE Transactions on Neural Networks and Learning Systems, PP:1-10, October 2018.
Irena Ivanova and Miroslav Kubat. Initialization of neural networks by means of decision trees. Knowledge- Based Systems, 8(6):333 344, 1995a. Knowledge- based neural networks.
Irena Ivanova and Miroslav Kubat. Decision-tree based neural network (extended abstract). In Ma- chine Learning: ECML-995, pp. 295-298, Berlin, Heidelberg, 1995b. Springer Berlin Heidelberg.
Cem Keskin and Shahram Izadi. Splinenets: Continuous neural decision graphs. In Advances in Neural Information Processing Systems, pp. 1994-2004, 2018.
Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Bulo. Deep neural decision forests. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
R. Krishnan, G. Sivakumar, and P. Bhattacharya. Extracting decision trees from trained neural networks. Pattern Recognition, 32(12):1999 2009, 1999.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. 2015.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann. lecun. com/exdb/nnist,, 2, 2010.
Zachary Chase Lipton. The mythos of model interpretability. corr abs/1606.03490 (2016). arXiv 2016.
preprint 1606vv:6006.000
SM Lundberg, G Erion, H Chen, A DeGrave, JM Prutkin, B Nair, R Katz, J Himmelfarb, N Bansal, and S-i Lee. From local explanations to global understanding with explainable ai for trees, nat. mach. intell., 2, 56-67, 2020.
Mason McGill and Pietro Perona. Deciding how to decide: Dynamic routing in artificial neural networks. In ICML, 2017.
Abdul Arfat Mohammed and Venkatesh Umaashankar. Effectiveness of hicrarchical softmax in large scale classification tasks. In 2018 International Conference Advances in Computing, Communications and Informatics (ICACCI), pp. 1090-1094. IEEE, 2018.
Calvin Murdock, Zhen Li, Howard Zhou, and Tom Duerig. Blockout: Dynamic model selection for hierarchical deep networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2583-2591, 2016.
Venkatesh N. Murthy, Vivek Singh, Terrence Chen, R. Manmatha, and Dorin Comaniciu. Deep decision network for multi-class image classification. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Vitali Petsiuk, Abir Das. and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In Proceedings of the British Machine Vision Conference (BMVC), 2018.
PPursabzi-Sangee D Goldstein, J Hofman, Vaughan, and Wallach. Manipulating and mea-
suring model interpretability. In MLConf, 2018.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263-7271, 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA. August 13-17, 2016, pp- 1135-1144, 2016.
Samuel Rota Bulo and Peter Kontschieder. Neural decision forests for semantic image labelling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 81-88, 2014.
11

---

Published as a conference paper at ICLR 2021
Anirban Roy and Sinisa Todorovic. Monocular depth estimation using neural regression forest. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5506-5514, 2016.
C Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. manuscript based on c. rudin please stop explaining black box ma- chine learning models for high stakes decisions. In Proceedings of NeurIPS 2018 Workshop on Critiquing and Correcting Trends in Learning, 2018.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 618-61 188626, 2017.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint
0tth::70010665. 2017.
Carlos N Silla and Alex A Freitas. A survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 22(1-2):31-72, 2011.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:/312.600. 2013.
Chapman Siu. Transferring tree ensembles neural networks. In Neural Information Processing, pp. 471-480, 2019.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The convolutional net. CORR, abs/1412.6606, 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. Inter- national Conference Machine Learning (ICML) 2017, 2017.
Ryutaro Tanno, Kai Arulkumaran, Daniel C. Alexander, Antonio Criminisi, and Aditya Nori. Adap- tive neural trees, 2019.
Ravi Teja Mullapudi, William R. Mark, Noam Shazeer, and Kayvon Fatahalian. Hydranets: Special- ized dynamic architectures for efficient inference. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In The European Conference on Computer Vision (ECCV), September 2018.
Mike Wu, M Hughes, Sonali Parbhoo, and F Doshi- Beyond sparsity: Tree-based regular- ization of deep models for interpretability. In In: Neural Information Processing Systems (NIPS) Conference. Transparent and Interpretable Machine Learning in Safety Critical Environments (TIML) Workshop, 2017.
Brandon Yang, Gabriel Bender, Quoe V Le, and Jiquan Ngiam. Condconv: Conditionally parameter- ized convolutions for efficient inference. In Advances in Neural Information P Processing Systems, pp. 1307-1318, 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision (ECCV), pp. 818-833. Springer, 2014.
Jianming Zhang, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. In European Conference on Computer Vision (ECCV), pp. 543- 559. Springer, 2016.
12


Reviews:
[{"review_id": "zwMJgzKxwzR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a neural-backed decision tree that aims to improve both the accuracy and the interpretability of deep learning models. Training under a newly introduced tree supervision loss, the authors show that NBDTs can outperform and out-generalize some modern architectures on several image datasets.\n\nOverall this paper is well written and established. The idea of using a differentiable oblique decision tree to replace the final linear layer is interesting. The authors provide clear illustration of the procedure and promising experimental results.\n\nQuestions:\n\n1. What is the main intuition that NBDTs can outperform the original network?\n2. Given the classes are in the leaves, does the ordering of classes in the leaf layer matter? How should one determine which two classes are in the same bottom subtrees?\n\nMinor comments:\n\n1. Figure 2 Step A: y_d -> y_k or y_4.\n2. Why are there many n/a results in Table 1?\n3. Section 3.1 in the Compute node probabilities paragraph, the definition of r_i seems confusing.\n4. What does NN mean in Table 1? How different is it from CNN-RNN?", "extra_scores": {}, "extra_text": {}}, {"review_id": "mlTIpDngMw0", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors did a fantastic job of answering questions, revising their manuscript in accord with reviewer feedback (Sec 3.4 title), and even adding new experimental results based on reviewer suggestions (mid-training hierarchy) and reflecting best practices in interpretability research. I was really impressed by their nimbleness and responsiveness. I will raise my score to a 7: I think this is a very solid paper and excellent research effort around a nascent idea. In particular, I think its impact is limited by\n\n- its close coupling to naturally hierarchical problems, e.g., multi-class classification with a taxonomy\n- its close coupling to image data and tasks\n- its heuristic nature: fully train neural net, infer hierarchy via clustering, retrain neural net, then map a priori labels onto inferred hierarchy\n\nThe \"10\" version of this paper (maybe future work?) would propose a way to infer the hierarchy on the fly and show how to apply it onto other kinds of data and problems with different structures.\n\n-----\n\nThis submission proposes a modification of neural networks that replaces the \"final linear layer with a decision tree.\" The term \"decision tree\" is applied somewhat loosely to a hierarchical neural architecture akin to a hierarchical softmax. In the current work (as I understand it), this hierarchy is induced from a pre-trained multi-output, e.g., multiclass, neural network via a hierarchical clustering and subsequent averaging of the output weights. At inference time, path probabilties can be computed based on the chain rule. Predictions can be made based on either a greedy traversal of the tree (choosing the most likely child at each step, a la hierarchical softmax) or by choosing the most probable leaf, which requires computing all path probabilities. Empirical results across three standard image datasets are suggestive, if not conclusive, and the paper concludes with some interesting, albeit cursory, examples of potential \"interpretability\" applications.\n\nThe submission summarizes its contributions at the end of Section 1 as follows:\n1. It proposes a tree-structured loss to augment supervised neural network training (predominantly for multiclass classification problems).\n2. It describes a heuristic to induce a hierarchy in the output weights of a pre-trained multi-output neural network, enabling decision tree-like inference and provide evidence it is more effective than other approaches for inducing hierarchies.\n3. It presents simple case studies of how the induced hierarchy can be used for traditional \"interpretability\" tasks, like debugging and generating explanations.\n\nI appreciate the idea at the center of this paper -- adding simple hierarchical structure to a multi-output neural network, with the aim of increased interpretability -- but I feel the work as it is presented is nascent and the manuscript itself is flawed. I lean toward rejection at the moment, but I could be persuaded to change my mind by some combination of solid revisions, convincing author response, or vociferous advocacy from other reviewers.\n\nI will briefly extol the paper's strengths before providing a longer discussion of what I consider to be its key weakness. First, I really like the last sentence in the paper:\n\n\"This challenges the conventional supposition of a dichotomy between accuracy and interpretability, paving the way for jointly accurate and interpretable models in real-world deployments.\"\n\nWeaknesses in the evaluation of its interpretability claims aside, I agree with this statement. I think the case studies presented do provide evidence of improved interpretability alongside small accuracy improvements. I think this paper does succeed in demonstrating that accuracy and interpretability are not necessarily competing objectives, at least for certain tasks (multiclass classification of images).\n\nA laundry list of other strengths:\n\n- The motivation is strong (modulo weakness discussed below): there is a growing need to provide human-understandable insights into decisions made by complex machine learning models.\n- The proposed approach is simple and elegant, easy to implement, and empirically effective. I'm quite impressed that the proposed tree loss appears to improve accuracy (!) on multiple tasks.\n- I also think this paper lays groundwork for a direction of research that the community could continue to build on.\n\nI think that the manuscript's largest flaw, ironically, regards interpretability, its primary motivation. The work's central claim is that the tree-structured decision layer delivers improved interpretability with comparable or slightly improved accuracy. In its discussion of this claim, the manuscript provides no precise definition of \"interpretable,\" making it difficult to verify the claim qualitatively or quantitatively. Section 5 presents a vignette of case studies, but the discussion of each is quite limited. In particular, none of the use cases is fully motivated or placed in the context of previous research on interpretability definitions [1][2]. The cursory presentation of results for each do the results a disservice by making it difficult for the reader to recognize and assess their significance.\n\nTo quote the introduction from Lipton's _The Mythos of Model Interpretability_ [1],\n\n\"Despite the absence of a definition, papers frequently make claims about the interpretability of various models. From this, we might conclude that either: (i) the definition of interpretability is universally agreed upon, but no one has managed to set it in writing, or (ii) the term interpretability is ill-defined, and thus claims regarding interpretability of various models may exhibit a quasi-scientific character.\"\n\nI believe the paper would be strengthened by focusing on one use case, e.g., debugging or human trust, using the ~1 page dedicated to Section 5 to motivate it more fully and to present the results in detail. If the primary use case is generalization or debugging, then I suggest designing a quantitative analysis so defend against claims of cherry picking the best results (a common problem in presenting \"example\" interpretability results).\n\nSection 5.4 includes a quantitative evaluation, but I question whether mere human preference is evidence of \"human trust.\" More recent research on trust appear to use more elaborate studies in which trust is measured by subjects' rate of success in performing a particular task aided by the machine learning model [3].\n\nI want to caveat the above: I really appreciate this line of work and think it has value. There is an ongoing discussion in our community about rewarding good ideas, rather than punishing imperfect or incomplete execution. I also acknowledge that I am far from an expert in the latest interpretability research. Nonetheless, my understanding is that interpretability researchers have grown more skeptical of interpretability claims about new methods absent a rigorous framework (definitional and/or experimental) for evaluating those claims.\n\nWhen I read this paper, I find it hard to escape the conclusion that its interpretability claims rest on the presupposition that trees are naturally more interpretable (and further that readers will accept this dogma). I disagree with this assertion (see below), but even if it were generally true, I still think the paper would be strengthened by adding a more rigorous discussion and analysis of its claims. Propose a definition or criterion (see [1][2] for ideas), ideally one that could be assessed qualitatively and evaluated empirically, then apply it.\n\nRegarding the claim about trees in Section 5: \"The interpretability of a decision tree is well-established when input features are easily understood (e.g. tabular data in medicine or finance).\"\n\nI would dispute that this is \"well-established\" for anything but the simplest decision tree models, with a single tree consisting of a small number of splits using a handful of features, which are rare in realistic settings. The most commonly used tree-structured models (gradient boosted decision trees and random forests) are not readily interpretable, even for tabular data and especially for high dimensional inputs. This has made research like SHAP [4] of great interest to practitioners.\n\nWhat is more, even for tabular data, the neural decision trees described in this paper are (to my understanding) basically a cascade of linear classifiers, with split each having access to all features at once. This does do not lend itself to the same kind of \"interpretation\" one gets for classic decision trees that use one feature per split. With even modestly deep hierarchies, the resulting \"explanations\" would rapidly become quite complex.\n\nI see one other weakness in the proposed method itself: as I understand things, it requires access to a pretrained neural network. At the very least, one needs pre-existing output weights to cluster in order to induce a hierarchy -- and the induced hierarchy is a necessary component in the presented results. This isn't a fatal flaw -- learning a hierarchy on the fly could be left for future work. Nonetheless, it limits the work's usefulness and potential impact.\n\nWhat is more, I don't think the manuscript is sufficiently clear about this requirement: on my first pass through the paper, I came away with the impression that there was a way to learn the hierarchy while training the neural net -- the inclusion of a section entitled \"Training with Tree Supervision Loss\" seems to imply this. I suggest revising the text to make it crystal clear that it is not possible to use the tree loss to train a neural net from scratch -- at least, not without a predefined hierachy (perhaps from a previous training run or prior knowledge).\n\nI will now summarize the improvements I suggest for strengthening the manuscript:\n1. Focus on one definition of interpretability and then analyze central claims through that lens. Introduce it early in the paper (introduction) and then dedicate Section 5 to it, rather than trying to cover lots of use cases superficially.\n2. Make the limitations of the proposed approach VERY clear. In particular, you need a predefined hierarchy to train with the tree loss, and you need pretrained neural net (or pre-existing weights, at least) to induce a hierarchy based on clustering.\n3. If the intention is for this approach to be used exclusively for finetuning or adapting an existing neural network, then this should be made clear in the text. Consider renaming Section 3.3.\n4. Justify (or reword) statements like \"the interpretability of a decision tree is well-established\" or \"neural features are visually interpretable\" (a single reference does not suffice...the Olah distill survey draws no such definitive conclusions).\n\nI have a few questions:\n- One thing that is not clear: when training with tree loss, are weights shared across nodes? In particular, the weight vector for an inner node is the average of its descendent leaf node weight vectors. When training with tree loss, do we then treat that inner node weight vector as a set of independent parameters with separate updates? Or do we continue to treat it as a sum of leaf parameters, so that leaf and inner node updates affect the same parameters, as in an RNN or recursive network.\n- Is there possibly a heuristic that could approximate \"learning the hierarchy?\" For example, train with a basic loss for enough iterations until the output weights start to converge, then pause training to induce the hierarchy and then resume training with the tree loss. Part of the heuristic could be guidance about how to detect when the output weights have sufficiently converged.\n- What are the key differences between this approach and a hierarchical softmax? My understanding is that they're basically equivalent at inference time (except maybe traditional hierarchical softmax uses hard decisions?). What about during training? Is it maybe the use of negative sampling for hierarchical softmax?\n- How would this approach perform for extremely high dimensional output spaces -- one of the primary motivations for hierarchical softmax? I imagine that for some output cardinality, \"soft\" inference becomes computationally infeasible.\n\n[1] Lipton. The Mythos of Model Interpretability. https://arxiv.org/abs/1606.03490.\n[2] Doshi-Velez and Kim. Towards A Rigorous Science of Interpretable Machine Learning. https://arxiv.org/abs/1702.08608.\n[3] Poursabzi-Sangdeh, et al. Manipulating and Measuring Model Interpretability. https://arxiv.org/abs/1802.07810.\n[4] Lundberg, et al. From local explanations to global understanding with explainable AI for trees. https://www.nature.com/articles/s42256-019-0138-9.", "extra_scores": {}, "extra_text": {}}, {"review_id": "jLwYNTMXJT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method to make neural networks more accurate and interpretable by replacing their final layers with a probabilistic decision tree. As a result, the network can produce a sequence of decisions that leads to the final classification result, given an input image. The method is trained with soft decisions by assigning probabilities to each leaf, which are associated with a single class. The tree decision hyperplanes are constructed automatically from the backbone networks final dense layer and finetuned. The fact that decisions are soft solves the differentiablility problem of decisions as in various other similar papers, cited or uncited (more below).\n\nThe paper is not written very clearly, so it would be hard to reproduce. It's not clear in places if indices correspond to nodes or classes, as it is used interchangeably. The text misses a proper mathematical formulation of the operations done in inner nodes, and this all makes it difficult to understand what the loss is and how it can update the decisions in the tree. Perhaps it's possible to understand all the details by re-reading the text several times, but the paper definitely lacks clarity. A nice comparison would be with the Deep Neural Decision Forest (DNDF) paper. I'd expect that level of clarity from an ICLR paper.\n\nThe justification behind using the hyperplanes from the last layer of the CNN for constructing inner nodes is not explained. The visualization in Figure 2 indicates that the averages used for the parent nodes act like clusters - and somehow averaging them forms bigger clusters, but what really happens is that the rows of the final layer, which are unnormalized hyperplanes, are averaged to form new hyperplanes that are assumed to cover both classes, which would not be the case most of the time. I'm not sure if this has a reasonable geometric meaning, but the visualization gives the wrong idea.\n\nThe paper makes the claim: \"Unlike previous decision trees or hierarchical classifiers, NBDTs use path probabilities for inference\".\nThere are certainly many papers that use path probabilities for inference. In fact it's been the norm for discrete decisions, since hard decisions are much harder to formulate in a differentiable manner. DFDN uses path probabilities, as do older papers like \"Decision Forests, Convolutional Networks and the Models in-Between\" by Ioannou et al. (uncited). I don't understand this claim.\nThe biggest difference here is that each class gets exactly one dedicated leaf, instead of each leaf storing a distribution. It is not clear to me why this is a good idea though - it's clearly not preferred in the decision tree literature.\n\nAnother interesting point - most papers I know on the subject actually try to enable sparse activation at test time for efficiency, which is the harder problem to solve. See CondConv (Neurips 2019 - uncited), SplineNets (Neurips 2018 - uncited), Outrageously Large Neural Networks by Shazeer et al. (uncited), Conditional Information Gain by Bicici et al., etc. Activating all the branches of the tree or graph becomes prohibitively costly for deeper trees.\n\nInterpretability is another major claim. The way it works is by using WordNet to assign higher level compound classes to images, such as Animal -> Turtle. This way by grouping similar leaves together, the inner nodes are assigned a meaning. This needs to be done when constructing the hierarchy, so the tree structure is manually given from what I understand. But then how are the pairs of nodes selected? It's not clear from the text, at least it is not explained clearly. In the Figure 5, a decision tree is given, which could only have been constructed by hand - i.e. with someone knowing that cats and dogs are the closest pairs. Where does this information come from for ImageNet when there are 1000 classes? WordNet?\n\nAnother concern about interpretability is that it is claimed that other good performing methods like DFDN are not interpretable. I don't see why that's the case if we use WordNet to assign meaning for the inner nodes of such solutions. I'd agree that it is not straightforward to do so, but with WordNet I can imagine how it could be done. So I'm not convinced about the claim that this solution is the most interpretable one when the paper does not explain why other solutions aren't in a persuasive manner.\n\nOne final concern is with the comparisons with similar models. The paper says these were reimplemented and tested with different backbones. Where are these implementations taken from, for instance for DFDN, which is quite complex? How can the reader trust these numbers?\n\nI find the claims unconvincing and the results unpersuasive. I think the paper needs i) a better mathematical formulation to clarify the method, ii) better explanation of how the trees are constructed (e.g. if WordNet is used), iii) better understanding of differences from similar work. Currently I don't think it meets the bar for ICLR.\n \n\nTreeSup result is likely wrong in Table 3.\n\nAfter Rebuttal:\n\nThe authors did a great job in addressing most of the issues I have, and made many changes that helped with the clarity of the paper. There are still some remaining issues like Figure 2 assigning a wrong geometric meaning to the clusters formed by taking means of hyperplanes, and the uncited references, which are simply added to the references section (which should be fixed). But I think the added survey results are a great addition and a persuasive proof about the increased interpretabililty of these models. Therefore I'll increase my score to 6.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ITf63wjbo4F", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Significance:\nThis article seems to be a useful contribution to the literature on interpretable deep networks. However, the paper could be strengthened by demonstrating and analyzing the interpretability of approaches to other types of data, such as sequential data.\n\nNovelty:\nThis paper's main contribution is to offer a new hybrid model that combines a deep neural network with a tree. The authors used the weights of the last layer of a DNN to build a tree from bottom to top, where each inner node in the tree is the average weight of each child. Then, the authors used softmax to compute the probability of  routing for each child. \n\nPotential Impact:\nThe approach presented in this paper is well-evaluated in computer vision but potentially useful in many other settings.\n\nTechnical Quality:\n+ The technical content of the paper appears to be correct, albeit there is some room for improvement. \n+ Page 2, the authors said, \"These models likewise limit interpretability by supporting no more than depth-2 trees.\" Having the depth-2 tree actually improves the interpretability since it is easier to follow the model prediction. For example, a tree of depth-2 considers more interpretable than a tree of depth-4. The authors should rephrase this sentence.\n+ While the authors claim that linearly increasing the weight in NBDT is the superior method, why the NBDT with the constant rate overperforms on CIFAR-10 as shown in Table 3?\n+ Based on the NBDT's explanation, all leaf nodes should have the same depth, but in the example shown in Figure 6 and supplement, the leaf nodes are in different depths. The authors need to explain why the final tree has leaf nodes with different depths.\n+ The authors did not compare their method's interpretability with a similar methods, such as Adaptive Neural Trees (Tanno et al. 2019).  I suggest running this experiment since Adaptive Neural Trees has a different interface than NBDT. . While the number of participants is noticeable in the interpretability study, it seems that participants only answer one question. Adding more questions could strengthen the paper. Further, the authors should provide a summary of participants (e.g., age, education, and gender).\n+ The paper will be strengthened if the authors run an experiment without using a pre-trained neural network on a small dataset like MNIST to demonstrate their algorithm's effectiveness. \n\nPresentation/Clarity\n+ While the paper is fairly readable, there is room for improvement in the clarity.\n+ Page 3, the last paragraph forgot a period after the parentheses. \"path (Figure 1 C, Appendix Table 6) This ...\"\n+ Page 7, figure 4, I believe that the authors mean \"without NBDT\" instead of \"without ResNet.\"\n+ While the authors explained how they label the tree nodes with WordNet in the supplement, there is no explanation in the main paper. Labeling the tree's nodes is an important part of the algorithm and should be included in the main paper.\n+ Page 5, the authors used \"Hierarchy Ablation\" and \"Loss Ablation\" subtitle in the paper. The word \"Ablation\" seems inappropriate in this context.\n\nReproducibility\nThe paper describes all the algorithms in full detail and provides enough information for an expert reader to reproduce its results. However, the authors did not discuss when they start to increase the W in equation 3, how to determine to stop the W from increasing, and with what rate the W should be increased.", "extra_scores": {}, "extra_text": {}}, {"review_id": "BBZBZrza9qP", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Aim to improve the interpretability and the accuracy of the neural network, this paper takes a step further on the integration of NN with a decision tree. It will replace the final linear layer of the NN with a decision tree induced by pre-trained model weights. It takes advantage of both hard and soft decision trees and designs suitable tree supervision loss thereon. Extensive experiments verify the design choice of the proposed components. On both small-scale and large-scale datasets, it beats the decision tree counterparts. Also, on the aspects of generalization and interpretability, it shows the strength compared to NN.\n\nThis work is a good try to combine the two techniques NN and decision tree. It finally makes the combination to achieve comparable accuracy with the NN and also enjoy the benefit in the aspects of generalization and interpretability. Recent SOTA of capsule networks which are based on the NN backbone and this work are both achieved comparable performance with NN. They show a promising direction for studying representation learning. Researchers can delve deeper based on this work to further exploit how to integrate decision tree into NN and the characteristics of the combination (e.g. adversarial examples). \n\nWith the decision tree, we can visualize the decision process the bring the benefits of interpretability. The paper proposes to label the decision nodes with WordNet and show the applications of zero-shot generalization, high-level concepts, dataset debugging, and improved human trust. There are lots to do on the aspects. Also, the zero-shot and high-level concept experiments are really intriguing. Using the pre-trained model weights to construct the tree and the proposed tree losses to train can help the generalization in such a significant way, though the performance would depend on the accuracy of the superclasses labeling and the agglomerative clustering. Where the benefits come from? The method is only used the same information as the NN and the tree is also constructed based on the pre-trained weights. Does the way of making hierarchy decisions help here? If you do not enforce the second term of the equation (3), will the phenomenon be the same?\n\nOverall, the paper is very easy to follow and the figures really help understanding. Extensive experiments help to know the performance, effectiveness of the proposed components, and also its unusual applications.\n\n----------------------------\nSome concerns and comments are listed below:\n\nWill you update the weights of the intermediate nodes? \n\nOn large-scale datasets, the paper currently only tests using the EfficientNet. The reviewer wonders if the author can use more advanced backbones to see the performance changes.\n\nThe reviewer is unsure of the specific way to label the decision nodes. Will you use the wordvec provided in the wordnet and compare it with the decision nodes' feature? Since your structure is different from the WordNet, how do you match the classes with the nodes?\n\n-----------------after rebuttal------------\n\nI would like to keep my origin score due to the pros listed above.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper initially received mixed ratings but after the rebuttal, all reviewers recommended acceptance. Reviewers appreciate the novel technical ideas and extensive experimental results. 
Decision: Accept (Poster)