Extracting Local Reasoning Chains of Deep Neural Networks
Authors: Haiyan Zhao
Tianyi Zhou
Guodong Long
Jing Jiang
Chengqi Zhang
Venue: Accepted by TMLR ()

Abstract: We study how to explain the main steps of inference that a pre-trained deep neural net (DNN) relies on to produce predictions for a (sub)task and its data.
This problem is related to network pruning and interpretable machine learning with the following highlighted differences: (1) fine-tuning of any neurons/filters is forbidden; (2) we target a very high pruning rate, e.g., ≥ 95%, for better interpretability; (3) the interpretation is for the whole inference process on a few data of a task rather than for individual neurons/filters or a single sample.
In this paper, we introduce NeuroChains to extract the local inference chains by optimizing differentiable sparse scores for the filters and layers, which reflects their importance in preserving the outputs on a few data drawn from a given (sub)task. 
Thereby, NeuroChains can extract an extremely small sub-network composed of critical filters exactly copied from the original pre-trained DNN by removing the filters/layers with small scores. 
For samples from the same class, we can then visualize the inference pathway in the pre-trained DNN by applying existing interpretation techniques to the retained filters and layers.
It reveals how the inference process stitches and integrates the information layer by layer and filter by filter. 
We provide detailed and insightful case studies together with several quantitative analyses over thousands of trials to demonstrate the quality, sparsity, fidelity and accuracy of the interpretation. In extensive empirical studies on VGG, ResNet, and ViT, NeuroChains significantly enriches the interpretation and makes the inner mechanism of DNNs more transparent. 

Guodong Long Published in Transactions on Machine Learning Research (11/2022)
Extracting Local Reasoning Chains of Deep Neural Networks
Haiyan Zhao University of Technology Sydney
Haiyen.Zhace www.oaaaaaaddee....... eduu.
Tianyi Zhou University of Moryland
zhou@ummiass. und edu
guodong long edu
University of Technology Sydney
Jing Jiang
jing. jinggj utts. edu
University of Technology Sydney
Chengqi Zhang
Cherggii. Zhangg Cutss edu.aw au
University of Technology Sydney
Reviewed on OpenReview: https: // opwp://veeeeewee....mmmm net/ forum? id= RP60787uD8
Abstract
We study how to explain the main steps of inference that pre-trained deep neural net (DNN) relies on to produce predictions for (sub)task and its data. This problem is related to network pruning and interpretable machine learning with the following highlighted differ- ences: (1) fine-tuning of any neurons/ neurons/filters /filters is forbidden; (2) we target a very high pruning rate, e.g., 2 95%, for better interpretabillty; (3) the interpretation is for the whole inference process on few data of task rather than for individual neurons/fill /fllttesss or a single sample. In this paper, we introduce NeuroChains to extract the local inference chains by optimiz- ing differentiable sparse scores for the filters and layers, which reflects their importance in preserving the outputs on few data drawn from a given (sub)task. Thereby, NeuroChains can extract an extremely small sub-network composed of critical filters exactly copied from the original pre-trained DNN by removing the filters/ /layers with small scores. For sam- ples from the same class, we can then visualize the inference pathway in the pre-trained DNN by applying existing interpretation techniques
the retained filters and layers. It reveals how the inference process stitches and integrates the information layer by layer and filter by filter. We provide detailed and insightful case studies together with several quan- titative analyses over thousands of trials to demonstrate the quality, sparsity, fidelity and accuracy of the interpretation. In extensive empirical studies on VGG, ResNet, and VIT, NeuroChains significantly enriches the interpretation and makes the inner mechanism of DNNs more transparent.
1 Introduction
Deep neural networks (DNNs) greatly reshape variety of tasks object classification, semantic segmenta- tion, natural language processing, speech recognition, robotics, etc. Despite theirs success vast majority of clean data, DNNs are also well-known to be sensitive to small amounts of adversarial noises. The lack of sufficient interpretability about their success or failure is one major bottleneck of applying DNNs to important areas such as medical diagnosis, public health, transportation systems, financial analysis, etc.
Interpretable machine lcarning has attracted growing interest a variety of areas. The forms of interpre- tation vary across different methods. For example, attribution methods (Bach et 2015 [Sundararajan et al. 2017 Shrikumar et al. 2017 Montavon al.] 2017 Kindermans et. al.] 2017 Smilkov et al. 2017)

---

Published in Transactions on Machine Learning Research (11/2022)
produce the importance score of each imput feature to the output prediction for given sample, while some
other methods Zeiler & Fergus 2014
Simonyan et al., 2013 Erhan et al. 2009) aim to explain the general functionality of each neuron/filte filter or an individual layer regardless of the input sample. Another line of
works [Ribeiro et al. 2016 Wu et al. 2018 Hou & Zhou 2018 cxplain DNNs in a local region of data space by training a shallow (e.g., linear) and easily interpretable model to approxinate the original pre-trained
DNN on some locally similar samples. Thereby, they reduce the problem to explaining the shallow model. These methods essentially reveal the neuron neuron correlations (e.g., input output, intermediate layer /neronn to output, etc), but they cannot provide an overview of the whole inference process occurring inside the complicated structure of DNNs.
In this paper, we study a more challenging problem: Can we unveil the major hidden sleps of inference in DNNs and present them in a succinct and human-readable form? Solving this problem helps to answer many significant questions, e-g. which process? Can two similar samples share most inference steps? Do all samples need the same number of neurons to locate the key information leading to their correct predictions? Do DNNs rely on entirely different neurons filters for different (sub)tasks? Some of them are related to other problems such network pruning (Han et al., 2015 Li ct al. 2016) and neural architecture search (NAS) (Zoph & Le 2017). For example, are winning tickets (Frankle & Carbin 2018 Liu et al., 2018) universal over different tasks or classes? Does the weight sharing scheme in recent NAS methods (Pham et al. 2018 Liu et al. 2019 9a Ying
plays the most/least important role in the inference
et al. 2019) limit the searching space or quality?
We develop an efficient tool called NeuroChains to extract the underlying inference chains of DNN only for subtask. A subtask of classification task can be defined classification a small subset of classes, which in this paper refers to a few data drawn from subset of 1000 classes in ImageNet. NeuroChains aims to extract a much smaller sub-network composed of a subset of neurons ceurons/fileers /filters exactly copied from the original pre-trained DNN and whose output for data from the subtask is consistent with that of the original pre-trained DNN. While the selected filters cxplain the key information captured by the original pre-trained DNN when applied to data from the same lask/clas information sequentially, i.e., step by step and layer by layer, and recover the major steps of inference that lead to the final outputs. We parameterize the sub-network as the original DNN with an additional score multiplied to each filter /laye's output featuremap. Thereby, we formulate the above problem of sub-network extraction as optimizing differentiable sparse scores of all the filters and layers to preserve the outputs on all given samples.
the architecture of the sub-network stitches these
The above problem can be solved by an efficient parameters. The objective is built upon the Kullback-Teiiber Leibler (KT.) divergence between the sub-network's output distribution and that of the original pre-trained DNN, along with an l regularization for sparse scores over filters. We further use sigmoid gate per layer to choose whether to remove the entire layer block. The gate plays an important role in reducing the sub-network size since many data and subtasks do not rely on all the layers. We extract chains from the sub-network and visualize their filters and featuremaps of the original network by existing methods (Zeiler & Fergus 2014 Erhan et al. 2009).
that only updates the scores with fixed filter
NeuroChains is a novel technique specifically designed for interpreting the local inference chains of DNNs. As aforementioned, it potentially provides an efficient tool to study other problems in related tasks. However, it has several fundamental differences to network pruning and existing interpretation tasks, which deter methods developed for these two problems from addressing our problem. Comparing to network pruning: (1) fine-tuning is not allowed in NeuroChains; (2) it targets a much larger pruning rate for succinct visualization, e.g., 2 95% for VGG-19 (Simonyan & Zisserman 2014). 2 99% for ResNet-50 (He et al. 2016) and 2 92% for ViT (Dosovitskiy et al. 2021 Touvron et al. 2021 on ImageNet. with S 200 filters or patches remained; (3) it is for a few samples drawn from a task instead of the whole data distribution. Comparing to mainstream interpretation tasks: (1) NeuroChains produces an interpretation of the entire inference process for a specified task (e.g., a subset of classes) rather than of one couron/filter or output on/
on/arouno /around a single sample; (2) NeuroChains provides complementary information to the importance of individual neurons ceurons/filters filters for a task.
2

---

Published in Transactions on Machine Learning Research (11/2022)
2 Related Works
Interpretable machine learning methods can be mainly categorized into the ones aiming to evaluate the importance of each input feature of a single sample and the ones explaining individual neurons/filtes Approaches in the first category usually rely on certain back paok-propagation from the DNN's output to derive an importance score for each input feature or hidden node. Earlier works are based on the back-propagated gradients, c.g.; doconvolution (Zeiler & Fergus, 2014), back-propagation (Simonyan et al., 2013) and guided back-propagation (Springenberg et al. 2014). (Sundararaaan et al. (2017) proposed to (approximately) caleu- late the integral of the gradients along path between baseline point and the input sample, which ensures the scnsitivity and implementation invariance lacking in some previous methods. More recent methods pro- pose novel back-propagation rules to directly derive the altribution scores of neurons from output to input, e.g. DeepLIFT (Shrikumar et al., 2017), deep Taylor decomposition MMnnavvnn et al. 2017]. and layer-wise relevance propagation (LRP) (Bach et al. 2015).
Methods in the second class treat DNNs as black boxes and seek simple models to explain how the DNN's output changes in local region. For example, Ancona et al.) (201)) add perturbations to different parts of the input to evaluate how the perturbations change the output, which reflect the importance of different parts. Zeiler& & Fergus (2014) covered different parts of the input with a gray square, which led to different prediction probabilities on the true class. Instead, Zintgraf et al. (2017) replaced each patch in the input image with the surrounding patch and tracked the induced changes in the output. In LIME, Ribeiro et al. (2016) trained a sparse linear model on noisy input-outpul pairs as a. local surrogale approximating the original pre-trained DNNs, where the sparse weights are used to explain the importance of input features. As mentioned before, our main difference to the above methods are we explain DNNs for (sub)task and its data and we further explain how DNNs step by step integrate the information of important filters/neurons
Network pruning Han et al. 2015 and Li et al. (2016) remove redundant neurons/nodes or connec- tions/ween hoons/eights from a pre-trained DNN and fine-tune the sub-network. Structural pruming removes whole aayers/channlls filters/ ceatters/nnuoonn /neurnns according to a certain norm of the associated weights (Li et al., 2016) or sparsity (Hu et al.] 2016). In contrast. Frankle & Carbin (2018) and Liu et al. (2018) prune a DNN during its training. Luo et al. (2017) apply pruning to two adjacent convolution layers at each time to take the dependency between the two layers into account. Liu et al. (2019b) and Guo et al. (2020) train sub-networks of different sizes in large DNN at the same time to satisty various constraints. Several recent works (Su ct al. 2020 Evci et al. 2020) empirically verify "lottery ticket hypothesis", i.c., there exists sub-networks (i.e., winning tickets) that can reach comparable generalization performance as the original pre-trained DNN if re-trained. In contrast, the sub-network extracted by NeuroChains cannot be fully re- re-trained since it has to preserve the original pre-trained DNN's filters, and our goal to retain the generalization performance only for a task with few data.
3 NeuroChains
3.1 Problem: Extract Sub-networks
Although the DNNs widely used nowadays are usually composed of hundreds of layers and millions to billions of hidden nodes. When applied to samples from subtask (e.g., composed of two classes), it is plausible that its inference process mainly relies on a small subset of layers and filters. In this paper, we verify this conjecture by developing an efficient and practical algorithm, i.e., NeuroChains, to extract the subset and its underlying architecture as a sub-network whose filters are selected and exactly copied from the original pre-trained DNN while its outputs for a given subset of data or classes retain the ones produced by the original pre-trained DNN. Although DNNs are usually non-smooth in definition if using a non-smooth piece- wise activation such as ReLU, when trained with the commonly used techniques, e.g., data augmentation, mix-up, dropout, the resulted DNNs are relatively smooth in sufficiently small local region.
In order to preserve the original inference chains, we do not allow any fine-tuning or re-training on any filter or the weight vector corresponding to any neuron: they pre-trained DNN. Let F(; [W'Je1::) (a mapping from input to output) denote the original pre-trained DNN, We represents the set of filters/weightt vectors in layer-f, and W represents the ith filter/ weightt
only be exactly copied from the original
3

---

Published in Transactions on Machine Learning Research (11/2022)
vector in layer-f. Any sub-network fulfilling above requirement can be defined and parameterized by an indicator vector M per layer, whose each entry is a {0, 13 value indicating whether retaining the associated filtr/neeron neuron in W We further define operator o
[.. 0,
M 1; M =
(W o M9
(1)
Thereby, [.M defines a qualified sub-network for inference chain and its weights are {W" o M' 1-=1-L where we extend the operator to make W M = (W ]=1:1 given the original pre-trained DNN's weights W JEm.L. Given a set of samples X' drawn from a specific subtask, we can formulate the problem of finding an inference chain as the following combinatorial optimization, which aims to find the most sparse indicator M (i.e., the sub-network with the fewest filters retained) that does not change the outputs of the original pre-trained DNN for Vo E X, i.e.,
Il s.t. F(e:)) o); Vr x. min (M (MCTe
However, it is impractical to directly solve this combinatorial optimization since the possible choices for M is exponential number. We relax the 0-1 indicator vector M to a nonnogative-valuee score vector 5th of the same size. We define operator @ applicd to W and its associated scores 5 as
(W' @ S s* W"]..
(3)
Note we limit entries in St within [0, 1] due to the possible redundancy among filters in the original pre- trained DNN, i.., there might be filters of similar functionality for the given samples and a preferred pruning should be able to only preserve one of them and multiply it by the number of those redundant filters in the sub-network. In addition, less constraints are casier to handle in optimization and helpful to find sub- network whose outputs are closer to that of the original pre-trained DNN, since the class of sub-networks with parameters S includes all the sub-networks with parameters WoM. Hence, we relax the challenging combinatorial optimization to the following optimization, i.e.,
min (emss
Σ
Σ ||s
(4)
X
where I(.--) is loss function aiming to minimize the distance between the original pre-trained DNN's output, F(;; W) and the sub-network's output F(a; W @ S). In our experiments, for classification, we use KT.ddivegeence between the output distributions over classes, where the two output distributions are computed by applying softmax to F(x; W) and F(x; W @ 5) respectively, i.e.,
[(F(;; W), F(x; W + S)) = (softmaxxx W)) (Fr
(5)
In addition, empirical evidence (Kuueger et al. 2017 Singh et al. 2016) show that for most samples there exist some layers that can be entirely removed without changing the final prediction. Hence, only a few hard and confusing samples need more delicate features, while most other samples can be correctly classified based on simple palterns from shallower layers. Therefore, in NeuroChains, we apply a sigmoid function with input score a a gate G determining whether to remove the entire layer-l during pruning, i.e.
G 1/ exp(-a'
where T is a lemperature parameter. With gate G applied after each layer-f whose input and output has the same size (which is common in many DNNs), we can recursively define the input H++1(.) to the next layer-(E + 1), i.c., He+1( (W O "a he=rt)
d C'F'(H o +1 c.).''
[w" os" a (e=-e))) if imput size output size w' os)
(7)
otherwise
where ('H:; W 3 8²) denotes the output of layer-f. The reason to gate here is that we expect to either remove the whole layer or retain it without adding an extra shortcut (which will change the original pre-trained DNN's architecture). Since prefer to remove non-informative layers, in the objective, we add
4

---

Published in Transactions on Machine Learning Research (11/2022)
another regularization a' to encourage the removal of entire layers (because decreasing reduces G and thus increase the chance layer removal). Therefore, the final optimization for NeuroChains is
(st bent [X| E ZEN (F(2;; W), H
H (x;
(8)
(W os aa be=1:))) + + E. |5 Xg c=1 a,
Our objective above is similar to the one used in Network Slimming (Liu et al. 2017) but we optimize it for subtask (so we can consider to remove layers) and we do not allow fine luning on weights W.
3.2 Algorithm
Our algorithm is simply a standard back- back- propagation for the optimization problem in Eq. which produces sparse scores for filters and gate values for layers. Note the weights in W are fixed and the backpropagation only updates S. We initialize the filter scores = I so W @ W at the beginning of optimization. We initialize the gate score a for all 1: so G 0.5 at the beginning, ie., the probabilities to remove or to retain layer is equal. For classification, we set loss ((--)) to be the KL-divergence between the output distributions of the original pre-trained DNN and the sub-network. After convergence of the optimization, we then apply simple thresholding to these scores to further remove more filters and layers: (1) we remove the filters with score under a threshold т; (2) we remove layer- if G < 0.5. This yields sufficiently small sub-network architecture. Given a sub-network produced by NeuroChains, we then visualize its architecture and scores as the structure the inference chains. Moreover, for samples from the same region, we visualize their inference pathway from the original pre-trained DNN by their activation patterns and featuremaps, respectively, using existing interpretation methods (Zeiler & Fergus 2014 Erhan al. 2009
4 Experiments
Statistics
NesNet-50
VGG-19
Statistics
(ie(TDiT-smaal))
Top-1 lest accuracy
76.5%
72.9%
Top=1 test accuracy
79.9%
Test images/sua /suns//subbbbwkeeee
10000/1681
10000/1746
Test
10000 /5500
Convolutional filters
26560
4480
Patches across blocks
2364
Parameters of Conv-layers
23454912
20018880
Parameters of MHAA-locks
7096320
Parameters of FC-layers
2048000
123633664
Parameters of FFN-blocks 14178816
Table 1: Information of pro-trained DNNs in this paper.
In experiments, we apply NeuroChains to extract the inference chains of widely- VGG-19, ResNet- 50, and ViT which are all pro-trained on ImageNet. We provide the basic information of the three DNNs Table ! In the following, we will present two quantitative analyses over hundreds of case studies, which show that (1) NeuroChains is capable to produce sub-networks retaining only 5% of filters and meanwhile preserve the outputs of the original pre-trained DNN in most cases; (2) the filters selected by NeuroChains with high scores are important to preserving the outputs since removing one will lead to considerable drop in performance. We also compare the capability of preserving the original neural network's outputs between NeuroChains and magnitude- based pruning and random pruning methods. We will then provide several detailed and insightful case studies and visualizations of extracted sub-networks for different subtasks.
4.1 Implementation Details
We implement NeuroChains by PyTorch (Paszke et al. 2017). In every case study, we firstly randomly sample classes ImageNet and then randomly sample 10 images from each class's images. Note that the sampled images may be wrongly classified to other classes by the original DNN. We apply inference on those 20 images and their outputs used in solving the optimization of Eqq (8) in order to extract the local
5

---

Published in Transactions on Machine Learning Research (11/2022)
0.010
0.015
0.008
0.006
0.010
0.004
0.002
0.000
100
150
200
250
300
350
150
200
250
350
Number filters pruning
filters after pruning 24560)
20
Denitty
100
Density
0
0.05 Decrease probability for the originali DNN prodicted dass
Decrease probability original predicted class
Figure Size and fidelity (how well the sub-networks preserve the original pre-trained DNN's outputs) of 1500 sub-networks extracted by NeuroChains for VGG-19(left) and (rihht-50rright) in different case studies (tasks). The x-axis in the top plot refers to the number of retained filters, while the x-axis in the bottom plot is the decrease of probability on the original-DNN predicted class. It shows that NeuroChains can usually find very small sub-networks and meanwhile preserve the original pre-trained DNN's outputs.
inference chain in the form of sub- sub-network. For models with shortcuts, e.g., ResNet-50, the sigmoid gate is applied prune a bottleneck block rather than layer. A layer inside block will be removed if the scores of all filters in the layer are nearly 0.
We use Adam optimizer for the optimization of Eq for filter/layer scores. We use a fixed learning rate of 0.005. We set temperature T = 0.2 in sigmoid gate (Eq. (6) to encourage the value G close to either or 1, and the threshold т to goal scores set to 0.1 so that the outputs of sub-networks are consistent. We only tried a limited number of choices on tens of experiments, and chose the best combination balancing the fidelity and sub-network size, and then applied it to all other experiments without further tuning, In particular, tried e {0.01, 1, 0.5), À {0.001, 0.005, 0.01, and Xg E (1,2,5). For different models, the weights of two penalties in Eq. (8) different. For VGG-19, choose À = 0.005 and Ag = 1 for ResNet-50. This choice performs consistently well and robust on all other experiments. The iteration steps of training is 300 and we stop training when the loss difference is quite small, i.e., less than 0.05. It costs only ~ 90s for VGG-19 and ~ 55s for ResNet-50 to extract a sub-network on a single RTX 6000 GPU since we only optimize a few number of scores.
À x= 0.005 and X9 2. While we
4.2 Quantitative Analyses
Melis & Jaakkola (2018) propose some criteria to evaluate the interpretation methods for DNNs. In this paper, extend some of their notations and present two quantitative analyses of NeuroChains over 1500 case studies for different subtasks, i.e., (1) Fidelity: does the sub-network preserve the original pre- treitraieed DNN's outputs on the given samples? how does it change for sub-networks of different sizes? (2) Faithfulness: how well does the importance score filter reflect the degeneration on the fidelity caused by removing the filter from the sub-network? In this paper, we evaluate the fidelity and faithfulness by the decreasing amount of probability on the original pre-trained DNN's predicted class when using the sub-network for inference. All the above metrics are averaged over 1500 sub-networks and across all the images used to extract each sub-network.
Figure I shows the statistics of the fidelity for sub-networks of different sizes (measured by the number of filters) that are extracted by NeuroChains. Most sub-networks only retain < 1%(5 5%) of ResNet-50(VGG- 19) for succinct visualization but they preserve the outputs of ResNet-50(VGG-19) with high fidelity.
Figure reports the faithfulness of extracted sub-networks, i.c., how a sub-network's performance preserv- ing the original pre-trained DNN's output degrade if removing one filter from it, and what is the relationship

---

Published in Transactions on Machine Learning Research (11/2022)
Decrease of pronabiity for origial-OW pradeted clees
0.7
0.6
0.5
Decraase probablity for origial DMN
0.5
0.4
0.4
0.3
0.3
0.2
0.2-
0.1
0.0
Descorec1. 1.5eccore2200 Score removed filter
1.0escore411 1.5eccoe4222 Score removed filter
Figure 2: Barplot of the faithfulness (filter score us. fidelity degeneration caused by removing the filter) of 783 sub-networks, each extracted by NeuroChains 20 uniform samples randomly drawn from two classes for (lef1 and Reeiet-50(right). The x-axis refers to the interval of filter scores, while the y-axis denotes the decrease of the sub-network's probability on the original-DNN predicted class after removing filter from the sub-network. It shows that the sub-networks suffer from more degeneration if removing a filter with higher score. Hence, the scores faithfully reflect the importance of filters explaining the original pre-trained DNNs.
between this degeneration and the score of the removed filter. The statistics on 1500 sub-networks in Fig- ure 2 show that the scores of filters increase, the fidelity degrades more. The degeneration and the score are strongly and positively correlated, indicating that our optimized scores laithfully reflect the importance of filters in explaining the original pre-trained DNN. Moreover, removing even only one highly scored filter from the sub-network can significantly degrade the explanation performance. Hence, NeuroChains usually find the smallest sub-networks without redundancy among retained filters/layers i.c., every critical inference step retained.
We also present another faithfulness study based on quotient metric defined below. Let P,q E A (A is the probability simplex for e classes) the output probability vectors of the original neural net and the extracted sub-network respectively for same input. We define a quotient metric to measure the change of class prediction between and q, ie.,
1/0l max g[z]
Q(v.g)
ply) max.cie) P[z] y arg maxp
(9)
zE|c]
where y is the predicted class by the original neural net, and Q(p,g) is the quotient of two probability differences computed respectively the original neural net and the sub-network. In particular, it computes the difference of probabilities for class y and the highest-rated other class. The sign of Q(p. indicates whether the predicted class changes (e.g., it changes if Q(,, 4) < 0) while the magnitude of Q(p, 4) measures the change in prediction confidence. The result is consistent with our above observations and is given in Figure I1 of Appendix.
We compare the capability of preserving the original neural network's outputs between NeuroChains and aagnitude-based pruning (removing the filters whose output featuremaps' average magnitude (L2 norm) over all considered samples is small) and random pruning (rby chance") in Figure 3 In particular, under the same setting of each experiment in the paper, prune the original VGG-19 and retain the filters with the largest fcaturemap magnitude in each layer or randomly, 180 in total (more than 157(mean) + 43(std) filters for sub-nelworks extracted by NeuroChains), and we then fine-tune the filters' scores. For Figure 3 the percentage of cases with KL-divergences 0.8 for NueroChians, magnitude-basee based pruning and random pruning are 54.3%, 38.8%, 21.1% 1% respectively, and 9.3%, 19.4%, 38.8% cases have KT-divergence2 1.5. Figure 3 shows the histogram of the KL divergence between the original output class distribution and the one produced by the sub-networks. For sub-networks generated by NeuroChains, the KT-divergence in most

---

Published in Transactions on Machine Learning Research (11/2022)
magnitue-bassed pruning featuremaps (Han 2015a]
NauroChains
Random pruning
0.8
Denaly
0.6
0.4
0.2
KL- diveraence between the outout distrbutions the oriinnal DNN and the sub network
Figure Comparison of different pruning methods on the capability of preserving the original network's output distribution (smaller KL divergence means better preservation) over 783x20 uniform samples.
cases stays close 0, while the output preserving capability of simple pruning is much worse. More complete quantitative analyses for both VGG-19 and ResNet-50 can be found in the Appendix.
4.3
Case Studies
We present three case studics of the sub-networks extracted by NeuroChains. For Figure I and Figure 5 the data points are from classes which are easy to tell apart while Figure images are sometimes mis-classified. The study of multi-classes classification task is shown in Figure 17 in Appendix. The visualization in each case study is composed of two parts: (1) the sub-network's architecture and filter scores; (2) original images from cach class and the visualization of the image's NeuroChains extracted from the original pre- trained model. The true class and predicted class of the sample are shown above the image. They show that NeuroChains considerably enrich the explanation details of DNN's inference process. By connecting the important filters from different layers, the extracted sub-network highlights the main steps leading to the output prediction.
On the sub-network's architecture, and "LO 1" to denote the first filter from this layer. For ResNet-50, we further use "L1B1" to denote the first sub-block in the first bottleneck block, "SC" for the shorteut connection and "C1" for the first convolution layer in the sub-block. The redder the node in the sub-network, the larger the scaling score, conversely, the bluer the node, the lower the score.
"LO" to denote the corresponding convolution layer in VGG-19
NeuroChains are stitched by filters step by step and the visualization of each filter and the corresponding featuremap generated by the original pre-trained model are displayed in each step. The visualization of each selected filter is achieved by maximizing its activation w.r.t. the input. Afterward, shows the patterns that the filter aims to detect which is independent of the input image. More case studies are given in Appendix.
4.4 Detailed Analysis of Case Studies
In Figure 4 and Figure 5] the studies of VGG-19 and Resnet-50 shown. For each sample from the subtask, two NeuroChains which present the inference process of the original pre-trained model are displayed.
In Figure | features extracted by the chains of strawberry gradually evolve from low-level to high-level, e.g., in the first chain, LIBISC_ L2B1SC_ 317, and L3B1C1 23 extract the low-level patterns of red- green color and texture, L3B1C2 18, L3B1C3 _18, and L4BICI_ 417 look like the abstract pattern of strawberry, and L4B1C2 328, L4B1C3 511 and L4B1SC 511 capture the pattern of the skin of strawberry. While the first chain of strawberry focus on the skin, the second chain extracts the pattern of the leaves.
8

---

Published in Transactions on Machine Learning Research (11/2022)
Subnet
Strawberry Strawberry)
convi_1
L281SC_317
L301C2_18
L10TC2_328
L403SC_511
L2815C
1381C2_ 190
L381C3_5
L481C1 127
1481CS 2004
Dalmatian Dalmatian)
convi.c 40
LIBISC 56
L2815C 424
L391C1 97
1381C2.4 4
13BIC3 902
L483C1 207
L4B1C2 48
L481E3 769
LAB3SC 763
convi. 62
L1B15C
L381C1 1381C2 119
L381C3
L4B1C1 289
L481C2
Figure 4: Inference chain by NeuroChains for ResNet-50 (pre-trained on ImageNet) when applied to 20 test images of "strawberry" and "dalmatian". The sub-network retains only 13/67 layers and 77 77/26560 filters of the ResNet-50. Refer to Section 4.4 for detailed analysis of extracted chains.
i.e., L3B1C1 154 and L3BIC3 The leaves in the featuremaps of L4B1CI 127 and L4B1C3 2004 are highlighted. L4B3C3 1760 extract the global patterns of strawberry. Both the two chains of dalmatian extract the global patterns of dalmatian's 's black and white fur. In shallow layers, L3B1C1 _9, L3B1C1 238. L3B1C2 4, and L3B1C2 119 capture a more local black spot pattern of the dalmatian while L4B1C1 289, L4B1C2 48, L4B1C3 769 and L4B3C3 524 reveal the global patterns.
In Figure 5 the featuremaps show that the first chain of kangaroo extracts the pattern of eyes and noses, while the second chain pays more attention the fur of kangaroos. L21 L21_296, L23 393, L32 L53, and L34 66 turn from round pattern to eye and nose pattern. T.21 24, T.32 463, and T.34 188 look like

---

Published in Transactions on Machine Learning Research (11/2022)
Subnet
08
08
19
105
40
13
335
555
203
203
211
177
202
231
121
177
198
421
451
322
991
321
395
499
903
L30
L30
125
Kangaroo (Kangaroo)
L0_31
15_63
L10_20
L19 106
121_296 296
123_ 1333393
L32 453
134 66
LO_21
15_28
130_22
L19_424
121_24 24
123_314
152 468
L34 188
Banana (Banana)
L5. 63
L21.
132
LG 18
L5.
L10 231
L19_455 452
L19.
196
123. 421
L32_458
L34_79
Figure 5: Inference chain by NeuroChains for VGG-19 when applied to images of "kangaroo" and "banana", The sub-network retains only 8/16 layers and 116/448) filters of the VGG-19. The scores for selected filters are represented using the colormap on the top-right. Refer Section for detailed analysis of extracted chains.
10

---

Published in Transactions on Machine Learning Research (11/2022)
the fur pattern of animals. In the first chain of bananas, L19_335, L21 111, 222__00, and L32_ 465 turn from the curved rectangle pattern to hand of bananas. The L322 458 and T.34 79 in the second chain of bananas seem like the pattern of toes and roots of bananas. The corresponding areas in the featuremaps are also highlighted. The two chains focus on bananas' global and local patterns, respectively.
In Figure G we first show the NeuroChains of correctly predicted castle and stone wall. Then, using the same filters from the extracted chains, we draw the featuremaps of a stone wall which is wrongly predicted as the castle by the original pre-trained model, to uncover the inner mechanism of false prediction. The featuremaps of the wrongly predicted stone wall from filters activated by castle show that the featuremaps in the mid-part of the chains are rarely activated, like L21_ 300 and L23 336, which is in line with our expectations. But in the deep layers, L32 15 and L32 45, which activate the roof and body of the castle, also highlight a wide area in the wrongly predicted stone wall image. This indicates that the wrongly predicted stone wall sample contains similar features as castle. The featuremaps of the wrongly predicted stone wall from filters activated by stone wall (correctly predicted) also shows the anomaly. L21 221_197, L25__ 26, and L32_260 which are critical to the correctly predicted stone wall, extract little information from the wrongly predicted stone wall. We find that L21 124 and L32 15 are both important to identify the castle and stone wall, which implics that castle and stone wall have similar patterns and are casily confused.
4.5 Applying NeuroChains to Vision Transformer
To show that NeuroChains can be applied to different types of neural networks for comparing their reasoning chains, we additionally apply NeuroChains to the vision transformer (ViT). Since recent work (Tang et al. 2022) discovered the effectiveness and advantages of patch pruning for VIT, our method produces a score for cach patch in different blocks of VIT. In particular, NeuroChains freezes all the paramcters in ViT and only optimizes the scores for each patch. At the end of patch pruning, patches with small scores are removed. The patch scores are not shared among training samples, i.e., each input sample has its own patch scores, the critical patch for cach sample can be preserved. In Fig. 7 we provide case study of NeuroChains ViT. The top figure show the patches (red squares) retained in each ViT block. In block1, the whole image is taken as the input. As the blocks get deeper, more patches are removed and only few patches are retained in the last few blocks. The pruning results imply that the patch preserved in the last block contains the most critical information to classify the input sample.
To study the inference process of ViT on the given sample, we generate the attribution heatmaps for both the class token and patch token in different blocks. The attribution scores are computed as following:
For ViT with L blocks and N patches, the attention score between patches in block- l can be calculated by Q.K])V where the Qr, Ke and Vz are queries, keys and values in the multi-head attention module of block-l. The attribution of each patch in block-f to the original patches in the input sample is computed as
Attention, = softmari Vduz
Attribution, = Attention, x Attribution, + Attribution
(10)
Attributiong = IN, indicating that in the input all patches only attributes to themselves. Since Attentiong is calculated based on the output patches of the previous block, in Eq. 10, we multiply the Attention, by Attribution;-1 to include the attribution to patches in the current block. We further add Attribution:-1 to reffect the shortcut operation in ViT blocks: the output patches of block-(E - 1), which encode the attribution of patches in block-(l -f- 1), are added directly to the output of block-f.
In the middle row of Fig. 7 we show the attribution heatmaps of the class token in different blocks of the pruned VIT. In the shallow blocks close to the input, it attends all patches with similar attribution scores. As the blocks gets deeper, the class token gradually focuses on the critical patches essential to classify the input image.
In this case study, only one patch token is retained in the last block of the pruned VIT. The attribution heatmaps of this patch are shown in the bottom row of Fig. 7 The patch token pays more attention to itself in the first few blocks but it starts to attend to more important patches in later blocks that are more relevant to the predicted class.
11

---

Published in Transactions on Machine Learning Research (11/2022)
(Castla)
1B
15 63
550 231
122 111
L19.85
D2 12
123
L35
L32.15
L3,63
15_0
130_22
132_15
119_35
121
123_366
125_405
L32_45
Stone Wall
(Castle)
L0.63
15.0
120.22
132_15
L19,35
300
123
125,405
132_45
09_35
Stone Wall (Castie)
15.18
LS 63
L10 22
110 183
131
134
133.48
D25 67
15
Figure 6: Inference chain by NeuroChains for VGG-19 when applicd to images of "Castle" and "Stone Wall" which confuse DNN models and may be mis-classified. The sub-network retains only 10/16 layers and 121 /4480 filters of the VGG-19. Due to space limitation, the subnet can be found in Appendix. Refer to Section 4.4 for detailed analysis of extracted chains.
12

---

Published in Transactions on Machine Learning Research (11/2022)
The results of applying NeuroChains to ViT in Fig. 7 show that the inference process of ViT and CNN are very different. In CNN, from the shallow layers to the deep ones, the model first extracts the low-level local patterns of the input image and then aggregates them to get high-level global patterns step by step towards classifying the input image. In ViT, the model can capture both the local and global patterns the input since the very boutom/early blocks/layers. During inference, the model gradually identifies patches important the classification task and mainly extracts patterns relevant the class token.
Preserved Patches in each block
Attribution heatmaps for class token
Attribution heatmaps for patch token retained in the last block
Block1
Block2
Block3
Block4
Block5
Block6
Block7
Block8
Block9
Block10
Block11
Block12
Figure A case study of applying NeuroChains to vision transformer (ViT). Top: the retained patches (red squares) in each block. Middle: attribution of the class token the original input patches. Bottom: allribution of the patch token (retained in the last block) to the original input patches.
5 Conclusion
We propose an efficient approach NeuroChains to extract the main inference chains/stops in a DNN when applied subtask. We learn sparse scoring of filters/layers extract a sub-network retaining a sub- set of Liters/layeri from the original pre-trained DNN. We provide case studies and quantitative analyses to demonstrate that NeuroChains produces more informative and reliable explanation to the inner infer- ence process of DNNs. Since local reasoning chain extraction is a novel and challenging problem even for classification, we will igate more complicated tasks like regression and detection in the future work.
References
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. A unilied view of gradient-based attribution methods for deep neural networks. In NIPS 2017- Workshop on Interpreting, Explaining and Visualizing Deep Learning. ETH Zurich, 2017.
Scbastian Bach, Alexander Binder, Grégoire Montavon, Froderick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openrevian wwtp://ww......ooooooooooooooooe
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher- features of deep network. Universily of Montreal, 134:1)): 2009.
13

---

Published in Transactions on Machine Learning Research (11/2022)
Utku Evci, T. Gale, Jacob Menick, P. S. Castro, and E. Elsen. Rigging the lottery: Making all tickets winners. ArXio, abs/1911.1113k, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 2018.
Tan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1tf:::441.66777 2014.
Zichao Guo, X. Zhang, H. Mu, Wen Heng, 7. Liu, Y. Wei, and Jian Sun. Single path one-shot neural architecture scarch with uniform sampling. In ECCV, 2020.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advonces in neural information processing systems, pp. 1135-1143, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In
Proceedings of the IEEE conference on computer vision and pallern recognilion, pp. 770 778, 2016.
Bo-Jian Hou and Zhi-Hua Zhou. Learning with interpretable structure from rmn arXiv preprint arXiv:1810.10000 2018.
Hengyuan Hu, Rui Peng, Yu- Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250 2016.
Pieter-Jan Kindermans, Kristof T Schütt, Maximilian Alber, Robert Müller, Dumitru Erhan, Been Kim, and Sven Dähne. Learning how to explain neural networks: Patternnet and patternattributton. arXiv preprint Rtf::770.0000 2017.
David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, Aaron C. Courville, and Chris Pal. Zoneout: Regulariz- ing rmns by randomly preserving hidden activations. In International Conference on Learning Represen- tations, 2017.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608. 100800000, 2016.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In Inter- national Conference on Learning Representations, 2019a. URL http:://oppnreviowwnnet/// wtt.:/rmwwtttoooommmmm S1eYHoC5FX
7. Liu, H. Mu, X. Zhang, Zichao Guo, X. Yang, K. Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp. 3295-3304, 2019b.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2736 2744, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value network pruning. arXiu preprint 1 810. 002000000000 2018
Jian Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 4058-5066, 2017.
A. Madry, Aleksandar Makelov, Ludwig Schmidt, D. Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ArXiv, abs/ 1117770608833 2018.
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self self-explaining neural net- works. In Advances in Neural Information Processing Systems, pp. 7775-7784, 2018.
14

---

Published in Transactions on Machine Learning Research (11/2022)
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojeiech Samek, and Robbrt Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65: 211 222, 2017.
T Nathan Mundhenk, Barry Chen, and Gerald Friedland. Efficient saliency maps for explainable ai. arXiv preprinl 1iv: 112993, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS- W, 2017.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing, In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 4095-01011 4104, 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144. ACM, 2016.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Reseorch, pp. 3145-311111 3153, International Convention Centre, Sydney, Australia, 06 11 Aug 2017. PMLR. URL http::/prroceeednggg mlr. wwr.pressvv0// shrikumar/aa.htm
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXit:1409.1550 1556, 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312, 2. 2.6034, 2013.
Saurabh Singh, Derek Hoiem, and David Forsyth. Swapout: Learning an ensemble of deep architectures. Advances in Neural Information Processing Systems 29, pp. 28-36 36. 2016.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arxiv 706. 008650 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint orXiv: 11..441.2.... 6806, 2014.
Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao, L. Wang. and J. Lec. Sanity-checking pruning methods: Random tickets can win the jackpot. ArXiv, abs/2009.11099 2020.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 94th International Conference on Machine Learning- Volume 70, pp. 3319-3328. JMLR. org, 2017.
Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 121744 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Je- gou. Training data-efficient image transformers & distillation through attention. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, vol- ume 139 of Proceedings of Machine Learning Research, pp. 10347-10351 PMLR, 18 24 Jul 2021. URL html
https://proceedd.g/ mlr. wttp.prees//13//torrrooo
Mike Wu, Michael Hughos, Sonali Parbhoo, Maurizio Zazzi, Volker Roth, and Finale Doshi- Beyond sparsity: Tree regularization of deep models for interpretability. In Thirty-Second AAAI Conference on Arlificial Intelligence, 2018.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS-bench- 101: Towards reproducible neural architecture search. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 7105 7114, 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833 Springer, 2014.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv: 002.2722.44555.. 2017.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. 2017. URL https:
//arxiv.. org/ www./bs/////o/oooooo...o...eeeeo 1611.0157
A Appendix
A.1 Ablation Studies of Layer Pruning
In Fig. 8 we compare the number of filters in sub-networrs extracted with or without layer pruning. When layer pruning is applied, for both VGG-19 and ResNet-50, most sub-networks sizes lie in a small range 0 200. Without layer pruning, much more redundant filters will be preserved the sub-networks. Hence, layer pruning is essential to build small, human interpretable, and effective sub-networks.
0.0175
with layer pruning
with layer pruning
wio layer pruning
0.008
w/o layer pruning
0.0150
0.0125
0.006
Densily
Density
0.0100
0.004
0.0075
0.0050
0.002
0.0025
0.0000
0.000
200
600
-250
250
500
750
1000 1250
1500
1750
Number of filters in each sub-network
Number of filters in each sub-network
Figure 8: Comparison between number of filters in sub-networks extracted with or without layer pruning on 19G-1 and ResNel-50(Right).
A.2 Effect of the Number of Classes
In Fig. 17 we show a case study of applying NeuroChains to 3-classes classification sub-task, demonstrating the capability and effectiveness of NeuroChains on multi-classes problem. However, as classes per sub- task increases, the number of preserved filters in the networkss also increases, which may weaken the interpretability. We test NeuroChains on the 10-classes classification sub-tasks. The averaged number of filters in the extracted sub-networks 795(1206) for VGG-I9((esNNet500) Though much smaller than the original network, the extracted sub-networks are too large to explain.
That being said, NeuroChains can explain the correlation among multiple classes by identifying the chains important to distinguish a class from the others. We conduct a binary-clasiffcation experiment shown in Fig. 10 In this sub-task, samples in one class are all eagles, while samples in another class are randomly selected from different classes, as shown in the top right of the figure.
From the figure, the sub-network extracts two characteristic patterns for eagles. The first chain detects the pattern of feathers, while the second chain extracts the pattern of eyes and heads. These two patterns are
16

---

Published in Transactions on Machine Learning Research (11/2022)
85
VGG-19
Accuracyle sub-nettorrs validation
ResNet-50
80
75
70
65
60
3
5
8
10
15
20
30
Number of samples for each class in sub-task
Figure 9: The validation-set accuracy of sub-networks vs. the number of training samples per class each sub-task.
enough recognize eagles. For images from other classes (a horse image in this case study), the key pattern of the object will be identified. In the figure, the first chain of the horse locates the elbow part. Filter L30 286 contains both the patterns of eyes and elbows. However, in the second chain of the eagle, it only activates the eyes; in the chain of the horse, the elbow of the horse is highlighted. The unique patterns in other objects will be extracted in order to distinguish them from cagles. image, we use the filters in the first chain of eagle to extract patterns for the horse image. The featuremaps show that the cagle filters are not activated on the horse image, so it can be distinguished from cagles.
the second chain the horse
A.3 Effect of the Number of Training Data
Fig. 9 shows how the sub-networks' validation-set accuracy when increasing the training samples per class in each sub-task. For both pre-trained VGG-19 and ResNet-50, when the number of samples class is too small (5 5), the sub-network tends to be overfitting and cannot generalize well to unseen validation data. However, when the number of samples per class increases to 10, the sub-network starts to achieve promising validation-set accuracy. further increasing the samples per class, the validation-sel accuracy quickly saturates. Therefore, in the experiments, 10 samples per class turn be the sweet spot extract sub-networks with sufficiently good generalization performance they can be interpreted as the inference chains of the pre-trained model given sub-tasks.
A.4 Details of the Heatmaps of Features
To obtain the heatmaps, each filter applied in the sub-network produces a featuremap for an input image. By applying an activation function to the featuremap, only a few pixels are activated and they highlight the regions in the featuremap, which represent what features ppttters of the input sample are detected by the features/patterns
filter. Since the featuremap size is always smaller than the input image, we apply the bilinear interpolation to upsample the featuremap up to the input image size. Finally, overlap the resized featuremap with the input image, which results in heatmap highlighting the regions of the input image represented by the corresponding filter.
A.5 More Quantitative Analysis
We performed 783 experiments each using 20 samples uniformly drawn from two classes. We evaluated these newly generated sub-networks using the quotient metric "A quotient of 'diff to highest scoring other class (extracted)) / "diff to highest scoring other class (original)" Eq. (9). We visualized the result in Figure 11
17

---

Published in Transactions on Machine Learning Research (11/2022)
Images of eagles
Subnet
Images randomly sampled
30
302
129
457
eagle (caple)
LO
L5 100
231
351
233
L30 228
L32
LO_6 6663
L5_122 122
L10 28
L19 47
L28 454
L30__88
234
horse (not eagle)
LO 20
L5 122
L10 22
L19 220
L28 93
L30 286
L32 119
L0_18 18
L5_100 100
L10 231
L19_351
L28 233
L30_228 L32_290
Figure 10: Inference chain by NeuroChains for VGG-19 when applied to tell apart images of "cagle" and other randomly selected samples. The sub-network retains only 7/16 layers and 110/4480 filters of the original VGG-19.
The left plot is the histogram of the quotient computed over all the 783x20 samples. The histogram shows that most samples keep the original predicted label after pruning, i..., NeuroChains preserve the original
18

---

Published in Transactions on Machine Learning Research (11/2022)
1.0
preportion of samples negatue quament (90)
1.0
0.8
0.8
0.6
Density
0.5
0.4
0.4
0.2
0.0
Quotient value (Eq. (9))
Score of removed filter
Figure 11:
Lest:Histogram of the quotient metric in Eq. 9 computed over all the 783x20 samples.
Right:Faithfulness of NeuroChains in terms of the quotient's sign.
pre-trained DNN's outputs in most cases. Moreover, the number of filters preserved in these sub-networks is 157(meam) = 43(sdd), which is small enough explain.
The right plot reports the Faithfulness of NeuroChains in terms of the quotient's sign. We remove each filter from each sub-network and report how many samples' predicted labels are changed after the removal, i.e., the quotient negative. Each point in the scatter plot corcesponds to sub-network, the x-axis the score of the removed filter given by NeuroChains, and the y-axis the proportion of samples with negative quotients. The plot shows strong linear correlation between the score of the removed filter and the degradation of faithfulness. Since removing filters with high scores results in more samples with predicted class changing after pruning, the score given by NeuroChains measures the importance of filters in DNN inference.
60
original samples
original samples
adverserial examples generated by FGSM attack adversarial exampics PGD attack
10
adversarial examples generated by FGSM attack adversarial examples generated by PGD attack
70
Denalty
Density
20
10
0.0
0.2
0.6
0.8
1.2 Probabiily ground-truh class produced by sub-networks extracted by NeuroChains
-0.2
0.2
Probabiiiyy of ground-rrth cless produced by the original model
Figure 12: Robustness of original model (Left) vs. extracted sub-networks (Right) under different attacks.
In order to evaluate NeuroChains on the subtasks in the raw input space, we extract sub-networks for uniformly random drawn samples and then evaluate the sub-networks on these samples' adversarial examples generated by two types of attacks: fast gradient sign method (FGSM) (Goodfellow et al. (2014) and projected gradient descent (PGD) et. al. 2018). Figure 12 compares the robustness of the original neural net
19

---

Published in Transactions on Machine Learning Research (11/2022)
LO
L5
L10
L19
L21
L23
L32
L34
Combine ALL
SMOE (Mundhenk et al., 2019) heatmaps of the sub-network extracted by NeuroChains
SMOE (Mundhenk al., 2019) heatmaps of the original model
LO
L2
L5
L10
L19
L23
L25
L32
L34
Combine ALL
SMOE (Mundhenk et al., 2019) heatmaps of the sub-network extracted by NeuroChains
SMOE (Mundhenk et al., 2019) heatmaps of the original model
Figure 13: Case studies of SMOE generated heatmaps for the original network and the NeuroChains extracted sub-network.
(Teft plot) and the extracted sub-networks (Right plot) under different altacks: each of the plots show the histogram of the output probability for the ground-truth class on those samples (original and adversarial). The left plot shows that the two types of adversarial attack are very effective the original neural net in reducing the probability of ground truth class. In contrast, the right plot shows that the NeuroChains extracted sub-networks are much more robust to the attacks, because the optimization in NeuroChains not only removes the irrelevant filters but also strengthens the important filters by assigning them weights > 1. This demonstrates the effectiveness of MeuroChain when applied to subtasks with non-smooth raw-imput space, and the extracted sub-setworks in this case significantly improves the robustness of the original model in defending adversarial attacks.
In Figure 13 show two case studies of comparing SMOE (Mndhenn et al., 2019) generated heatmaps for the original network and the NeuroChains extracted sub-network. We can see that the patterns extracted by the two networks are consistent and are all critical patterns for the class, c.g., the eyes and fists of kangaroos and the feet and face of the horse. However, compared with the original network, these patterns are strengthened in much shallower layers the sub-network, producing better interpretations. This observation is also consistent with the result of analysis on adversarial attacks in Figure
Figure 14: The 20 images from 2 classes used to extract. the sub-network of case study in Figure
20

---

Published in Transactions on Machine Learning Research (11/2022)
Subnet
20
10
1.0
42
0.8
0.6
90
109
220
272
139
07
164
05.
364
150 m
28
224
203
111
05
25
95
300
113
197
313
336
541
174
197
374
184
179
0.0
439
383
405
351
333
250
300
209
350
400
478
478
419
220
10
L10
123
125
L32
The extracted sub-network and 20 images from 2 classes used to extract the sub-network of case study in Figure
Figure 15:
A.6 Case Studies
In Figure [4 the 20 images from 2 classes used to extract the sub-network of the case study in Figure 4 are displayed. All the NeuroChains are extracted from the original pre-trained model based on a few data of a subtask. The extracted sub-network and 20 images from 2 classes used to extract the sub-network of the case study in Figure 0 are shown in Figure
NeuroChains also be used to extract the sub-networks for tasks with multi-classes, and the extracted sub-networt is still small enough to be analyzed and explained. In Figure 16 and Figure 17 the sub-network and the NeuroChains of a 3-classes subtask are presented. The sub-network contains 8 layers and 158 filters, which is easy to visualize. The NeuroChains corresponding to the images in the three classes are displayed.
The chains of orangulans extract features from different parts the orangutan. The top chain orangulan extracts the pattern in the face. L23 and T,25 27 show the round and black node pattern. Then L30 42 and L32 453, these nodes evolve into the pattern of eyes and noses. In L34 112, these features are integrated into the entire face of orangutan. The second chain of orangutan mainly focus the fur and shape of arms. In I.25 220, L330 439 and T.32 376, the elbow part in the featuremap is highlighted, and the visualization of filters also show the pattern of curved shape of the joint part. In the first chain of tram, the pattern of the arranged windows are extracted in L25 202, L30 329 and L32 491 and the windows in the featuremap are highlighted. From the featuremap of the second chain, the bottom part of tram is highlighted and the pattern I.32 seems like wheels, but the visualization of other filters is not readable, which indicate that this visualization method still has limitations. In the chains of broccoli, the head part is extracted by the first chain while the stem part is captured by the second chain. From layer 10 to layer 34 of the first chain, L19 482, L23 83 and L25 247 show the pattern of dense semisphere, and L30_ 440, L32_28 and L.3_428 428 look like the pattern of the head of broccoli. In the second chain, the pattern of closely arranged columns are shown in L30 421, L32 388 and L34 323, which is consistent with the shape the stem of broccoli. From these analysis, we can find that despite the small number of filters in the sub-network, for samples from different classes, more than one critical patterns are captured identify the samples. shows the strong expression ability of neural network and the interpretation ability of NeuroChains.
21

---

Published in Transactions on Machine Learning Research (11/2022)
10
Subnet
132
127
109
126
22
64
183
201
24
35
58
83
196
96
202
223
236
250
184
16
63
50
192
221
182
220
218
253
46
100
105
287
426
270
354
31.
202
119
526
376
435
375
443
388
433
067
454
454
400
Figure 16:
The extracted sub-network and 30 images from classes used to extract the net-neework of case
study in Figure 17
In Figure 18 another case study to classify pineapple and lcopard on Resnet-50 shown. In the first chain, the shallow layers extract low-level patterns like edges L3B1C2 162. L4B1C2 485, L4B1C3 384 and L4B3SC _925 from deep layers evolve these low-level patterns into the crown. In the second chain, the mid-part of the network like L3B1C1 45, L3B1C2 10, L3BIC3 541 and L4B1C1 308 show the pattern of the skin of the pineapple. Filters in the last block, ie. L4B1C2 4, L4B1C3 1010, and L4B3SC 1106 extract the global pattern which look like the main body of pincapples. In the chains of leopard, the body and head of the leopard are lit up respectively. But the visualization of filters in the 2 chains seem similar. The skin marked with black spots is its most obvious pattern like L3BIC2 221, L4B1C1 274, L3B1C3 857 L4B1C3 462 and L4B1C3 769. _69.
In Figure 19 a case study to distinguish indigo bunting from horse on VGG-19 is displayed. The chains of indigo bunting extract the local head pattern and global pattern of indigo bunting respectively. In the first chain, the beak and eye are lit up by L19 497, L23 375 and L25 199. L32 352 and L34 140 integrate previous pattern to the whole head. L19 44, L23 497 and L25 81 in the second chain extract the pattern of feathers. In L32 178 and L34 483, the pattern may also introduce patterns in the first chain and the global pattern including the head are lit up. Similar to indigo bunting, the chains of horse extract the head pattern and body pattern of horse respectively. L32 94 and L34 509 show the pattern like striped muscles. The whole back of the horse highlighted. The second chain focus on the head part of the horse. L25_317 captures the mouth while L.32 30 and L34 156 captures the eye. This study shows that for samples like indigo bunting, the global pattern is learnd by the model to identify the samples. For samples like horse, some local patterus like the body and eyes are enough.
197
22

---

Published in Transactions on Machine Learning Research (11/2022)
Orangutan (Orangutan)
15_63
L1O_83 83
L19_201 204
123_82
125_27
L30 42
132 453
L34 112
L0_58
L5_148
L10_105
L19_287
123_353
L25_220
L30_439
L32_376
L34_445
Tram
(Tram)
28
_5_85
L10 231
L19 368
L23 227
125 202
130_30 329
L32_491 491
134 134_343
LO_58
L5_
L10_5
L19_506
L23_327
L25_95
L30_349
132_5
L34_452
Broccoli (Braccali)
LD_
15_
L10_96 96
L19 482
L2
L25 247
L30 440
L32 _32_281 281
L34
L10_99
192
L25_
Figure 17: Inference chain by NeuroChains for VGG-19 when applied to images of 'orangutan", "tram" " and "broccoli". The sub-network retains only 8/16 layers and 158/4480 filters of the VGG-19 is shown in Figure 16 Refer to Section A.6 for detailed analysis of extracted chains.
23

---

Published in Transactions on Machine Learning Research (11/2022)
110
10.0 8
Subnet
208
208
48
29.0
30.0
11
C4B
20:0
3018
1930
Pineapple
(Pineapple)
Conv1
L1B1SC 195
L261SC_77
L3B1C1 236
L3B1C2
162
L381C3
L4B1C1 292
L4B1C2 485
L4B1C3
L4B3SC 925
Conv1_35
L1B15C 109
L2B1SC
L3B1C1_45 45
L3B1C2_10 10
L381C3 L4B1C1 308 L4B1C2_4
L4B1C3 1010
L4B3SC 1106
Leopard (Leopard)
Conv1 63
L1B1SC 233
L2B1SC_ 324 L3B1C1 12
L3BiC2 221
L3ICC3 331
L4B1C1 274
L4B1C2_300
226
L4B1C3 769
L4B3SC 765
Conv1_43 L1B1SC L281SC L381C1 L381C2 L381C3_85 L4B1C1 L4B1C2 L4B1C3 462 L4B3C2
Figure 18: Inference chain by NeuroChains for ResNet-50 (pre-trained on ImageNet) when applied to 20 test images of "pineapple" and "leopard". The sub-network retains only 16/67 layers and 133/26560 filters of the ResNet-50. Refer to Section A.6 for detailed analysis of extracted chains.
24

---

Published in Transactions on Machine Learning Research (11/2022)
Subnet
111
1350
155
10
1788
1798
24.8
218
234
1950
235
34.8
275
3150
232
241
121
2138
53
1760
4200
indigo bunting
(indlo bunting)
L5
L10 113
L19 497
L23 375
L25 199
L32 352
L34 L34_140 140
L5
L10_76
L19_44
L23 497
L25
L32, 178
L34 483
(Horse)
LO 20
L5 63
L10 22
L19 424
L23 294
L25 398
L32_94
L34_509
LO
LS 28
233
L19 481
L23 432
L25 317
L32
L34 156
Figure 19: Inference chain by NeuroChains for VGG-19 when applied to images "indigo bunting" and "horse". The sub-network retains only 8/16 layers and 107/4480 filters of the VGG-19. Refer to Section A.6 for detailed analysis of extracted chains.
25


Reviews:
[{"review_id": "9SdKN7b8mG", "reviewer": "Reviewer_nEr7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- The paper presents an interesting alternative setting (and method) to explainable AI. Specifically it studies explaining network prediction on a subtask, rather than on a single example as prior works.\n- Detailed analysis on common architectures are presented.\n\n\nWeakness / Concerns:\n- My main conern is that, since the pruning objective is defined on a subtask, is it possible that the pruning process \"overfits\" to the subtask? If so, this might affect the reliability of the resulting subnetworks in representing the model's actual reasoning chain. To what extend can we measure and trust the representability of these subnetworks? Although the weights of these subnetworks directly come from the original model, the pruning process along might still induce biases. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new setting for explainable AI. Concretely, the authors propose to explain the inference process of pretrained deep learning via network pruning. Concretely, pruning the network down to a tiny subnetwork, whose intermediate feature maps can be used to study and explain the reasoning process of the original network. Different from compression, this paper 1). does not allow fine-tuning and only match the output distribution between original and sub networks on subtasks, defined as a subset of labels in this work. The specific pruning method is to optimizer a learnable mask on the model weight, and perform pruning based on the resulting scores.\n\nLeveraging the proposed tool, the author provides case studies on ResNet and VGG to visualize their inference process.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "N/A", "broader_impact_concerns": "Since the subnetworks will be used to analyze and interpret the original model, perhaps the author could discuss any potential biases that might be introduced in the process of generating these subnetworks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7PiSP1kBFO", "reviewer": "Reviewer_JDVr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. NeuroChain is a well-executed and documented methodology for finding sparse subnetworks that can elucidate the inference which a neural network performs during a forward pass.\n\n2. The approach could potentially be applied to any neural network. Thus, it is a general method in spirit.\n\n3. While simple, the method is distinguished from the existing literature.\n\nWeaknesses:\n\n1. The paper only studies ResNet-50 and VGG-19. I would argue that for a modern analysis of neural networks, it would also be interesting to study transformers, e.g. ViT. Trained ViTs are publicly accessible, so I think a study on ViTs (and comparison with ResNet-50 and VGG-19) should be feasible.\n \n2. The Introduction should be more specific what is meant by (sub)tasks. \n\n3. Right now the study is focused on particular neural networks in isolation. It would have been more interesting to consider comparison between different architectures using your tool. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "NeuroChains optimizes differentiable scores on filter outputs and neural network layers, to select a sub-network of a neural network that has predictions matching that of the neural network. There is no fine-tuning of the weights of the neural net.\n\nThe hypothesis of the papers is that while neural networks can be large, when applied to a sample, the \"inference\" relies only a small subset of layers and filters.\n\nThe authors confirm that hypothesis by evaluating trained VGG-19 and ResNet-50 neural networks. The authors show the importance of filters, for example removing only one highly scored filter from the sub-network can significantly degrade performance. Then the authors confirm with analysis that neural networks are making gradually more and more concrete features for a particular class.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "\n1. In general, I think it would be interesting to see more discussion about what are meaningful tasks to be studied with NeuroChains.\n\n2. Right now your method is focused on categorical datasets. Could you elaborate on potentially other types of datasets?\n\n3. Please make a comparison between the inferences of different types of neural networks (in the way they are designed or trained). I think the neural networks should not be studies in isolation, when NeuroChains is applied.\n\n4. How does your method relate to the Lottery Ticket Hypothesis, which asks you to fine-tune the weights from the original initialization? Do you think you are finding new types of Lottery Tickets here? And if so, how are they different from the tickets we are familiar with? Could you elaborate?\n\nMinor: \n\n* in abstract: extra space before a period\n\n* in first paragraph of Intro: despite its <- despite theirs \n\n* 2nd paragraph of Intro: an given <- a given\n\n* whether removing the entire layer <- whether to remove the entire layer\n\nThere are more grammatical errors and typos throughout the text. Please, proofread the text carefully.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pyF1eOuuqQ", "reviewer": "Reviewer_pnfj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the paper is quite well written and easy to understand. The method is simple and probably it is easy implement for a large portion of deep learning researchers and practitioners. I think it is quite surprising that only such a small subnetwork suffices to retain accuracy on a two-way-classification problem. \nMy problem with this is that I did not gain too much out of the visualisations / case studies proposed by the authors, on the other hand \nit is possibly of limited interest how a large network distinguishes only two classes of the large dataset it was trained on although I imagine it being very difficult to do this for a larger amount of classes. See requested changes. \n\nI am not familiar with the related work but except from insights through the case studies, which were very limited for me, I think the proposed method is \"the first thing\" that people would try when given a couple of training points available for pruning in contrast to unsupervised methods. Nevertheless, I think the study is quite well executed and I believe the results. Also I think the study is valuable for a larger audience and of interest to the community. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose to learn masks to either remove filters or whole layers in pre-trained deep convolutional neural networks. This is done on a small subset of a two way classification problem. They find that with their method only a low percentage of filters and layers is needed to retain accuracy. Given these very small subnetwork, they visualise the features and aim to interpret the neural networks computation throughout the layer, for this \"subtask\".   ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. I would propose to include \"convolutional\" in the title since the study, for now, is limited to this architecture type. \n2. Can you include ablation studies how much your whole layer pruning adds to the sparsity / vice versa. \n3. Can you show what happens to your algorithm if you prune on more classes? I think it is interesting how the NeuroChains change when doing so. Identifying chains that are important to distinguish classes from all the others for example. \n4. What happens to the algorithm if you add (remove) more data? It is very surprising to me that you need so little data to train the subnetworks. \n5. Can you include a detailed description of you you compute the feature scores and the heatmaps in the appendix. Or did I missed this? It is not clear to me how these are computed.  ", "broader_impact_concerns": "Does not apply here I think. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a framework for interpreting deep neural networks focused on extracting a very small (e.g. 150 filters out of 4480 filters of VGG-19) sub-network critical dedicated to some aspect of the training task, such as distinguishing between two selected classes from the dataset.

After discussions, the reviewers were unanimously supportive of accepting the paper. They highlighted several strong aspects of the submission:

1. It has the potential to become a new useful tool in the interpretability toolbox. Importantly, experiments are thorough enough to warrant optimism about this direction. As written by Reviewer nEr7 “I think this paper presents a novel aspect to the XAI community. The experiments and discussions are sufficient to justify the objective of this paper.”. Reviewer JDVr also appreciated that the framework seems to be novel. 
2. As highlighted by reviewer pnfj, the result that only a very small network is sufficient for distinguishing between two classes is quite interesting on its own. 
3. The method can be in principle applied to any architecture. In particular, the Authors have added the revision experiments on ViT. The experiments suggest that ViT and CNN arrive at the final decision using a significant difference reasoning mechanism, which again is an interesting result on its own (even if likely known in the field).

Weaker sides of the submission include the lack of head to head comparison to other interpretability methods. While it is challenging to compare interpretability methods, there are certain experiments one could do. For example, one could use the interpretability method in some downstream task (e.g. identifying issues in a dataset). The experiments are mostly narrowed down to CNNs (exception is the ViT experiments in the Appendix). 

All in all, it is my pleasure to recommend acceptance of the paper. Thank you for submitting your work to TMLR. Please make sure you address all comments by the reviewers. I would also encourage, though it is not of course required, to promote ViT experiments to the main text, as these experiments highlight the strong sides of the framework.
Decision: Accept (poster)