Extract Local Inference Chains of Deep Neural Nets
Authors: Haiyan Zhao
Tianyi Zhou
Guodong Long
Jing Jiang
Chengqi Zhang
Venue:  (2021)
Keywords: Model Interpretability
Model Pruning
Attribution
Model Visualization

Abstract: We study how to explain the main steps/chains of inference that a deep neural net (DNN) relies on to produce predictions in a local region of data space. This problem is related to network pruning and interpretable machine learning but the highlighted differences are: (1) fine-tuning of neurons/filters is forbidden: only exact copies are allowed; (2) we target an extremely high pruning rate, e.g., $\geq 95\%$; (3) the interpretation is for the whole inference process in a local region rather than for individual neurons/filters or on a single sample. In this paper, we introduce an efficient method, \name, to extract the local inference chains by optimizing a differentiable sparse scoring for the filters and layers to preserve the outputs on given data from a local region. Thereby, \name~can extract an extremely small sub-network composed of filters exactly copied from the original DNN by removing the filters/layers with small scores. We then visualize the sub-network by applying existing interpretation technique to the retained layer/filter/neurons and on any sample from the local region. Its architecture reveals how the inference process stitches and integrates the information layer by layer and filter by filter. We provide detailed and insightful case studies together with three quantitative analyses over thousands of trials to demonstrate the quality, sparsity, fidelity and accuracy of the interpretation within the assigned local regions and over unseen data. In our empirical study, \name~significantly enriches the interpretation and makes the inner mechanism of DNNs more transparent than before. 

Under review as a conference paper at ICLR 2021
NEUROCHAINS: EXTRACT LOCAL REASONING CHAINS OF DEEP NEURAL NETS
Anonymous authors Paper under double-blind review
ABSTRACT
We study how to explain the main steps/chains of inference that deep neural net (DNN) relies on to produce predictions in local region of data space. This prob- lem is related to network pruning and interpretable machine learning but the high- lighted differences are: (1) fine-tuning of neurons/filters is forbidden: only exact copies are allowed; (2) we target an extremely high pruning rate, e.g.. 2 95%; (3) the interpretation is for the whole inference process in a local region rather than for individual neurons/filters or single sample. In this paper, we introduce efficient method, NeuroChains, to extract the local inference chains by optimizing differentiable sparse scoring for the filters and layers preserve the outputs given data from local region. Thereby, NeuroChains can extract an extremely small sub-network composed of filters exactly copied from the original DNN by removing the filters/layers with small scores. We then visualize the sub-network by applying existing interpretation technique the retained layer/filter/neurons and on any sample from the local region. Its architecture reveals how the inference process stitches and integrates the information layer by layer and filter by filter. We provide detailed and insightful case studies together with three quantitative analyses over thousands of trials to demonstrate the quality. sparsity, fidelity and accuracy of the interpretation within the assigned local regions and over unseen data. In our empirical study, NeuroChains significantly enriches the interpretation and makes the inner mechanism of DNNs more transparent than before.
INTRODUCTION
Deep neural networks (DNNs) greatly reshape variety of tasks - object classification, semantic segmentation, natural language processing, speech recognition, robotics, etc. Despite its success on vast majority of clean data, DNNs are also well-known to be sensitive to small amounts of adversarial noises. The lack of sufficient interpretability about their success or failure is one major bottleneck of applying DNNs to important areas such as medical diagnosis, public health, trans- portation systems, financial analysis, etc.
Interpretable machine learning has attracted growing interest in a variety of areas. The forms of interpretation vary across different methods. For example, attribution methods (Bach et al.) 2015
Sundararajan al. 2017 Shrikumar al.) 2017 Montavon et al.) 2017 Kindermans et al. 2017
Smilkov et al. (2017) produce the importance score of each input feature to the output prediction for an given sample, while some other methods (Zeiler Fergus 2014 Simonyan al.
2013 Erhan
et al. 2009) aim to explain the general functionality of each neuron filter or an individual layer re- gardless of the input sample. Another line of works (Ribeiro et al., 2016 Wu et al. 2018 Hou & explain DNNs in a local region of data space by training shallow (e.g., linear) and eas- ily interpretable model to approximate the original DNN on some locally similar samples. Thereby, they reduces the problem to explaining the shallow model. These methods essentially reveal the neuron to neuron correlations (c.g.. input to output, intermediate layer/ncuron to output, etc), but they cannot provide an overview of the whole inference process occurring inside the complicated structure of DNNs.
In this paper, study more challenging problem: Can unveil the major hidden steps of in- ference in DNNs and present them in a succinct and maman--eadaale form? Solving this problem helps answer many significant questions, e.g., which (nner(s))) plays the most/least im-

---

Under review as a conference paper at ICLR 2021
portant role in the inference process? Do two similar samples really share most inference steps? Do all samples need the same number of neurons/layers to locate the key information leading to their correct predictions? How/when/where does failure happen during the inference neuron(s / ( labels differ? Are DNNs using entirely different parts for data from different local regions? Some of them are related to other problems such as network pruning (Han et al. [2015] Li et al.) 20166) and neural architecture search (NAS) Zoph & Le 2017). For example, are winning tickets (Frankle & Carbin, 2018 Liu et al., 2018) universal over different local regions and classes? Does the weight sharing scheme recent NAS methods (Pham al.) 2018 Liu et 2019
DNN? Which are shared by different samples from the same region even when their
Ying et al., 2019) limit
the searching space or quality?
We develop an efficient tool called Neu- roChains to extract the underlying inference chains of a DNN for a given local data re- gion. Specifically, we aim to extract a much smaller sub-network composed of a subset of neurons/filters exactly copied from the original DNN and whose output for data from the local region stays consistent with that of the origi- nal DNN. In experiments, we assume the data from the same classes reside in a local region. While the selected filters explain the key infor- mation captured by the original DNN when ap- plied to data from the local region, the archi- tecture of the sub-network stitches these infor- mation sequentially, i.e., step by step and layer by layer, and thus recover the major steps of in- ference that lead to the final outputs. Despite its combinatorial nature, we parameterize the nub-network as the original DNN with ad- ditional score multiplied to each filter/layer's output featuremap. Thereby, we formulate the above problem of sub-network extraction as op- timizing differentiable sparse scoring of all the filters and layers in order to preserve the outputs on all given samples.
Figure 1: Inference chain by NeuroChains for ResNet- 50 (pre-trained on ImageNet) when applied to 20 test images of "dalmatian" and "strawberry". Top: The nub-network retains only 13/67 layers and 75/26560 filters the ResNet-50.. The scores for selected fil- ters are represented using the colormap on the top- right. Middle: The per-layer fcaturemaps gener- ated by SMOE (Mundhenk et al.] 2019) show a clear trends of firstly extracting the local patterns (dots on dalmatian and strawberry) and gradually covering a global shapes of the classes. Bottom: Filters with the largest scores are visualized using the method by Er- han et al. (2009). In shallower layers, L4B1C2_48
The above problem can be solved by an ef- ficient back-propagation that only updates the scores with fixed filter parameters. The objec- tive is built upon the Kullback-Leibler (KL) di- vergence between the sub-network's ' output dis- tribution and that of the original DNN, along with f regularization for sparse scores over filters. We further use a sigmoid gate per layer to choose whether removing the entire layer. The gate plays an important role in reducing the sub-network size since most local regions do not rely on all the layers. In practice, we fur- ther apply a thresholding to the sparse scores to obtain an even smaller sub-network and em- ploy an additional fine-tume fine-tuning to the filter scores on the sub-network. We illustrate the sub- network's architecture and visualize its filters and intermediate- ing methods (Mundhenk et al. 2019 Erhan et al.
L4BIC5 C3_524 capture a more local black spot pat- tern of the dalmatian, L3BIC3 668 captures the eyes and nose patterns; L3BIC3 1331C33_818 818 extracts the local color pattern of strawberry. In the last bottleneck layer, L4B3SC. and L4B3SC 1457 capture the global pat- terns of dalmatian's 's black and white fur: L4B3C3 456 and L4B3SC_511 captures the main shape and color of strawberry. It shows an inference chain for straw- berry: LIBISSC_17 1.2BISC_75 and I.2B1SC_342 L.3BIC3 818 + L4B1C2 328 L4B3C3 456 and L4B3SC.511.
featuremaps by exist-
2009).
NeuroChains is a novel pruning technique specifically designed for interpreting the local inference chains of DNNs. As aforementioned. it potentially provides an efficient tool to study other problems in related tasks. However, it has several fundamental differences to network pruning and exist-
2

---

Under review as a conference paper at ICLR 2021
ing interpretation tasks, which deter methods developed for these two problems from addressing
our problem. Comparing to network pruning: (1) fine-tuning is not allowed in NeuroChains; (2) it targets much larger pruning rate for succinct visualization, e.g., 2 95% for VGG-19 and
2 99% for ResNet-50 on ImageNet with < 200 filters remained; (3) it is for data from a local region instead of the whole data distribution. Comparing to mainstream interpretation tasks: (1) NeuroChains produces an interpretation of the entire inference process in local region (which may contain different classes) rather than of one neuron/filter or output on/around single sample; (2) the sub-netwand 's architecture provides orthogonal information to the importance of individual neurons/filters.
2 RELATED WORKS
Interpretable machine learning methods can be mainly categorized into the ones aiming to eval- uate the importance of each input feature of single sample and the ones explaining individual neurons/filters. Approaches in the first category usually rely on certain back-propagation from the
DNN's output to derive an importance score for each input feature or hidden node. Earlier works
are based on the propappopa gradients, e.g., deconvolution (Zeiler & Fergus 2014). back-
propagation (Simonyan et al. 2013) and guided pack-propagation (Springenberg et al. 20144. Sun-
dararajan et al. (2017) proposed to (approximately) calculate the integral of the gradients along a path between baseline point and the input sample, which ensures the sensitivity and implemen- tation invariance lacking in some previous methods. More recent methods propose novel back- propagation rules to directly derive the attribution scores of neurons from output input, e.g., DIe (Shrikumar et al. 2017), deep Taylor decomposition (Montavon et al. 2017). and layer- wise relevance propagation (LRP) (aach et al. 2015).
Methods in the second class treat DNNs as black boxes and seek a simple model to explain how the DNN's output changes in local region. For example, they add perturbations to different parts of the input to evaluate how the perturbations change the output (Ancona et al.) (017), which reflect the importance of different parts. Zeiler & Fergus (2014) covered different parts of the input image with gray square, which led to different prediction probabilities on the true class. Instead, Zintgraf jet al. 2017) replaced each patch in the input image with the surrounding patch and tracked the induced changes in the output. In LIME (Ribeiro et al. 2016), Ribeiro et al. trained sparse linear model on noisy input-output pairs as a local surrogate approximating the original DNNs, where the sparse weights are used to explain the importance of input features. As mentioned before, our main difference to the above methods are we explain DNNs for a local region of multiple samples and we further explain how DNNs step by step integrate the information of important filters/neurons.
2015 Li et al. 2016) remove redundant neurons/nodes or connec- tions/weights from pre-trained DNN and fine-tune the sub-network. Structural pruning removes according to a certain norm of the associated weights (Li et al., 2016) or sparsity (Hu et al. 2016). In contrast, Frankle & Carbin (2018;; Liu et al. (2018) prune a DNN during its training. Luo et al. (2017) apply pruning to two adjacent convolution layers at each time to take the dependency between the two layers into account. Several recent works em- pirically verify "lottery ticket hypothesis", i.e., there exists sub-networks (i.e., winning tickets) that can reach comparable generalization performance as the original DNN if re-trained. In contrast, the sub-network extracted by NeuroChains cannot be fully re-trained since it has to preserve the original DNN's filters, and our goal is to retain the generalization performance only for a local region.
Network pruning (Han et al.)
whole layerr/channels/iitrrrs/eeee
NEUROCHAINS
3.1 PROBLEM: EXTRACT LOCAL INFERENCE CHAINS AS NETWORKS
Although the DNNs widely used nowadays are usually composed of hundreds of layers and millions to billions of hidden nodes. When applied to samples from a local region in data space, it is plausible that its inference process mainly relies on a small subset of layershneroossfilterr. In this paper, verify this conjecture by developing an efficient and practical algorithm, i.c., NeuroChains, to extract the subset and its underlying architecture as a sub-network whose weights/filters are selected and exactly copied from the original DNN while its outputs in given local region retain the ones
3

---

Under review as a conference paper at ICLR 2021
produced by the original DNN. Although DNNs are usually non-smooth in definition if using non- smooth piece-wise wiie activation such as ReLU, when trained with the commonly used techniques, e.g., data augmentation, mix-up, dropout, the resulted DNNs are relatively smooth in sufficiently small local region.
In order to preserve the original inference chains, we do not allow any fine-tuning re-training on any filter or the the weight vector corresponding to any neuron: they can only be exactly copied from the original DNN. Let F(;; [W")r= (a mapping from input output) denote the original DNN, W' represents the set of filters/weight vectors in layer-,, and W [ii represents the j filter/weight vector in layer-f. Any sub-network fulfilling our above requirement can be defined and parame- terized by an indicator vector M per layer, whose each entry is a {0, 1 value indicating whether retaining the associated filter/neuron W We further define operator as
[.], 0,
M 1: M = 0.
(W M
(1)
Thereby, Je=:L defines a qualified sub-network for inference chain and its weights are {W' M' JE=LLL where we extend the operator o to make W o M = { o M Jt=12 given the original Given a set of samples X' drawn from local region of data
DNN's weights W = space, we can formulate the problem of finding an inference chain as the following combinatorial optimization, which aims to find the most sparse indicator M (i.e., the sub-network with the fewest filters retained) that does not change the outputs of the original DNN for Va E X, i.e.
L
El1M s.t, F(;;W) = F(;; W o M), E. Σ
min {M' br
However, directly solving this combinatorial optimization is impractical since the possible choices for M is of exponential number. In this paper, we relax the 0-1 indicator vector M to a real-valued score vector S of the same size. We define an operator @ applied to W and its associated scores S as
(W @ S é [' [W.
(3)
Note we do not limit entries in S within [0, 1] due to the possible redundancy among filters in the original DNN, i.e., there might be filters of similar functionality for the given samples and preferred pruning should be able to only preserve one of them and multiply it by the number of those redundant filters in the sub-network. In addition, less constrains are easier to handle in optimization and helpful to find sub-network whose outputs DNN, since the class of sub-networks with parameters W @ S includes all the sub-networks with parameters W M. Hence, we relax the challenging combinatorial optimization to the following unconstrained continuous optimization, i.e.,
closer to that of the original
1 X
Σ (FF(:;W), F(x; S)) + Σ || DEN
min {S HImL
where I(-,-) is a loss function aiming to minimize the distance between the original DNN's output F(x; W) and the sub-network's 's output F(x; W @ S). In our experiments, for classification, use KL-divergence between the output distributions over classes, where the two output distributions are computed by applying softmax to F(x; W) and F(x; W @ respectively, i...
(5) In addition, empirical evidences (Krueger et al.) 2017 Singh et al.) 2016) show that for most samples there exist some layers that can be entirely removed without changing the final prediction. Hence, only few hard and confusing samples need more delicate features, while most other samples can be correctly classified based on simple patterns from shallower layers. Therefore, in NeuroChains, we apply a sigmoid function with input score a gate G determining whether removing the entire layer-l during pruning, i.e.,
((F(;;W) F(;; W @ S)) = Dx (sof(sssfxax ((:; W)) /softmax F(x;; W O S))).
G² = [1 +exx(xxxaa/m /T)),
(6)
where is a temperature parameter. With a gate G applied after each layer-l whose input and output has the same size (which is common in many DNNs), we recursively define the input
4

---

Under review as a conference paper at ICLR 2021
(+)(() to the next layer-(e + 1), ie., H 1 (z; (We os a )e=1:)
G CF we o + + (1- (2 (We o se, a je-ne-1), if input size output size (F
otherwise
(7) where (H,;; w o 5t) denotes the output of layer-f. The reason to use gate here is that we expect to either remove the whole layer or retain without adding an extra shortcut (which will change the original DNN's architecture). We apply a T < to sharpen the gate values. Since we prefer to remove non-informative layers, in the objective, we add another regularization a to encourage the removal of entire layers (because decreasing a reduces G and thus increase the chance of layer removal). Therefore, the final optimization for NeuroChains is
L a² (8)
I {
Σ (FF((;;;) W), H (a; (W os (e=)))=))) FEX
min (5t,0))(-L
E=1
Our objective above is similar the used in Network Slimming (Liu et al. 2017) but optimize for local region (so we can consider to remove layers) and we do not allow fine tuning on weights W.
3.2 ALGGORITHM
Our algorithm is simply a stan- dard pacpppropagation for the optimization problem in Eq. (8), which produces sparse scores for filters and gate values for layers. Note the weights in W
Table I: Information of pre-trained DNNs in this paper.
STATISTICS
RESNET-50 VGG-19
TOP-1 TEST ACCURACY
76.5%
72.9%
TEST MAGGS/SUB-NETWWOOO
10000/1688
10000/1746
CONVOLUTIONAL FILTERS PARAMETERS OF CONY-LAYERS
26560
4480
are fixed and the backpropaga- tion only updates S. We initial- ize the filter scores S
23454912
20018880
PARAMETERS OF FC-LAYERS
2048000
123633664
W @ S = W at the beginning of optimization. We initialize the gate score a = 0 for all f= 1 L so G 0.5 the beginning, i.c., the probabilities to remove or retain layer is equal. For clas- sification, we set loss ((--)) to be the KL-divergence between the output distributions of the original DNN and the sub-network. After convergence of the optimization, we then apply simple thresh- olding to these scores to further remove more filters and layers: (1) we remove the filters with score under a threshold r; (2) we remove layer-f if G < 0.5. This yields a sufficiently small sub-network architecture but the scores might be sub-optimal in preserving the original DNN's outputs within the local region. Therefore, we further fine-tune the nonzero scores in S (note filters are always fixed), ie, the scores for the retained filters/layrrs, by minimizing Eq. (8) without the two regularizations, the first term in Eq. (8) excceds a threshold Given a sub-network produced by NeuroChains, then visualize its architecture and scores the structure of the inference chains. Moreover, visualize the retained filters and layers on the chains by their activation patterns and featuremaps, respectively, using existing interpretation methods (Mundhenk et al. 2019 Erhan al. 2009).
EXPERIMENTS
In experiments, we apply NeuroChains extract the inference chains of two widely-adopted CNNs pre-trained on ImageNet, i.e., VGG-19 and ResNet-50. We provide the basic information of the two DNNs in Table I In the following, We will present three quantitative analyses over hundreds of case studies, which show that (1) NeuroChains is capable to produce sub-networks retaining only < 5% of filters and meanwhile preserve the outputs of the original DNN in most cases; (2) every filter selected by NeuroChains is important to preserving the outputs since removing one will leads to considerable drop in performance; (3) the sub-network extracted based on a finite number of samples can be generalized to unseen samples in nearby regions. To demonstrate the effectiveness of NeuroChains in local regions in the non- smooth raw- raw-input space, we evaluate the sub-netue network on the adversarial examples of each sample. We also study the firing behaviour of filters in the original network and the sub-network. We will then provide several detailed and insightful case studies and visualizations of extracted sub-networks for different local regions.
5

---

Under review as a conference paper at ICLR 2021
1.0
0.8
orggnn
0.6-
0.8
0.4-
0.2
100
200
300
400
500
600 700 500
Number of filters after pruning
Score filter
Figure 2: Left: Size and fidelity (how well the sub-networks preserve the original DNN's outputs) of 1500 sub-networks extracted by NeuroChains for ResNet-50 in different studies (local regions). The x-axis refers to the number of retained filters, while the y-axis is induced decrease of probability on the original- DNN predicted class. It shows that NeuroChains can usually find very small sub-networks and meanwhile preserve the original DNN's outputs. Right: Faithfulness (filter score and degeneration by removing the filter) of 783 sub-networks, each extracted by NeuroChains on 20 uniform samples randomly drawn from two classes. The x-axis refers to the filter scores, while the y-axis denotes the decrease the sub-network's probability on the original-DNN -NNN predicted class after removing a filter from the sub-network. shows that the sub-networks suffer from more degeneration if removing a filter with higher score (magenta line and the shaded areas show strong linear correlation between the two). Hence, the scores faithfully reflect the importance of filters in explaining the original DNNs.
4.1
IMPLEMENTATION DETAILS
0.8
We implement NeuroChains by Py- Torch (Paszke al.) 2017). In every study, we firstly randomly sample classes and then randomly sample 10 images from each class's images that the original DNN has high confidence, c.g., 95%. Note high confidence does not guarantee the correctness since it can be associated with a wrong class. We choose images with high confidence since the inference on them mostly represent the original DNN's inference chains in the region. We apply inference on those 20 images and their outputs are used in solving the optimization of Eq (8) in order to extract the local inference chain the form of sub-network. For models with shortcuts, e.g., ResNet-50, the sigmoid prune bottleneck block rather than a layer. A layer inside a block will be removed if the scores of all filters in the layer are nearly 0.
0.5
0.4
0.3
6.1
0.0
62
120
140
202
180
200
220
240
262
Numbber nearest nelgrbours
Figure 3: Averaged test fidelity (stability) of 1500 sub-networks extracted by NeuroChains for VGG-19 when applied unseem images nearby local regions. For each sub-network and every image used to extract it. we rank images from the validation set of [Deng] al. (2009) their distance to the image (in VGG- 19 penultimat-llayer output space) and obtain its K- nearest neighbors. We evaluate the test fidelity (accu- racy of preserving the original DNN's predicted class) of the sub-network over these unscem K-NN images for all the images used extract it (duplicates removed). x-axis refers to 20K since each sub-network is ex- tracted for 20 samples, while the y-axis is the test fi- delity averaged over all the 1500 sub-networks.
gate applied
We use Adam optimizer for both the optimiza- tion of Eq (8) and fine-tune phases for fil- ter/layer scores. We use a fixed learning rate 0.005 the former and 0.01 in the latter. We set temperature T = 0.2 in the sigmoid gate (Eq. (6) to encourage the gate value G close to either 0 or 1, and the threshold 7 to goal scores is set to 0. so that the outputs sub-networks are as consistent as possible. We only tried limited number of choices combination balancing the fidelity and neb-network size, and then applied to all other experiments without further tuning. In particular, and Ag E 21,2,5) 5). For different models, the weights of two penalties in Eq. 8 are different. For VGG-19, we use 0.007 and Ag 2. While choose À 0.0065 and X 0.9 for ResNet-50.
tens of experiments, and chose the best
tried E {0.01, 0.1, 0.5). À {0.001, 0.005. 0.01, 0.1),

---

Under review as a conference paper at ICLR 2021
This choice performs consistently well and robust on all other experiments. We fine-tune S in the pruned network if the KL-divergence after training is greater than threshold = 0.1. The iteration steps of training and fine-tune is 300 and 50 respectively. We stop training fine- fine-tuning when the loss difference is less than 0.02. It costs only ~ 90s for VGG-19 and ~ 55s for ResNet50) to extract sub-network single GPU since we only optimize few number of scores.
4.2 QUANTITATIVE ANALYSES
In Melis & Jaakkola 2018), they propose three criteria to evaluate the interpretation methods for DNNs, i.e., faithfulness, stability and explicitness. In this paper, extend some of their notations and present three quantitative analyses of NeuroChains over 1500 case studies for different local regions, i.e. (1) Fidelity: does the sub-network preserve the original DNN's outputs on the given samples? how does it change for sub-networks of different sizes? (2) Faithfulness; how much de- generation on the fidelity will be caused if removing one filter from the sub-network?
(3) Stability: what is the fidelity (the accuracy of preserving the original DNN's prediction) of the sub-network on unseen test images from nearby regions? In this paper, we evaluate the fidelity and faithfulness by the decreasing amount of probability the original DNN's predicted class when using the sub- network for inference; we evaluate the stability by evaluating the test fidelity of the sub-network image's 's K-nearest neighbor images from validation set (details below Figure 14]. All the above metrics are averaged over 1500 sub-networks and across all the images used to extract each sub-network.
119
49
500
10
220
Figure 4: Inference chain by NeuroChains for VGG-19 19 when applied to images of "Indigo bunting" and "sorrel". Left:: The sub-network retains only 9/16 layers and 118/4480 filters of the VGG-19. Middle: In the SMOE featuremaps, the eyes of both indigo hunting and sorrel, the feathers of indigo bunting, and the legs of sorrel are gradually located as the key features. Right: In shallow layers, filters extract local patterns like eyes (L23 198, L23 205 and L25 506) and feathers (L25 294 and L25 413). In the last two layers, L32 30 captures the eyes of sorrel (big and with eyelids) while L32_352 shows the eyes and the whole head of the bird; L32_455 captures the pattern of sorrel legs; L34_490 captures the contour of sorrel's 's main body. It implies inference chain for Indigo bunting: L10_105 L19_103 + L23_200 + L25 5556 and L25 2952294222 L32_352 + L33_483.
The left plot of Figure 2 shows the fidelity for sub-networks of different sizes (measured by the number of filters) that extracted by NeuroChains. Note most sub-networks only retain < 1% of ResNet-50 for succinet visualization but they preserve the outputs ResNet-50 with high fidelity.
The right plot in Figure reports the faithfulness of extracted sub-networks, i.e.. how sub- network's ' performance in preserving the original DNN's output degrade if removing one filter from it, and what is the relationship between this degeneration and the score of the removed filter. The
7

---

Under review as a conference paper at ICLR 2021
statistics on 1500 sub-networks in Figure rightt show that removing even only one filter from the sub-network can significantly degrade the explanation performance. Hence, NeuroChains usually find the smallest sub-networks without redundancy among retained filters/layers, i.c., every critical inference step is retained. Moreover, the degeneration and the score are strongly and positively cor- related, indicating that our optimized scores faithfully reflect the importance of filters in explaining the original DNN. We also present another faithfulness study based on a quotient metric defined below. Let p. q E A (A e is the probability simplex for e classes) be the output probabílity vectors of the original neural net and the extracted sub-network respectively for a same input. We define a quotient metric to measure the change of class prediction between p and q. i.e.,
Q(p. g) = glvl max.. 9[2]
E arg max =E|c]
(9)
plu] jdl P(z]
where y is the predicted class by the original neural net, and (p, q) is the quotient of two probability differences computed respectively on the original neural net and the sub-network. In particular, computes the difference of probabilities for class and the highest-rated other class. The sign of
Q(,. q) indicates whether the predicted class changes (e.g.. it changes if g). < 0) while the magnitude of Q(p.s) 4) measures the change in prediction confidence. The result is given in Figure g of Appendix.
We report the average stability of 1500 sub-networks extracted by NeuroChains in Figure 14 of Appendix. It shows that each sub-network can still achieve high fidelity (the accuracy of preserving the original DNN's output) within a relatively large local region (c.g.. 100 unseen samples) around the 20 samples used to extract the sub-network. It therefore indicates that the inference chains extracted for finite number of samples by NeuroChains can be applied and generalized other unseen samples from the same region or even the nearby regions. So NeuroChains is an effective solution to the problem raised in the introduction.
In order to evaluate NeuroChains on the local regions in the raw input space, we extract sub-networks for uniformly random drawn samples and then evaluate the sub-networks on these samples' adver- sarial examples generated by two types of attacks: fast gradient sign method (FGSM) and projected gradient descent (PGD). Figure [ compares the robustness of the original neural net (LEFT plot) and the extracted sub-networks (RIGHT plot) under different attacks: each of the plots show the histogram of the output probability for the ground-trute class on those samples (original and adver- sarial). The left plot shows that the two types of adversarial attack are very effective on the original neural net in reducing the probability of ground truth class. In contrast, the right plot shows that the NeuroChains extracted sub sub-networks are much more robust to the attacks, because the optimization in NeuroChains not only removes the irrelevant filters but also strengthens the important filters by assigning them weights > 1. This demonstrates the effectiveness of NeuroChains when applied to local regions in the non-smooth raw-input space, and the extracted sub-networks in this case significantly improves the robustness of the original model in defending adversarial attacks.
ong ter
3 51 excrpiies gensioeec FORM stock a2vara ecampies ganamnac by PGD stack
genersted PGD atruck
80
30
20
2
0.0
0.2 0.4
0.5 0.5
0
Probabiitty of gww.nd--uut class produced orginaal modal
Photaaall ofg.raadddnnuuu
wab-naaoorra sotructed NaureCham
Figure 5: Robustness of original model (LEFT) vs. extracted sub-networks (RIGHT) under different attacks. The firing behaviour of the same filter in the original network and the sub-network is usually differ- ent since NeuroChains combines similar filters by assigning scores/weights > to some filters. For this reason, one filter's 's featuremap produced in the sub-network is usually a combination of the fea- turemaps produced by multiple (pruned) filters from the original network, so the correlation between the same filter's featuremaps in the two networks can be small. In this way, we can better compress the networks to sub-networks with much fewer filters that are casier to interpret while preserving the original network's inference process. To this phenomenon, we present two case studies in
8

---

Under review as a conference paper at ICLR 2021
Figure 6 It shows that each featuremap of the original network's filter only represents part of the object, while the featuremap of the preserved filter in the sub-network combines all the parts to form complete representation of the object in the image. More complete quantitative analyses for both VGG-19 19 and ResNet-50 can be found in the Appendix.
Acandder femonme scling
Biss Eonelaye
Faskrenus
such Dhe M011
forare
10 of sntvort
Figure 6: Firing behaviour (featuremaps) of filters in the original network and the sub-network.
4.3
CASE STUDIES
We present three case studies of the sub-networks extracted by NeuroChains For Figure and Figure 4 the data points from classes which are easy tell apart while in Figure images are always mis-classified. More case studies are given in Appendix. The visualization in each case study is composed of five parts: (1) the sub-network' 's architecture and filter scores; (2) the 20 images from 2 classes used to extract the sub-network. The sub-network is supposed to explain the inference chains on images from the same local region of those 20 images; (3) one image from each class and the visualization of the image's per-layer featuremaps in the sub-network, which is produced by SMOE (Mundhenk et al. 2019): (4) for comparison, we also present the interpretations produced by two recent works (Wang al. 2019 Nam et al.] 2019) of each filter retained the sub-nect network using Erhan et al. (2009) after zoomed-in. In the caption below each case study, we highlight some filturs/fctturemass and their related class. We also provide examples of inference chains composed of filters stitched by the sub-network's architecture. They show that NeuroChains considerably enrich the explanation details of DNN's inference process in a local data region. By connecting the important filters from different layers, the extracted sub- network highlights the main steps leading to the output prediction.
the two images; (5) visualization
244
1:3
524
2246
L33
D21
576
433
6570
125
100
100
500
13
10
1640
21
L1
12
2944
114
116
100
251
Irout Iaggingsssn mangsannnnaaaaaa
1234
Figure 7: Inference chain by NeuroChains for VGG- 19 when applied to images of "Castle" and "Stone Wall"
which confuse DNN models. Left: The sub-network retains only 10/16 layers and 138/4480 filters of the VGG- 19. Middle: In the SMOE featuremaps, the wall of castle and the windows of the wall are gradually located
the wrong key features. Right: For castle: L.4 96 - I.19 145 + 123 216 and 1.23 220 - 1.28 283 + L32 351: For stone wall: L14 20 L19 364 - L23 449 L28 378 L32 45. Filters like L19 364, L23_216 and L28_378 confuse the model and cause wrong predicts since they capture the patterns of both
classes.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. A unified view of gradient- based attribution methods for deep neural networks. In NIPS 2017-Workshop on Interpreting, Explaining and Visualizing Deep Learning. ETH Zurich, 2017.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Rober Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, :0014,0, 2015.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Dumitru Erhan, Yoshua Bengio. Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 1341(3):1, 2009.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 2018.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015.
Bo-Jian Hou and Zhi-Hua Zhou. Learning with interpretable structure from mn. arXiv preprint arXiv:18100 10708, 2018.
Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03225 2016.
Pieter-Jan Kindermans, Kristof T Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Erhan, Been Kim, and Sven Dähne. Learning how to explain neural networks: Patternnet and patternat- tribution. arXiv preprint arXiv:7005.05598, 2017.
David Krueger, Tegan Maharaj. János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, Aaron C. Courville, and Chris Pal. Zone- out: Regularizing rnns by randomly preserving hidden activations. International Conference on Learning Representations, 2017.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 16Xiv: 08710, 2016.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019. URL. /tt.hhtttts:://eeeeeeii
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2736-2744, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05277 2018
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruming method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5066, 2017.
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems, pp. 7775-7784, 2018.
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65:211-222. 2017.
10

---

Under review as a conference paper at ICLR 2021
T Nathan Mundhenk, Barry Y Chen, and Gerald Friedland. Efficient saliency maps for explainable ai. arXiv preprint arXiv:1911.112220 2019.
Woo-Jeoung Nam, Shir Gur, Jaesik Choi, Lior Wolf, and Seong-Whon Lee. Relative attributing propagation: Interpreting the comparative contributions of individual units in deep neural net- works, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitito Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, W, 2017.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In Proceedings of the 35th International Conference Machine Learning, volume 80, pp. 4095-4104, 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144. ACM, 2016.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3145-3153, International Convention Centre, Sydney, Australia, 06- 11 Aug 2017. PMLR. URL http:ttt///c http::/pproeeedigggg/e/// mlr html
www.:mw....reseeemmmmmmmmm wrw.s/v70 shr kumar la.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: networs. Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
Saurabh Singh, Derek Hoiem, and David Forsyth. Swapout: Learning ensemble of deep archi- tectures. In Advances in Neural Information Processing Systems 29, pp. 28-36. 2016.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03880 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 1422,6806 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference Machine Learning- Searning-Volume 70, pp. 3319- 3328. JMLR. org, 2017.
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks, 2019.
Mike Wu, Michael C Hughes, Sonali Parbhoo, Maurizio Zazzi, Volker Roth, and Finale Doshi- Velez. Beyond sparsity: Tree regularization of deep models for interpretability. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS- bench-101: Towards reproducible neural architecture search. In Proceedings of the 36th Interna- tional Conference on Machine Learning, volume 97, pp. 7105-7114, 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint 17Xv::700..04955 2017.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. 2017. URL httpss:/erre http::///rwvvv.o//o/oo////////// www./os.g/////oooooooooooooo 115178

---

Under review as a conference paper at ICLR 2021
5 APPENDIX
5.1 QUANTITATIVE ANALYSIS
We performed 783 experiments each using 20 samples uniformly drawn from two classes (not only the high-confidence samples) and achieved 783 new sub-networks by NeuroChains on VGG-19. We evaluated these newly generated sub-networks using the quotient metric "A quotient of "diff to highest scoring other class (extracted)" "diff highest scoring other class (original)" Eq. (9) We visualized the result in Figure The left plot is the histogram of the quotient computed over all the 783 x2 samples. The histogram shows that most samples keep the original predicted label after pruning, i.e., NeuroChains can preserve the original DNN's outputs in most cases. Moreover, the number of filters preserved these sub-networks is 157 (meem)) = (3(dd)), which is small enough to explain. The right plot reports the Faithfulness of NeuroChains in terms of the quotient's sign. We remove each filter from each sub-network and report how many samples' predicted labels are changed after the removal, i.e., the quotient is negative. Each point in the seatter plot corresponds to sub-network, the x-axis is the score of the removed filter given by NeuroChains, and the y-axis the proportion of samples with negative quotients. The plot shows strong linear correlation between the score of the removed filter and the degradation of faithfulness. Since removing filters with high scores results in more samples with predicted class changing after pruning, the score given by NeuroChains measures the importance of filters in DNN inference.
1.-
Frozertion of samples with mosaavv quartent (Ec. 1991
1.0
03-
0.8
02-
-6
Quobient value (cc, (91)
0.0
Score of removed fiber
Figure 8: Histogram of the quotient metric in Eq. 9 computed over all the x0 samples (LEFT). Faithfulness of NeuroChains in terms of the quotient's 's sign (RIGHT).
We evaluate the stability of NeuroChains using the nearest neighbours from the penultimate-layer representation space. Because ReLUU pattern does not provide an ideal metric to measure the distance of samples, even in the raw input space: (1) the number of ReLU linearity zones grows exponentially with the number of hidden nodes. Most ReLU linearity zones are empty and do not contain any real sample; (2) For the few ReLU linearity zones that do contain samples, each only contains one sample and by large chance its neighboring linearity zones are empty, and this is true for most practical cases as empirical studies suggested. So same ReLUU linearity zone even close in their ReLU patterns of the first layer; (3) For two ReL.U linearity zones that are only different in one facet of their polyhedra (i.e., only one digit of their ReLU patterns flips), their corresponding linear models can still be very different (the linear model is an extreme case of sub-network). Therefore, we speculate that samples close to each other in terms of their ReLU patterns do not share sufficiently small sub-networt preserving their original predictions.
almost impossible to find two samples sharing the
That being said, evaluated each NeuroChains extracted sub-network of V9G-19 using 20 sam- ples randomly drawn from two classes on the K-th nearest neighbour (NN) of each sample by sorting the Hamming distance on their ReL.U patterns. The K-NN samples' prediction cannot be well pre-
12

---

Under review as a conference paper at ICLR 2021
L2 distance the K-t nearest neighbor outpul space penuttmale layer
80000
60000
40000
20000
The K-th nearest neighbor in output space of penultimate layer
The K-th nearest neighbor in raw input space
K
Figure 9: Med of L2 distance in the the penultimate-layer representation space between each sample and its K-nearest neighbours from the penultimate-layy representation space (blue) and ReLU pattern space (red).
served on the sub-networks, because the nearest neighbors in terms of ReLU patterns have very different semantic concepts or classes from the samples that the sub-networks are extracted for. Hence, the local region of ReLUU patterns not local region on the smooth data manifold. To see this, in Figure 9 for each sample, computed its L2 distance to the ReLU pattern K-NN sample's penultimate-layer representation for K = 1, 2, 10 (the red curve reports mean: and we compared them with the L2 distance the K-NN in the penultimate- layer representation space (the blue curve reports mean shows that the ReLU pattern K-NN has a much larger L2 distance in the semantic space (i.e.. penultimate-layer representation), the original sample. Moreover, we show some examples of the ReLU pattern K-NN images and the penultimate-layer K-NN images for the sample in Figure TO which show that ReLU pattern K-NN images are much less related to the original sample.
it is very different in concepts to
show case studies of comparing SMOE generated heatmaps for the original network and the NeuroChains extracted sub-network. We can see that the patterns extracted by the two networks are consistent and are all critical patterns for the class, e.g., the eyes and fists of kanga- roos and the feet and face of the horse. However, compared with the original network, these patterns are strengthened much shallower layers of the sub-network, producing better interpretations. This observation is also consistent with the result of analysis on adversarial attacks in Figure
In Figure
we compare the capability preserving the original neural network's s outputs between NeuroChains and magnitude-based pruning (removing the filters whose output featuremaps' average magnitude (L2 norm) over all considered samples small). In particular, under the same setting of each experiment in the paper, we prune the original VGG-19 and retain the filters with the largest featuremap magnitude each layer, 180 in total (more than 157 (mean) = 43(std) filters for sub- networks extracted by NeuroChains). and then fine-tune the filters' scores/ woores/weights as we did for NeuroChains. Figure shows the histogram of the KI. divergence between the original output class distribution and the produced the sub-networks For sub-networks generated NeuroChain, the KL-divergence in most cases stays close to O, while the output preserving capability of simple pruning is much worse.
In Figure
13

---

Under review as a conference paper at ICLR 2021
Original image
10 nearest neighbors in output space of penultimate layer
neighbors input space
Original image
nearest neighbors in output space penultimate layer
10 nearest neighbors in raw input space
Original image
nearest neighbors output space of penultimate layer
10 nearest neighbors in raw input space
Figure 10: Case studies of an image, its 10-nearest neighbours in the output space of penultimate layer (Top), and its 10-nearest neighbours in the input space in terms of Hamming distance between first-layer ReI.U patterns (Bottom).
LO
L23
L25
L34
Combine ALL
SMOE [Munddennke et al., 2019) hes Imaps of the sub-network extracted by NeuroChsins
SMOE (Mundhenk al,, 2019) heatmaps
original
LO
L5
L10
L19
L21
L23
L32
L34
Combine ALL
SMOE (Mundhenk et al., 2019) heatmaps of the sub-network extracted by NeuroChains
SMOE (Mundhenk et al., 2019) heatmaps of the original model
Figure 11: Case studies of SMOE generated heatmaps for the original network and the NeuroChains extracted sub-network.
14

---

Under review as a conference paper at ICLR 2021
L2 magnitude-based pruning of featuremaps (Han 201
1.0
NeuroChains
0.8
Density
0.6
0.4
0.2
0.0
KL- KL-ddiergeenc between the output distributions of the original DNN and the sub-network
Figure 12: Comparison of NeuroChains and magnitude-based pruning the capability of preserv- ing the original network's ss output distribution (smaller KL divergence means better preservation) over 783 x20 uniform samples.
Logarioon Ro divergencc
Logant om 1o.
50
100
130
200
250
300
350
400
100
200
500
400
300
800
700 800
Number of Bibars after pruning
Number of fibars after pruning
Figure 13: Statistics of the output discrepancy between sub-networks extracted by NeuroChains and the original network: VGG-19 (Left) and ResNet-50 (Right). The x-axis refers to the number of filters in sub-DNNS, the y-axis is the logrithm of KL-divergence between the output distribu- tions produced by the sub-networks and the original network. The KL-divergence for most samples small, indicating the sub-networks preserve the original network's output distribution for most samples.
15

---

Under review as a conference paper at ICLR 2021
0.8
0.7
0.6
Test Testlid fidelity
0.5
0.4
0.3
0.2
20
40
60
80
100
120
140
160
180
200
220
240
Number nearest neighbours
Figure 14: Histogram stability of sub-networks extracted by NeuroChains for ResNet-50. The x-axis refers 20K for the K-nearest neighbours of the 20 samples used to extract the sub-network) in the penultimate-layer representation space, while the y-axis is the test fidelity (averaged over all sub-networks), i.c., the accuracy of sub-networks in preserving the predicted class by the original network on the unseem K-nearest neighbours.
16

---

Under review as a conference paper at ICLR 2021
Decraze of prolabiitty for the ofiginal ONN presicced closs
0.6
0.4
6.
0.0
fiter
Figure 15: Left: Scatterplot with a jointly density estimate of the performance of sub- nub-networks ex- tracted by NeuroChains for VGG-19. Each point corresponds to sample. The x-axis refers to the number of filters the sub-network, the y-axis measures the decrease of probability on the origi- nal network predicted class. For VGG-19, most sub-networks' output probabilities drop very little regardless of how many filters are retained. Right: Scatterplot with a jointly density estimate of faithfulness of sub-DNNs extracted by NeuroChains for VGG-19, The x-axis refers to the scaling score removed filter, the y-axis is the decrease average probability for the original-DNN pre- dicted class compared with the complete sub-DNNS For VGG-19, it seems the higher the score, the more the probability drops. The slope of the magenta line the linear (Pearson) correlation, while the shaded around the line represents the confidence interval.
Logartthn of viveeeeeee
Loganihhm XL divergonce
Score
Score removved fitee
Scatterplot with a jointly density estimate of faithfulness of sub-networks extracted by NeuroChains for VGG-19 (Left) and ResNet50 (Right). The x-axis refers to the scaling score (weight) of removed filter in the sub-networks, the y-axis is the logrithm of KL-divergence between the outputs of the new sub-networks (after removal) and the original sub-networks (before removal). For both VGG- 19 and Resnet-50, it shows that the higher the score, the higher the KL-divergence.
Figure 16:
17

---

Under review as a conference paper at ICLR 2021
5.2 MORE DETAILS ABOUT CASE STUDIES
On the sub-network's s architecture, we use "LO" to denote the corresponding convolution layer in VGG-19 and "LO_1" to denote the first filter from this layer. For ResNet-50, we further use "LIBI" to denote the first sub-block in the first bottleneck block, "SC" for the shorteut connection and "C1" for the first convolution layer in the sub-block. The redder the node in the sub-network, the larger the scaling score, conversely, the bluer the node, the lower the score. More case studies can be found in the Appendix. SMOE, Mundhenk et propose to measure information at the end of every feature scale and then combined them into saliency map. We apply this technique to each layer of the sub-network since each layer may prefer different features. In each of our case study, featuremap-overlad input image is shown for each layer and for the whole sub-network which is marked as "Combine All". The visualization of each selected filter is achieved by maximizing its activation w.r.. the input. Afterward, shows the patterns that the filter aims detect which is independent of the input image.
5.3 CASE STUDIES
18

---

Under review as a conference paper at ICLR 2021
136
336
21.0
14
155
04
138
138m
87.0
130
2920
11.0
214
238
163
204
293
253
412
1050
430
233
300
505
335
266
307
455
4349
4970
375
6220
Figure 17: Inference chain by NeuroChains for VGG-19 when applied to images of "bald eagle" and "castle".
Top: The sub-network retains only 10/16 layers and 118/4480 filters. Middle: The per-layer featuremaps generated by SMOF. Since there is nothing similar hetween
eagles and castles, it's easy for VGG-19 to tell them apart. Different types of feathers are an important feature of eagle and the contour of the castle is highlighted. Bottom: Filters with the largest scores. In shallower layers, L23_314 and L19_69 capture the patterns of feathers and eyes of eagle, which different from other species. L28_277 in the deeper layer combines the above two patterns. L25_81 identify the half circle of feathers around neck be key pattern of cagle. L32_24 and L34_18c can be explained as detectors of the whole head and nock of cagle that combines all the patterns detected in previous layers. L23 shows the pattern of small room with windows. L28 220,,10, L28_283, L30 62 and L32 24 extract clear patterns of castle. It shows inference chain for eagle: L.00.83 83 + L19_69, L.2,3144 and 1.55 224 L28_277 L300414 L.22,18 L34_184. shows inference chain
for castle: L10.125 + L19.31 L23.92 - L.8 210 L28 283 - L.30 and L30.468 + L.32 24
438.
19

---

Under review as a conference paper at ICLR 2021
La
62
25.0
102
89
27.8
59.6
100
1466
181
111
22
15
256
7266
212
200
165
1758
152
1886
136
07
104
364
352
206
198
3338
1550
1216
422.9
019
2009
1156
2926
502
261
015.
400
350
390
300
117
Inference chain by NeuroChains for VGG- 196 when applied to images of "kangaroo" and "banana". Top: The sub-network retains only 8/16 layers and 148/4480 filters. Middle: The per-layer featuremaps generated by SMOE. The black ball-shape pattern exists both kangaroos and bananas. The hands eyes of kangaroos are highlighted, and the ends of bananas are lit up. These parts are all black and round in images. Bottom: Filters with the largest scores. L21_312 L23 269, L23 233393 are all related to the black round pattern. L21_312 shows the basic black round pattern. L23_269 looks like the eyes and
Figure 18:
of animals while in better distinguish this two class, VGG-19 19 whole image of hands bananas. L34_66 combine the previous features and show the pattern of animal faces. It shows an inference chain for
L23_393 these nodes are closely arranged like a hand of bananas.
introduces some key patterns
each class. L.32_465 and I.34]9 depict
kangaron: L10 105 - I.19 358 + 1.21 312 and 1.21 484 - I.23 269 - 1.32 403 L334 66.
20

---

Under review as a conference paper at ICLR 2021
image
SconoCam
RAPIA
CA
78
L350C5 257
LABSSC 1558
Conv1_37
Canv1 LIBICI 30
LIIC 156
IIBIC 1000
LAIIC 107
Figure 19: Inference chain by NeuroChains for Resnet-50 when applied images of "pineapple" and "leop- ard". Top: The sub-network retains only 17/67 layers and 157/26560 filters. Middle: The per-layer fea- turemaps generated by SMOE. Both the body and leaves of the pincapple are highlighted. The special skin texture is enough for ResNet-50 to identify lcopard. Bottom: Filters with the largest scores. By observ- ing the patterns in the activation maximization result and the highlighted regions in the featuremap, find that L4BIC3_1O1 capture the texture and the color the main body, L.3ICII capture the patterns of leaf part. It is interesting to see that L4B3C2_107 is the accurate descriptor the main body and the leaf parts and thus provide nearly orthogonal features. For leopard, the skin marked with black spots is its most obvious feature. L4B1C1 .244 extracts the basic texture and color while L4B3SC_1538 and L4B3C1 2247 really show the skin pattern of the leopard. shows inference chain for pineapple: I.2BISC_344 L3BIC1 154 - L4BIC3 1010 + L4B3C2 107.
can
filters extract different local patterns appearing different parts of pineapple. For example,
L3B1C C3_806 and
21

---

Under review as a conference paper at ICLR 2021
+10
Basellnes
L351C2 165
LABICI 278
L4B25C
L483C3 1726
12515C
Comv1_45
L1BICL150
12813C_396
136163_774
LABIC2 323
L492SC_1796 L483C2447
Inference chain by NeuroChains ResNet-50 when applied to images of "chain" and "volcano". Top: The sub-network retains only 18/67 layers and 114/26560 filters. Middle; In the SMOE featuremaps, not only the main body of the volcano, but the crater is also highlighted as key features identify volcanos. Bottom: In the first several layers, L3B IC__165, L3BIC3_72 and L3B CC3_488 extract basic patterns such as curved steel bars and the of mountains, whilst deeper layers focus more global patterns such different orientations of folded strata (L.48 C1_278 and L4B2SC_570) and chains (LAB2SC_179) and L4B3C2_466 L4B3C2 487 captures the features when lava crupts from volcanos as in the penultimate image. It reveals an inference chain for volcano: L2BISC_IA 145 - L3BIC3 488 - L4BICI 278 + L4B2SC 570 - L4B3C2 487.
Figure 20:
22


Reviews:
[{"review_id": "rSVuZ4JpnHz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe submission considers the task of extracting a small sub-network of a pre-trained neural network that can explain a local region of the data space (a small number of the same classes). The resulting sub-network can be interpreted as the inference path moving from the input to the prediction and the filters and associated weights along this path can be visualised. The key of the proposed approach is to apply a multiplicative weight to each filter and layer in the network and enforce these weights to be sparse. The subnetwork is then selected by thresholding the weights. Some quantitative and qualitative analyses were provided.\n\nAssessment:\n\nThe paper considers an interesting interpretation and pruning setting: we wish to prune a pre-trained network to explain how the network performs prediction for a class by training on examples from this class only whilst keeping the pre-trained weights fixed. The resulting final sub-network is very small and thus can be easily visualised. The proposed objective is also very fast to train since the local region used is very small. Despite the similarity to existing post-hoc network pruning/slimming work, I think the goal of finding this small network is very different from that of pruning so thus this work could be of interest to the ICLR community.\n\nThe quantitative analysis of the fidelity and the faithfulness of the objective in preserving the output is useful to understand the proposed approach.\n\nThe paper is also generally well-written. \n\nConcerns:\n\n+ since the target of the final subnetwork is explainability/visualisation, could this be compared to existing methods in some quantitative form? Could this be used to identify problems and provide more insights on how the network works on out-of-distribution examples or examples with wrong predictions that other methods do not provide? How do we know if the final subnetworks are in general more explainable/interpretable, and what can we do with these sub-networks? (I’m trying to be pedantic here to generate discussion and for me to understand the goal this paper is trying to address)\n\n+ the proposed algorithm has several hyperparameters that seem to have been manually selected (?). How were these values selected and how sensitive is the visualisation/final sub-networks to different hyperparameters?\n\nOverall I think this is a good submission. Adding a comparison to alternatives and discussing more about how this could be used/useful in practice will greatly strengthen the submission.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ecikc9tBn6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission extracts a very sparse subnetwork from an RNN. The extraction tries to preserve the computation of the original network in the subnetwork so the behavior of the original network can be analyzed on this subnetwork. \n\nThe submission proposes to multiply every filter with a weight/gate and optimize these weights to extract the subnetwork. Up on that, the method also multiple weights to entire network layers so a network layer can also be pruned. \n\nI have a few questions/comments about the proposed methods\n1. I understand that it is not easy to optimize binary variables, but can you limit the value of gates in (0, 1) by applying sigmoid transformations of free variables? Is it possible to use some simple rules to prune filters, e.g. remove non-activating entries in max-pooling before running the proposed algorithm? \n\n2. When you remove one layer, do you change behaviors later filters because they actually sees different feature maps in the subnetwork. If this is the case, is the analysis of these filters still applicable to the original network? \n\n3. It seems that gate values of the subnetwork are part of the subnetwork in predictions. \"we further fine-tune the nonzero scores\" Since feature maps are multiplied to scores, which is not in the original neural network. The multiplication will change feature maps and thus be likely to change behaviors of later filters. Actually, some important filters may not even be selected. \n\nTo answer question 2&3, is it possible to compute the correlation between the firing behaviors of filters in the subnetwork and those in the original network? If they are highly correlated, then the claim of that the subnetwork preserves the computation is more consolidated. \n\n4. In figure 3 right, \"It shows that the sub-networks suffer from more degeneration if removing a filter with higher score \" -- I don't see a strong correlation. The line shows the direction but not the strength of the correlation. The data points are far from this line. \n\n5. I don't see many unexpected findings from this analysis. The visualization of filters and feature maps are somewhat similar to previous analysis. What are new conclusions from the analysis if I omit the points you want to make?\n\nIn general, I feel that the research in the direction might be able to provide interesting tools for analyzing CNNs, but there seems to be a few gaps to claim that fidelity of the analysis. \n \n\n\n \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HjtMqmtlS9P", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "content:\nIt is about pruning for explanation. The goal of the methods presented is, given a sample x, to extract a network, which is\nan unmodified subset of the original network,\nand has similar predictions to the original network in a region around x.\n\nThe authors derive a gradient-based optimization procedure and do a heuristic threshold-cutoff postprocessing to remove layers and filter channels after the optimization.\n\nThe authors evaluate faithfulness in the sample itself. \nFaithfulness for the region is evaluated using the nearest neighbors from the dataset and in a high level feature map metric.\n\nstrength: \npaper concept is well explained. Clarity of the idea.\nThe outcome are sample-dependent very small sub-networks.\n\n\n\nweaknesses:\n\n--The validation of the method.\n\nFirst of all, they do not validate the impact on the region of a sample x sufficiently, and that must be done because it is a central claim of the paper. The not satisfactorily attendance to that claim is the main reason to reject this paper at the moment. page3: \"(3) it is for data from a local region instead of the whole data distribution.\"\n\nThat is relevant even more so as the input space has usually some adversarial samples nearby with respect to a metric in the input space.\n\n\nFigure 4 gives a partial result - but for a very high level feature space notion of neighbors rather than with respect to a metric in the input space. \n\nUsing the last layer for defining the metric to obtain the nearest neighbors may result in rather \"semantic\" neighbors with similar high level structure, but very different low level structure, which does not conform to the idea of a \\eps-lp-metric Ball around x in input space. It is not local with respect to the input space.\n\nIn that sense the evaluation of local faithfulness is not complete.\n \nFurthermore using nearest neighbors from the dataset also does not guarantee that they are locally close to x (for regions with low data density this may not be guaranteed)..\n\n\n--one needs to perform evaluation with some kind of sampling of points within a ball around a sample, close to the local ReLU linearity zone and evaluate faithfulness for those sampled points -- adressing the notion of points being close wrt.~a metric in the input space.\n\nThe reviewer would be satisfied, if that would be done for a few hundred test points if 1500 x2 networks costs too much time.\n\n--one needs to perform evaluation on what happens with predicted labels of adversarial samples close to x. At least to consider how likely do they switch back to the original, non-adversarial label when looking at the extracted subnet. \n\nThe reviewer would be satisfied, if that would be done for one adversarial per test point but over hundreds of different test points.\nOptionally to consider how likely do they switch to another wrong label (this relevant for targeted attacks only, thus optional). \n\n-- any test to be done also for both nets\n\n\nSecondly,\n\nthey do evaluate faithfulness in the sample point. The reviewer is not fully satisfied with the metric.\n\nFig 3 decrease in probability for predicted class does not tell if it changes the predicted label. They circumvent this problem by using only high confidence samples, however this creates a biased or limited evaluation (namely how this method works on the most confident samples).\n\nit would be better to measure two things:\n\n-- do these evaluations for all samples on their predicted label (a few hundred ...), not only the very confident ones.\n\n--the change in difference to the highest scoring other class ( this gets negative if the predicted label switches ) when comparing original and extracted net. \n\nFor example by A quotient of  \"diff to highest scoring other class (extracted)\" / \"diff to highest scoring other class (original)\" - this is signed and gets negative if on the extracted net the highest scoring other class is the flipped predicted label\n \n--the probability that the predicted label switches when looking at the subnet \n\n\n\ncentral suggestions for improvement:\n\nrun experiments: \n\n--one needs to perform evaluation with random sampling of points within a ball around a sample, close to the ReLU linearity zone and evaluate faithfulness for those sampled points -- adressing the notion of points being close wrt.~a metric in the input space.\n\n--one needs to perform evaluation on what happens with predicted labels of adversarial samples close to x. At least to consider how likely do they switch to the original, non-adversarial label. \n\n\nFig3 with:\n--the change in difference to the highest scoring other class ( this gets negative if the predicted label switches ) when comparing original and extracted net. \n\nFor example by A quotient of  \"diff to highest scoring other class (extracted)\" / \"diff to highest scoring other class (original)\" - this is signed and gets negative if on the extracted net the highest scoring other class is the flipped predicted label (as it works on the predicted class on the original net, the denominator is always positive)\n \n--the probability that the predicted label switches when switching at the subnet \n\n-- do not perform the experiments only on high confidence samples but on all samples. This may constitute a bias towards \"easy\" samples otherwise.\n\nThis further also allows to answer the question: are difficult samples with lower confidence less sparsely represented than high confidence samples ?\n\n\ntechnical problem:\n\neq (7) seems to have a typo. The idea is understood, to consider to drop a layer and use the output of the layer before, if input and output size matches, which may make sense for NNs with residual connections.\n\nHowever if one looks at the case otherwise in eq.7, then it is F^l(x,W^{1:l}) without any filter masks S. In accordance to eq5 this is the mask-less original NN, which is likely a mistake.\n\nIt seems that for both cases (incl. the first case) F^l(x,W^{1:l}) needs to be changed to include the masks S \nF^l(x,(W^{l'}  \\odot S^{l'}, \\alpha)^{l'})_{l'=1:l}\n\nThat is a fixable problem that does not affect the paper rating.\n\nTechnical Questions to the authors:\n\nSection 3.2 algorithms:\n-- if the loss is above a certain threshold, it is understood that then you perform the finetuning on the S values. In that case, do you roll back then the filter removal using tau ? In that case, do you roll back then the filter removal by G ?\n-- how do you estimate \\tau,  \\lambda and \\lambda_g  ?\n\nGeneral questions to the authors: \n\nThe authors contribution in Fig5 and Fig6 seems to be the sparse network extraction, as the SMOE visualization and the filter activation maximization are known. It seems that these visualizations, while nice to show, are not really central to the questions raised by the authors. However, this should not be misunderstood as a wish from the reviewer to remove them. Rather to remark that the paper novelty is the left side in these figures.\n\nThe reviewer takes the value of the method so far by: \"When applied to samples from a local region in data space, it is plausible that its inference process mainly relies on a small subset of layers/neurons/filters.\"\n\n--Do the authors have a suggestion how the sparse network extraction can be used for any kind of analysis or insight beyond showing the sparse network itself ? \n\n-- SMOE can also be applied on the original, large network. What is the difference (or value) between applying SMOE visualization on the original network to at first pruning and then applying SMOE on the pruned net?\n\ntypos:\n--three quantitative analysis over --> three quantitative analyses over (Plural)\n\n--Fig2 is a table\n\nminor suggestions:\n\n--\"we remove layer-$\\ell$ if G^l < 0.5\" is simpler\n--remove the top subfig in figure 1, as it is anyway shown in Fig 5 and Fig6 in better resolution. I think you need the space for more interesting content.\n\nPost review: The reviewer thinks that the authors did a thorough job of addressing the reviews. The results are interesting in several aspects, for example Fig 5 and 8. That said, regarding the question \"Do the authors have a suggestion how the sparse network extraction can be used for any kind of analysis or insight beyond showing the sparse network itself?\" The reviewer has doubts that non-ML expert end users (e.g. M.D.s) could make use of a sparse network as explanation mode (as this would assume that they can make sense of what a neuron has learnt). The reviewer updated his rating upwards. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-P2BhUt6lmk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method called NeuroChains for extracting a sub-network from a deep neural network (DNN) that can accurately match the outputs of the full network for inputs in a small region of the input space. The goal is to be able to explain the important steps that a DNN takes to get from inputs in a small region of the input space, to its predictions for those inputs. NeuroChains initialises the sub-network as the original DNN except each weight/filter is multiplied with a real-valued score. The L1 norm of these scores is minimised while penalising any deviation of the output of the sub-network from the output of the original network. After the minimisation, weights/filters with a score below some threshold value are removed, leaving a sub-network. In the paper there are experiments that aim to verify three claims: (1) NeuroChains can find sub-networks containing less than 5% of the filters in a deep convolutional neural network while preserving its outputs in some small region of the input space; (2) every filter selected by NeuroChains is important for preserving the outputs and removing one of them leads to considerable drop in performance; (3) the sub-networks extracted for small regions of the input space can be generalized to unseen samples in nearby regions.\n\nStrengths:\n1. The experimental results suggest that NeuroChains can, as claimed, extract sparse sub-networks that match the outputs of the original network for some small region of the input space. These sub-networks are easier to analyse than the full network, which should help with interpretability.\n\n2. The descriptions of the method and the experiments were very clear. As a result, the experimental results could be reproduced based only on the information in the paper.\n\n3. The distinction between NeuroChains and related methods was made clear. In particular, comparisons were made to pruning methods and work on interpretable machine learning. \n\nSuggestions and questions:\n1. In the paper it says that scores are not limited to [0,1] due to possible redundancy among filters in the original DNN. If there are redundant filters, their scores could be set to 0 so that these redundant filters are ignored. Why does the possibility of redundant filters motivate unconstrained scores?\n\n2. The KL divergence between the full network output distribution and the sub-network output distribution was used to penalise the scores. Why was KL divergence used over cross-entropy? The KL divergence is equal to the cross-entropy of the sub-network output distribution relative to the full network output distribution minus the entropy of the full network output distribution. The entropy of the full network output distribution is independent of the scores. Therefore, cross-entropy and KL divergence have the same minima and gradient with respect to the scores. Since the cross-entropy is a little cheaper to compute than the KL divergence, wouldn’t it be better to minimise the cross-entropy instead of the KL divergence?\n\n3. In section 4, paragraph 1, the paper states \"(2) every filter selected by NeuroChains is important to preserving the outputs since removing one will leads to considerable drop in performance\". However, in Figure 3 (right) there are many filters with scores greater than the threshold tau = 0.1 that appear to cause almost 0 change in the predicted probabilities when removed. Therefore, the claim that all filters found by NeuroChains are important is not well supported by the experimental results.\n\n4. In Figure 3 (right), the magenta line appears to be a line of best fit, which does not by itself imply correlation. However, in the paper it says \"magenta line implies strong correlation between the two”. If the goal is to demonstrate strong correlation, wouldn’t it be better to report the Pearson correlation coefficient or Spearman correlation coefficient instead of the line of best fit?\n\n5. On page 8, paragraph 1, the paper references Figure 4 and states that sub-networks extracted for local regions of the input space can generalise to nearby regions because their test fidelity (accuracy with which the output of the full network can be reproduced by the sub-network) remains high when the number of nearest images in the validation set is below 100. However, in my opinion, Figure 4 seems to suggest that as the number of nearest images increases, the test fidelity starts to decrease immediately and at a roughly constant rate until 180 nearest images is reached. Therefore, I don't think that Figure 4 provides strong evidence that the sub-networks extracted for local regions of the input space generalise well to nearby regions.\n\nOverall, the NeuroChains method appears to do a good job at extracting sub-networks from large DNNs and these sub-networks make predictions of the original DNN significantly easier to interpret. However, I don’t fully understand some of the decisions made in the design of the algorithm (questions 1. and 2.) and I am not totally convinced by some of the conclusions drawn from the experimental results (questions 3., 4. and 5.).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, this seems like a neat idea and well-done work. Main principle is to extract a very sparse net that does a good job at locally "explaining" a given example. The NeuroChains idea does this with a diffentiable sparse objective. I think this work is well-positioned and has nice properties: (1) retains a very small percentage of "filters", (2) it appears that all the selected filters are actually needed/useful (3) there are some generalization properties wrt to unseen samples that are close to the sample of interest.

I appreciate that the authors responded with very detailed rebuttals to the concerns of the reviewers. I'm still worried, like AnonReviewer4, about the generalization around local regions though the follow-up experiments satisfy me for the most part. There is a genuine concern that while this method has the *potential* to produce useful outputs that could be useful for downstream experts to analyze the underlying network, the paper itself doesn't really show this. In other words, while I agree that on the technical side of things, the work passes the bar, it's not clear that the work passes the bar from the impact side of things.

This did make for a genuinely a borderline case in terms of decisions and unfortunately this work landed on the reject side this time around.
Decision: Reject