Switched linear projections and inactive state sensitivity for deep neural network interpretability
Authors: Lech Szymanski
Brendan McCane
Craig Atkinson
Venue:  (2020)
Keywords: deep learning
interpretability
artificial neural networks

Abstract: We introduce switched linear projections for expressing the activity of a neuron in a ReLU-based deep neural network in terms of a single linear projection in the input space. The method works by isolating the active subnetwork, a series of linear transformations, that completely determine the entire computation of the deep network for a given input instance. We also propose that for interpretability it is more instructive and meaningful to focus on the patterns that deactive the neurons in the network, which are ignored by the exisiting methods that implicitly track only the active aspect of the network's computation. We introduce a novel interpretability method for the inactive state sensitivity (Insens). Comparison against existing methods shows that Insens is more robust (in the presence of noise), more complete (in terms of patterns that affect the computation) and a very effective interpretability method for deep neural networks

Under review as a conference paper at ICLR 2020
SWITCHED LINEAR PROJECTIONS AND INACTIVE STATE SENSITIVITY FOR DEEP NEURAL NETWORK IN- TERPRETABILITY
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce switched linear projections for expressing the activity of a neuron in a ReLU-based deep neural network in terms of single linear projection in the input space. The method works by isolating the active subnetwork. series of lin- car transformations, that completely determine the entire computation of the deep network for a given input instance. We also propose that for interpretability it is instructive and meaningful to focus on the patterns that deactive the neurons in the network, which are ignored by the exisiting methods that implicitly track only the active aspect of the network's computation. We introduce a novel interpretabil- ity method for the inactive state sensitivity (Insens). Comparison against existing methods shows that Insens is robust (in the presence of noise), complete (in terms of patterns that affect the computation) and a very effective interpretability method for deep neural networks.
INTRODUCTION
It is notoriously hard to interpret how deep networks accomplish the tasks for which they are trained. At the same time, due the pervasiveness of deep learning in numerous aspects of computing, it is increasingly important to gain understanding of how they work. There are risks associated with the possibility that neural network might not be "looking" at the "right" patterns (Nguyen et al. Geirhos et al.] 2019). as well as opportunities to learn from the network's ' capable of better than human performance (Sadler & Regan 2019). Hence, there is ongoing effort to improve the interpretation and interpretability of the internal representation of neural networks.
What makes this interpretation the inside of neural network hard is the high dimensionality and the distributed nature of its internal computation. Aside from the first hidden layer, neurons operate in an abstract digh--imensional space. that was not hard enough, the analysis of individual components of the network (such activity of individual neurons) is rarely instructive, since it is the intricate relationships and interplay of those components that contain the "secret sauce". The two broad approaches to dealing with this complexity is either use simpler interpretable models to approximate what a neural network does, or to trace back the elements of the computation into the input space in order to make the internal dynamics relatable to the input. In the latter approach we are typically interested in neurons' sensitivity how the changes in network input affect their output, and decomposition how different components of the input contribute to the output.
In this paper we propose straightforward and elegant method for expressing the computation of an arbitrary neuron's is activity to single linear projection in the input space. This projection consists of a switched weight vector and a switched bias that easily lend themselves to sensitivity analysis (analogous to gradient-based sensitivity) and decomposition of the internal computation. We also in- troduce new approach for interpretability analysis, called inactive state sensitivity (Insens), which uses switched linear projections to aggregate the contribution of patterns in the input that deactivate neurons in the network. We demonstrate provides a comprehensive picture of a deep network's 's internal computation. The only constraint for the proposed methods is that the network must ReL.U activation functions for its hidden neurons.
several networks and image- based datasets that Insens

---

Under review as a conference paper at ICLR 2020
T1
31
T1
Σ
Σ
022
22
322
(a)
(b)
(c)
Figure I: Let's assume that for particular input [x] 22] going into the ReLU network shown in (a) the white neurons inactive; then, for this partieular input, the network from (a) is equivalent to network in (b) where the inactive neurons are treated as dead and the active ones operate in the linear part of their ReLU activation fuction; which makes both of these networks equivalent one in (c); the grey hidden neurons form the active subnetwork.
2 RELATED WORK
Previous work on deep learning interpretability is extensive with wide variety of methods and approaches SSionnyan al. (2014); Zeiler Fergus Bach et al. (2015); Mahendran Vedaldi Montavon al. (2017): Sundararajan most prominent efforts in this area. Our work on the single linear projection follows the approach akin to Lee et al. and Erhan et al. (2009, where the objective was to interpret the computation performed by an arbitrary neuron for a particular input vector as a projection in the input space. However, whereas these previous attempts were based on Deep Belief Nets (Hinton et al.) (006) and required an approximation of the said projection, our method is forward computation that gives the neuron's ' activity in terms of linear projection in the input space. It works for any neural network, including convolutional ones, as long as all hidden neurons use piecewise linear activation functions.
al. (2017); Zhou et al. (2019) being just selection of the
All existing methods for interpretability of deep learning, due to the nature ReLU computation, necessarily provide information only about the active subnetwork of the ReLU-based architecture. Our own observations, well as other evidence showing that practice neural networks produce relatively low number of activation regions (Hanin & Rolnick, 2019), lead us to the hypothesis that the analysis of the patterns the input that switch neurons off gives an excellent picture of net- work's sensitivity. We also take the view that too much interpretation in interpretability introduces risk of showing us what we expect to see and not what the network is actually focussing on. For instance, in Deep Taylor Decomposition (Montavon et al. (017)) choices of different root-points for the decomposition of the relevance function lead to different rules for Layerwise Relevance Propaga- tion (Lac) et al., 2015), which can lead to different interpretations of what is important in the input. The LRP-o180 rule, for example, emphasises the computation over the positive weights the network while discounting the relevance of the information passing through the negative weights. This rule is justified by assumptions about desired properties of the explanation, but this comes with risk of confirmation bias. Insens is attempt take into account the patterns in the input that cause the neurons inside the network to produce zero output. The information related to the inactive network may seem irrelevant, since inactive neurons do not directly contribute the computation of the overall output. However, there is something in the input that switches particular set of neurons off, thus regulating the active computation, and as we show in this paper, this something carries lot of meaningful information.
3 SWITCHED LINEAR PROJECTIONS
The basis of the switched network concept is the fact that neurons that produce output of zero do not contribute to the computation the overall output of the network. The notion dead neurons, that is neurons that always output zero, is not new, nor is the realisation that these neurons, along with their connecting weights, can be taken out the network without any impact on the computation. In switched projection,
treat the zero-output neurons as temporarily dead for given instance of input. We refer to these neurons as inactive, since they may become active for a different network input. Thus we isolate the subnetwork of the active neurons in given computation. As it happens,
2

---

Under review as a conference paper at ICLR 2020
for ReLU neuron the active neurons those that pass their activity, the weighted of its inputs plus bias, directly to its outpu["] This means that subnetwork of active ReLU neurons is just series of linear transformations, which is equivalent to single linear transformation. As result, we can express the computation performed by any neuron in a ReLU network as a projection onto switched weight vector in the input space plus the switched bias. The term switched indicates that this weight and bias vector changes when the state of the network changes, the state corresponding the particular combination of the active and inactive neurons the network. Figure [ iillustrates the concept graphically, and a formal description is given in the following proposition:
Proposition 1 (Switched linear projections). Let x E Rd be a vector of inputs, Wis E R the weight vector, and bú, E R the bias neuron in layer (with Ut-1 inputs from the previous layer). Let the activity of neuron in layer be defined as:
vr((x)= = (.x)=((..o(0xxxWW bb) ( bb) .....::w + byi
... Witr. T denotes transpose, b, (v) = max(v, 0) is the ReLU activation function. If define input-dependent state of the nenwork as
where Wi
(bn bwil and
w(* [ōr( (x))(()))))) (x))w? ... o, (vu, (x)) w[er] and
(x) (vn (x))xx) ...
ār (vu, (x) bu)il
where (v) da.lo) thenfor de
(x((x) = w(x
w/ and
ba(x) = bi w! w by w
(x w + ............................ bi + bis we have
t (x)
+ (x).
(2)
The proof is provided Appendix definition for step function, so that
Note that the ReLU derivative, o, (t),, is just a convenient
.820
(v) 0
(3)
otherwise.
To simplify the notation, whenever referring to the parameters of the switched projection w, as well as activity v, we will drop the explicit dependency
While Figure | illustrates the switching concepts on small fully connected network, switched be computed for networks with convolutional as well pooling layers. A convolutional layer is just a special case a fully connected layer with many weights being zero and groups of neurons constrained share the weight values on their connections. For max pooling, the neurons that do not win the competition, and thus their output does not affect the computation from then on, are deemed be inactive regardless of the output they produce.
linear projections
3.1 SENSITIVITY
Equation 2 makes it obvious that given neuron's switched weight vector is just the derivative of its activity with respect to the network input, w de(x) Thus, the switched weight vector is analogous to gradient- based sensitivity analysis. Figure shows the heatmaps of the switched weight sensitivity for the same set of hidden neurons with different inputs from the MNIST-trained 2CONV neural network (for details the network architecture featured in this paper see Appendix B). In this visualisation we show normalised w. with intensity of red corresponding to the larger positive value, and the intensity of blue the negative value. The neurons were chosen from the 2nd convolutional layer and the penultimate fully connected layer respectively such that for the four
In our terminology, activity denotes output before the activation function and an active neuron is one that produces non-zero output after the activation function; for ReLU neuron the active and inactive neurons those that have positive and negative activity respectively.
3

---

Under review as a conference paper at ICLR 2020
0.3
15.9
-00.2
- 13.3
-0.7
85.6
95.7
-510.
-271.8
1515
label: pred:
-0.1
61-1.8
6.6
7 1.4
-6.1
13.9
34.7
3 25.0
2.4
7
y 0.3
== -92.1
o= -1.0
== 773.6
08.0
== 0.074
== 382.3
235.4
-63.5
80.1
label:
pred:
= -0.1
= 9.4
== 0.1
= 7.8
8= 4.7
8 0.074
6= 16.00
8 -718
8 2.5
i 12.9
2
Figure 2: The heatmaps of w of arbitrary neurons from different MNIST inputs (first column), in the second convolutional layer (next 5 columns) and the first fully connected layer (last 5 columns) of the 2CONV neural network; normalised intensity of red and blue in each heatmap corresponds to the magnitude of the positive and negative value (with white indicating O); the activity xw +b and the switched bias are shown above the heatmap of each neuron.
considered inputs some neurons were always active, some inactive, and others sometimes active and sometimes not.
Figure2 it clear that a given neuron is not necessarily sensitive to the same pattern for different network inputs this is most evident in the sensitivity the neurons of the fully connected layer. Also note that some neurons in the convolutional layers are active despite the fact that they only "see" the part of the input image that is "empty" (all pixels are black). This leads us to the conclusion that given neuron
necessarily detector of particular pattern in the input space, which is often the underlying assumption of the existing interpretability techniques.
Something also apparent in switched projection analysis, though not evident from Figure is that for a given input most of the neurons in the network are inactive. On average, only 17% of the neurons were active for a given MNIST input in this architecture. The fact that only subset of neurons are active in given computation is not quirk of one specific network, observed by (Hanin Rolnick 2019). Switched linear projections give us an interpretation of a deep network as a set of linear, input-dependent, transformations. Something about the input activates subset of neurons, but also keeps all the remaining neurons inactive. Traditional sensitivity analysis, as well as the one shown in Figure 2 shows direction (or magnitude) of the gradient of the input that would increase neurons activity provided the same state of the network remained unchanged. However, often in these networks the state does change even after small perturbations of the input, often resulting in same classification, but different input gradient. We reason that the analysis the state, including information about what makes the neurons inactive, provides missing and likely very important aspect, to interpretability than analysis of the active subnetwork alone.
3.2 DECOMPOSITION
Switched linear projections can decompose the activity of neuron into contributions from its input, in our case the pixels. For this we propose another re- interpprettatoon of the computation of the output that will allow us to distribute the bias over the attributes of the input vector. Note that for linear projection
2 == xw + = (x (x-c)
(4)
where c = x ww w, cw"" and c E Rdd The vector can be thought of as a translation of the coordinate system to neuron-centered one, where w goes through the origin at c. Montavon cet al. (2017) call this vector the nearest root point, but we will refer to it as the neuron's ' centre. Figure provides geometrie interpretation of in simple 2D scenario. Since Rd we can break down the computation of e such that
d
Σ (x) - -c-), j=1
(5)
and take (z, cy)u to be the contribution of the input component j (in our examples, a single pixel) to the computation of neuron's activity. Since the switched linear projection is equivalent to
4

---

Under review as a conference paper at ICLR 2020
11
W
x
XwT + = v
C zwi + m 0
C2
Figure 3: Graphical representation of the concept of neuron's 's centre in 2D scenario, where input vector is x = [r) #2]: the diagram shows input-space coordinate system - the neuron-specific one would have its origin at c.
weights and bias of a single neuron, can compute the switched centre
(6)
The switched centre è is related the concept of reference in DeepLIFT (Shrikumar et al. 2017) that gets subtracted from the input in order to extract a pattern of interest. However, whereas in DeepLIFT the reference is external to the model, and used for examination of perturbations it induces in the output, our proposed centre is a component of the actual computation of the network's 's output; one could say, is given neuron's 's inherent reference.
Visualisations based on decomposition of active neurons do not offer anything more than existing methods. However, the concept of neuron's centre will be used in the method we propose next, for interpretability based on the sensitivity of the inactive network.
4 INACTIVE STATE SENSITIVITY
Since switched linear projection can be found for any neuron in the network, it can just easily relate what it about the input that drives neuron into the negative just as well as positive. Tracking the patterns in the input that make the neuron more inactive tells us about the aspects of the input that would ensure the stability of the network's 's state. The bigger the magnitude of the negative activity, switch on, and thus change the switched projection. Some
the less likely the inactive neurons of the inactive neurons are closer and others further away from the point where they would activate. Hence, we propose definition of inactive sensitivity based switched linear projection that takes the magnitude of activity into account,
@ - U/
W.
The difference x - C; can be thought of as the component of the input that projects onto neuron's 's switched weight vector W; the coordinate system centred č; Note that w; is still a vector the direction of w, and so it measure of neuron's 'sso sensitivity, but its magnitude is proportional to the absolute value activity. In an attempt to capture the information about the state the network, propose averaging the sensitivity of all inactive neurons in given layer of the network:
(x)
mi N
(
BET
where I₁ is a set of all inactive neurons, Vi S in layer of the network (excluding the output neurons), and è; is the switched centre of neuron i; m; is the number of times neuron i is on when the network is evaluated over the set of N training samples. We refer state sensitivity (Insens) of the network with respect to the input x. The m./N factor weights the importance of neuron based on its activity over the training set; neuron that never on, m, 0, might represent an arbitrary pattern that has no useful information, while the neuron that is usually
②{*)) Rd the inactive
on, but not for the input for which ② E Rd is computed, is taken carry more weight.
5

---

Under review as a conference paper at ICLR 2020
Integrated Gradients
Integrated Gradients
Input
Gradient
DeepTaylor
MAP-PresetAfia
Insens
Input
Gradient
DeepTaylor
LAP-PresetAFlat
Insens
label:
label:
pred: 7
7
pred:
label: 2
2
label:
pred:
pred:
label: 7
label: 7
pred: 3
pred:
label: 2
label:
pred: 3
pred: 2
Figure 4: Visualisations for a set of interpretability methods of a single 2CONV neural network trained on the clean MNIST data in response to clean as well as noisy input; top-left shows clean MNIST, tip-right noisy bordered MNIST, bottom-left noisy background MNIST and bottom-right noisy framed MNIST; the intensity of red and blue correspond respectively to the magnitude of the positive and negative values in the heatmaps.
4.1 EVALUATION
We evaluated Insens interpretability against plain gradient sensitivity, Deep Taylor decomposition
(Montavon et al. 2017). Integrated gradients 4Sundararajan et al. 2017) and Layerwise relevance propagation (Bach et al. 2015) as implemented by the iNNvestigate toolbox (Alber et al. 2018).
For visualisations of the Insens-based interpretability we simply show 5xxx) as heatmap. with intensity of the red pixels relating the magnitude of its positive components, and the intensity of blue the magnitude of the negative component of the vector. Since the individual W; gives a weighted gradient with respect to the input, and thus the change vector that would make neuron i less prone
to become active, we take the average 2(x) E Rd to be indicative of the pattern in the input that is related to the stability of the network's 's state in layaer I induced by x. all evaluations we examine visualisation of 2CONV neural network (described in Appendix B
First, we examine visualisations of ( E Rd, that is the inactive sensitivity over the penultimate layer of the network trained on the MNIST dataset (Lecun et al. 1998). We reason (and confirm below with saliency checks) that the last layer before the softmax output provides most information related to network's ' decision making out of all the hidden layers. Figure 4 shows visualisations from existing methods against the Insens heatmaps for different instances of the input and the same network. In the first instance we use clean MNIST input (on which test accuracy of 2CONV is over 99%). Note that Insens visualisation shows something that other methods hint at but do not show explicitly- that the network is sensitive to the black and white contrast of the digit and its outline. The red and blue areas in the Insens heatmaps suggest that respectively lightening and darkening these regions would make the current network state more stable (though in this case it is not possible to make black pixels darker nor white pixels any lighter). To verify the information provided by Insens, added different types of noise to the images, also shown in Figure 4 and used them as input to the same network. We wanted to confirm that adding noise to the background, but not the digit outline, as suggested by Insens, does not impact performance. Indeed, testing with images that contain random gaussian noise in the background, but not within the 3-pixel outline of the digit (Figure top-right) still gives 87% accuracy over 10000 test images. If we use images with random gaussian background everywhere (except the digit itself) or images with the same number of clean pixels as in the 3-pixel outline, but located around the frame of the image (Figure blttom-left and bottom-right), the accuracy drops to 32% and 45% respectively. This confirms that Insens patterns for the penultimate layer provide meaningful information about the network's is sensitivity. Also note that the other methods tend to lose the patterns of the digits in the visualisations over noisy images, whereas Insens still shows the digits, to which the network is still sensitive, albeit also being affected by the noise.

---

Under review as a conference paper at ICLR 2020
Integrated Gradients
Integrated Gradients
LRP PresetAFiit Insens
LRP Pr-PPreetAFlat
Input
Gradient
DeepTaylor
Input
Gradient
DeepTaylor
Insens
label: Plane
label: Dog
pred: Plane
pred: Dog
label: Plane
label: Cat
pred: Animal
pred: Deer
label: Animal
label: Ship
pred: Animal
pred: Ship
label: Truck
label: Deer
pred: Truck
pred: Deer
Figure 5: Visualisations for set of interpretability methods on two 2CONV neural networks one trained on the smallNORB dataset (left) and the other on the CIFARIO dataset (right); the intensity of red and blue correspond respectively to the magnitude of the positive and negative values in the heatmaps.
fig
labal: Plane
labal:
label: Plane
pred. Animal
predi
label: Animal
label: Ship
ped: Animal
pred Ship
labe: Truck
labe: Deer
pred Truck
pred: Dear
Figure 6: Visualisations for Insens (2,00 for layers 1=1, 3, and 5 of the 2CONV network trained on smallNORB dataset (left) and CIFARIO datasets (right); the intensity of and blue correspond respectively to the magnitude of the positive and negative values in the heatmaps.
In Figure 5 we show evaluations of (x) -based Insens and other methods for two additional 2CONV networks one trained on smallNORB (LeCun et al.) and the other on CIFARIO dataset Krizhevskyy 2009) (for details of the training see Appendix B Insens visualisation for the penul- timate layer of smallNORB and CIFARIO IO datasets might not seem all that different from other methods. However, with Insens obtaining information about what the network pays attention to at different stages of its computation. In Figure 6 we show Insens visualisation for convolutional layers (1 and 1 3) as well as the penultimate, fully connected layer (1 = 5) of the 2CONV network trained on smalINORB and CIFARIO 10. There are Insens visualisations for the max-pooling layers, as these will contain only either active, or the arbitrary inactive chosen neurons by the max-pooling operation.
examine the network state of specific layer in the network,
Finally, provide an objective measure of the quality of Insens visualisations, perform sanity and saliency checks as prescribed by Adebayo et al. (2018). In these tests we measure correlation between interpretability visualisations for different networks; in each instance 2CONV network trained on given dataset against an untrained (randomly initialised) network, and in the second instance the same dataset trained network against a network trained on identical input with ran- domly shuffled target labels. High correlation between visualisations from the same input between the trained and either randomly initialised or random label trained network suggests that an inter- pretability method is model agnostic, most likely showing echo of the input. In Figure we show average Spearman rank-order correlation coefficients between visualisations from 20 instances of independently trained 2CONV networks, for each using 100 randomly chosen images from the CI- FAR10 dataset. In the first instance, we check the correlation the penultimate layer-based based Insens against other methods. Insens correlation is low, on par with the Gradient and Integrated Gradi-

---

Under review as a conference paper at ICLR 2020
Rod of Spearan rack rank areer corrletio cpenicies
vs.
cffa110 2COM
untrained 2C0NV
1.0
offar10 2CONV
0.6
0.6
0.5
0.6
0.4
0.4
0.2
0.0
Input
gredkent
Integrated creds
LAN Po--PreottFial
Insens
0.0
toput
Gradient
Drep Tayior
Integrated Grads
Lae- PresatAfial
Irown
0.0
Irpitt
0.0
Input
fife
Doup
Figure 7: Mean spearman rank-order correlation between visualisations derived from random sam- ple of CIFAR- 10 imput for different methods including Insens 527 between trained and randomly initialised 2CONV network (first left), between trained and random label CIFAR- 10 trained 2CONV network (second left); the same correlation is shown over Insens based different layers of the 2- CONV network (two figures on the right); Insens correlations are shown in red; vertical black lines show the variance.
methods. Note that the correlations between visualisation of the DeepTaylor method, which arguably provides the most visually striking visualisations, are very high, suggesting that most of the provided information is independent of the network's state. the second instance we examine the correlations over Insens from different layers of the 2CONV architecture. It is very obvious that this correlation increases the closer the layer is to the input. believe this not an artefact of the Insens method, but an indication of the initial passing and then gradual filtering of information passing through the network.
CONCLUSION
The switched linear projection is an interpretation of the computation inside ReLU network that distinguishes between the active and inactive parts of the deep neural network architecture. The active subnetwork tends to be a smaller subset of the deep network (see Appendix [. for details) and the linear projection it provides somewhat arbitrary, in the sense that does not matter what the orientation of the switched weight vector is, long it produces the desired output. Hence interpretability analysis based on the active subnetwork may not give the full picture of the patterns from the input that the network relies in order produce its computation.
Inactive state sensitivity (Insens), the proposed method for interpretability ReLU networks, ag- gregates weighted sensitivity over layer of inactive neurons. This sensitivity relates the gradient of the input that would potentially drive the activity of the inactive neurons further away from zero, thus corresponding
the patterns in particular input that would keep the state of the particular layer stable and the output decision the same. As the name implies the method isolates the patterns in the input to which the network, sense, insensitive. Our evaluations show that these patterns give comprehensive picture of what and at what point of the computation the network reacts to patterns given input.
It is worth noting that it is possible to extend Insens by summing over all, active and inactive, neurons Equation g In our experiments found that inclusion of active neurons did change the visualisations in significant way, and we reasoned that the effect of these neurons is already accounted for in the computation of the switched linear projections on which the Insens evaluation based However, might be worth investigating in the future whether there cireumstances, datasets, or types of problems where inclusion of active state affects the sensitivity visualisations.
Since switched linear projections are just interpretation of the computation inside neural net- work, they may also become useful tool for complexity analysis deep networks. For instance, it might be possible to develop centres of the neurons the network. remains to be investigated how the nature of the inactive subnetwork, and potential ways of manipulating it during training, would affect generalisation.
regularisation methods based switched weights, biases and
8

---

Under review as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X GPU used for this research.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 9505- 9515. 2018
Maximilian Alber, Sebastian Lapuschkin, Philipp Seegerer, Miriam Hägele, Kristof T. Schütt, Grégoire Montavon, Wojciech Samek, Klaus-Robert Müller, Sven Dähne, and Pieter-Jan Kin- dermans. iNNvestigate neural networks! arXiv reprint: arXiv 1808. 042660 2018.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Rober Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wist wise relevance propagation. PLOS ONE, 10(7):1-46 07 2015.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. Technical Report 1341, University of Montreal, 2009.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. taagenet-traines CNNs are biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Conference on Learning Representations, 2019.
Boris Hanin and David Rolnick. Deep relu networks have surprisingly few activation patterns. arXiv reprint: arXiv 1906.00904, 2019.
Geoffrey F. Hinton, Simon Osindero, and Yee- Whye Teh. fast learning algorithm for deep belief nets. Neural Computation, 18(7):1227-1554 2006.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, Fu Jie Huang, and Léon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pp. 2278-2324, 1998.
Honglak Lee, Chaitanya Ekanadham, and Andrew Ng. Sparse deep belief net model for visual area v2. J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis (eds.), Advances in Neural Informa- tion Processing Systems 20, pp. 873-880.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp- 5188-5196.
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65(C):211-22 2017.
Anh Mai Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 427 436.
M. Sadler and N. Regan. Game Changer: AlphaZero's Groundbreaking Chess Strategies and the Promise of AI. New in Chess, 2019.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70, pp. 3145-3153, 2017.

---

Under review as a conference paper at ICLR 2020
K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations, 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pp. 3319-3328, 2017.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp.
B. Zhou, D. Bau, A. Oliva, and A. Torralba. Interpreting deep visual representations via network dissection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(9):2131-2155, 2019.
A PROOF OF PROPOSITION
1
Proposition 1 (Switched linear projections). Lel x E Rd be a vector of inpuls, Wis E RUi- the weight vector, and bis E R the bias of neuron in layer (with Ut-1 inputs from the previous layer). Let the activity of neuron in layer be defined as:
vx((x) (...0._06,,, b1) ... - + bia
(1)
where Wt = WH whir.. T denotes transpose, b; = [b bwid and
(v) = max(v; 0) is the Rel.U activation function. If we define an input-dependent state of the network
W. (x) = (1n(())))) (x))w?i
or (ww,, (x)) w[,,] and or (.w, bru))
of [vn (x))br
where (,) dorlo) then du
(T((x) w( w/x)
w wI. and
bx(x) ((x) ( w
by (
W/ w// +... bi-1 + bis we have
u/ (s(x) x (x) + (x).
Proof. By definition from Equation I the activity of neuron in layer / is
U, Σ /=1
oia (x)
or (x)
was + bi.
(9)
where Way is the weight the connection between neuron of layer I - 1 and neuron of layer and by is the bias of neuron in layer I.
Since or (v) = oo
v > O, otherwise,
and ā,(w) = do,(e)
[1 v >> otherwise,
we have
du
Ui 15 (x((x)r
> O, otherwise,
(x) was (
10
and thus
or (x)) wirk = u1-1x(x)0, (0) (0L-1((x) wikk.
As result
Ut U1-15u11 bis mx(x) Σ U| "111ōr ( wrss bis Σ
(10)
DEA
10

---

Under review as a conference paper at ICLR 2020
Table 1: 2CONV network (conv convolution; fc fully connected)
Layer
Type
Filters/ neurons
Window
Stride
Activation
conv 32
5x5
1xl
relu
2
maxpool
2x2
2x2
relu
3
conv
64
5x5
1xl
relu
4
maxpool
2x2
2x2
relu
5
fc
512
relu
5
fc
K
where is the set of neurons with activity v > O, the active neurons. Substituting the expression
for activity from Equation 10 into its recursive definition in Equation 9 where fos(x) 2ii reveals that the overall computation is a series of linear transformations of x equivalent to single linear transformation
(x((x) xw + (x).
where W (x) W(x) ((x)
w w/ and
bx(x)
x bj
b2 W
w wI +...+ Mn-iw/ + bu-
B 2CONV ARCHITECTURE
This network consists of convolutional layers each followed by maxpool layer that down- samples the feature map, followed by single fully connected layer of 512 neurons and K output neurons, where K is the number of classes in the dataset of interest (see Table [ for details). For training this network we used Adam optimiser minimising the softmax cross-entropy without regu- larisation. all training portion of the training set was set aside for validation for early stopping.
For the first, MNIST evaluation, the network was trained on cleaned MNIST data to test accuracy of 99.4%. The network was then tested on several variant of the MNIST dataset: MNIST Gauss images were given random Gaussian background (test accuracy of 31.7% )), MNIST outline images with same Gaussian background except for outline within pixels of the digit (test accuracy of 86.0%), and MNIST frame with Gaussian background and the same number of clean pixels as in the MNIST outline version given image around the frame (test accuracy of 44.9%).
For the second, smalINORB evaluation, the network was trained four times variants of the dataset differing the encoding of the images: smalINORB with grayscale intensity given by value between 0 and for black to white pixel respectively, smalINORB neg with grayscale intensity between 0 and for white to black pixel, smalINORB mean with grayscale between -1 and +1 for black to white pixel and smalINORB neg sifciation accuracy on these four networks tested on 24300 image of each dataset was respectively 84.3%, 87.2%, 81.4% and 83.6%.
with grayscale between -1 and for white black pixel. The clas-
For the third, CIFAR-I0 evaluation, single network was trained on augmented (by random contrast adjustment and crop a 24x24 size) images of the CIFAR-10 dataset giving test accuracy of 81.0%.
B. NUMBER OF ACTIVE NUEURONS
Table 2 shows the average percentage of neurons that are active during input-output mappings for the trained networks evaluated in this paper.
C MULTIPLE OBJECTS OF INTEREST AND ADVERSARIAL EXAMPLES
We evaluate Insens against other methods on a classification task over images with two objects one to of interest and the another to be ignored. For this experiment created dataset where the

---

Under review as a conference paper at ICLR 2020
Table 2: Average percentage of active neurons in a neural network
Dataset
Network
Total # neurons
Average % active neurons
MNIST
2CONV
38144
17%
MNIST gauss
2CONV
38144
25%
MNIST outline
2CONV
38144
27%
MNIST border
2CONV
38144
23%
smallNORB
2CONV
442880
20%
CIFAR-10
2CONV
28160
19%
input consists of an even and odd MNIST digit concatenated in random order. The training label identifies the even digit, making it the object of interest. We call this dataset MNISTdbl. Figure Insens and other methods for a 2CONV network trained on this dataset achieve 99.5% test accuracy. All the methods seem emphasise the object of interest mostly by making the even digit brighter and more prominent in the visualisation. LRP visualisation seems also differentiate the two objects by colour with the object of interest highlighted in red and the other in blue.
shows heatmap visualisations for penultimate-lay
For deeper evaluation of these interpretability methods, we obtain visualisations from the same 2CONV network (trained on the MNISTdbl images) using different types of adversarial examples: with only the object to ignore present (Figure 9). two objects of interest in a single image (Figure 10) and two objects to ignore in single image (Figure TT.. Through the same red/blue colour highlighting scheme as in the previous example, LRP-based visualisations tend to communicate rather well which objects in the image are those of interest and which ones are the ones the network has learned to ignore. It does not provide too much explanation of the predictions that the network outputs for these examples. DeepTaylor and Integrated Gradients methods highlight the digit used for prediction in the two even digits case. Insens visualisation might be providing information about the decisions with respect to the output predictions, but other than the fact that the objects of interest tend to get better outlines, it is hard to say what the nature of these decisions might be. Most intuitive expectation for the predicted digit show up the visualisation. However, this is making an assumption about the way the network makes its decisions, and, as other adversarial examples show, it's not necessarily the case that the network perceives the shape of the shape of the digits like we do.
Finally, we include an evaluation of the penultimate layer Insens and comparison to other methods on gradient-based adversarial examples, where a correctly classified image has been modified in the direction of the input gradient obtained from a trained 2CONV network to increase the activ- ity the output corresponding to the wrong label. Figure 12 shows visualisations from different interpretability methods over different adversarial examples derived from the same image for the MNIST and smallNORB trained 2CONV networks. Figure 13 shows the difference between the visualisations from the real and adversarial input. Once again, the predicted digit does not show up in any of the visualisations, though again, Insens does indicate subtle changes, especially in the MNIST examples, with parts of the digits missing and some extra bits showing up.
12

---

Under review as a conference paper at ICLR 2020
Integrated Gradients
LRP-PresetAFlat
Gradient
DeepTaylor
Input
Insens
label:
pred:
>
y
label:
8 3
pred:
label:
3 2
pred:
label:
2
pred:
label:
0 9
pred:
label:
pred:
3 0
Figure 8: Visualisations of the 2CONV architecture trained MNISTdbl dataset, where input is composite of an odd and even digit from the MNIST database and the target label corresponds to the even digit; input images shown test images from the MNISTdbl dataset.
Integrated Gradients
LRP-PresetAFlat
DeepTaylor
Gradient
Input
Insens
pred: 8
pred:
7
pred: 8
9
pred: 3
pred: 8
5
3
pred: 2
Figure 9: Visualisations of the 2CONV architecture trained on MNISTabl dataset, where input is composite of odd and even digit from the MNIST database and the target label corresponds the even digit; input images shown are ad alversarial examples where only the odd digit is present.
13

---

Under review as a conference paper at ICLR 2020
Integrated Gradients
LRP- HMP-PresetAFlat
DeepTaylor
Gradient
Input
Insens
pred:
0
8
4
6
pred:
O y
pred:
pred: 2 '8
pred: 6 8
pred: 20
0
Figure 10: Visualisations of the 2CONV architecture trained on MNISTabl dataset, where input is a composite of an odd and even digit from the MNIST database and the target label corresponds to the even digit; input images shown are adversarial examples with two even digits.
Integrated Gradients
LRP- HMP-PresetAFlat
DeepTaylor
Input
Gradient
Insens
3
5
pred:
pred: 6
/ 9
7 5
pred: 2
pred: 8
7 3
pred: 4
7 7
pred: 3 3
3 3
Figure 11: Visualisations of the 2CONV architecture trained on MNISTabl dataset, where input is a composite of an odd and even digit from the MNIST database and target label corresponds to the even digit; input images shown are adversarial examples with two odd digits.
14

---

Under review as a conference paper at ICLR 2020
Integrated Gradients
Intagrated Gradients
Inpul
Gradient
DeepTaylor
LRP-PresetAFial
Insene
Input
Gradient
DeepTaylar
LRR-PresetAFiat
Insens
2
Animo
prod.
predi Animal
pred:
peed: Plane
Visualisations for a set of interpretability methods of single 2CONV neural network trained on the MNIST dataset (left) and smalINORB dataset (right) in response to advesrarial ex- amples; the intensity of red and blue correspond respectively to the magnitude of the positive and negative values in the heatmaps.
Figure 12:
Integrated Gradients
Integrated Gradients
LRP. Pre-PrestAFist
LAP PresersFFaa
Input
Gradient
DeepTaylor
Insens
Input
Gradient
DeepTaylor
Insens
pred:
Pane
pred: 6
presi
pred-9
predi Truck
Figure 13: Visualisations of the difference between the visualisations from Figure 12 the differ-
ence shown between visualisation based on the image from the dataset (top row Figure [2 and respectively each adversarial example visualisation (second, thirds and fourth rows of Figure [2 the intensity of red and blue correspond respectively to the magnitude of the positive and negative differences of values in the heatmaps.
15


Reviews:
[{"review_id": "rklJ1VSbjH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript introduces a novel method to explain activities of ReLU-based deep networks by constructing a linear subnetwork which only contains neurons activated by the input. The status of each neuron can be obtained given any input sample. Moreover, the author applies the notion of “neuron’s center”, which is a neutral data point that is similar to actual input x, but with differences in particular objects to cause f(x) be positive. The activity of each neuron can be decomposed into the attribution of each input pixel, and this decomposition can also be used to measure the contribution of each pixel to the network stability. Overall, the proposed methodology is intuitive and distinctive to the state-of-the-art interpretability methods.\n\nHowever, the application constraint on the ReLU-based deep neural network prevents this method from being a model agnostic approach: the problem formulation would be much different if other non-linear activation functions are used. Although the experiment part visualizes the superiority of switched linear projections over other prevalent approaches, the evaluations contain mostly subjective assessment and the arguments are monotonous. I would suggest adding more experiments with quantitative analysis, or mathematically demonstrate why the proposed method is better than, say purely gradient-based method, in the linear case. In addition, additional experiments on a broader set of input data (e.g., tabular, text) could avoid the evaluations look cherry-pick. \n\nMinor issues: \n\n1. In figure 2, I think it would be better to write down explicitly the connections between v, \\hat{b} and \\hat{w} for each neuron given any input. Just seeing v and \\hat{b} on top of each subfigure is a bit confusing. \n2. I spent a long time to understand the \"neuron’s center\" concept, it might be better to add some background or mathematical formulation.\n3. In figure 4, when the digits get misclassified, the Insens explanation should highlight the patterns of wrongly predicted digits, but the patterns of neurons' inactive state sensitivity still look like the correct digits.\n4. It would also be interesting to show how the Insens explanation would change when the input is under various kinds of adversarial attacks rather than adding simple Gaussian noise.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Syl-oOHTYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work basically introduced a new way of looking at interpretability; instead of focusing on the source of activations in the network for a given input image, focus on the source of stability (non-active) neurons (in a ReLU network). The work starts by proving (although it is trivial) that in a ReLU (more generally any piece-wise linear) network, for a given input image, there is a locally linear relationship between a given neuron's activation and the image: v= w^T x + b. As the authors correctly mention, focusing on 'w' as the sensitivity analysis is basically the vanilla gradient method. The contribution, however, is focusing on the projection of bias and the introduced notion of 'centre'. With this provided notion, one can focus on the deactivated neurons in the network and how each input pixel is responsible for it. In other words, unlike previous work that focuses on the activation map, the authors correctly refer to the deactivated neurons as another source of the network's prediction.\n\nI'm have reasons for both accepting and rejecting this work.  The work provides a new perspective and asks a very interesting question. The introduced method, although quite simple and trivial, is useful and the authors do a very good job of making valid and reasonable claims about their work's contribution and how it connects to the existing literature. The main drawback of the paper, however, is whether the contributions are enough for this venue. The paper does not convince me that the introduced method would result in better interpretability of deep networks compared to what is already there. Another minor (or for some people in the field major) issue is the experimental setup.  All of the experiments are focused on subjective examples and no objective measure of the introduced method is provided (and the field has many of those objective measures). Providing a few examples of the method in comparison with other methods is not sufficient. Anyhow, the experiment where they prove the usefulness of the method by adding background noise is interesting. I would personally suggest the authors to expand this experiment to testing the method's sanity using the sanity measures provided in previous work: https://arxiv.org/abs/1810.03292 The claims made about the results on smallNORB can be controversial as the authors interpret their method's flipping of importance to be the reality of what's happening in the network and the other method's focus on the edges as false; this is not clear to be true. My score would be subject to change if better experimental results are provided (and the other way round).\n\nA few suggestions and questions:\n* One very important issue with the method is that it considers all of the inactive neurons. We know that a substantial percentage of inactive neurons are just dead neurons the stability of which does not matter. How would the method address the issue?\n* There definitely needs to be an objective measure of the introduced method's performance compared to previous work.\n* The work seems very related to DeepLIFT while there is no mentioning of it.\n* I'm not a fan, but adding results on a SOTA ImageNET paper always helps with making the experiments section crisper.\n* The authors claim that even small perturbations will change the activation pattern. This is not a small claim and is definitely in need of more evidence.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SkxlgZMOjH", "reviewer": "AnonReviewer7", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to capture patterns of “off” neurons using a newly proposed metric. While the authors have considered only linear networks, the setup is still relevant because how often these networks can give meaningful results, and can possibly pave the way for future research into more general networks. \n\nPros: The idea itself is interesting, the related works are discussed well, and MNIST experiments are very interesting. \n\nCons/comments : The writing needs a lot of improvement if to be considered for a top venue like iclr.  Other than the MNIST experiments, which show and indicate the importance of relative contrast and boundary, I am not sure how other experiments are meaningful. CIFAR and smallnorb experiments are merely presented, without discussions on what the interpretation shows or helps over the existing methods. Infact, the other methods seem to capture a lot more information than the proposed method. I would suggest adding more discussions and more experiments that show interpretation that this method/metric helps with to make this work stronger.\n\nHave the authors considered  the metric to consider “on” neurons instead of “off”neurons ? Is it possible to have a general metric that combines the two in some way ? Intuitively, its unclear to me why only the off patterns can help (except in specific cases as shown in MNIST experiments). \n\n “and thus is responsible for the activity vi” – This is unclear to me. I understand the projection part though, but I cant make sense of this statement.\n\n“interpretation and interpretability ” in the introduction – the writing is too informal. Making use of italicized phrases like “switched linear projection” does not help with the understanding at all, especially because “switched” is defined after using the term atleast thrice. \n\nConfirmation bias <-> “information we want to get”. \n\nThe same issue right after eq 6. “Reference subtracted from ….” where the first word is italicized to probably imply some intuitive explanation, but for someone not familiar with what reference is just tends to confuse the reader.  \nPlease fix missing references.\n\nEq 7 seems written incorrectly, with the where “v= …”. Please fix. \n\nWhat is the variance of saliency checks ? In other words, if the experiment of 100 random samples is repeated (say) 20 times, how different are the corresponding coefficients across these repeated runs ?\n\nFigure 3 is waste of space (move to appendix?)\n\nI might be splitting the hairs but Theorem 1 does not warrant a theorem. The result/proof is too straightforward to be a theorem and is already known in some form in the folklore. \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lRpmI9Kr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Notes: \n\n  -Goal is to study the \"active subnetwork\" of Relu based networks for interpretability.  \n\n  -The question of interpretation seems rather thorny.  \n\n  -In Figure 4, the result for Insens seem alright, although it's weird that the data is just mnist digit / noise.  I feel like something with multiple objects would make it much clearer if there is an actual improvement?  For example on Figure 5 I'm not really sure if Insens is better.  The results often look worse to me than \"DeepTaylor\", especially on CIFAR10.  \n\nReview: This paper proposes to improve the interpretation of relu based networks by considering the \"inactive network\" which could potentially become activated by local perturbations instead of just considering the active part of the network (which is locally linear).  I think this is a step in the right direction for the interpretation of relu based networks, although the results are somewhat borderline.  \n\nAdditionally the tasks could be much better, to show situations where an object is present but which is not related to the labels.  This would provide a much clearer test of the model's capabilities.  \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review:  This paper proposes a method to capture patterns of the so called “off” neurons using a newly proposed metric. The idea is interesting and worth pursuing. However, the paper needs another round of modification to improve both writing and experiments. 
Decision: Reject