STEP-WISE SENSITIVITY ANALYSIS: IDENTIFYING PARTIALLY DISTRIBUTED REPRESEN-
Year: 2018

Abstract:
In this paper, we introduce a novel method, called step-wise sensitivity analysis, which makes three contributions towards increasing the interpretability of Deep Neural Networks (DNNs). First, we are the first to suggest a methodology that aggregates results across input stimuli to gain model-centric results. Second, we linearly approximate the neuron activation and propose to use the outlier weights to identify distributed code. Third, our method constructs a dependency graph of the relevant neurons across the network to gain fine-grained understanding of the nature and interactions of DNN’s internal features. The dependency graph illustrates shared subgraphs that generalise across 10 classes and can be clustered into semantically related groups. This is the first step towards building decision trees as an interpretation of learned representations.

heading: 1 INTRODUCTION
text: Deep Neural Networks (DNNs) have impressed the scientific community because of their performance and the variety of domains to which they can be applied. However, DNNs are difficult to interpret because of their highly complex non-linear and interconnected nature. The lack of transparency is a threefold problem. First, it inhibits adoption, especially in industries under heavy regulation and with a high cost of errors. Second, it makes it difficult to debug existing models and hampers development progress. Third, it prevents us from utilising the insights gained from the models for further knowledge discovery.
DNN interpretability is loosely defined, and it is also referred to as Explanatory AI, Understandable Machine Learning, and Deep Visualisation. DNN interpretability can be gained from a humaninterpretable explanation of the reasons behind the network’s choice of output (Ribeiro et al., 2016; Doshi-Velez & Kim, 2017). In a DNN the basis for a decision is encoded in features either as one neuron – local representation; or as a set of neurons – partially-distributed representation (PDR) (Li et al., 2016; Fong & Vedaldi, 2018).
The identification of PDRs and their interactions remains the main hindrance to end-to-end interpretability systems (Olah et al., 2018). Once identified, PDRs will enable us to give much finergrained explanations (e.g., an image is a shark because the network detected a sea, sharp teeth, a long fin, etc.). In this paper, we introduce our novel technique, step-wise sensitivity analysis (SSA), with which we make 3 contributions towards this vision: SSA 1) identifies PDRs; 2) illustrates interactions between related PDRs; 3) applies a novel perspective for interpretability – statistics across multiple input stimuli (instance-specific) to gain a general understanding of the DNN operation (model-centric).
The method produces statistical topological interpretability. That is, we analyse the network’s properties as a directed graph over various inputs to produce a dependency graph between neurons. The dependency graph highlights the relationships between adjacent layers that are pertinent to the decision, and how these relationships are formed in each layer and across all layers to form a feature representation.
The remainder of this paper is organised as follows: Section 2 reviews related work; in Section 3 we introduce our technique for building the cross-input layer-wise dependency graph; Section 4
illustrates three novel neuron relevance metrics stemming from the dependency graph and how they can be used to determine and interpret neurons of interest; Section 5 makes concluding remarks and suggests how our work can be applied and extended.

heading: 2 BACKGROUND
text: In this section, we organise the existing effort dedicated to DNN interpretability, based on its three main limitations: functional vs. topological, instance-specific versus model-centric, single neuron versus layer analysis. We then compare step-wise sensitivity analysis with other works that address the same limitations.
First, recent attempts focus primarily on the input-output relationship, considering the network as a black-box function. These methods are classified as functional since they treat the entire network as a black-box function with an input, output and parameters, while the topological approach considers the structure of the network. One of the most investigated areas in the functional vein is sensitivity analysis, which produces a heatmap, illustrating the input parts relevant to the output decision. Examples include deconvolution (Zeiler & Fergus, 2014), sensitivity image-specific class saliency visualisations (Simonyan et al., 2013), guided-back propagation (Springenberg et al., 2014), and predictive difference analysis (Zintgraf et al., 2017). In contrast to these functional methods, Section 3 demonstrates how the sensitivity analysis technique can be modified to gain more granular information across layers.
The second limitation is that there are two completely opposite types of visualisation approaches for interpretability – model-centric or instance-specific. The model-centric approaches, such as activation maximisation (Erhan et al., 2009) and inversion (Mahendran & Vedaldi, 2015), synthesise a prototype image to explain a neuron. This can be generalised to every data point, but does not contain enough details to reason about particular mistakes or edge-cases. On the other hand, instance-specific methods operate on the level of a single instance, but this fine-granularity cannot be used to elicit principles applicable across a wider set of instances (e.g., the relevance of regions in a single image). Our methodology iterates over instance-specific results, and we compare the results for different classes to illustrate similar “thought patterns”, that is, the relevance of filters is shared across classes. Thus, our approach can be viewed as both, instance-specific and model-centric.
Third, current approaches either explore a single neuron in isolation, such as sensitivity analysis and activation maximisation, or the entire layer, such as inversion (Mahendran & Vedaldi, 2015). The former approach assumes purely local representations, while the latter assumes fully-distributed representations. However, recent findings (Li et al., 2016; Fong & Vedaldi, 2018; Agrawal et al., 2014; Bau et al., 2017) suggest that every layer consists of a mixture of local and partially-distributed representations. Our approach addresses the last two limitations, in particular, it interprets the internal DNN structure across layers of the network to identify the different types of representations in each layer.

heading: 2.1 RELATED WORK

heading: 2.1.1 RELEVANCE SCORE TECHNIQUES
text: Net2Vec (Fong & Vedaldi, 2018) builds on network dissection (Bau et al., 2017) to propose a method for selecting and combining relevant filters, and providing an explanation for filters in conjunction with each other, thus identifying and interpreting PDRs. This method determines the relevance of neurons by optimising the combinations of filters for classification and segmentation on proxy ad hoc tasks. In contrast, our method can ascertain the neuron relevance using the original data set, which makes it more generalisable as it is not necessary to compile explanatory datasets for various problems. On the other hand, similarly to our approach, in Landecker et al. (2013) it is argued that computing the relevance in a layer-by-layer fashion yields more fine-grained results, and draws the distinction between functional and topological approaches. In Bach et al. (2015) this idea is incorporated into a sensitivity analysis technique - Layer-wise-relevance propagation (LRP). Deep Taylor Decomposition (Montavon et al., 2017) generalises the output of such sensitivity analysis techniques to an importance score, which is computed in a topological layer-wise fashion. DeepLift (Shrikumar et al., 2017) proposes an alternative functional approximation method for computing the relevance score by comparing the activation of a neuron to a reference one. Our work is similar to the rel-
evance score techniques in that it uses the original sensitivity analysis approach as an importance score metric. In addition and similarly to the topological approaches, we suggest computing the importance score at each layer. However, we propose that the network’s decision is driven by a partially-distributed representation rather than the entire layer. Hence, instead of distributing the relevance across all neurons, we only redistribute the relevance to a small number of outliers.

heading: 2.1.2 CONSTRAINED REDISTRIBUTION
text: Our work is comparable to excitation backpropagation (Zhang et al., 2016), which distributes the relevance to a subset of neurons. There are two important differences. First, excitation backpropagation focuses on improving the heatmap quality, while we investigate how to discover PDRs. Second, while excitation backpropagation uses a probabilistic winner-take-all sampling approach that is limited to neurons with ReLU activations and positive weight connections between adjacent layers, we deploy a more generalisable linear Taylor approximation and statistical analysis over multiple inputs to restrict the relevant neurons.

heading: 2.1.3 VISUALISATION
text: Our method resembles the approach in (Liu et al., 2017) in that we also generate a DAG and augment it with additional visualisations to produce an explanation. The main difference resides in that they use clustering to reduce the visual clutter, and group neurons together. In contrast, we propose a novel method to select only the relevant paths for investigation.

heading: 3 STEP-WISE SENSITIVITY ANALYSIS
text: Our interpretability method is based on the sensitivity analysis technique by Baehrens et al. (2010) that was first applied to DCNNs by Simonyan et al. (2013) to produce image-specific class saliency visualisations. Formally, given an image I0, a representation function Φ : RH×W×C → Rd such that Φ(I) = o, and a neuron n – approximate the activation of on with a linear function. In the neighbourhood of Ii, this is achieved by computing the first-order Taylor expansion:
on = Φn(I) ≈ ωT I + b (1) where ω is the gradient of Φnwith respect to an image I. The function is evaluated at image Ii:
ω = ∂Φn ∂I ∣∣∣ Ii
(2)
This formulation allows us to interpret the magnitude of the values of ω as an importance metric corresponding to each pixel. In other words, these values indicate which pixels need to be changed the least to change Φ(I) such that on (corresponding to a classification decision) is increased the most.

heading: 3.1 SINGLE BACK-PROPAGATION STEP
text: Sensitivity analysis performs a complete back-propagation pass to compute ω in equation 2. The end result is a class saliency map, which is particularly useful to identify the image regions most pertinent to the network’s decision. However, this is a very coarse-grained explanation since it only analyses the input-output relationship. There is no information regarding the effect of particular layers and neurons on the selection of the relevant image regions. We propose a much more finegrained analysis based on the simple hypothesis that sensitivity analysis can be used in an analogous way to determine the relevance between adjacent layers.
Instead of trying to approximate on directly, we consider Φ to be defined as the successive composition of smaller functions that represent the transformations of data between layers:
Φ(I) = f l(Φl−1(I))
= f l ◦ f l−1 ◦ f l−2... ◦ f1(I) (3)
where l = 1...L, L is the network’s depth, and each layer denoted as f l : Rd′ → Rd represents the operation applied by layer l, when d′ is the output dimensionality of the input layer f l−1 and d is
the output dimensionality of layer l. Starting with neuron n from the top layer f ln, we conduct only the last step of back-propagation to compute the ωl−1 ∈ Rd′ values, where the node of the network under consideration is f l−1 ∈ Rd′ . We call our approach Step-wise sensitivity analysis (SSA) since it iteratively applies sensitivity analysis through the computation of a single back-propagation step. The result of a single back-propagation step can be seen as performing one step of step-wise sensitivity analysis between a higher layer l and a lower one j:
ωln,i,: =
∂f ln
( Φj(I) ) ∂I ∣∣∣∣∣ Ii
(4)
The ω values now represent the neurons which have to be changed the least to affect the upper layer’s neuron activation the most, and as such they can be treated as the relevance scores for f ln of all lower layer neurons f l−1. A large positive ωln,i,k value means the neuron f l−1 k contributes substantially to the activation of f ln, while a large negative ω l n,i,k value inhibits the activation.
The ability to identify the positively and negatively contributing neurons in layer-by-layer fashion is the first step towards understanding the internal representation structure and identifying partiallydistributed representations. This can be accomplished by using Step-wise sensitivity analysis and the resulting ωl to guide a hierarchical traversal of the network layers. Next, we formally describe our method.

heading: 3.2 THE METHOD
text: The main contribution of our algorithm is the granularity with which it can illuminate the internal workings of a DNN. We generate a directed acyclic graph that spans the entire network. We call it a dependency graph because it increases the understanding of exactly how the activation of higher output layers depends on lower input layers. It illustrates how the input is transformed in each of the network’s layers and highlights “paths” that, when followed, identify the building blocks of higher level features. For example, our method can take us much closer to the ability to say that the concept of a shark is encoded as combination of a fin, body, and tail feature (see Fig. 1a).
Our step-wise sensitivity analysis method is separated into two parts. The first (wrapper) part traverses the network and maintains the input to iteratively apply the second part of the method. The second (step) part applies step-wise sensitivity analysis to identify relevant neurons, as specified in Algorithm 1.
Algorithm 1 Step-wise sensitivity analysis: Identifying partially-distributed representations
INPUT: DNN classifier Φ, a layer f l ∈ Rd from Φ, a set of relevant neurons n ∈ S, and a set of images Ii ∈ I. STEP 1: Compute relevance of neurons in layer f l−1 for each n and Ii so that if f l−1 is a:
1. Fully-connected layer: stack results into a relevance tensor ωln,:,: ∈ R|S|×K ; 2. Convolutional layer: spatially average the output volume tensor ωln,i,... into a relevance
tensor ωln,i,: ∈ R|S|×|I|×K ; 3. Pooling-layers: directly compute for l − 2: ωl = ∇fl−2f l|Ii
STEP 2: Select outliers as relevant neurons using 1.5× Inter Quartile Range STEP 3: Rank relevant neurons based on their relevance frequency across images (ωl,:n,k). STEP 4: Select top b relevant neurons, where b is a branching factor. OUTPUT: b relevant neurons for each distinct n in S.
The basic idea of our step-wise sensitivity analysis is illustrated in Fig. 1a. Given a DNN classifier Φ and a set of relevant neurons n ∈ S, start from the top layer and follow Algorithm 1 to produce a set of b relevant neurons – Bn, for each distinct n in S. Next, set S to the union of all relevant neurons Bn (S← ⋃ n Bn) for the lower layer, and repeat until the input layer. For computational efficiency, the magnitude of b is a threshold for the cardinality of each Bn, thus discarding a proportion of potential relevant neurons. We believe that b is an important hyper-parameter since it limits the size of potential PDRs, which recent studies indicate to be typically between 8 and 50 neurons (Fong & Vedaldi, 2018). We detail each of the steps in Algorithm 1 next.
STEP 1: COMPUTE RELEVANCE TENSOR
Input: This step requires a network (Φ), a layer f l, a neuron n ∈ f l, and an image Ii. Output: Computes the relevance score of neurons in layer f l − 1 with respect to a neuron n in layer f l as a gradient at Ii using Equation equation 4. Essentially, this produces the relevance of all neurons in layer f l − 1 to the activation of neuron n. Method: The relevance for DCNN is computed differently depending on the type of layer f l.
If f l is fully-connected, the result is a relevance vector ωln,i,: ∈ R|f l−1|. Repeating this process for all images and neurons in S yields a relevance tensor ωl.
If l is a convolutional layer, the result of Equation equation 4 is a 5D relevance tensor ωln,i,... ∈ RH×W×K , where H , W , K are respectively the height, width, and number of activation maps in l − 1. Since every activation map k is produced by convolving identical weights onto a lower layer activation map p, k represents the existence of an identical feature across p. Hence, the vector ωln,i,h,w,: represents the relevance of all lower level activation maps (features) at a location (h,w) to the activation of n. Since we are interested in the relative importance of a feature, we perform spatialaveraging over all locations (h,w) to convert ωf l n
i into a relevance vector ω f ln i ∈ RK , where each
dimension indicates the relative importance of an activation map across locations. This formulation enables us to repeat the process for all images, neurons and again obtain a 3D relevance tensor ωl.
The pooling layers can be seen as a filter of their predecessors since df l
dIi = c × df
l−1
dIi , where
c ∈ {0, 1}. Hence, if f l−1 is a pooling layer we compute the relevance tensor directly w.r.t l − 2: ωl = ∇f l−2f l|Ii .
STEP 2: OUTLIER DETECTION
Input: This steps requires a relevance tensor ωl. Output: The result identifies neurons relevant for neuron n in layer l − 1. Method: Our preliminary experiments indicated that each row ωln,i,: follows a normal distribution, and consistently exhibits a small number of outliers across i (see Figure 1b). Therefore, we make two simplifying assumptions. First, we assume these outliers are the only relevant neurons. Second, we choose to focus on only the positive ω outlier values and leave the analysis of negative outliers
for future work. Finally, we use the Tukey’s fences (1.5× Inter-Quartile Range) outlier detection method (Tukey, 1977) to select relevant neurons from each row ωln,i,:. Figure 1b illustrates this step.
STEP 3: RANKING
Input: Outliers in ωln,:,:. Output: Relevance ranking of each lower-layer neuron. Method: We use the outlier detection procedure to detect relevant neurons in every row. We rank the neurons based on the number of i columns of ωln,:,k in which they appear as relevant, resulting in a relevance ranking for each k ∈ f l−1.
STEP 4: SELECT
Input: Relevance ranking for each k ∈ f l−1. Output: b relevant neurons for each distinct n in S. Method: Select top b most frequent neurons B ⊂ ∀kk ∈ ωln,:,k as relevant, where b = |B| is a branching factor.

heading: 4 RESULTS AND DISCUSSION
text: We now demonstrate how our method can be applied to the ImageNet dataset (Russakovsky et al., 2015) for the 16-layer VGG network (VGG16) (Simonyan & Zisserman, 2014). We use the publicly available pre-trained model implemented in the deep learning framework keras (Chollet et al., 2015) and modify the keras-vis (Kotikalapudi & contributors, 2017) implementation of sensitivity analysis using the Guided-backpropagation algorithm (Springenberg et al., 2014). We perform experiments with 10 classes and select 100 images per class from the training set for which VGG16 the probability mass of the correct class is above 99%. The execution takes approximately 12 hours on NVIDIA Tesla K80 GPU to traverse the entire network for one class with a branching factor of 3. However, most of the time is spent within the convolutional layers, which have a larger number of unique neurons. Since our approach’s time complexity can be reduced to worst-case O(bd), where b is the branching vector and d is the depth (d = 22 in the case of VGG16) we constrain b due to computational limitations. The approach is still practical since it is not designed to be executed every time that an explanation is necessary, just as a network is not retrained every time before a prediction. In Section 4.1 we show the particular occurrence of outliers within the ωl values to demonstrate how our method chooses relevant neurons.
Furthermore, we provide in Section 4.1 a quantitative justification behind the choice of heuristics to detect relevant outliers, while in Section 4.2 we demonstrate our methodology on two classes, demonstrate that the results generalise across 10 class, and illustrate the importance of statistical interpretability.
The novelty of our approach is that it generates a dependency graph of relevant neurons. In contrast to other approaches, we study the process of feature formation and the interdependence of relevant features by following a path through the graph. This allows us to selectively visualise relevant activation maps through sensitivity analysis. We demonstrate not only an increase in the granularity of the state-of-the-art interpretability capabilities, but also that an analysis on a single image could lead to false assumptions about the operation of the network, as demonstrated in Section 4.2. We use ffc to refer to a fully-connected layer, and f bicj to refer to blocknconvj convolutional layer.

heading: 4.1 QUANTITATIVE JUSTIFICATION OF OUTLIERS AS RELEVANT NEURONS
text: According to the ω values, there is a consistent presence of a small number of relevant outlier neurons (less than 6%) – see Figure 2b. The outlier neurons are not identical for different input stimulus of the same class, as can be seen in Figure 2a. This indicates that our approach is not identical to simply selecting the neurons with highest weights, which would yield constant results across images. On the contrary, the frequency of relevance follows a power-law distribution. This suggests that the most frequently occurring neurons could be the main “drivers” (the most pertinent) for the class activation, while the other relevant neurons pick-up nuances or modulate the main drivers. In other words, the most relevant neurons form a basis, which is transformed by the less
relevant neuron. In some sense, we are finding the inverse of Szegedy et al. (2013)’s adversarial examples – high probability low-dimensional “pockets” in the manifold. What is surprising about Figure 2b is that it is not only consistent with the power law distribution result, but also that different images of the same class share peaks and troughs at exactly the same neurons. We hypothesise that Figure 2b is a visualisation of part of the PDR for a Hammerhead shark in layer ffc2. To investigate this hypothesis, we generate dependency graphs of relevant neurons for 10 different classes, which we describe next.

heading: 4.2 DEPENDENCY GRAPH
text: Figure 3 illustrates two examples of the output of Step-wise sensitivity analysis with two important observations. First, graphs in Figures 3a & 3b for different classes may share significant similarities. For example, the subgraphs of both Hammerhead shark and Egyptian cat starting from ffc21820 reveal similar most relevant neurons (e.g., ffc21820,f fc1 3116,f fc1 2053), identical subgraph structure (e.g., f fc1 3116 and its top 3 most relevant neurons) and similar subgraph structure (e.g., ffc12053, only one relevant child neuron is shared – f b5c349 ). Interestingly, the two classes share 6 out of the 8 most relevant activation maps in block_5_conv3– 41, 49, 155, 256, 313, 335. This implies that the dependency graphs open the frontier for pattern matching and analysis of network motifs (Milo et al., 2002) across classes. We hypothesise that the emerging network motifs will give a strong indication of the various PDRs positions within a layer and how upper PDRs are leveraging them. In Section 4.2.1 we present further evidence about the existence of network motifs.
Second, Figures 3a & 3b show that both dependency graphs share multiple incoming connections to the very same neuron (f b5c3155 ). It is surprising that this is the first time a neuron is shared within a class. Consequently, the interpretation of the dependency graph enables us to infer an additional relevance metric for a neuron – its inter-connectedness according to the number of incoming edges.
Therefore, step-wise sensitivity analysis allows researchers to focus analysis and interpretation efforts on the most pertinent regions of a DNN. For example, Figures 4b & 4b display targeted visualisation through sensitivity analysis of the especially relevant neuron f b5c3155 . Had we relied on a single visualisation, we would have erroneously presumed that the neuron perfectly encodes either the idea of a shark or of a cat. However, step-wise sensitivity analysis exposes that the neuron is equally important for both classes, and forms a part of a shared sub-structure. Therefore, it must encode a more abstract concept.
Exploring a neuron or activation map in isolation is simplistic. In reality, the semantics are expressed within the combination of neurons within the PDR. In future work, we will apply activation maximisation Erhan et al. (2009) of an entire PDR to investigate its semantic properties.

heading: 4.2.1 SHARED DEPENDENCY GRAPHS
text: In order to investigate the generalisability of the results in Figure 3, we transformed the 10 dependency graphs into bag-of-nodes features representations. Then we performed the ward method (Murtagh & Legendre, 2011) for hierarchical agglomerative clustering with cosine distance similarity to group the dependency graphs.
Interestingly, closer inspection of Figure 4a reveals three clusters of most similar dependency graphs – 1) hammer head and tiger shark; 2) African and Indian elephant; 3) German Sheppard and great white shark. Naturally, the first two clusters consist of the most semantically and visually similar classes – this supports the validity of our approach. Surprisingly, cluster 3) suggests an unnatural similarity between animals. One possible explanation could be that both of these classes share a PDR encoding sharp teeth. Another interesting observation is that the graphs are separated into two general clusters – one consisting of the all the sharks, the German Shepherd, and the Persian cat; and another consisting of the elephants, the Labrador Retriever and the Egyptian cat. One natural semantic separation between the two groups could be the degree of “danger”.
Finally, as expected, most of the lower layer activation maps are shared across all classes since they encode very abstract features. At the same time, while there is a large proportion of abstract neurons shared across all dependency graphs, the upper dendrogram in Figure 4a depicts other more specialised neuron clusters, which are idiosyncratic to their semantic groupings.
These three observations support the hypothesis that the dependency graphs reveal semantically meaningful groups of neurons across classes that form PDRs. Identifying and analysing such specific sub-graphs is a non-trivial graph theory problem, which we leave for future work. Once we are able to accurately extract the shared sub-graphs, we will be able to provide hierarchical explanation behind a particular decision. For example, the classification was a shark because the network detected a sea, sharp teeth, a fish tail and a long fin.

heading: 5 CONCLUSIONS & FUTURE WORK
text: In this paper we make three contributions to the area of interpreting learned representations. First, we are the first to propose a statistical DNN interpretability method that aggregates results of an instance-specific method to gain model-centric results. Second, we build a dependency graph of the relevant neurons to gain finer-grained understanding of the nature and interactions of a DNN’s internal features. Third, we propose three new relevance metrics to identify salient neurons: 1) the outlier weights of a linear approximation of the neuron activation; 2) the ranking score of a neuron based on its frequency as an outlier across multiple input stimuli; 3) the interconnectedness of a neuron within the dependency graph. Our method modifies sensitivity analysis into Step-wise sensitivity analysis that applies the same linear approximation – but on a layer-by-layer basis as opposed to the usual output-input basis. We demonstrate that the results generalise by illuminating shared subgraphs across 10 classes. These subgraphs can be grouped into semantic clusters since they contain the quintessential neurons of a class.
Step-wise sensitivity analysis opens an opportunity for further and more focused explorations of the internal operations of DNNs. Although it can still be applied on a single image to gain instancespecific interpretability, we argue that to gain a statistically viable result it is important to conduct analysis both, across classes and images. In the future, we will apply the approach to facilitate error explanation and decision justification on a much lower level by providing semantic interpretation of the discovered PDRs through visualisation approaches. We will demonstrate the features that make the difference between semantically similar classes and quantify the interpretability of the resulting PDRs using concept segmentation as in (Fong & Vedaldi, 2018; Bau et al., 2017). Further, we will investigate the suitability of our approach to defend against adversarial attacks. Finally, we will explore the possibility to use the dependency graphs to prune the network in order to perform network compression, or the extract binary classifiers for particular classes in the form of dependency graphs to distil the network.

heading: 6 APPENDIX

heading: 6.1 INVESTIGATION OF THE SHARED FEATURES
text: An important claim of our paper is that a comparison of activation maps across images can lead to a better understanding of the effect of a feature. For instance, rows 1 & 5, column 256 in Figure 5a could lead to the erroneous conclusion that f b5c3256 detects shark tails, while it also activated for the shark head and also for front and rear parts of cats.
Figure 5b suggests that neurons f b5c349 and f b5c3 335 are complementary. They activate for similar regions; however, they each capture different parts (e.g., a cat’s ear – row 1; the region of f b5c3335 is contained within that of f b5c349 , but has a much sharper boundary). In future work we will explore the exact relationship between such neurons.

Review summaries:
This paper introduces step-wise sensitivity analysis (SSA), which is a modification of saliency maps (Baehrens et al. 2010, Simonyan et al. 2013) to a per-layer implementation. Instead of only measuring the importance of input nodes (e.g. pixels) to the classification, SSA measures the importance of all nodes at each layer. This allows for a way to find the important sub-nodes for each node in the tree given a particular sample. It is then straightforward to aggregate results across different input samples and output a dependency graph for nodes.