Step-wise Sensitivity Analysis: Identifying Partially Distributed Representations for Interpretable Deep Learning

1 INTRODUCTION . Deep Neural Networks ( DNNs ) have impressed the scientific community because of their performance and the variety of domains to which they can be applied . However , DNNs are difficult to interpret because of their highly complex non-linear and interconnected nature . The lack of transparency is a threefold problem . First , it inhibits adoption , especially in industries under heavy regulation and with a high cost of errors . Second , it makes it difficult to debug existing models and hampers development progress . Third , it prevents us from utilising the insights gained from the models for further knowledge discovery . DNN interpretability is loosely defined , and it is also referred to as Explanatory AI , Understandable Machine Learning , and Deep Visualisation . DNN interpretability can be gained from a humaninterpretable explanation of the reasons behind the network ’ s choice of output ( Ribeiro et al. , 2016 ; Doshi-Velez & Kim , 2017 ) . In a DNN the basis for a decision is encoded in features either as one neuron – local representation ; or as a set of neurons – partially-distributed representation ( PDR ) ( Li et al. , 2016 ; Fong & Vedaldi , 2018 ) . The identification of PDRs and their interactions remains the main hindrance to end-to-end interpretability systems ( Olah et al. , 2018 ) . Once identified , PDRs will enable us to give much finergrained explanations ( e.g. , an image is a shark because the network detected a sea , sharp teeth , a long fin , etc. ) . In this paper , we introduce our novel technique , step-wise sensitivity analysis ( SSA ) , with which we make 3 contributions towards this vision : SSA 1 ) identifies PDRs ; 2 ) illustrates interactions between related PDRs ; 3 ) applies a novel perspective for interpretability – statistics across multiple input stimuli ( instance-specific ) to gain a general understanding of the DNN operation ( model-centric ) . The method produces statistical topological interpretability . That is , we analyse the network ’ s properties as a directed graph over various inputs to produce a dependency graph between neurons . The dependency graph highlights the relationships between adjacent layers that are pertinent to the decision , and how these relationships are formed in each layer and across all layers to form a feature representation . The remainder of this paper is organised as follows : Section 2 reviews related work ; in Section 3 we introduce our technique for building the cross-input layer-wise dependency graph ; Section 4 illustrates three novel neuron relevance metrics stemming from the dependency graph and how they can be used to determine and interpret neurons of interest ; Section 5 makes concluding remarks and suggests how our work can be applied and extended . 2 BACKGROUND . In this section , we organise the existing effort dedicated to DNN interpretability , based on its three main limitations : functional vs. topological , instance-specific versus model-centric , single neuron versus layer analysis . We then compare step-wise sensitivity analysis with other works that address the same limitations . First , recent attempts focus primarily on the input-output relationship , considering the network as a black-box function . These methods are classified as functional since they treat the entire network as a black-box function with an input , output and parameters , while the topological approach considers the structure of the network . One of the most investigated areas in the functional vein is sensitivity analysis , which produces a heatmap , illustrating the input parts relevant to the output decision . Examples include deconvolution ( Zeiler & Fergus , 2014 ) , sensitivity image-specific class saliency visualisations ( Simonyan et al. , 2013 ) , guided-back propagation ( Springenberg et al. , 2014 ) , and predictive difference analysis ( Zintgraf et al. , 2017 ) . In contrast to these functional methods , Section 3 demonstrates how the sensitivity analysis technique can be modified to gain more granular information across layers . The second limitation is that there are two completely opposite types of visualisation approaches for interpretability – model-centric or instance-specific . The model-centric approaches , such as activation maximisation ( Erhan et al. , 2009 ) and inversion ( Mahendran & Vedaldi , 2015 ) , synthesise a prototype image to explain a neuron . This can be generalised to every data point , but does not contain enough details to reason about particular mistakes or edge-cases . On the other hand , instance-specific methods operate on the level of a single instance , but this fine-granularity can not be used to elicit principles applicable across a wider set of instances ( e.g. , the relevance of regions in a single image ) . Our methodology iterates over instance-specific results , and we compare the results for different classes to illustrate similar “ thought patterns ” , that is , the relevance of filters is shared across classes . Thus , our approach can be viewed as both , instance-specific and model-centric . Third , current approaches either explore a single neuron in isolation , such as sensitivity analysis and activation maximisation , or the entire layer , such as inversion ( Mahendran & Vedaldi , 2015 ) . The former approach assumes purely local representations , while the latter assumes fully-distributed representations . However , recent findings ( Li et al. , 2016 ; Fong & Vedaldi , 2018 ; Agrawal et al. , 2014 ; Bau et al. , 2017 ) suggest that every layer consists of a mixture of local and partially-distributed representations . Our approach addresses the last two limitations , in particular , it interprets the internal DNN structure across layers of the network to identify the different types of representations in each layer . 2.1 RELATED WORK . 2.1.1 RELEVANCE SCORE TECHNIQUES . Net2Vec ( Fong & Vedaldi , 2018 ) builds on network dissection ( Bau et al. , 2017 ) to propose a method for selecting and combining relevant filters , and providing an explanation for filters in conjunction with each other , thus identifying and interpreting PDRs . This method determines the relevance of neurons by optimising the combinations of filters for classification and segmentation on proxy ad hoc tasks . In contrast , our method can ascertain the neuron relevance using the original data set , which makes it more generalisable as it is not necessary to compile explanatory datasets for various problems . On the other hand , similarly to our approach , in Landecker et al . ( 2013 ) it is argued that computing the relevance in a layer-by-layer fashion yields more fine-grained results , and draws the distinction between functional and topological approaches . In Bach et al . ( 2015 ) this idea is incorporated into a sensitivity analysis technique - Layer-wise-relevance propagation ( LRP ) . Deep Taylor Decomposition ( Montavon et al. , 2017 ) generalises the output of such sensitivity analysis techniques to an importance score , which is computed in a topological layer-wise fashion . DeepLift ( Shrikumar et al. , 2017 ) proposes an alternative functional approximation method for computing the relevance score by comparing the activation of a neuron to a reference one . Our work is similar to the rel- evance score techniques in that it uses the original sensitivity analysis approach as an importance score metric . In addition and similarly to the topological approaches , we suggest computing the importance score at each layer . However , we propose that the network ’ s decision is driven by a partially-distributed representation rather than the entire layer . Hence , instead of distributing the relevance across all neurons , we only redistribute the relevance to a small number of outliers . 2.1.2 CONSTRAINED REDISTRIBUTION . Our work is comparable to excitation backpropagation ( Zhang et al. , 2016 ) , which distributes the relevance to a subset of neurons . There are two important differences . First , excitation backpropagation focuses on improving the heatmap quality , while we investigate how to discover PDRs . Second , while excitation backpropagation uses a probabilistic winner-take-all sampling approach that is limited to neurons with ReLU activations and positive weight connections between adjacent layers , we deploy a more generalisable linear Taylor approximation and statistical analysis over multiple inputs to restrict the relevant neurons . 2.1.3 VISUALISATION . Our method resembles the approach in ( Liu et al. , 2017 ) in that we also generate a DAG and augment it with additional visualisations to produce an explanation . The main difference resides in that they use clustering to reduce the visual clutter , and group neurons together . In contrast , we propose a novel method to select only the relevant paths for investigation . 3 STEP-WISE SENSITIVITY ANALYSIS . Our interpretability method is based on the sensitivity analysis technique by Baehrens et al . ( 2010 ) that was first applied to DCNNs by Simonyan et al . ( 2013 ) to produce image-specific class saliency visualisations . Formally , given an image I0 , a representation function Φ : RH×W×C → Rd such that Φ ( I ) = o , and a neuron n – approximate the activation of on with a linear function . In the neighbourhood of Ii , this is achieved by computing the first-order Taylor expansion : on = Φn ( I ) ≈ ωT I + b ( 1 ) where ω is the gradient of Φnwith respect to an image I . The function is evaluated at image Ii : ω = ∂Φn ∂I ∣∣∣ Ii ( 2 ) This formulation allows us to interpret the magnitude of the values of ω as an importance metric corresponding to each pixel . In other words , these values indicate which pixels need to be changed the least to change Φ ( I ) such that on ( corresponding to a classification decision ) is increased the most . 3.1 SINGLE BACK-PROPAGATION STEP . Sensitivity analysis performs a complete back-propagation pass to compute ω in equation 2 . The end result is a class saliency map , which is particularly useful to identify the image regions most pertinent to the network ’ s decision . However , this is a very coarse-grained explanation since it only analyses the input-output relationship . There is no information regarding the effect of particular layers and neurons on the selection of the relevant image regions . We propose a much more finegrained analysis based on the simple hypothesis that sensitivity analysis can be used in an analogous way to determine the relevance between adjacent layers . Instead of trying to approximate on directly , we consider Φ to be defined as the successive composition of smaller functions that represent the transformations of data between layers : Φ ( I ) = f l ( Φl−1 ( I ) ) = f l ◦ f l−1 ◦ f l−2 ... ◦ f1 ( I ) ( 3 ) where l = 1 ... L , L is the network ’ s depth , and each layer denoted as f l : Rd′ → Rd represents the operation applied by layer l , when d′ is the output dimensionality of the input layer f l−1 and d is the output dimensionality of layer l. Starting with neuron n from the top layer f ln , we conduct only the last step of back-propagation to compute the ωl−1 ∈ Rd′ values , where the node of the network under consideration is f l−1 ∈ Rd′ . We call our approach Step-wise sensitivity analysis ( SSA ) since it iteratively applies sensitivity analysis through the computation of a single back-propagation step . The result of a single back-propagation step can be seen as performing one step of step-wise sensitivity analysis between a higher layer l and a lower one j : ωln , i , : = ∂f ln ( Φj ( I ) ) ∂I ∣∣∣∣∣ Ii ( 4 ) The ω values now represent the neurons which have to be changed the least to affect the upper layer ’ s neuron activation the most , and as such they can be treated as the relevance scores for f ln of all lower layer neurons f l−1 . A large positive ωln , i , k value means the neuron f l−1 k contributes substantially to the activation of f ln , while a large negative ω l n , i , k value inhibits the activation . The ability to identify the positively and negatively contributing neurons in layer-by-layer fashion is the first step towards understanding the internal representation structure and identifying partiallydistributed representations . This can be accomplished by using Step-wise sensitivity analysis and the resulting ωl to guide a hierarchical traversal of the network layers . Next , we formally describe our method .

Summary: This paper introduces step-wise sensitivity analysis (SSA), which is a modification of saliency maps (Baehrens et al. 2010, Simonyan et al. 2013) to a per-layer implementation. Instead of only measuring the importance of input nodes (e.g. pixels) to the classification, SSA measures the importance of all nodes at each layer. This allows for a way to find the important sub-nodes for each node in the tree given a particular sample. It is then straightforward to aggregate results across different input samples and output a dependency graph for nodes.