Switched linear projections and inactive state sensitivity for deep neural network interpretability

We introduce switched linear projections for expressing the activity of a neuron in a ReLU-based deep neural network in terms of a single linear projection in the input space . The method works by isolating the active subnetwork , a series of linear transformations , that completely determine the entire computation of the deep network for a given input instance . We also propose that for interpretability it is instructive and meaningful to focus on the patterns that deactive the neurons in the network , which are ignored by the exisiting methods that implicitly track only the active aspect of the network ’ s computation . We introduce a novel interpretability method for the inactive state sensitivity ( Insens ) . Comparison against existing methods shows that Insens is robust ( in the presence of noise ) , complete ( in terms of patterns that affect the computation ) and a very effective interpretability method for deep neural networks . 1 INTRODUCTION . It is notoriously hard to interpret how deep networks accomplish the tasks for which they are trained . At the same time , due to the pervasiveness of deep learning in numerous aspects of computing , it is increasingly important to gain understanding of how they work . There are risks associated with the possibility that a neural network might not be “ looking ” at the “ right ” patterns ( Nguyen et al . ; Geirhos et al. , 2019 ) , as well as opportunities to learn from the network ’ s capable of better than human performance ( Sadler & Regan , 2019 ) . Hence , there is ongoing effort to improve the interpretation and interpretability of the internal representation of neural networks . What makes this interpretation of the inside of a neural network hard is the high dimensionality and the distributed nature of its internal computation . Aside from the first hidden layer , neurons operate in an abstract high-dimensional space . If that was not hard enough , the analysis of individual components of the network ( such as activity of individual neurons ) is rarely instructive , since it is the intricate relationships and interplay of those components that contain the “ secret sauce ” . The two broad approaches to dealing with this complexity is to either use simpler interpretable models to approximate what a neural network does , or to trace back the elements of the computation into the input space in order to make the internal dynamics relatable to the input . In the latter approach we are typically interested in neurons ’ sensitivity – how the changes in network input affect their output , and decomposition – how different components of the input contribute to the output . In this paper we propose a straightforward and elegant method for expressing the computation of an arbitrary neuron ’ s activity to a single linear projection in the input space . This projection consists of a switched weight vector and a switched bias that easily lend themselves to sensitivity analysis ( analogous to gradient-based sensitivity ) and decomposition of the internal computation . We also introduce a new approach for interpretability analysis , called inactive state sensitivity ( Insens ) , which uses switched linear projections to aggregate the contribution of patterns in the input that deactivate neurons in the network . We demonstrate on several networks and image-based datasets that Insens provides a comprehensive picture of a deep network ’ s internal computation . The only constraint for the proposed methods is that the network must use ReLU activation functions for its hidden neurons . 2 RELATED WORK . Previous work on deep learning interpretability is extensive with a wide variety of methods and approaches – Simonyan et al . ( 2014 ) ; Zeiler & Fergus ; Bach et al . ( 2015 ) ; Mahendran & Vedaldi ; Montavon et al . ( 2017 ) ; Sundararajan et al . ( 2017 ) ; Zhou et al . ( 2019 ) being just a selection of the most prominent efforts in this area . Our work on the single linear projection follows the approach akin to Lee et al . and Erhan et al . ( 2009 ) , where the objective was to interpret the computation performed by an arbitrary neuron for a particular input vector as a projection in the input space . However , whereas these previous attempts were based on Deep Belief Nets ( Hinton et al. , 2006 ) and required an approximation of the said projection , our method is a forward computation that gives the neuron ’ s activity in terms of a linear projection in the input space . It works for any neural network , including convolutional ones , as long as all hidden neurons use piecewise linear activation functions . All existing methods for interpretability of deep learning , due to the nature of ReLU computation , necessarily provide information only about the active subnetwork of the ReLU-based architecture . Our own observations , as well as other evidence showing that in practice neural networks produce a relatively low number of activation regions ( Hanin & Rolnick , 2019 ) , lead us to the hypothesis that the analysis of the patterns in the input that switch neurons off gives an excellent picture of a network ’ s sensitivity . We also take the view that too much interpretation in interpretability introduces the risk of showing us what we expect to see and not what the network is actually focussing on . For instance , in Deep Taylor Decomposition ( Montavon et al. , 2017 ) choices of different root-points for the decomposition of the relevance function lead to different rules for Layerwise Relevance Propagation ( LRP ) ( Bach et al. , 2015 ) , which can lead to different interpretations of what is important in the input . The LRP-α1β0 rule , for example , emphasises the computation over the positive weights in the network while discounting the relevance of the information passing through the negative weights . This rule is justified by assumptions about desired properties of the explanation , but this comes with a risk of confirmation bias . Insens is an attempt to take into account the patterns in the input that cause the neurons inside the network to produce zero output . The information related to the inactive network may seem irrelevant , since inactive neurons do not directly contribute to the computation of the overall output . However , there is something in the input that switches a particular set of neurons off , thus regulating the active computation , and as we show in this paper , this something carries a lot of meaningful information . 3 SWITCHED LINEAR PROJECTIONS . The basis of the switched network concept is the fact that neurons that produce output of zero do not contribute to the computation of the overall output of the network . The notion of dead neurons , that is neurons that always output zero , is not new , nor is the realisation that these neurons , along with their connecting weights , can be taken out the network without any impact on the computation . In a switched projection , we treat the zero-output neurons as temporarily dead for a given instance of input . We refer to these neurons as inactive , since they may become active for a different network input . Thus we isolate the subnetwork of the active neurons in a given computation . As it happens , for a ReLU neuron the active neurons are those that pass their activity , the weighted sum of its inputs plus bias , directly to its output1 . This means that a subnetwork of active ReLU neurons is just a series of linear transformations , which is equivalent to a single linear transformation . As a result , we can express the computation performed by any neuron in a ReLU network as a projection onto a switched weight vector in the input space plus the switched bias . The term switched indicates that this weight and bias vector changes when the state of the network changes , the state corresponding to the particular combination of the active and inactive neurons in the network . Figure 1 illustrates the concept graphically , and a formal description is given in the following proposition : Proposition 1 ( Switched linear projections ) . Let x ∈ Rd be a vector of inputs , wli ∈ RUl−1 the weight vector , and bli ∈ R the bias of neuron i in layer l ( with Ul−1 inputs from the previous layer ) . Let the activity of a neuron i in layer l be defined as : vli ( x ) = ( . . . σr ( σr ( xW1 + b1 ) W2 + b2 ) . . . ) wli + bli , ( 1 ) where Wl = [ wTl1 . . . w T lUl ] , T denotes transpose , bl = [ bl1 . . . blUl ] and σr ( v ) = max ( v , 0 ) is the ReLU activation function . If we define an input-dependent state of the network as W ( x ) l = [ σ̇r ( vl1 ( x ) ) wTl1 . . . σ̇r ( vlUl ( x ) ) wTlUl ] and b ( x ) l = [ σ̇r ( vl1 ( x ) ) bl1 . . . σ̇r ( vlUl ( x ) ) blUl ] , where σ̇r ( v ) = dσr ( v ) dv , then for ŵTli ( x ) = W ( x ) 1 W ( x ) 2 . . .W ( x ) l−1w T li and b̂li ( x ) = b ( x ) 1 W ( x ) 2 . . .W ( x ) l−1w T li + b ( x ) 2 W ( x ) 3 . . .W ( x ) l−1w T li + . . .+ bl−1w T li + bli , we have vli ( x ) = xŵ T li ( x ) + b̂li ( x ) . ( 2 ) The proof is provided in Appendix A . Note that the ReLU derivative , σ̇r ( v ) , is just a convenient definition for a step function , so that σ̇r ( v ) w = { w , v > 0 0 , otherwise . ( 3 ) To simplify the notation , whenever referring to the parameters of the switched projection ŵ , b̂ , as well as activity v , we will drop the explicit dependency on x . While Figure 1 illustrates the switching concepts on a small fully connected network , switched linear projections can be computed for networks with convolutional as well as pooling layers . A convolutional layer is just a special case of a fully connected layer with many weights being zero and groups of neurons constrained to share the weight values on their connections . For max pooling , the neurons that do not win the competition , and thus their output does not affect the computation from then on , are deemed to be inactive regardless of the output they produce . 3.1 SENSITIVITY . Equation 2 makes it obvious that a given neuron ’ s switched weight vector is just the derivative of its activity with respect to the network input , ŵ = ∂v ( x ) x . Thus , the switched weight vector is analogous to gradient-based sensitivity analysis . Figure 2 shows the heatmaps of the switched weight sensitivity for the same set of hidden neurons with different inputs from the MNIST-trained 2CONV neural network ( for details on the network architecture featured in this paper see Appendix B ) . In this visualisation we show normalised ŵ , with intensity of red corresponding to the larger positive value , and the intensity of blue the negative value . The neurons were chosen from the 2nd convolutional layer and the penultimate fully connected layer respectively such that for the four 1In our terminology , activity denotes output before the activation function and an active neuron is one that produces non-zero output after the activation function ; for a ReLU neuron the active and inactive neurons are those that have positive and negative activity respectively . pred : 1 b̂ = −0.1 b̂ = −2.9 b̂ = −0.5 b̂ = 7.2 b̂ = −0.1 b̂ = −7.9 b̂ = −18.3 b̂ = 6.7 b̂ = 10.6 b̂ = −9.6 label : 0 pred : 0 v = 0.3 b̂ = −0.1 v = 39.1 b̂ = −4.2 v = 21.5 b̂ = −2.3 v = 28.6 b̂ = −3.0 v = −4.1 b̂ = 0.3 v = 88.5 b̂ = 5.7 v = −122.8 b̂ = 7.6 v = 399.6 b̂ = −39.2 v = 180.8 b̂ = −22.4 v = −14.7 b̂ = 3.1 considered inputs some neurons were always active , some inactive , and others sometimes active and sometimes not . Figure 2 makes it clear that a given neuron is not necessarily sensitive to the same pattern for different network inputs – this is most evident in the sensitivity of the neurons of the fully connected layer . Also note that some neurons in the convolutional layers are active despite the fact that they only “ see ” the part of the input image that is “ empty ” ( all pixels are black ) . This leads us to the conclusion that a given neuron is not necessarily a detector of a particular pattern in the input space , which is often the underlying assumption of the existing interpretability techniques . Something also apparent in our switched projection analysis , though not evident from Figure 2 , is that for a given input most of the neurons in the network are inactive . On average , only 17 % of the neurons were active for a given MNIST input in this architecture . The fact that only a subset of neurons are active in a given computation is not a quirk of one specific network , as observed by ( Hanin & Rolnick , 2019 ) . Switched linear projections give us an interpretation of a deep network as a set of linear , input-dependent , transformations . Something about the input activates a subset of neurons , but also keeps all the remaining neurons inactive . Traditional sensitivity analysis , as well as the one shown in Figure 2 , shows a direction ( or magnitude ) of the gradient of the input that would increase neurons activity provided the same state of the network remained unchanged . However , often in these networks the state does change even after small perturbations of the input , often resulting in same classification , but different input gradient . We reason that the analysis of the state , including information about what makes the neurons inactive , provides a missing and likely very important aspect , to interpretability than analysis of the active subnetwork alone .

Summary: The work basically introduced a new way of looking at interpretability; instead of focusing on the source of activations in the network for a given input image, focus on the source of stability (non-active) neurons (in a ReLU network). The work starts by proving (although it is trivial) that in a ReLU (more generally any piece-wise linear) network, for a given input image, there is a locally linear relationship between a given neuron's activation and the image: v= w^T x + b. As the authors correctly mention, focusing on 'w' as the sensitivity analysis is basically the vanilla gradient method. The contribution, however, is focusing on the projection of bias and the introduced notion of 'centre'. With this provided notion, one can focus on the deactivated neurons in the network and how each input pixel is responsible for it. In other words, unlike previous work that focuses on the activation map, the authors correctly refer to the deactivated neurons as another source of the network's prediction.