Bias Also Matters: Bias Attribution for Deep Neural Network Explanation

The gradient of a deep neural network ( DNN ) w.r.t . the input provides information that can be used to explain the output prediction in terms of the input features and has been widely studied to assist in interpreting DNNs . In a linear model ( i.e. , g ( x ) = wx+ b ) , the gradient corresponds solely to the weights w. Such a model can reasonably locally linearly approximate a smooth nonlinear DNN , and hence the weights of this local model are the gradient . The other part , however , of a local linear model , i.e. , the bias b , is usually overlooked in attribution methods since it is not part of the gradient . In this paper , we observe that since the bias in a DNN also has a non-negligible contribution to the correctness of predictions , it can also play a significant role in understanding DNN behaviors . In particular , we study how to attribute a DNN ’ s bias to its input features . We propose a backpropagation-type algorithm “ bias back-propagation ( BBp ) ” that starts at the output layer and iteratively attributes the bias of each layer to its input nodes as well as combining the resulting bias term of the previous layer . This process stops at the input layer , where summing up the attributions over all the input features exactly recovers b . Together with the backpropagation of the gradient generating w , we can fully recover the locally linear model g ( x ) = wx + b . Hence , the attribution of the DNN outputs to its inputs is decomposed into two parts , the gradient w and the bias attribution , providing separate and complementary explanations . We study several possible attribution methods applied to the bias of each layer in BBp . In experiments , we show that BBp can generate complementary and highly interpretable explanations of DNNs in addition to gradient-based attributions . 1 INTRODUCTION . Deep neural networks ( DNNs ) have produced good results for many challenging problems in computer vision , natural language processing , and speech processing . Deep learning models , however , are usually designed using fairly high-level architectural decisions , leading to a final model that is often seen as a difficult to interpret black box . DNNs are a highly expressive trainable class of non-linear functions , utilizing multi-layer architectures and a rich set of possible hidden non-linearities , making interpretation by a human difficult . This restricts the reliability and usability of DNNs especially in mission-critical applications where a good understanding of the model ’ s behavior is necessary . The gradient is a useful starting point for understanding and generating explanations for the behavior of a complex DNN . Having the same dimension as the input data , the gradient can reflect the contribution to the DNN output of each input dimension . Not only does the gradient yield attribution information for every data point , but also it helps us understand other aspects of DNNs , such as the highly celebrated adversarial examples and defense methods against such attacks ( Szegedy et al. , 2013 ) . When a model is linear , the gradient recovers the weight vector . Since a linear model locally approximates any sufficiently smooth non-linear model , the gradient can also be seen as the weight vector of that local linear model for a given DNN at a given data point . For a piecewise linear DNN ( e.g. , a DNN with activation functions such as ReLU , LeakyReLU , PReLU , and hard tanh ) the gradient is exactly the weights of the local linear model1 . 1This is true except when the gradient is evaluated at an input on the boundary of the polyhedral region within which the DNN equals to the local linear model . In such case , subgradients are appropriate . Although the gradient of a DNN has been shown to be helpful in understanding the behavior of a DNN , the other part of the locally linear model , i.e. , the bias term , to the best of our knowledge , has not been studied explicitly and is often overlooked . If only considering one linear model within a small region , the bias , as a scalar , seems to contain less information than the weight vector . However , this scalar is the result of complicated processing of bias terms over every neuron and every layer based on the activations , the non-linearity functions , as well as the weight matrices of the network . Uncovering the bias ’ s nature could potentially reveal a rich vein of attribution information complementary to the gradient . For classification tasks , it can be the case that the gradient part of the linear model contributes to only a negligible portion of the target label ’ s output probability ( or even a negative logit value ) , and only with a large bias term does the target label ’ s probability becomes larger than that of other labels to result in the correct prediction ( see Sec 5 ) . In our empirical experiments ( Table 1 ) , using only the bias term of the local linear models achieves 30-40 % of the performance of the complete DNN , thus indicating that the bias term indeed plays a substantial role in the mechanisms of a DNN . In this paper , we unveil the information embedded in the bias term by developing a general bias attribution framework that distributes the bias scalar to every dimension of the input data . We propose a backpropagation-type algorithm called “ bias backpropagation ( BBp ) ” to send and compute the bias attribution from the output and higher-layer nodes to lower-layer nodes and eventually to the input features , in a layer-by-layer manner . Specifically , BBp utilizes a recursive rule to assign the bias attribution on each node of layer ` to all the nodes on layer ` − 1 , while the bias attribution on each node of layer ` − 1 is composed of the attribution sent from the layer below and the bias term incurred in layer ` − 1 . The sum of the attributions over all input dimensions produced by BBp exactly recovers the bias term in the local linear model representation of the DNN at the given input point . In experiments , we visualize the bias attribution results as images on a DNN trained for image classification . We show that bias attribution can highlight essential features that are complementary from what the gradient-alone attribution methods favor . 2 RELATED WORK . Attribution methods for deep models is an important modern research in machine learning since it is important to complement the good empirical performance of DNNs with explanations for how , why , and in what manner do such complicated models make their decisions . Ideally , such methods would render DNNs to be glass boxes rather than black boxes . To this end , a number of strategies have been investigated . Simonyan et al . ( 2013 ) visualized behaviors of convolutional networks by investigating the gradients of the predicted class output with respect to the input features . Deconvolution ( Zeiler & Fergus , 2014 ) and guided backpropagation ( Springenberg et al. , 2014 ) modify gradients with additional constraints . Montavon et al . ( 2017 ) extended to higher order gradient information by calculating the Taylor expansion , and Binder et al . ( 2016 ) study the Taylor expansion approach on DNNs with local renormalization layers . Shrikumar et al . ( 2017 ) proposed DeepLift , which separated the positive attribution and negative attribution , and featured customer designed attribution scores . Sundararajan et al . ( 2017 ) declared two axioms an attribution method needs to satisfy . It further developed an integrated gradient method that accumulates gradients on a straightline path from a base input to a real data point and uses the aggregated gradients to measure the importance of input features . Class Activation Mapping ( CAM ) ( Zhou et al. , 2016 ) localizes the attribution based on the activation of convolution filters , and can only be applied to a fully convolutional network . Grad-CAM ( Selvaraju et al. , 2017 ) relaxes the all-convolution constraints of CAM by incorporating the gradient information from the non-convolutional layers . All the work mentioned above utilizes information encoded in the gradients in some form or another , but none of them explicitly investigates the importance of the bias terms , which is the focus of this paper . Some of them , e.g . Shrikumar et al . ( 2017 ) and Sundararajan et al . ( 2017 ) , consider the overall activation of neurons in their attribution methods , so the bias terms are implicitly taken into account , but are not independently studied . Moreover , some of the previous work ( e.g . CAM ) focus on the attribution for specific network architectures such as convolutional networks , while our approach generally applies to any piece-wise linear DNN , convolutional or otherwise . 3 BACKGROUND AND MOTIVATION . We can write the output f ( x ) of any feed-forward deep neural network in the following form : f ( x ) =Wmψm−1 ( Wm−1ψm−2 ( . . . ψ1 ( W1x+ b1 ) . . . ) + bm−1 ) + bm . ( 1 ) Where Wi and bi are the weight matrix and bias term for layer i , ψi is the corresponding activation function , x ∈ X is an input data point of din dimensions , f ( x ) is the network ’ s output prediction of dout dimensions , and each hidden layer i has di nodes . In this paper , we rule out the last softmax layer from the network structure ; for example , the output f ( x ) may refer to logits ( which are the inputs to a softmax to compute probabilities ) if the DNN is trained for classification tasks . The above formalization of DNN generalizes many widely used DNN architectures . Clearly , Eq . ( ( 1 ) ) can represent a fully-connected network of m layers . Moreover , the convolution operation is essentially a matrix multiplication , where every row of the matrix corresponds to applying a filter from convolution on a certain part of the input , and therefore the resulting weight matrix has tied parameters and is very sparse , and typically has a very large ( compared to the input size ) number of rows . The average-pooling is essentially a linear operation and therefore is representable as a matrix multiplication , and max-pooling can be treated as an activation function . Batchnorm ( Ioffe & Szegedy , 2015 ) is a linear operation and can be combined into the weight matrix . Finally , we can represent a residual network ( He et al. , 2015 ) block by appending an identity matrix at the bottom of a weight matrix so that we can keep the input values , and then add the kept input values later through another matrix operation . 3.1 PIECEWISE LINEAR DEEP NEURAL NETWORKS . In this paper , we will focus on DNNs with piecewise linear activation functions , which cover most of the successfully used neural networks in a variety of application domains . Some widely used piecewise linear activation functions include the ReLU , leaky ReLU , PReLU , and the hard tanh functions . A general form of a piecewise linear activation function applied to a real value z is as follows : ψ ( z ) = c ( 0 ) · z , if z ∈ ( η0 , η1 ] c ( 1 ) · z , if z ∈ ( η1 , η2 ] · · · , · · · c ( h−1 ) · z , if z ∈ ( ηh−1 , ηh ) ( 2 ) In the above , there are h linear pieces , and these correspond to h predefined intervals on the real axis . We define the activation pattern φ ( z ) of z as the index of the interval containing z , which can be any integer from 0 to h− 1 . Both ψ ( z ) and φ ( z ) extend to element-wise operators when applied to vectors or high dimensional tensors . As long as the activation function is piecewise linear , the DNN is a piecewise linear function and is equivalent to a linear model at each input point x . Specifically , each piece of the DNN ( associated with an input point x ) is a linear model : f ( x ) =W xmW x m−1 · · ·W1x+ ( W xmW xm−1 · · ·W x2 bx1 + · · ·+W xmW xm−1bxm−2 +W xmbxm−1 + bm ) = m∏ i=1 W xi x+ m∑ j=2 m∏ i=j W xi b x j−1 + bm = ∂f ( x ) ∂x x+ bx . ( 3 ) This holds true for all the possible input points x on the linear piece of DNN . We will give a more general result later in Lemma 1 . Note W xi and b x i in the above linear model are modified from Wi and bi respectively and have to fulfill xi+1 = ψi ( Wixi + bi ) =W x i xi + b x i , ( 4 ) where xi is the activation of layer i ( x1 is the input data ) and bxi is an xi-dependent bias vector vector . In the extreme case , no two input training data points share the same linear model in Eq . ( 3 ) . Note in this case , the DNN can still be represented as a piecewise linear model on a set of input data points , and each local linear model is only applied to one data point . Given xi , W xi and b x i can be derived from Wi and bi according to the activation pattern vector φ ( Wixi + bi ) . In particular , each row of W xi is a scaled version of the associated row of Wi , and each element in bxi is a scaled version of bi , i.e. , W xi [ p ] = c ( φ ( Wixi+bi ) [ p ] ) ·Wi , and bxi [ p ] = c ( φ ( Wixi+bi ) [ p ] ) · bi . ( 5 ) For instance , if ReLU ψReLU ( z ) = max ( 0 , z ) is used as the activation function ψ ( · ) at every layer i , we have an activation pattern φ ( Wixi + bi ) ∈ { 0 , 1 } di , where φ ( Wixi + bi ) [ p ] = 0 indicates that ReLU sets the output node p to 0 or otherwise preserves the node value . Therefore , at layer i , W xi and b x i are modified from Wi and bi by setting the rows of Wi , whose corresponding activation patterns in φ ( Wixi + bi ) are 0 , to be all-zero vectors 0 , and setting the associated elements in bxi to be 0 while other elements to be bi . We can apply the above process to deeper layers as well , eliminating all the ReLU functions to produce an x-specific local linear model representing one piece of the DNN , as shown in Eq . ( 3 ) . Since the model is linear , the gradient ∂f ( x ) ∂x is the weight vector of the linear model . Also , given all the weights of the DNN , each linear region , and the associated linear model can be uniquely determined by the ReLU patterns { φ ( xi ) } mi=2 , which are m binary vectors .

Summary: This paper proposes a method for attributing the output of a neural network to bias terms. The method is restricted to networks that have piecewise-linear activation functions. Computation is recursive, starting from bias attribution to the activation of the penultimate layer, such that the final attribution is of the same size as the input data point and sums up to the bias term when the network is written as a linear function (for that input).