Hierarchical interpretations for neural network predictions

1 INTRODUCTION . Deep neural networks ( DNNs ) have recently demonstrated impressive predictive performance due to their ability to learn complex , non-linear , relationships between variables . However , the inability to effectively visualize these relationships has led DNNs to be characterized as black boxes . Consequently , their use has been limited in fields such as medicine ( e.g . medical image classification ( Litjens et al. , 2017 ) ) , policy-making ( e.g . classification aiding public policy makers ( Brennan & Oliver , 2013 ) ) , and science ( e.g . interpreting the contribution of a stimulus to a biological measurement ( Angermueller et al. , 2016 ) ) . Moreover , the use of black-box models like DNNs in industrial settings has come under increasing scrutiny as they struggle with issues such as fairness ( Dwork et al. , 2012 ) and regulatory pressure ( Goodman & Flaxman , 2016 ) . To ameliorate these problems , we introduce the use of hierarchical interpretations to explain DNN predictions . Our proposed method , agglomerative contextual decomposition ( ACD ) 1 , is a general technique that can be applied to a wide range of DNN architectures and data types . Given a prediction from a trained DNN , ACD produces a hierarchical clustering of the input features , along with the contribution of each cluster to the final prediction . This hierarchy is optimized to identify clusters of features that the DNN learned are predictive ( see Fig 1 ) . ∗Equal contribution , order determined by coin flip 1Code and scripts for running ACD and experiments available at https : //github.com/csinva/ acd The development of ACD consists of two novel contributions . First , importance scores for groups of features are obtained by generalizing contextual decomposition ( CD ) , a previous method for obtaining importance scores for LSTMs ( Murdoch et al. , 2018 ) . This work extends CD to arbitrary DNN architectures , including convolutional neural networks ( CNNs ) . Second , most importantly , we introduce the idea of hierarchical saliency , where a group-level importance measure , in this case CD , is used as a joining metric in an agglomerative clustering procedure . While we focus on DNNs and use CD as our importance measure , this concept is general , and could be readily applied to any model with a suitable measure for computing importances of groups of variables . We demonstrate the utility of ACD on both long short term memory networks ( LSTMs ) ( Hochreiter & Schmidhuber , 1997 ) trained on the Stanford Sentiment Treebank ( SST ) ( Socher et al. , 2013 ) and CNNs trained on MNIST ( LeCun , 1998 ) and ImageNet ( Russakovsky et al. , 2015 ) . Through human experiments , we show that ACD produces intuitive visualizations that enable users to better reason about and trust DNNs . In particular , given two DNN models , we show that users can use the output of ACD to select the model with higher predictive accuracy , and that overall they rank ACD as more trustworthy than prior interpretation methods . In addition , we demonstrate that ACD ’ s hierarchy is robust to adversarial perturbations ( Szegedy et al. , 2013 ) in CNNs . 2 BACKGROUND . Interpreting DNNs is a growing field ( Murdoch et al. , 2019 ) spanning a range of techniques including feature visualization ( Olah et al. , 2017 ; Yosinski et al. , 2015 ) , analyzing learned weights ( Tsang et al. , 2017 ) and others ( Frosst & Hinton , 2017 ; Andreas et al. , 2016 ; Zhang et al. , 2017 ) . Our work focuses on local interpretations , where the task is to interpret individual predictions made by a DNN . Local interpretation Most prior work has focused on assigning importance to individual features , such as pixels in an image or words in a document . There are several methods that give feature-level importance for different architectures . They can be categorized as gradient-based ( Springenberg et al. , 2014 ; Sundararajan et al. , 2017 ; Selvaraju et al. , 2016 ; Baehrens et al. , 2010 ) , decompositionbased ( Murdoch & Szlam , 2017 ; Shrikumar et al. , 2016 ; Bach et al. , 2015 ) and others ( Dabkowski & Gal , 2017 ; Fong & Vedaldi , 2017 ; Ribeiro et al. , 2016 ; Zintgraf et al. , 2017 ) , with many similarities among the methods ( Ancona et al. , 2018 ; Lundberg & Lee , 2017 ) . By contrast , there are relatively few methods that can extract the interactions between features that a DNN has learned . In the case of LSTMs , Murdoch et al . ( 2018 ) demonstrated the limitations of prior work on interpretation using word-level scores , and introduced contextual decomposition ( CD ) , an algorithm for producing phrase-level importance scores from LSTMs . Another simple baseline is occlusion , where a group of features is set to some reference value , such as zero , and the importance of the group is defined to be the resulting decrease in the prediction value ( Zeiler & Fergus , 2014 ; Li et al. , 2016 ) . Given an importance score for groups of features , no existing work addresses how to search through the many possible groups of variables in order to find a small set to show to users . To address this problem , this work introduces hierarchical interpretations as a principled way to search for and display important groups . Hierarchical importance Results from psychology and philosophy suggest that people prefer explanations that are simple but informative ( Harman , 1965 ; Read & Marcus-Newhall , 1993 ) and include the appropriate amount of detail ( Keil , 2006 ) . However , there is no existing work that is both powerful enough to capture interactions between features , and simple enough to not require a user to manually search through the large number of available feature groups . To remedy this , we propose a hierarchical clustering procedure to identify and visualize , out of the considerable number of feature groups , which ones contain meaningful interactions and should be displayed to the end user . In doing so , ACD aims to be informative enough to capture meaningful feature interactions while displaying a sufficiently small subset of all feature groups to maintain simplicity . 3 METHOD . This section introduces ACD through two contributions : Sec 3.1 proposes a generalization of CD from LSTMs to arbitrary DNNs , and Sec 3.2 explains the main contribution : how to combine these CD scores with hierarchical clustering to produce ACD . 3.1 CONTEXTUAL DECOMPOSITION ( CD ) IMPORTANCE SCORES FOR GENERAL DNNS . In order to generalize CD to a wider range of DNNs , we first reformulate the original CD algorithm into a more generic setting than originally presented . For a given DNN f ( x ) , we can represent its output as a SoftMax operation applied to logits g ( x ) . These logits , in turn , are the composition of L layers gi , such as convolutional operations or ReLU non-linearities . f ( x ) = SoftMax ( g ( x ) ) = SoftMax ( gL ( gL−1 ( ... ( g2 ( g1 ( x ) ) ) ) ) ) ( 1 ) Given a group of features { xj } j∈S , our generalized CD algorithm , gCD ( x ) , decomposes the logits g ( x ) into a sum of two terms , β ( x ) and γ ( x ) . β ( x ) is the importance measure of the feature group { xj } j∈S , and γ ( x ) captures contributions to g ( x ) not included in β ( x ) . gCD ( x ) = ( β ( x ) , γ ( x ) ) ( 2 ) β ( x ) + γ ( x ) = g ( x ) ( 3 ) To compute the CD decomposition for g ( x ) , we define layer-wise CD decompositions gCDi ( x ) = ( βi , γi ) for each layer gi ( x ) . Here , βi corresponds to the importance measure of { xj } j∈S to layer i , and γi corresponds to the contribution of the rest of the input to layer i . To maintain the decomposition we require βi + γi = gi ( x ) for each i . We then compute CD scores for the full network by composing these decompositions . gCD ( x ) = gCDL ( g CD L−1 ( ... ( g CD 2 ( g CD 1 ( x ) ) ) ) ) ( 4 ) Previous work ( Murdoch et al. , 2018 ) introduced decompositions gCDi for layers used in LSTMs . The generalized CD described here extends CD to other widely used DNNs , by introducing layerwise CD decompositions for convolutional , max-pooling , ReLU non-linearity and dropout layers . Doing so generalizes CD scores from LSTMs to a wide range of neural architectures , including CNNs with residual and recurrent architectures . At first , these decompositions were chosen through an extension of the CD rules detailed in Murdoch et al . ( 2018 ) , yielding a similar algorithm to that developed concurrently by Godin et al . ( 2018 ) . However , we found that this algorithm did not perform well on deeper , ImageNet CNNs . We subsequently modified our CD algorithm by partitioning the biases in the convolutional layers between γi and βi in Equation 5 , and modifying the decomposition used for ReLUs in Equation 10 . We show the effects of these two changes in Supplement S7 , and give additional intuition in Supplement S1 . When gi is a convolutional or fully connected layer , the layer operation consists of a weight matrix W and a bias b . The weight matrix can be multiplied with βi−1 and γi−1 individually , but the bias must be partitioned between the two . We partition the bias proportionally based on the absolute value of the layer activations . For the convolutional layer , this equation yields only one activation of the output ; it must be repeated for each activation . βi =Wβi−1 + |Wβi−1| |Wβi−1|+ |Wγi−1| · b ( 5 ) γi =Wγi−1 + |Wγi−1| |Wβi−1|+ |Wγi−1| · b ( 6 ) When gi is a max-pooling layer , we identify the indices , or channels , selected by max-pool when run by gi ( x ) , denotedmax idxs below , and use the decompositions for the corresponding channels . max idxs = argmax idxs [ maxpool ( βi−1 + γi−1 ; idxs ) ] ( 7 ) βi = βi−1 [ max idxs ] ( 8 ) γi = γi−1 [ max idxs ] ( 9 ) Finally , for the ReLU , we update our importance score βi by computing the activation of βi−1 alone and then update γi by subtracting this from the total activation . βi = ReLU ( βi−1 ) ( 10 ) γi = ReLU ( βi−1 + γi−1 ) − ReLU ( βi−1 ) ( 11 ) For a dropout layer , we simply apply dropout to βi−1 and γi−1 individually , or multiplying each by a scalar . Computationally , a CD call is comparable to a forward pass through the network f . 3.2 AGGLOMERATIVE CONTEXTUAL DECOMPOSITION ( ACD ) . Given the generalized CD scores introduced above , we now introduce the clustering procedure used to produce ACD interpretations . At a high-level , our method is equivalent to agglomerative hierarchical clustering , where the CD interaction is used as the joining metric to determine which clusters to join at each step . This procedure builds the hierarchy by starting with individual features and iteratively combining them based on the interaction scores provided by CD . The displayed ACD interpretation is the hierarchy , along with the CD importance score at each node . More precisely , algorithm 1 describes the exact steps in the clustering procedure . After initializing by computing the CD scores of each feature individually , the algorithm iteratively selects all groups of features within k % of the highest-scoring group ( where k is a hyperparameter , fixed at 95 for images and 90 for text ) and adds them to the hierarchy . Each time a new group is added to the hierarchy , a corresponding set of candidate groups is generated by adding individual contiguous features to the original group . For text , the candidate groups correspond to adding one adjacent word onto the current phrase , and for images adding any adjacent pixel onto the current image patch . Candidate groups are ranked according to the CD interaction score , which is the difference between the score of the candidate and original groups . ACD terminates after an application-specific criterion is met . For sentiment classification , we stop once all words are selected . For images , we stop after some predefined number of iterations and then merge the remaining groups one by one using the same selection criteria described above . Algorithm 1 is not specific to DNNs ; it requires only a method to obtain importance scores for groups of input features . Here , we use CD scores to arrive at the ACD algorithm , which makes the method specific to DNNs , but given a feature group scoring function , Algorithm 1 can yield interpretations for any predictive model . CD is a natural score to use for DNNs as it aggregates saliency at different scales and converges to the final prediction once all the units have been selected . Algorithm 1 Agglomeration algorithm . ACD ( Example x , model , hyperparameter k , function CD ( x , blob ; model ) ) # initialize tree = Tree ( ) # tree to output scoresQueue = PriorityQueue ( ) # scores , sorted by importance for feature in x : scoresQueue.push ( feature , priority=CD ( x , feature ; model ) ) # iteratively build up tree while scoresQueue is not empty : selectedGroups = scoresQueue.popTopKPercentile ( k ) # pop off top k elements tree.add ( selectedGroups ) # Add top k elements to the tree # generate new groups of features based on current groups and add them to the queue for selectedGroup in selectedGroups : candidateGroups = getCandidateGroups ( selectedGroup ) for candidateGroup in candidateGroups : scoresQueue.add ( candidateGroup , priority=CD ( x , candidateGroup ; model ) -CD ( x , selectedGroup ; model ) ) return tree

Summary: This paper proposes a novel approach to explain neural network predictions by learning hierarchical representations of groups of input features and their contribution to the final prediction. The proposed method is a straightforward extension of the contextual decomposition work by (Murdoch et. al. 2018) which estimates feature interpretability for LSTMs. This work extends (Murdoch et. al. '18) to more general NN architectures and further employs agglomerative clustering to identify groups of features-- as opposed to individual features--that are predictive of the output.