Towards Hierarchical Importance Attribution: Explaining Compositional Semantics for Neural Sequence Models

1 INTRODUCTION . Recent advances in deep neural networks have led to impressive results on a range of natural language processing ( NLP ) tasks , by learning latent , compositional vector representations of text data ( Peters et al. , 2018 ; Devlin et al. , 2018 ; Liu et al. , 2019b ) . However , interpretability of the predictions given by these complex , “ black box ” models has always been a limiting factor for use cases that require explanations of the features involved in modeling ( e.g. , words and phrases ) ( Guidotti et al. , 2018 ; Ribeiro et al. , 2016 ) . Prior efforts on enhancing model interpretability have focused on either constructing models with intrinsically interpretable structures ( Bahdanau et al. , 2015 ; Liu et al. , 2019a ) , or developing post-hoc explanation algorithms which can explain model predictions without elucidating the mechanisms by which model works ( Mohseni et al. , 2018 ; Guidotti et al. , 2018 ) . Among these work , post-hoc explanation has come to the fore as they can operate over a variety of trained models while not affecting predictive performance of models . Towards post-hoc explanation , a major line of work , additive feature attribution methods ( Lundberg & Lee , 2017 ; Ribeiro et al. , 2016 ; Binder et al. , 2016 ; Shrikumar et al. , 2017 ) , explain a model prediction by assigning importance scores to individual input variables . However , these methods are not ideal for explaining phrase-level importance , as phrase importance is often a non-linear combination of the importance of the words in the phrase . Contextual decomposition ( CD ) ( Murdoch et al. , 2018 ) and its hierarchical extension ( Singh et al. , 2019 ) go beyond the additive assumption and compute the contribution solely made by a word/phrase to the model prediction ( i.e. , individual contribution ) , by decomposing the output variables of the neural network at each layer . Using the individual contribu- 1Project page : https : //inklab.usc.edu/hiexpl/ tion scores so derived , these algorithms generate hierarchical explanation on how the model captures compositional semantics ( e.g. , stress or negation ) in making predictions ( see Figure 1 for example ) . However , despite contextual decomposition methods have achieved good results in practice , what reveals extra importance that emerge from combining two phrases has not been well studied . As a result , prior lines of work on contextual decomposition have focused on exploring model-specific decompositions based on their performance on visualizations . We identify the extra importance from combining two phrases can be quantified by studying how the importance of the combined phrase differs from the sum of the importance of the two component phrases on its own . Similar strategies have been studied in game theory for quantifying the surplus from combining two groups of players ( Fujimoto et al. , 2006 ) . Following the definition above , the key challenge is to formulate the importance of a phrase on it own , i.e. , context independent importance of a phrase . However , we show contextual decomposition do not satisfy this context independence property mathematically . To this end , we propose a formal way to quantify the importance of each individual word/phrase , and develop effective algorithms for generating hierarchical explanations based on the new formulation . To mathematically formalize and efficiently approximate context independent importance , we formulate N -context independent importance of a phrase , defined as the difference of model output after masking out the phrase , marginalized over all possible N words surrounding the phrase in the sentence . We propose two explanation algorithms according to our formulation , namely the Sampling and Contextual Decomposition algorithm ( SCD ) , which overcomes the weakness of contextual decomposition algorithms , and the Sampling and OCclusion algorithm ( SOC ) , which is simple , model-agnostic , and performs competitively against prior lines of algorithms . We experiment with both LSTM and fine-tuned Transformer models to evaluate the proposed methods . Quantitative studies involving automatic metrics and human evaluation on sentiment analysis and relation extraction tasks show that our algorithms consistently outperform competitors in the quality of explanations . Our algorithms manage to provide hierarchical visualization of compositional semantics captured by models , extract classification rules from models , and help users to trust neural networks predictions . In summary , our work makes the following contributions : ( 1 ) we identify the key challenges in generating post-hoc hierarchical explanations and propose a mathematically sound way to quantify context independent importance of words and phrases for generating hierarchical explanations ; ( 2 ) we extend previous post-hoc explanation algorithm based on the new formulation of N -context independent importance and develop two effective hierarchical explanation algorithms ; and ( 3 ) experiments demonstrate that the proposed explanation algorithms consistently outperform the compared methods ( with both LSTM and Transformer as base models ) over several datasets and models . 2 PRELIMINARIES . 2.1 POST-HOC EXPLANATIONS OF NEURAL SEQUENCE MODELS . We consider a sequence of low-dimensional word embeddings x1 : T : = ( x1 , x2 , ... , xT ) , or denoted as x for brevity , as the input to a neural sequence model , such as standard RNNs , LSTM ( Hochreiter & Schmidhuber , 1997 ) and Transformers ( Vaswani et al. , 2017 ) . These neural models extract hidden representations h1 : T from the input sequence x , and feed these hidden representations to a prediction layer to generate prediction scores in the label space ( e.g. , sentiment polarity of a sentence ) . For LSTM , we use the last hidden state hT to give unnormalized prediction scores s ( x ) ∈ Rdc over dc label classes as follows . s ( x ) =WlhT , ( 1 ) whereWl ∈ Rdc×dh is a trainable weight matrix . For Transformers , the representation corresponding to the [ CLS ] token at the final layer is fed to the prediction layer to generate scores s ( x ) . Towards post-hoc explanation of s ( x ) , a notable line of work , additive feature attribution methods ( Ribeiro et al. , 2016 ; Shrikumar et al. , 2017 ; Sundararajan et al. , 2017 ; Lundberg & Lee , 2017 ) , measure word-level importance to the model prediction s ( x ) by attributing an importance score φ ( xi , x ) to each word in the input sequence xi ∈ x . However , the additive assumption hinders these methods from explaining the complex interactions between words and compositional semantics in a sentence ( e.g. , modeling negation , transition , and emphasis in sentiment classification ) , as shown in Figure 1 . 2.2 HIERARCHICAL EXPLANATIONS VIA CONTEXTUAL DECOMPOSITION . To caputure non-linear compositional semantics , the line of work on contextual decomposition ( CD ) ( Murdoch et al. , 2018 ) designs non-additive measures of importance from individual words/phrases to the model predictions , and further extend to agglomerative contextual decomposition ( ACD ) algorithm ( Singh et al. , 2019 ) for generating hierarchical explanations . Given a phrase p = xi : j in the input sequence x , contextual decomposition ( CD ) attributes a score φ ( p , x ) as the contribution solely from p to the model ’ s prediction s ( x ) . Note that φ ( p , x ) does not equal to the sum of the scores of each word in the phrase , i.e. , φ ( p , x ) 6= ∑ xi∈p φ ( xi , x ) . Starting from the input layer , CD iteratively decomposes each hidden state h of the model into the contribution solely made by p , denoted as β , and the contributions involving the words outside the phrase p , denoted as γ , with the relation h = β+γ holds . The algorithm also keeps the contribution from the bias term , denoted as ζ , temporally before element-wise multiplication . For a linear layer h =Wixt+ bi with input xt , when xt lies in the given phrase p , the contribution solely from xt to h is defined as β = Wixt when xt is part of the phrase ( i.e. , xt ∈ p ) , and the contribution involving other words in the sentences ( denoted as x\p ) is defined as γ = 0 . The contribution of the bias term ζ is thus bi . When xt lies outside of the phrase , γ is quantified as Wixt and β is 0 . In the cases when CD encounters element-wise multiplication operations h = ha hb ( e.g. , in LSTMs ) , it eliminates the multiplicative interaction terms which involve the information outside of the phrase p. Specifically , suppose that ha and hb have been decomposed as ha = βa + γa + ζa and hb = βb + γb + ζb , CD computes the β term for ha hb as β = βa βb + βa ζb + ζa βb . When dealing with non-linear activation h′ = σ ( h ) , CD computes the contribution solely from the phrase p as the average activation differences caused β supposing γ is present or absent , β′ = 1 2 [ σ ( β + γ + ζ ) − σ ( γ + ζ ) ] + 1 2 [ σ ( β + ζ ) − σ ( ζ ) ] . ( 2 ) Following the strategies introduced above , CD decomposes all the intermediate outputs starting from the input layer , until reaching the final output of the model hT = β + γ . The logit score Wlβ is treated as the contribution of the given phrase p to the final prediction s ( x ) . As a follow-up study , Singh et al . ( 2019 ) extends CD algorithm to other families of neural network architectures , and proposes agglomerative contextual decomposition algorithm ( ACD ) . The decomposition of activation functions is modified as β′ = σ ( β ) . For a linear layer h′ = Wh + b with its decomposition h = β + γ , the bias term b is decomposed proportionally and merged into the β′ term of h′ , based on β′ =Wβ + |Wβ|/ ( |Wβ|+ |Wγ| ) · b . 3 METHODOLOGY . In this section , we start by identifying desired properties of phrase-level importance score attribution for hierarchical explanations . We propose a measure of context-independent importance of a phrase and introduce two explanation algorithms instantiated from the proposed formulation . 3.1 PROPERTIES OF IMPORTANCE ATTRIBUTION FOR HIERARCHICAL EXPLANATION . Despite the empirical success of CD and ACD , no prior works analyze what common properties a score attribution mechanism should satisfy to generate hierarchical explanations that reveal compositional semantics formed between phrases . Here we identify two properties that an attribution method should satisfy to generate informative hierarchical explanations . Non-additivity . Importance of a phrase φ ( p , x ) should not be a sum over the importance scores of all the component words xi ∈ p , i.e. , φ ( p , x ) 6= ∑ xi∈p φ ( xi , x ) , in contrast to the family of additive feature attribution methods . The property is also suggested by Murdoch et al . ( 2018 ) . Context Independence . For deep neural networks , when two phrases combine , their importance to predicting a class may greatly change . The surplus by combining two phrases can be quantified by the difference between the importance of the combined phrase and the sum of the importance of two phrases on its own . It follows the definition of marginal interactions ( Fujimoto et al. , 2006 ) in game theory . According to the definition , the importance of two component phrases should be at least evaluated independently to each other . Formally , if we are only interested in how combining two phrases p1 and p2 contribute to a specific prediction for an input x , we expect for input sentences x̃ where only p2 is replaced to another phrase , the importance attribution for p1 remains the same , i.e. , φ ( p1 , x ) = φ ( p1 , x̃ ) . In our bottom-up hierarchical explanation setting , we are interested in how combining a phrase and any other contextual words or phrases in the input x contribute . Therefore , we expect φ ( p , x ) = φ ( p , x̃ ) given the phrase p in two different contexts x and x̃ . Limitations of CD and ACD . Unfortunately , while CD and ACD try to construct decomposition so that β terms represent the contributions solely from a given a phrase , the assigned importance scores by these algorithms do not satisfy the context independence property mathematically . For CD , we see the computation of β involves the γ term of a specific input sentence in Eq . 2 ( see Figure 2 ( a ) for visualization ) . Similarly , for ACD , the decomposition of the bias term involves the γ terms of a specific input sentence . As a result , the β terms computed by both algorithms depend on the context of the phrase p. Given the limitation of prior works , we start by formulating a importance measure of phrases that satisfies both non-additivity and context independence property .

Summary: The paper addresses the problem of hierarchical explanations in deep models that handle compositional semantics of words and phrases. The paper first highlights desirable properties for importance attribution scores in hierarchical explanations, specifically, non-additivity and context independence, and shows how prior work on additive feature attribution and context decomposition doesn’t accurately capture these notions. After highlighting the said properties in context of related work, the authors propose an approach to calculate the context-independent importance of a phrase by computing the difference in scores with and without masking out the phrase marginalized over all possible surrounding word contexts (approximated by sampling surrounding context for a fixed radius under a language model). Furthermore, based on the above, the authors propose two more score attribution approaches -- based on integrating the above sampling step with (1) the contextual decomposition pipeline and (2) the input occlusion pipeline. Experimentally, the authors find that the attribution scores assigned by the proposed approach are more correlated with human annotations compared to prior approaches and additionally, the generated explanations turn out to be more trustworthy when humans evaluate their quality.