Attention Interpretability Across NLP Tasks

1 INTRODUCTION . Attention is a way of obtaining a weighted sum of the vector representations of a layer in a neural network model ( Bahdanau et al. , 2015 ) . It is used in diverse tasks ranging from machine translation ( Luong et al. , 2015 ) , language modeling ( Liu & Lapata , 2018 ) to image captioning ( Xu et al. , 2015 ) , and object recognition ( Ba et al. , 2014 ) . Apart from substantial performance benefit ( Vaswani et al. , 2017 ) , attention also provides interpretability to neural models ( Wang et al. , 2016 ; Lin et al. , 2017 ; Ghaeini et al. , 2018 ) which are usually criticized for being black-box function approximators ( Chakraborty et al. , 2017 ) . There has been substantial work on understanding attention in neural network models . On the one hand , there is work on showing that attention weights are not interpretable , and altering them does not significantly affect the prediction ( Jain & Wallace , 2019 ; Serrano & Smith , 2019 ) . While on the other hand , some studies have discovered how attention in neural models captures several linguistic notions of syntax and coreference ( Vig & Belinkov , 2019 ; Clark et al. , 2019 ; Tenney et al. , 2019 ) . Amid such contrasting views arises a need to understand the attention mechanism more systematically . In this paper , we attempt to fill this gap by giving a comprehensive explanation which justifies both kinds of observations . The conclusions of Jain & Wallace ( 2019 ) ; Serrano & Smith ( 2019 ) have been mostly based on text classification experiments which might not generalize to several other NLP tasks . In Figure 1 , we report the performance on text classification , Natural Language Inference ( NLI ) and Neural Machine Translation ( NMT ) of two models : one trained with neural attention and the other trained with attention weights fixed to a uniform distribution . The results show that the attention mechanism in text classification does not have an impact on the performance , thus , making inferences about interpretability of attention in these models might not be accurate . However , on tasks such as NLI and NMT uniform attention weights degrades the performance substantially , indicating that attention is a crucial component of the model for these tasks and hence the analysis of attention ’ s interpretability here is more reasonable . In comparison to the existing work on interpretability , we analyze attention mechanism on a more diverse set of NLP tasks that include text classification , pairwise text classification ( such as NLI ) , and text generation tasks like neural machine translation ( NMT ) . Moreover , we do not restrict ourselves to a single attention mechanism and also explore models with self-attention . For examining the interpretability of attention weights , we perform manual evaluation . Our key contributions are : 1 . We extend the analysis of attention mechanism in prior work to diverse NLP tasks and provide a comprehensive picture which alleviates seemingly contradicting observations . 2 . We identify the conditions when attention weights are interpretable and correlate with feature importance measures – when they are computed using two vectors which are both functions of the input ( Figure 1b , c ) . We also explain why attention weights are not interpretable when the input has only single sequence ( Figure 1a ) , an observation made by Jain & Wallace ( 2019 ) , by showing that they can be viewed as a gating unit . 3 . We validate our hypothesis of interpretability of attention through manual evaluation . 2 TASKS AND DATASETS . We investigate the attention mechanism on the following three task categories . 1 . Single Sequence tasks are those where the input consists of a single text sequence . For instance , in sentiment analysis , the task is to classify a review as positive or negative . This also includes other text classification tasks such as topic categorization . For the experiments , in this paper , we use three review rating datasets : ( 1 ) Stanford Sentiment Treebank ( Socher et al. , 2013 ) , ( 2 ) IMDB ( Maas et al. , 2011 ) and ( 3 ) Yelp 20171 and one topic categorization dataset AG News Corpus ( business vs world ) .2 2 . Pair Sequence tasks comprise of a pair of text sequences as input . The tasks like NLI and question answering come under this category . NLI involves determining whether a hypothesis entails , contradicts , or is undetermined given a premise . We use Stanford Natural Language Inference ( SNLI ) ( Bowman et al. , 2015 ) and Multi-Genre Natural Language Inference ( MultiNLI ) ( Williams et al. , 2018 ) datasets for our analysis . For question answering , similar to Jain & Wallace ( 2019 ) , we use CNN News Articles ( Hermann et al. , 2015 ) and three tasks of the original babI dataset ( Weston et al. , 2015 ) in our experiments , i.e. , using one , two and three supporting statements as the context for answering the questions . 3 . Generation tasks involve generating a sequence based on the input sequence . Neural Machine translation is an instance of generation task which comprises of translating a source text to a target language given translation pairs from a parallel corpus . For our experiments , we use 1from www.yelp.com/dataset challenge 2www.di.unipi.it/ gulli/AG corpus of news articles.html three English-German datasets : Multi30k ( Elliott et al. , 2016 ) , En-De News Commentary v11 from WMT16 translation task3 and full En-De WMT13 dataset . 3 NEURAL ATTENTION MODELS . In this section , we give a brief overview of the neural attention-based models we analyze for different categories of tasks listed in Section 2 . The overall architecture for each category is shown in Fig 1 . 3.1 SINGLE SEQUENCE MODELS : . For single sequence tasks , we adopt the model architecture from Jain & Wallace ( 2019 ) ; Wiegreffe & Pinter ( 2019 ) . For a given input sequence x ∈ RT×|V | , where T and |V | are the number of tokens and vocabulary size , we first represent each token with its d-dimensional GloVe embedding Pennington et al . ( 2014 ) to obtain xe ∈ RT×d . Next , we use a Bi-RNN encoder ( Enc ) to obtain an m-dimensional contextualized representation of tokens : h = Enc ( xe ) ∈ RT×m . Then , we use the additive formulation of attention proposed by Bahdanau et al . ( 2015 ) for computing attention weights αi for all tokens defined as : ui = tanh ( Whi + b ) ; αi = exp ( uTi c ) ∑ j exp ( u T j c ) , ( 1 ) where W ∈ Rd′×m , b , c ∈ Rd′ are the parameters of the model . Finally , the weighted instance representation : hα = ∑T i=1 αihi ∈ Rm is fed to a dense layer ( Dec ) followed by softmax to obtain prediction ŷ = σ ( Dec ( hα ) ) ∈ R|Y| , where |Y| denotes the label set size . We also analyze the hierarchical attention model ( Yang et al. , 2016 ) , which involves first computing attention over the tokens to obtain a sentence representation . This is followed by attention over sentences to obtain an instance representation hα , which is fed to a dense layer for obtaining prediction ( ŷ ) . At both word and sentence level the attention is computed similar to as defined in Equation 1 . 3.2 PAIR SEQUENCE MODELS : . For pair sequence , the input consists of two text sequences : x ∈ RT1×|V | , y ∈ RT2×|V | of length T1 and T2 . In NLI , x indicates premise and y is hypothesis while in question answering , it is the question and paragraph respectively . Following Bowman et al . ( 2015 ) , we use two separate RNNs for encoding both the sequences to obtain { hx1 , ... , hxT1 } and { h y 1 , ... , h y T2 } . Now , similar to Jain & Wallace ( 2019 ) , attention weight αi over each token of x is computed as : ui = tanh ( W1h x i +W2h y T2 ) ; αi = exp ( uTi c ) ∑ j exp ( u T j c ) , ( 2 ) where similar to Equation 1 , W1 , W2 ∈ Rd×d ′ denotes the projection matrices and c ∈ Rd′ is a parameter vector . Finally , the representation obtained from a weighted sum of tokens in x : hα =∑T i=1 αih x i is fed to a classifier for prediction . We also explore a variant of the above attention proposed by Rocktäschel et al . ( 2016 ) . Instead of keeping the RNN encoders of both the sequences independent , Rocktäschel et al . ( 2016 ) use conditional encoding where the encoder of y is initialized with the final state of x ’ s encoder . This allows the model to obtain a conditional encoding { h′y1 , ... , h ′y T2 } of y given the sequence x . Moreover , unlike the previous model , attention over the tokens of x is defined as follows : M = tanh ( W1X +W2h ′y T2 ⊗ eT1 ) ; α = softmax ( wTM ) , ( 3 ) where X = [ hx1 , ... , h x T1 ] , eT1 ∈ RT1 is a vector of ones and outer product W2h ′y T2 ⊗ eT1 denotes repeating linearly transformed h′yT2 as many times as words in the sequence x ( i.e . T1 times ) . 3http : //www.statmt.org/wmt16/translation-task.html 3.3 GENERATION TASK MODELS : . In this paper , for generation tasks , we focus on Neural Machine Translation ( NMT ) problem which involves translating a given source text sentence x ∈ RT1×|V1| to a sequence y ∈ RT2×|V2| in the target language . The model comprises of two components : ( a ) an encoder which computes a representation for each source sentence and ( b ) a decoder which generates a target word at each time step . In this work , we utilize RNN based encoder and decoder models . For each input sentence x , we first obtain a contextualized representation { h1 , ... , hT1 } of its tokens using a multi-layer Bi-RNN . Then , at each time step t , the decoder has a hidden state defined as ct = f ( ct−1 , yt−1 , h t α ) , where h t α = T1∑ i=1 αt , ihi . In our work , we compute αt , i as proposed by Bahdanau et al . ( 2015 ) and Luong et al . ( 2015 ) . The former computes attention weights using a feed-forward network , i.e. , αt , i = wT tanh ( W [ ct ; hi ] ) while the latter define it simply as αt , i = cTt hi . 3.4 SELF-ATTENTION BASED MODELS : . We also examine self-attention based models on all three categories of tasks . For single and pair sequence tasks , we fine-tune pre-trained BERT ( Devlin et al. , 2019 ) model on the downstream task . In pair sequence tasks , instead of independently encoding each text , we concatenate both separated by a delimiter and pass it to BERT model . Finally , the embedding corresponding to [ CLS ] token is fed to a feed-forward network for prediction . For neural machine translation , we use Transformer model proposed by Vaswani et al . ( 2017 ) with base configuration .

Summary: This paper investigates the degree to which we might view attention weights as explanatory across NLP tasks and architectures. Notably, the authors distinguish between single and "pair" sequence tasks, the latter including NLI, and generation tasks (e.g., translation). The argument here is that attention weights do not provide explanatory power for single sequence tasks like classification, but do for NLI and generation. Another notable distinction from most (although not all; see the references below) prior work on the explainability of attention mechanisms in NLP is the inclusion of transformer/self-attentive architectures.