Attention Interpretability Across NLP Tasks

1 INTRODUCTION . Attention is a way of obtaining a weighted sum of the vector representations of a layer in a neural network model ( Bahdanau et al. , 2015 ) . It is used in diverse tasks ranging from machine translation ( Luong et al. , 2015 ) , language modeling ( Liu & Lapata , 2018 ) to image captioning ( Xu et al. , 2015 ) , and object recognition ( Ba et al. , 2014 ) . Apart from substantial performance benefit ( Vaswani et al. , 2017 ) , attention also provides interpretability to neural models ( Wang et al. , 2016 ; Lin et al. , 2017 ; Ghaeini et al. , 2018 ) which are usually criticized for being black-box function approximators ( Chakraborty et al. , 2017 ) . There has been substantial work on understanding attention in neural network models . On the one hand , there is work on showing that attention weights are not interpretable , and altering them does not significantly affect the prediction ( Jain & Wallace , 2019 ; Serrano & Smith , 2019 ) . While on the other hand , some studies have discovered how attention in neural models captures several linguistic notions of syntax and coreference ( Vig & Belinkov , 2019 ; Clark et al. , 2019 ; Tenney et al. , 2019 ) . Amid such contrasting views arises a need to understand the attention mechanism more systematically . In this paper , we attempt to fill this gap by giving a comprehensive explanation which justifies both kinds of observations . The conclusions of Jain & Wallace ( 2019 ) ; Serrano & Smith ( 2019 ) have been mostly based on text classification experiments which might not generalize to several other NLP tasks . In Figure 1 , we report the performance on text classification , Natural Language Inference ( NLI ) and Neural Machine Translation ( NMT ) of two models : one trained with neural attention and the other trained with attention weights fixed to a uniform distribution . The results show that the attention mechanism in text classification does not have an impact on the performance , thus , making inferences about interpretability of attention in these models might not be accurate . However , on tasks such as NLI and NMT uniform attention weights degrades the performance substantially , indicating that attention is a crucial component of the model for these tasks and hence the analysis of attention ’ s interpretability here is more reasonable . In comparison to the existing work on interpretability , we analyze attention mechanism on a more diverse set of NLP tasks that include text classification , pairwise text classification ( such as NLI ) , and text generation tasks like neural machine translation ( NMT ) . Moreover , we do not restrict ourselves to a single attention mechanism and also explore models with self-attention . For examining the interpretability of attention weights , we perform manual evaluation . Our key contributions are : 1 . We extend the analysis of attention mechanism in prior work to diverse NLP tasks and provide a comprehensive picture which alleviates seemingly contradicting observations . 2 . We identify the conditions when attention weights are interpretable and correlate with feature importance measures – when they are computed using two vectors which are both functions of the input ( Figure 1b , c ) . We also explain why attention weights are not interpretable when the input has only single sequence ( Figure 1a ) , an observation made by Jain & Wallace ( 2019 ) , by showing that they can be viewed as a gating unit . 3 . We validate our hypothesis of interpretability of attention through manual evaluation . 2 TASKS AND DATASETS . We investigate the attention mechanism on the following three task categories . 1 . Single Sequence tasks are those where the input consists of a single text sequence . For instance , in sentiment analysis , the task is to classify a review as positive or negative . This also includes other text classification tasks such as topic categorization . For the experiments , in this paper , we use three review rating datasets : ( 1 ) Stanford Sentiment Treebank ( Socher et al. , 2013 ) , ( 2 ) IMDB ( Maas et al. , 2011 ) and ( 3 ) Yelp 20171 and one topic categorization dataset AG News Corpus ( business vs world ) .2 2 . Pair Sequence tasks comprise of a pair of text sequences as input . The tasks like NLI and question answering come under this category . NLI involves determining whether a hypothesis entails , contradicts , or is undetermined given a premise . We use Stanford Natural Language Inference ( SNLI ) ( Bowman et al. , 2015 ) and Multi-Genre Natural Language Inference ( MultiNLI ) ( Williams et al. , 2018 ) datasets for our analysis . For question answering , similar to Jain & Wallace ( 2019 ) , we use CNN News Articles ( Hermann et al. , 2015 ) and three tasks of the original babI dataset ( Weston et al. , 2015 ) in our experiments , i.e. , using one , two and three supporting statements as the context for answering the questions . 3 . Generation tasks involve generating a sequence based on the input sequence . Neural Machine translation is an instance of generation task which comprises of translating a source text to a target language given translation pairs from a parallel corpus . For our experiments , we use 1from www.yelp.com/dataset challenge 2www.di.unipi.it/ gulli/AG corpus of news articles.html three English-German datasets : Multi30k ( Elliott et al. , 2016 ) , En-De News Commentary v11 from WMT16 translation task3 and full En-De WMT13 dataset . 3 NEURAL ATTENTION MODELS . In this section , we give a brief overview of the neural attention-based models we analyze for different categories of tasks listed in Section 2 . The overall architecture for each category is shown in Fig 1 . 3.1 SINGLE SEQUENCE MODELS : . For single sequence tasks , we adopt the model architecture from Jain & Wallace ( 2019 ) ; Wiegreffe & Pinter ( 2019 ) . For a given input sequence x ∈ RT×|V | , where T and |V | are the number of tokens and vocabulary size , we first represent each token with its d-dimensional GloVe embedding Pennington et al . ( 2014 ) to obtain xe ∈ RT×d . Next , we use a Bi-RNN encoder ( Enc ) to obtain an m-dimensional contextualized representation of tokens : h = Enc ( xe ) ∈ RT×m . Then , we use the additive formulation of attention proposed by Bahdanau et al . ( 2015 ) for computing attention weights αi for all tokens defined as : ui = tanh ( Whi + b ) ; αi = exp ( uTi c ) ∑ j exp ( u T j c ) , ( 1 ) where W ∈ Rd′×m , b , c ∈ Rd′ are the parameters of the model . Finally , the weighted instance representation : hα = ∑T i=1 αihi ∈ Rm is fed to a dense layer ( Dec ) followed by softmax to obtain prediction ŷ = σ ( Dec ( hα ) ) ∈ R|Y| , where |Y| denotes the label set size . We also analyze the hierarchical attention model ( Yang et al. , 2016 ) , which involves first computing attention over the tokens to obtain a sentence representation . This is followed by attention over sentences to obtain an instance representation hα , which is fed to a dense layer for obtaining prediction ( ŷ ) . At both word and sentence level the attention is computed similar to as defined in Equation 1 . 3.2 PAIR SEQUENCE MODELS : . For pair sequence , the input consists of two text sequences : x ∈ RT1×|V | , y ∈ RT2×|V | of length T1 and T2 . In NLI , x indicates premise and y is hypothesis while in question answering , it is the question and paragraph respectively . Following Bowman et al . ( 2015 ) , we use two separate RNNs for encoding both the sequences to obtain { hx1 , ... , hxT1 } and { h y 1 , ... , h y T2 } . Now , similar to Jain & Wallace ( 2019 ) , attention weight αi over each token of x is computed as : ui = tanh ( W1h x i +W2h y T2 ) ; αi = exp ( uTi c ) ∑ j exp ( u T j c ) , ( 2 ) where similar to Equation 1 , W1 , W2 ∈ Rd×d ′ denotes the projection matrices and c ∈ Rd′ is a parameter vector . Finally , the representation obtained from a weighted sum of tokens in x : hα =∑T i=1 αih x i is fed to a classifier for prediction . We also explore a variant of the above attention proposed by Rocktäschel et al . ( 2016 ) . Instead of keeping the RNN encoders of both the sequences independent , Rocktäschel et al . ( 2016 ) use conditional encoding where the encoder of y is initialized with the final state of x ’ s encoder . This allows the model to obtain a conditional encoding { h′y1 , ... , h ′y T2 } of y given the sequence x . Moreover , unlike the previous model , attention over the tokens of x is defined as follows : M = tanh ( W1X +W2h ′y T2 ⊗ eT1 ) ; α = softmax ( wTM ) , ( 3 ) where X = [ hx1 , ... , h x T1 ] , eT1 ∈ RT1 is a vector of ones and outer product W2h ′y T2 ⊗ eT1 denotes repeating linearly transformed h′yT2 as many times as words in the sequence x ( i.e . T1 times ) . 3http : //www.statmt.org/wmt16/translation-task.html 3.3 GENERATION TASK MODELS : . In this paper , for generation tasks , we focus on Neural Machine Translation ( NMT ) problem which involves translating a given source text sentence x ∈ RT1×|V1| to a sequence y ∈ RT2×|V2| in the target language . The model comprises of two components : ( a ) an encoder which computes a representation for each source sentence and ( b ) a decoder which generates a target word at each time step . In this work , we utilize RNN based encoder and decoder models . For each input sentence x , we first obtain a contextualized representation { h1 , ... , hT1 } of its tokens using a multi-layer Bi-RNN . Then , at each time step t , the decoder has a hidden state defined as ct = f ( ct−1 , yt−1 , h t α ) , where h t α = T1∑ i=1 αt , ihi . In our work , we compute αt , i as proposed by Bahdanau et al . ( 2015 ) and Luong et al . ( 2015 ) . The former computes attention weights using a feed-forward network , i.e. , αt , i = wT tanh ( W [ ct ; hi ] ) while the latter define it simply as αt , i = cTt hi . 3.4 SELF-ATTENTION BASED MODELS : . We also examine self-attention based models on all three categories of tasks . For single and pair sequence tasks , we fine-tune pre-trained BERT ( Devlin et al. , 2019 ) model on the downstream task . In pair sequence tasks , instead of independently encoding each text , we concatenate both separated by a delimiter and pass it to BERT model . Finally , the embedding corresponding to [ CLS ] token is fed to a feed-forward network for prediction . For neural machine translation , we use Transformer model proposed by Vaswani et al . ( 2017 ) with base configuration .

Summary: I use (unqualified) “self-attention” to refer to attention of tokens in a sequence to other tokens in the same sequence, as described by [some corrected version of] Eq (1) and the paragraph following it (citing Bahdanau et al. 2015). This contrasts with “Transformer self-attention” and “cross-sequence attention”.