Attention Interpretability Across NLP Tasks
Authors: Shikhar Vashishth
Shyam Upadhyay
Gaurav Singh Tomar
Manaal Faruqui
Venue:  (2020)
Keywords: Attention
NLP
Interpretability

Abstract: The attention layer in a neural network model provides insights into the model’s reasoning behind its prediction, which are usually criticized for being opaque. Recently, seemingly contradictory viewpoints have emerged about the interpretability of attention weights (Jain & Wallace, 2019; Vig & Belinkov, 2019). Amid such confusion arises the need to understand attention mechanism more systematically. In this work, we attempt to fill this gap by giving a comprehensive explanation which justifies both kinds of observations (i.e., when is attention interpretable and when it is not). Through a series of experiments on diverse NLP tasks, we validate our observations and reinforce our claim of interpretability of attention through manual evaluation.

Under review as a conference paper at ICLR 2020
ATTENTION INTERPRETABILITY ACROSS NLP TASKS
Anonymous authors
Paper under double-blind review
ABSTRACT
The attention layer in a neural network model provides insights into the model's s reasoning behind its prediction, which are usually criticized for being opaque. Re- cently, seemingly contradictory viewpoints have emerged about the interpretabil- ity attention weights (Jain & Wallace 2019; Vig Belinkov 2019). Amid such confusion arises the need to understand attention mechanism more systematically. In this work, we attempt to fill this gap by giving a comprehensive explanation which justifies both kinds of observations (i.e., when is attention interpretable and when it is not). Through a series of experiments on diverse NLP tasks, we validate observations and reinforce claim of interpretability of attention through manual evaluation.
INTRODUCTION
Attention is way of obtaining weighted sum of the vector representations of layer in neural network model (Bahdanau et al.] 2015). It is used in diverse tasks ranging from machine transla- tion (Luong et al.) 2015),, language modeling (Liu & Lapatal 2018) to image captioning (Xu et al.; and object recognition (Ba et al. 2014). Apart from substantial performance benefit ( (Vaswani al. 2017), attention also provides interpretability to neural models (Wang et al., 2016 Lin al.,
2015)
2017
Ghaeini et al. (0188) which are usually criticized for being black-box function approximators
(Chakraborty al. 2017).
There has been substantial work on understanding attention in neural network models. On the one hand, there is work on showing that attention weights are not interpretable, and altering them does not significantly affect the prediction (Jain & Wallace, 2019 Serrano & Smith 2019). While on the other hand, some studies have discovered how attention in neural models captures several linguistic notions of syntax and coreference (Vig & Belinkov 2019; Clark et al.) 2019) Tenney et al.) 2019). Amid such contrasting views arises a need to understand the attention mechanism more systemati- cally. In this paper, attempt to fill this gap by giving comprehensive explanation which justifies both kinds of observations.
The conclusions Jain & Wallace 2019)) Serrano & Smith (019) have been mostly based on text classification experiments which might not generalize to several other NLP tasks. Figure [ we report the performance text classification, Natural Language Inference (NLI) and Neural Ma- chine Translation (NMT) of two models: one trained with neural attention and the other trained with attention weights fixed to a uniform distribution. The results show that the attention mechanism in text classification does not have an impact on the performance, thus, making inferences about inter- pretability of attention in these models might not be accurate. However, on tasks such as NLI and NMT uniform attention weights degrades the performance substantially, indicating that attention is a crucial component of the model for these tasks and hence the analysis of attention's 's interpretability here is more reasonable.
In comparison to the existing work on interpretability, we analyze attention mechanism on a more diverse set of NLP tasks that include text classification, pairwise parwiss text classification (such as NL.), and text generation tasks like neural machine translation (NMT). Moreover, we do not restrict ourselves to a single attention mechanism and also explore models with self-attention. For examining the interpretability of attention weights, we perform manual evaluation. Our key contributions are:
1. We extend the analysis of attention mechanism in prior work to diverse NLP tasks and provide comprehensive picture which alleviates seemingly contradicting observations.

---

Under review as a conference paper at ICLR 2020
hu
ha
22
04
Premise
h2
h22
hs
h2
h2
h
Hypothesis
X2
Input text
ht
hg
hg
Source text
Target text
Attention
Uniform
20
45
20
40
60
80
15
Accuracy
Accuracy
BLEU Score
(a) Text Classification
(b) Natural Language Inference
(c) Neural Machine Translation
Figure 1: Comparison of performance with and without neural attention on text classification (IMDB), Natural Language Inference tasks (SNLI) and Neural Machine Translation (News Com- mentary). Here, and denote attention weights and context vector respectively. The results show that attention does not substantially effect performance on text classification. However, the same does not hold for other tasks.
2. We identify the conditions when attention weights are interpretable and correlate with feature importance measures- when they are computed using two vectors which are both functions of the input (Figure Db. c). We also explain why attention weights are not interpretable when the input has only single sequence (Figure [a), observation made by Jain & Wallace (2019), by showing that they can be viewed as a gating unit.
3. We validate our hypothesis of interpretability of attention through manual evaluation.
2 TASKS AND DATASETS
We investigate the attention mechanism on the following three task categories.
1. Single Sequence tasks those where the input consists of single text sequence. For in- stance, in sentiment analysis, the task is to classify a review as positive or negative. This also includes other text classification tasks such topic categorization. For the experiments, in this paper, we use three review rating datasets: (1) Stanford Sentiment Treebank (Socher et al., 2013). (2) IMDB Maas al. 2211)) and (3) Yelp 201 and one topic categorization dataset AG News Corpus (business world)
2. Pair Sequence tasks comprise of a pair of text sequences as input. The tasks like NLI and question answering come under this category. NLI involves determining whether hypoth- esis entails, contradicts, or is undetermined given a premise. We use Stanford Natural Lan- guage Inference (SNLI) (Bowman al.) 2015) and Multi-Genre Natural Language Inference (MultiNLI) (Williams et al.) (018)) datasets for our analysis. For question answering, similar to Jain Wallace (2019). we use CNN News Articles (Hermann et al. 2015) and three tasks of the original babl dataset (Weston et al. 2015) in our experiments, i.e., using one, two and three supporting statements as the context for answering the questions.
3. Generation tasks involve generating a sequence based on the input sequence. Neural Machine translation is an instance of generation task which comprises of translating source text to target language given translation pairs from parallel corpus. For our experiments, we use
from www.yelp.com/datasset challenge www..i.unippi.it gulli/AG corpus of atticles.html
2

---

Under review as a conference paper at ICLR 2020
three English- German datasets: Multi30k (Elliott al. 2016), En-De News Commentary v11 from WMT16 translation task and full En-De WMT13 dataset.
English-German
and
NEURAL ATTENTION MODELS
In this section, we give a brief overview of the neural attention-based models we analyze for different
categories of tasks listed in Section 2
The overall architecture for each category is shown in Fig
3.1 SINGLE SEQUENCE MODELS:
For single sequence tasks, we adopt the model architecture from Jain & Wallace (019); Wiegreffe & Pinter 2019). For given input sequence E Txx where T and are the number of tokens and vocabulary size, we first represent each token with its d-dimensional GloVe embedding Pennington et al. (2014) to obtain 2e Rx'' Next, we use a Bi-RNN encoder (Enc) to obtain an m-diimnssional contextualized representation of tokens: h Enc(æ, IR''xm Then, use the additive formulation attention proposed by [Bahdanau et al. (2015) for computing attention weights ag for all tokens defined as:
exp(u c) u; = tanh Wh; + b); 04 Σ, exp(uf,c)
where W E Rdd xm b, E Rdd are the parameters of the model. Finally, the weighted instance representation: ha = Σi= o;h; E R is fed to a dense layer (Dec) followed by softmax to obtain prediction ý = aDec E R!'!, where [}] denotes the label set size.
We also analyze the hicrarchical attention model (Yang et al.) 2016). which involves first computing attention over the tokens to obtain sentence representation. This is followed by attention over sen- tences obtain an instance representation ha, which fed a dense layer for obtaining prediction (0). At both word and sentence level the attention is computed similar to as defined in Equation [
3.2 PAIR SEQUENCE MODELS:
For pair sequence, the input consists of two text sequences: E T y RTxx of length Ti and T2. In NLI, x indicates premise and y is hypothesis while in question answering. it is the question and paragraph respectively. Following [oomman et al. (215)) we use two separate RNNs for encoding both the sequences to obtain {hf,... h J and {h ... h% J. Now, similar to Jain & Wallace 2019), attention weight a, over each token of æ is computed as:
exp(u. c) 0; Σ, exp(
u tanh hh + W2h
where similar Equation W1. W2 E ddxdd' denotes the projection matrices and IRd is parameter vector. Finally, the representation obtained from a weighted sum of tokens in x: ho Einash
aghę is fed to a classifier for prediction.
We also explore variant of the above attention proposed by Rocktäschel al. (2016). Instead of keeping the RNN encoders of both the sequences independent, Rocktäschel et al. (2016) use condi- tional encoding where the encoder of y is initialized with the final state of x's encoder. This allows the model to obtain a conditional encoding (h" .... J of given the sequence æ. Moreover, unlike the previous model, attention over the tokens of defined as follows:
M- tanh W, X + Wyh @ eT,))
softmax(w" M),
where X [hf .... 1 en, E T is vector of ones and outer product Wah @ er, denotes repeating linearly transformed h4 as many times as words the sequence x (i.e. Ti times).
hhtp://www.sttam.oog//www oata
3

---

Under review as a conference paper at ICLR 2020
3.3 GENERATION TASK MODELS:
In this paper, for generation tasks, we focus on Neural Machine Translation (NMT) problem which involves translating a given source text sentence a E IT x|Vi to a sequence y E R in (a) encoder which computes representation for each source sentence and (b) a decoder which generates a target word at each time step. In this work, we utilize RNN based encoder and decoder models. For each input sentence æ, we first obtain a contextualized representation thi....,TTT of its tokens using a multi-layer Bi-RNN. Then, each time step the decoder has a hidden state defined as
the target language. The model comprises of two components:
T
C₁ = f(4 1,, ha)), where h
= O4 h-
In our work, we compute 0ı,i as proposed by Bahdanau et al. (2013) and Luong et al. (2015). The former computes attention weights using a feed-forward network, i.c. at, w''ann [ [[c h,]) while the latter define it simply as as = h.
3.4 SELF-ATTENTION BASED MODELS:
We also examine self-attention based models on all three categories of tasks. For single and pair sequence tasks, we fine-tune pre-trained BERT (Devlin et al. 2019) model on the downstream task. In pair sequence tasks, instead of independently encoding each text, we concatenate both separated by delimiter and pass to BERT model. Finally, the embedding corresponding [CLS] token fed to a ferd-forward network for prediction. For neural machine translation, we use Transformer model proposed by Vaswani et al. (017) with base configuration.
IS ATTENTION AN EXPLANATION?
In this section, we attempt address the question: Is attention an explanation? through series of experiments which involve analyzing attention weights in variety of models (33) on multiple tasks ($2)) Following Jain Wallace (201)), we take the definition of explainability of attention
as: inputs with high attention weights are responsible for model output. Jain & Wallace (2019); Serrano Smith (2019) have extensively investigated this aspect for certain class of problems and have shown that attention does not provide explanation. However, another series of work (Vig & Belinkov| 2019; Clark et al. [219 Tenney et al.] 2019) has shown that attention does encode several linguistic notions. In our work, we claim that the findings of both the line of work consistent. We note that the observations the former works can be explained based on the following proposition.
Proposition 4.1. Attention mechanism as defined in Equation as
c)
= t=ttannWW bb;;
exp((u.
for single sequence tasks can be interpreted as a gating unit in the network.
Proof: The attention weighted averaging computed in Equation for single sequence tasks can be interpreted as gating proposed by Dauphin et al. (2017) which is defined as
h((r)= = (x) x (g(x),
where æ E I the input and x denotes product between transformed input f(x) E I and its computed gating scores (g(x)) R. Equation | can be reduced to the above form by taking as an identity function and defining g(x) cd tanh W x + b) E R and replacing a with softmax. We note that the same reduction does hold in the case of pair sequence and generation tasks where attention along with input also depends on another text sequence Y and current hidden state Ct, respectively. Thus, attention mechanism for these tasks take the form
h y) (() (g(x, v)),
which does not reduce the above equation for gating unit.
4

---

Under review as a conference paper at ICLR 2020
SST
IMDB
AG News
YELP
Bahdanau et al. 2015
83.4 0.5
90.7 0.7
96.4 0.1
66.7 0.1
Uniform (Train+Infer/ Infer)
-1.0/-0.8
0.8 0.3
0.1/-0.7
0..5/6.3
Random (Train-Infer Infer)
-191/-0.9
0.6/-6.4
--0
-044/664
Permute (Infer)
-177
-0.9
-7.8
Yang al. (016))
83.2 0.5
89.7 + 0.6
96.1 + 0.2
65.8 + 0.1
Uniform (Traintinfer/ Inler)
-1.0/-0.8
+0.2/
1-5
0-7/-8.0
Random (Train-Infer/ Infer)
-0.9/-1.0
-1.2/-8.2
-0.1/-1.8
--.0/-10.2
Permute (Infer)
-1.8
-5.1
-0.7
-10.7
Table 1: Evaluation results on single sequence tasks. We report the base performance of attention models and absolute change accuracy for its variant. We note that across all datasets, degradation in performance on altering attention weights during inference is more compared to varying them during both training and inference. Overall, the change in performance less compared other tasks. Please refer to 44.1 for more details.
Based on the above proposition. we argue that weights learned in single sequence tasks cannot be in- terpreted as attention, and therefore, they do not reflect the reasoning behind the model's prediction. This justifies the observation that for the single sequence tasks examined in Jain & Wallace (2019); Serrano & Smith (2019), attention weights do correlate with feature importance measures and permuting them does not change the prediction of the model. In light of this observation, we revisit the explainability of attention weights by asking the following questions.
4.1 How DOES ALTERING ATTENTION WEIGHTTS AFFECT MODEL OUTPUT ON TASKS?
In this section, we compare the performance of various attention mechanism described in S for different categories tasks listed in 2 For each model, we analyze its three variants defined as:
Uniform denotes the case when all the inputs are given equal weights, i.c., a; 1/T, E (1..., T). This is similar the analysis performed by Wiegreffe & Pinter (2019). However, we consider two scenarios when the weights are kept fixed both during training and inference (Train+Infer) and only during inference (Infer).
Random refers to the variant where all the weights are randomly sampled from uniform distribution: a; ~ (0, 1), Vi E {1,.... T}. this is followed by normalization. Similar to Uniform, analyze both Traen and Infer.
Permute refers to the case when the learned attention weights are randomly permuted during inference, i.e., a = shuffle(a.). Unlike the previous two, here we restrict our analysis to only permuting during inference Tensorflow currently does not support backpropagation with shuffle operation
Effect on single sequence tasks: The evaluation results on single sequence datasets: SST, IMDB, AG News, and YELP are presented in Table [ We observe that Train+Infer case of Uniform and Random attentions gives around 0.5 and 0.9 average decrease in accuracy compared to the base model. However, in Infer scenario the degradation on average increases to 3.9 and 4.5 absolute points respectively. This is so because the model becomes more robust to handle altered weights in the former case. The reduction in performance from Permute comes around to 4.2 across all datasets and models. The results support the observation of Jain & Wallace (2019); (Serrano & Smith (2019) that alternating attention in text classification task does not have much effect on the model output. The slight decrease in performance can be attributed to corrupting the existing gating mechanism which has been shown to give some improvement (Oord et al. 2016 Dauphin et al.
2017 Marcheggiani & Titov 2017
The results on pair sequence and generation tasks summarized in Table 2 and respectively. Overall, we find that the degradation in performance from altering attention weights in case of pair sequence and generation tasks much more substan- tial than single sequence tasks. For instance, in Uniform (Train+ Infer), the average relative decrease
Effect on pair sequence and generation tasks:
5

---

Under review as a conference paper at ICLR 2020
SNLI
MultNLII
CNN
babl 0
babl
babl 2
Bahdanau et al.
2013
75.7 0.3
6..1
63.4 # 0.8
96.1 : 4.3
95.8 + 0.3
92.8 + 0.1
Uniffrm (Tnin=lafor (nnfer)
-41.8/442.9
26.6 26.6/-28.7
30.8/-55.9
-44.4/ 14.4/-63.4
-47.4/66044
-484 48.4/-62.1
Random (Triinllnfer (Infr))
-11.6/43..1
26.77 /-2866
30.9/-56.9
45.0/-62.0
47.3/-6044
-9.9 /-6.22
Permute (Infer)
-41.0
-27.6
-54.8
-67.8
-68.3
-66.7
Rocktäschel al. 2016)
78.1 +:
62.4
63.6 + 0.6
98.6 1.6
96.2
93.2 + 0.1
Uniform (Inin+Inter Inter)
-11.2/1554
/-33/330.3
30.8 1-1111
447.7 1-.7/-67.8
-17.9/622.
19.8 /-6099
Random (Train-nnfer/ Infor)
44.3/-44.9
27.99 /-28.3
00.6/-43.3
-4..5/66.9
18.4/-63.3
49.8 /-6.99
Permute (Infer)
-11.7
-29.2
-4.9
-68.8
-6.33
-65.2
Table The performance comparison of attention based models and their variants on pair sequence tasks. We find that the degradation in performance is much more than single sequence tasks.
Dataset
Multi30k
News Commentary
Bahdanau et al. 2015
31.3 1 0.1
12.6 + 0.1
Uniform (Train4Infer Infer)
- -10.4 /-2
-877 11.8
Random (Train+Infer/ Infer)
10.1 /9.
-8.8/ 1-19-11.9
Permute (Infer)
29.77
-12.1
Luong ct al. 2015))
31.5 + 0.2
12.7 + 0.2
Uniform (Truin+Infer / Infer)
10.66-29.
-8.8 /-12.0
Random (Train+Infer/ Infer)
10.3/-29.8
-8.9 /-12.0
Permute (Infer)
-30.1
-12.2
Table 3: Evaluation results on neural machine translation. Similar to pair sequence tasks, we find that the deterioration in performance is much more substantial than single sequence tasks. Please refer to 44.1 for more details.
in performance of single sequence tasks is 0.1% whereas in case of pair sequence and generation tasks it is 49.5% and 51.2% respectively. The results thereby validate our Proposition 4. and show that altering attention does affect model output for task where the attention layer cannot be mod- eled as gating unit in the network.
Visualizing the effect of permuting attention weights: To further reinforce our claim, similar to Jain & Wallace 2019), we report the median of Total Variation Distance (TVD) between new and original prediction on permuting attention weights for each task. The TVD between two predictions E 1ú d2s1. where |}] denotes the total number of classes the problem. We use TVD for measuring the change in output distribution on permuting the attention weights. In Figure 2 we report the relationship between the maximum attention value and the median induced change in model output over 100 permutations on categories of tasks. For NMT task, we present change in output at the Sitth-prrenttlee length of sentences for both datasets. Overall, we find that for single sequence tasks even with the maximum attention weight in range J0.75, 1.0]. the change in prediction is considerably small (the violin plots are to the left of the figure) compared to the pair sequence and generation tasks (the violin plots are to the right of the figure).
인1 and ÿ/2 is defined as: TVD(Ú1: (02) 1
4.2 Do ATTENTION WEIGHTS CORRELATE WITH FEATURE IMPORTANCE MEASURES?
In this section, similar to the analysis of Serrano Smith 2019). we investigate the importance of attention weights only when one weight is removed. Let d" be the input corresponding to the highest attention weights and let r be any randomly selected input. We denote the original model's prediction as p and output after removing i" and input q(i-) and q(r) respectively. Now, to measure the impact removing i' relative to any randomly chosen input r on the model output, we compute the difference of Jensen-Shannon (JS) divergence between JS(p, q(: and JS(p, q())) given as: AJS JS(p, ))))) JS(p. {(r)). ). The relationship between the difference of attention weights corresponding to and r, i.c., ar+ a, and AJSS for different tasks is presented in Figure In general, we found that for single sequence tasks, the change JS divergence is small even for cases

---

Under review as a conference paper at ICLR 2020
10.00
06.00
1040
Mer ottentin
0.501
0.50)
0.50
18.50
10.50
10.50
0.751
0.75
3.750
19.75
[6.75
1.000
3.0
0.0
Median Output Difference
(a) SST
(c) MultiNLI
(e) Multi30k
10.00
10.00
10.00
0.50)
0.500
0.500
0.50.
00.50
10300
0.751
1.00)
1.00]
1.000
0.8
(b) IMDB
(d) babl2
News Commentary
Single Sequence tasks
Pair Sequence tasks
Generation tasks
Figure 2: Relationship between maximum attention weight and median change in output per- muting attention weights. For single sequence tasks, M. indicate negative and positive class. For MultiNLI,
denotes contradiction, entailment and neutral respectively. The results rein- force the claim that altering attention weights in single sequence tasks does not have much effect on performance while the same does not hold with other tasks. Refer to [4. for details.
Single Sequence (IMDB)
[b) Sequence (babl2)
[c] Generation Task (News Commentary)
6.8
04
ISGn ) [S
6.2
611
69
0.0
9.2
Difference zeroed Attention weights magnitude (a. (K,)
Figure 3: Analysis of correlation between attention weights and feature importance measure. We report relationship between difference in zeroed attention weights and corresponding change in JS divergence for different tasks. Please refer to 44.2 for more details.
when the difference in attention weight is considerable. However, for pair sequence and generation tasks, there is substantial change in the model output.
4.3 How PERMUTING DIFFERENT LAYERS SELF-ATTENTION BASED MODELS AFFECT PERFORMANCE?
In this section, we analyze the importance of attention weights on the performance of self-attention based models as described in (3.4] We report the accuracy on single, and pair sequence tasks and BLEU score for NMT on WMT13 dataset on permuting the attention weights of layers cumulatively. For Transformer model, we analyze the effect of altering attention weights in encoder, decoder, and across encoder-decoder (denoted by Across). The results are presented in Figure
Overall, we find that unlike the pattern observed $4.1 and 44.2 for single sequence tasks, altering weights in self-attention based models does have substantial effect the performance. We note that this is because while computing attention weights over all tokens with respect to a given token, Proposition 4.1 does not hold. Thus, altering them does have an impact across all three tasks. We note that in the case of transformer model, altering the weights in the first step of Decoder and in Across has maximum effect it almost stops the flow of information from encoder to decoder.

---

Under review as a conference paper at ICLR 2020
Encoder
Decoder
80
IMDB MNLI
Accuracy
BLEU Score
29
20
10
BERT Layers
(b) Transformer Layers
Figure 4: Performance comparison with permuting attention weights for different layers in self- attention based models. The results are reported on representative instance of single sequence, pair sequence and generation tasks. The dotted lines denote the base performance on the task. Refer
4.3 for details.
Attention Moaningful
Attention Meaningful
Correct Prediction
No
Yes
No
Yes
No Yos
79.5
11.5
No 18
9.5
74.5
No 1
97
No YE2
9.5
14.5
4.5
2.5
13.5
1.5
0.5
12.5
63.5
Original
Permuted
Original
Permuted
(a) Single Sequence (IMDB)
(b) Pair Sequence (babl)
Figure 5: Manual evaluation of interpretability of attention weights single and pair sequence tasks. Although with original weights the attention does remain interpretable on both tasks but in the case of single sequence tasks making it meaningless does not change the prediction substantially. However, the same does not hold with pair sequence tasks.
4.4 ARE ATTENTION WEIGHTS HUMAN INTERPRETABLE?
To determine if attention weights are human interpretable, here, we address the question of inter- pretability of attention weights by manually analyzing them on representative dataset of single and pair sequence task. For each task, we randomly sample 100 samples with original attention weights and 100 with randomly permuted weights. Then, we shuffle all 200 samples together and present them to annotators for deciding whether the top three highest weighted words are relevant for the model's 'ss prediction.
The overall results are reported in Figure 5 Cohen's kappa score of inter-annotator agreement (Co- hen, 1960) on IMDB and babl is 0.84 and 0.82, respectively, which shows near-perte pear-perfect agreement (Landis & Koch 1977). We find that in both single and pair sequence tasks, the attention weights in samples with original weights do make sense in general (highlighted with blue color). However, in the former case, the attention mechanism learns give higher weights to tokens relevant both kinds of sentiment. For instance, in "This is a great movie. Too bad it is not available on home video.". tokens great, too, and bad get the highest weight. Such examples demonstrate that the attention mechanism in single sequence tasks works like gating unit, as shown 44.1
For permuted samples, in the case of single sequence, the prediction remains correct in majority although the attention weights were meaningless. For example, in "This movie was terrible the acting was lame, but it 's hard to tell since the writing was so had." the prediction remains the same on changing attention weights from underlined to bold tokens. However, this does not hold with the pair sequence task. This shows that attention weights single sequence tasks do not provide reason for the prediction, which in the case of pairwise tasks, attention do reflect the reasoning behind model output.
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
In this paper, we addressed the seemingly contradictory viewpoint over explainability of attention weights in NLP. On the one hand, some works have demonstrated that attention weights are interpretable, and altering them does not affect the model output while several others have shown that attention captures several linguistic notions in the model. We extend the analysis of prior works to diverse NLP tasks and demonstrate that attention weights are interpretable and are correlated with feature importance measures. However, this holds only for cases when attention weights are essential for model's prediction and cannot simply be reduced to gating unit. Through battery of experiments, we validate our claims and reinforce them through manual evaluation.
REFERENCES
Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual attention. In Proc. of ICLR, 2014.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR, 2015.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. In Proc. of EMNLP, 2015.
S. Chakraborty, R. Tomsett, R. Raghavendra, D. Harborne, M. Alzantot, F. Cerutti, M. Srivastava, A. Preece, S. Julier, R. M. Rao, T. D. Kelley, D. Braines, M. Sensoy, J. Willis, and P. Gurram. Interpretability of deep learning models: A survey of results. In Proc. of UbiComp, 2017.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of bert's attention. In Proc. of BlackBoxNLP, 2019.
J. Cohen. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Mea- surement, 20(1):37, 1960.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proc. of ICML, 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL, 2019.
Desmond Elliott, Stella Frank, Khalil Sima' an, and Lucia Specia. Multi30k: Multilingual english- german image descriptions. In Proc of the 5th Workshop on Vision and Language, 2016.
Reza Ghaeini, Xiaoli Z. Fern, and Prasad Tadepalli. Interpreting recurrent and attention- based neural models: case study on natural language inference. In Proc. of EMNLP. 2018
Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proc. of NIPS, 2015.
Sarthak Jain and Byron C. Wallace. Attention is not Explanation. In Proc. of ACL, 2019.
Richard Landis and Gary G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33(1), 1977.
Zhouhan Lin, Minwei Feng, Cícero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In Proc. of ICLR, 2017.
Yang Liu and Mirella Lapata. Learning structured text representations. Transactions of the Associ- ation for Computational Linguistics, 6:63-75, 2018. ISSN 2307-387X.
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Proc. of EMNLP, 2015.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proc. of ACI,, 2011.

---

Under review as a conference paper at ICLR 2020
Diego Marcheggiani and Ivan Titov. Encoding sentences with graph convolutional networks for semantic role labeling. In Proc. of EMNLP, 2017.
Aäron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelenn decoders. In Proc. of NIPS, 2016.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Proc. of EMNLP. 2014.
Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, and Phil Blunsom. Reasoning about entailment with neural attention. In Proc. of ICLR, 2016.
Sofia Serrano and Noah A. Smith. Is attention interpretable? In Proc. of ACL, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. of EMNLP, 2013.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline, In Proc. of ACL, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of NIPS, 2017.
Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in transformer language model. In Proc. of BlackBoxNLP, 2019.
Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. Attention-based LSTM for aspect-level sentiment classification. In Proc. of EMNLP. 2016.
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. In Proc. ICLR, 2015.
Sarah Wiegreffe and Yuval Pinter. Attention is not explanation. In Proc. of EMNLP, 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proc. of NAACI, 2018.
Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Proc. ICML, 2015.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Proc. of NAACL, 2016
10


Reviews:
[{"review_id": "r1xry0J6tB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the degree to which we might view attention weights as explanatory across NLP tasks and architectures. Notably, the authors distinguish between single and \"pair\" sequence tasks, the latter including NLI, and generation tasks (e.g., translation). The argument here is that attention weights do not provide explanatory power for single sequence tasks like classification, but do for NLI and generation. Another notable distinction from most (although not all; see the references below) prior work on the explainability of attention mechanisms in NLP is the inclusion of transformer/self-attentive architectures. \n\nOverall, this is a nice contribution that unifies some parallel lines of inquiry concerning explainability, and attention mechanism for different NLP tasks. The contrast in findings between single sequence (classification) and other NLP tasks and the implications for explainability is interesting, and provides a piece missing from the analyses conducted so far. \n\nThe main issue I have with the paper is that I'm not sure I follow the rationale in Section 4 arguing that because attention \"works as a gating unit\", it follows that we cannot view single sequence tasks \"as attention\". Can the authors elaborate on why the conclusion follows from the premise? In other words, why is attention inherently *not gating*? This seems like an interesting connection, but again I do not see why attention acting as a gate implies that we should not view it as attention at all. Perhaps the authors could elaborate on their reasoning. \n\nSome additional references the authors might consider. Note that the latter two papers, I think, would seem to broadly disagree regarding self-attention and tasks aside from single sequence (classification): \n- \"Fine-grained Sentiment Analysis with Faithful Attention\": https://arxiv.org/abs/1908.06870\n- \"Interrogating the Explanatory Power of Attention in Neural Machine Translation\": https://arxiv.org/abs/1910.00139\n- \"On Identifiability in Transformers\": https://arxiv.org/abs/1908.04211", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hke1Vxpg9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "CONTRIBUTIONS:\nI use (unqualified) “self-attention” to refer to attention of tokens in a sequence to other tokens in the same sequence, as described by [some corrected version of] Eq (1) and the paragraph following it (citing Bahdanau et al. 2015). This contrasts with “Transformer self-attention” and “cross-sequence attention”.\nC1. Self-attention is gating. (Prop. 4.1, Sec. 4)\nC2. Gating cannot provide explanations in the way that attention is alleged to do.\nC3. Single input sequence models deploy only self-attention, so by C1-C2, attention cannot be used for explanation of these models; attention in two input sequence models is not limited to self-attention and is not equivalent to gating, so attention can provide explanations in these models.\nC4. Outputs are sensitive to alteration of attention weights for two-sequence but not one-sequence models. (Tables 2-3 vs. 1; Figs. 2-3) \nC5. Human judgments (“manual evaluation”) of the intuitive importance, for the output, of items with highest attention weights show that such intuitive importance is found for both one- and two-sequence models. (Fig. 5)\nRATING: Reject\nREASONS FOR RATING (SUMMARY). Because the modeling experiments closely follow previous work, the primary contribution rests on the account provided of why explanation-by-attention works only sometimes --- on the basis of the Proposition identifying attention with gating. But the reasoning and the math as presented are problematic. There is a worthwhile contribution from the human judgment experiment, but this is not sufficient to overcome the weakness with the main argument of the paper. (I also have reasons to question whether one- vs two-sequence inputs is the right distinction that needs to be accounted for.) \nREVIEW\nC1. If this point were made in plain English, ‘attention weights and gating units both multiply activation values’, I would say, “yes, obviously”. But the point is stated as a Proposition, with a Proof, so the math given should be correct. I don’t see a way to make it correct, and that shouldn’t be the job of the reader anyway. There are several errors in the dimensions of the matrices which make the equations incoherent. Eq. (1) contains W*h, a matrix product, where the dimensions stated are W in R^{d x d’} and h in R{T x m}; these cannot be multiplied. This unclarity about matrix dimensions propagates into Prop. 4.1. In the definition of g(X), we have WX + b, where b is presumably a vector. Addition then requires that WX also be a vector, but X is stated to be in R^{n x d}, so WX cannot be a vector. Whether WX + b is actually a vector or a matrix, g(X) = c^T tanh(WX + b) is not a matrix: it is either a scalar or a vector. But this can’t be. The definition of h uses the elementwise product, which requires that both arguments have the same dimensions, so g(X) must have the same dimensions as f(X). We’re told f is the identity function, so f(X), like X, must be a matrix. Furthermore, the statement of Prop. 4.1 says that self-attention can be interpreted as *a* gating unit. By the standard definition of ‘unit’, this should mean that self-attention is a scalar.\nThroughout the paper, we are never told what kind of “RNN” is being assumed. If the RNN unit contains gates, as in an LSTM or a GRU, I can imagine that the intention is for Prop. 4.1 to say that (*) “the effect of self-attention can be reproduced without attention by adjusting the weights in the gating mechanism already present in the RNN”, so that attention doesn’t increase the capacity of the model. But what I see in the paper does not convince me that (*) is true. (Because of the kind of global normalization required by the softmax, I actually suspect it is not.)\nC2. I don’t see why (formally or intuitively) gating is not a legitimate candidate for explaining the behavior of networks containing gates; I would assume just the opposite, actually. How can it *not* be part of a satisfactory explanation? And why should changing a name from “attention” to “gating” have any bearing on whether it (whatever it is called) can potentially serve for explanation of network behavior?\nC3. Leaving the formalism aside, I don’t see intuitively why, whatever an analysis of self-attention might entail about explanation, the same implication shouldn’t apply to straightforward (not Rocktaeschel) attention when two sequences are present. Why can’t we just treat the concatenation of the input sequences as a single input sequence, as standardly done for example for the Transformer? If the formal content of Prop. 4.1 were clear, perhaps this could be justified, but it is simply asserted without justification in the proof that “the same reduction does not hold in the case of pair sequence”.\nC4. Claims C1-C3 attempt to give an account for why various tests of the explanatory role of attention turn out positive for two-sequence but not one-sequence tasks, a pattern previously reported and verified with new results in the present paper. I fear however that one- vs two- is not the correct empirical generalization about attention that one should try to account for. Messing about with attention weights would not be expected to alter outputs if the output is determined by the input considered as a bag of words. And there is a troubling confound in the tasks compared: the one-sequence tasks are sentiment and topic classification, where a BOW model should do very well – and I suspect that is the real reason why these tasks don’t show strong sensitivity to attention weight distribution. But the two-sequence tasks are NLI and QA, where (ideally) BOW models should not do nearly so well: paying attention to the right tokens should be important. The same is true of translation. So the confound in the tasks examined leaves it undetermined whether the crucial factor to account for is one- vs two-sequence or BOW-friendly vs BOW-unfriendly tasks.\nC5. Put together, the human judgments and the sensitivity-to-altering-attention-weights seem to indicate that attention tends always to be allocated to intuitively important tokens, and that matters for the output of the two-sequence models but not for the one-sequence models. This is what we’d expect if attention is always being allocated appropriately, but for BOW-friendly tasks that doesn’t make much difference.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1l-cYdUFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Motivated by an existing paper, the paper analyzes the interpretability of attention mechanism over three NLP tasks. The previous paper claims that attention mechanism is not interpratable. The paper makes incremental contribution by showing that attentions are not interpratable when they mimic gating units (single sequence task) but are interpretable for two sequence and generation tasks. Experimental results are given to support the claims, which can also help readers to gain insights into the attention mechanism. The paper is well written and all claims are supported. I also have some questions below for clarification. If I get reasonable answers to these questions, I tend to accept the paper. \n\n1. Jain & Wallace (2019) show that for the SNLI problem, attentions weakly correlate with the output based on Kendall's correlation and JSD which contradicts to your observation. Could you explain how this happens? are there any model settings different?\n2. in figure 5, for the single sequence (original), most of attentions leading to correct predictions are labeled meaningful. Does this mean that even the attention doesn't necessarily contribute too much to the prediction correctness but they are also interpretable if they are allowed to be trained. Also, in Table 1, with modified attentions, all scores go down a little. This means that attention still can contribute to the final prediction but not significant enough (some like yelp are significant). I am gussing that the sequence encoding already present useful features for the final prediction. Did you check the distance of different word encodings? Are these encodings all very similar?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the degree to which we might view attention weights as explanatory across NLP tasks and architectures. Notably, the authors distinguish between single and "pair" sequence tasks, the latter including NLI, and generation tasks (e.g., translation). The argument here is that attention weights do not provide explanatory power for single sequence tasks like classification, but do for NLI and generation. Another notable distinction from most (although not all; see the references below) prior work on the explainability of attention mechanisms in NLP is the inclusion of transformer/self-attentive architectures. 

Unfortunately, the paper needs work in presentation (in particular, in Section 3) before it is ready to be published.
Decision: Reject