SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\begin{array}{l} \frac {d a _ {e}}{d \tau} = \underbrace {\frac {1}{m \lambda^ {2}} \left(\boldsymbol {V} ^ {\top} \boldsymbol {W} \cdot \lambda\right) \pi (e)} _ {(D)} \\ + \underbrace {\frac {1}{m \lambda^ {2}} \left\| \boldsymbol {V} \right\| _ {2} ^ {2} \rho (e)} _ {(E)} \tag {27} \\ + \underbrace {\frac {1}{m \lambda^ {2}} \sum_ {(t , j)} y ^ {(t)} \beta^ {(t)} \alpha_ {j} ^ {(t)} e ^ {\top} e _ {j} ^ {(t)} \left(s _ {j} ^ {(t)} - s ^ {(t)}\right)} _ {(F)}. \\ \end{array}
$$
Let us now understand the influence of these terms respectively:
- Term  $D$ . When  $\boldsymbol{V}^{\top}\boldsymbol{W} > 0$ , the positive tokens will receive a positive update whereas the negative tokens will receive a negative update from this term after each step. When  $\boldsymbol{V}^{\top}\boldsymbol{W} < 0$ , the influence is the other way around. It does not influence the attention scores of the neutral tokens much as the corresponding  $\pi(e)$  is approximately zero. When it is not close to zero, this term can lead to a gap between the final attention scores of the positive tokens and negative tokens.
- Terms  $E$  and  $F$ . Based on our analysis,  $E > 0$ , and  $F \geq 0$  for polarity tokens, and  $E \approx 0$  and  $F \approx 0$  for neutral tokens. This means for the positive tokens and negative tokens, their attention scores will likely receive a positive value from this term after each update when approaching a local minimum. Their corresponding attention scores may end up with large positive scores eventually. For the neutral tokens, this term does not have much influence on their attention scores.
From here we can observe that when  $V^{\top}W\cdot \lambda$  is small, the polarity tokens will likely end up with larger attention scores than the neutral tokens. This is actually a desirable situation - polarity tokens are likely more representative when used for predicting the underlying class labels, and therefore shall receive more "attention" in general.
However, we note that if the scaling factor  $\lambda$  is too large, the term  $D$  may be significant. This means the sign of  $\mathbf{V}^{\top}\mathbf{W}$  will then play a crucial role - when it is non-zero and when  $\lambda$  is very large, positive tokens and negative tokens will likely have
Dataset	AvgLength	VocabSize	Train	Size Dev	Test
SST	18	16174	3610/3310	444/428	909/912
IMDB	183	63311	8539/8673	2113/2191	2174/2189
20News I	185	17584	624/612	156/154	195/192
20News II	187	29433	794/790/716	91/70/79	84/100/90
Table 1: Datasets are all split into training, dev and test sets, respectively and are all balanced. The first 3 datasets are for binary classification (positive/negative), and the last is for 3-class classification (rec.motorcycles/sci.med/talk.politics.guns).
attention scores of opposite signs. This may not be a very desirable situation as the attention scores would be less interpretable in that case. On the other hand, as we have discussed in the previous section, the scaling factor  $\lambda$  should not be too small too. Otherwise term  $A$  in Equation 19 would not be close to 0 - as a result the conclusions on the polarity scores for the tokens stated at end of Sec 4.1 may not hold.
In conclusion, if we would like to observe the desirable behavior as discussed for the attention mechanism, it is important for us to choose an appropriate  $\lambda$  value or we shall possibly find ways to control the value of  $V^{\top}W^{6}$ . We will conduct experiments on real datasets to verify our findings.
Besides the above analysis, we have also analyzed polarity scores and attention scores from the model with additive attention, the model with an affine input layer and the model for multi-class classification respectively. There are terms that have similar effects on polarity and attention scores during training. Due to space limitations, we provide such details in the supplementary material.
We conducted experiments on four text classification datasets<sup>7</sup>. The statistics of the datasets are shown in Table 1. We followed the work of Jain and Wallace (2019) for pre-processing of the datasets<sup>8</sup>, and lower-cased all the tokens.
- Stanford Sentiment Treebank (SST) (Socher et al., 2013). The original dataset that consists of 10,662 instances with labels ranging from 1 (most negative) to 5 (most positive). Similar to the work of Jain and Wallace (2019), we removed neutral instances (with label 3), and regarded instances with label 4 or 5 as positive and instances with the label 1 or 2 as negative.
- IMDB (Maas et al., 2011). The original dataset
$^6$ We have further discussions on  $V^\top W$  in the supplementary material.
7We also conducted analysis on synthetic datasets. The results can be found in the supplementary material.
<sup>8</sup>https://github.com/successar/AttentionExplanation
See the supplementary material for more details.
λ	SST	λ	20News I
DP	DP-L	DP-A	AD	DP	DP-L	DP-A	AD
0.001	55.3	79.8	67.9	62.8	0.001	54.8	88.6	78.6	49.4
1	74.4	81.2	73.4	73.4	1	88.4	93.0	85.3	87.6
10	82.2	81.7	80.8	80.3	10	92.8	91.2	92.8	92.0
20	81.4	80.9	81.0	81.2	20	93.5	92.2	93.5	91.2
50	80.8	82.0	81.5	79.9	50	93.3	92.3	92.2	91.7
100	81.2	81.1	80.7	80.8	100	92.8	91.2	92.8	93.3
10000	79.6	81.4	79.3	80.8	10000	92.8	92.0	93.0	92.0
λ	IMDB	λ	20News II
DP	DP-L	DP-A	AD	DP	DP-L	DP-A	AD
0.001	55.5	87.7	73.3	69.8	0.001	31.8	90.1	64.6	59.2
1	79.5	88.2	85.4	83.7	1	85.4	92.3	88.3	86.7
10	89.2	87.8	89.6	88.2	10	93.4	93.4	91.7	90.0
20	89.6	88.1	89.6	89.6	20	94.9	94.2	93.3	92.1
50	89.8	87.2	89.1	88.5	50	94.9	92.3	92.9	93.8
100	89.3	88.3	89.2	88.8	100	94.9	93.1	92.9	92.9
10000	89.3	88.4	88.9	88.9	10000	94.5	93.8	92.5	92.9
Table 2: Test set results in accuracy  $(\%)$ . Models were chosen based on the highest accuracy on the dev sets.  $L_{2}$ -regularization was adopted on DP-L, DP-A and AD.
that consists of 50,000 movie reviews with positive or negative labels.
- 20Newsgroup I (20News I). The original dataset that consists of around 20,000 news-group correspondences. Similar to the work of Jain and Wallace (2019), we selected the instances from these two categories: "rec.sport.hockey" and "rec.sport.baseball", and regarded the former as positive instances and the latter negative.
- 20Newsgroup II (20News II). This is a dataset for 3-class classification. We selected instances from these three categories: "rec.motorcycles", "sci.med" and "talk.politics.guns".
Our analysis focused on the ideal case (e.g., positive tokens only appear in positive documents). To be as consistent as possible with our analysis, we only examined the tokens of strong association with specific labels and the tokens that could be seen almost evenly across different types of instances based on their frequencies (note that we only selected these tokens for examination after training, but no tokens were excluded during the training process). We defined a metric  $\gamma_{e}$  to measure the association between the token  $e$  and instance labels:
$$
\gamma_ {e} = \frac {f _ {e} ^ {+} - f _ {e} ^ {-}}{f _ {e} ^ {+} + f _ {e} ^ {-}}, \tag {28}
$$
where  $f_{e}^{+}$  and  $f_{e}^{-}$  refer to the frequencies in the positive and in the negative instances respectively. If  $\gamma_{e} \in (0.5, 1)$  and  $f_{e}^{+} > 5$ , the token will be regarded as a "positive token". If  $\gamma_{e} \in (-1, -0.5)$
and  $f_{e}^{-} > 5$ , the token will be regarded as a "negative token". If  $\gamma_{e} \in (-0.1, 0.1)$  and  $|f_{e}^{+} - f_{e}^{-}| < 5$ , the token will be regarded as a "neutral token".<sup>10</sup>
We ran the experiments using different scaling factors  $\lambda$  on the models with the scaled dot-product attention (DP) and additive attention (AD) respectively. For the former, we also investigated the performances on the models with a LSTM (DP-L) or an affine transformation layer (DP-A) as the input encoder.[11] The Adagrad optimizer (Duchi et al., 2011) was used for gradient descent. Dropout (Srivastava et al., 2014) was adopted to prevent overfitting. All the parameters were learned from scratch to avoid the influence of prior information. For the same reason, while we may be able to use pretrained word embeddings, we chose to initialize word embeddings with a uniform distribution from -0.1 to 0.1 with a dimension  $d = 100$ .
The results are shown in Table 2. For the scaled dot-product attention, which is our focus in this work, it can be observed that when the scaling factor  $\lambda$  is small (1 or 0.001), the test set results appear to be worse than the case when  $\lambda$  is set to a larger value. The optimal results may be obtained when  $\lambda$  is set to a proper value. However, setting  $\lambda$  to a very large value does not seem to have a significant impact on the performance – in this case, from Equations 1 and 2 we can see that the attention weights will be close to each other for all input tokens, leading to an effect similar to mean pooling. Results on using LSTM or the affine transformation layer as the input encoder are similar – setting a proper value for  $\lambda$  appears to be crucial.
Figure 2 shows the results for polarity scores and attention scores for the first 3 datasets, when  $\lambda$  is set to a moderate value of 10 (i.e.,  $\sqrt{d}$ ). These results are consistent with our analysis. It can be observed that generally positive tokens have positive polarity scores while negative tokens have negative polarity scores. Neutral tokens typically have polarity scores around zero. It can also be observed that both the positive and negative tokens generally have larger attention scores than the neutral tokens.
We also examined whether there would be an obvious gap between the attention scores of the polarity tokens when  $\lambda$  is large. As we can see from Figure 3b, when  $\lambda$  is set to 100, the resulting attention scores for the positive tokens are smaller than those of the neutral (and negative) tokens. In
For multi-class classification, we determined the polarity of each token based on the relative frequency of each token with respect to each label. For each token, we calculated the frequency distribution across the labels that they appear in. If the largest element of the distribution is above a given threshold, we will regard the token as a polarity one.
Example selected tokens from these datasets can be found in the supplementary material.
11 More results from these models can be found in the supplementary material. For each model, we only reported one set of the results with a random initialization as we found the patterns were similar with different initializations.
(a) SST: polarity scores  $(\lambda = 10)$
(b)IMDB:polarity scores  $(\lambda = 10)$
(c) 20News I: polarity scores  $(\lambda = 10)$
(d) SST: attention scores  $(\lambda = 10)$
(e)IMDB:attention scores  $(\lambda = 10)$
(f) 20News I: attention scores  $(\lambda = 10)$
Figure 2: Polarity (top) and attention scores (bottom). Scaled dot product attention is used with  $\lambda = 10$ .
(a) SST: polarity scores  $(\lambda = 100)$
Figure 3: Polarity (left) and attention (middle) scores for SST with scaling factor  $\lambda$  set to 100. Attention scores (right) for 20News II, with scaling factor  $\lambda$  set to 10. Scaled dot product attention is used.
(b) SST: attention scores  $(\lambda = 100)$
(c) 20News II: attention scores  $(\lambda = 10)$
this case, the resulting attention scores appear to be less interpretable. However, as we discussed above, when  $\lambda$  is very large, the attention mechanism will effectively become mean pooling (we can also see from Figure 3b that attentions scores of all tokens are now much smaller), and the overall model would be relying on the average polarity scores of the word tokens in the sentence for making prediction. Interestingly, on the other hand, as we discussed before at the end of Section 4.1, when  $\lambda$  is large, the polarity tokens will likely end up with polarity scores of large magnitudes - a fact that can also be empirically observed in Figure 3a. It is because of such healthy polarity scores acquired, the model is still able to yield good performance in this case even though the attention scores do not appear to be very interpretable.
We also tried to set a constraint on  $V^{\top}W$  by introducing a regularization term to minimize it in the learning process. We found doing so will generally encourage the attention model to produce more interpretable attention scores – for example,
even when  $\lambda$  was large, both the positive and negative tokens ended up with positive attention scores that were generally larger than those of the neutral tokens. However, empirically we did not observe a significant improvement in test performance. See the supplementary material for details.
We examined the attention scores on the 20News II dataset which consists of 3 labels. As shown in Figure 3c, polarity tokens that are strongly associated with specific labels are still likely to have larger attention scores than those of neutral tokens.
To understand whether there are similar patterns for the polarity and attention scores when using the additive attention models, we replaced the scaled dot-product attention layer with the additive attention layer and ran experiments on the SST dataset. The results are shown in Figure 4, which are similar to those of our scaled dot-product attention model.
Furthermore, we analyzed the relationship between the global attention scores and the local attention weights. We collected all the attention weights on the test set of SST for the positive, negative and
Figure 4: Polarity and attention scores when additive attention is used (on SST,  $\lambda = 10$ ).
Figure 5: Distributions of average attention weights for positive, negative and neutral tokens. The minimum, maximum, median, first and third quartiles are displayed for tokens of each type. Circles are outliers.
neutral tokens, and calculated the average weight for each token. Next we plot in Figure 5 the distribution of such average attention weights for tokens of these three types separately. As we can observe, generally, the polarity tokens are more likely to have larger attention weights than the neutral tokens. However, the positive tokens seemed to receive lower scores than the negative tokens in terms of the attention weights. This is consistent with the attention scores shown in Figure 2d: the attention scores of the positive tokens were generally lower than those of the negative tokens. Meanwhile, we could see that there were some outliers of large weights for the neutral tokens (circles that appear outside the boxes are outliers). We looked into the case, it was due to that all of the three tokens in the short instance "is this progress" had negative attention scores, and the last token "progress" somehow had a relatively larger one, making its corresponding attention weight the largest amongst the three. This can be explained by the fact that
attention weights only capture relative significance of tokens within a local context.
These empirical results support our analysis as well as our belief on the significance of the attention scores. When certain hyperparameters are properly set, the attention mechanism tends to assign larger attention scores to those tokens which have strong association with instances of a specific label. Meanwhile, the polarity scores for such tokens tend to yield large absolute values (of possibly different signs, depending on the polarity of the tokens), which will be helpful when predicting instance labels. By contrast, neutral tokens that appeared evenly across instances of different labels are likely assigned small attention scores and polarity scores, making them relatively less influential.
In this work, we focused on understanding the underlying factors that may influence the attention mechanism, and proposed to examine attention scores – a global measurement of significance of word tokens. We focused on binary classification models with dot-product attention, and analyzed through a gradient descent based learning framework the behavior of attention scores and polarity scores – another quantity that we defined and proposed to examine.
Through the analysis we found that both quantities play important roles in the learning and prediction process and examining both of them in an integrated manner allows us to better understand the underlying workings of an attention based model. Our analysis also revealed factors that may impact the interpretability of the attention mechanism, providing understandings on why the model may still be robust even under scenarios where the attention scores appear to be less interpretable. The empirical results of experiments on various real datasets supported our analysis. We also extended to and empirically examined additive attention, multi-label classification and models with an affine input layer, and observed similar behaviors.
There are some future directions that are worth exploring. Specifically, we can further examine the influence of using pre-trained word embeddings – whether similar words can help each other boost their polarity and attention scores. Moreover, we can also examine the influence of using deep contextualized input encoders such as ELMo (Peters et al., 2018) or BERT (Devlin et al., 2018).
We would like to thank the anonymous reviewers for their thoughtful and constructive comments. We also thank Rui Qiao for his help on proofreading. This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISGRP-2019-012), and Ministry of Education, Singapore, under its Academic Research Fund (AcRF) Tier 2 Programme (MOE AcRF Tier 2 Award No: MOE2017-T2-1-156). Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of National Research Foundation, Singapore and AI Singapore or the views of the Ministry of Education, Singapore.
Leila Arras, Ahmed Osman, Klaus-Robert Müller, and Wojciech Samek. 2019. Evaluating recurrent neural network explanations. Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL.
John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(Jul):2121-2159.
Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In Proceedings of NAACL.
Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. In Proceedings of NeurIPS.
Jiwei Li, Xinlei Chen, Eduard H. Hovy, and Dan Jurafsky. 2016a. Visualizing and understanding neural models in NLP. Proceedings of NAACL.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016b. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220.
Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of EMNLP.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of ACL.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Proceedings of NeurIPS.
Tomas Mikolov, Kai Chen, G.s Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Proceedings of Workshop at ICLR.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Proceedings of NeurIPS.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL.
Ning Qian. 1999. On the momentum term in gradient descent learning algorithms. Neural networks, 12(1):145-151.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI.
Damien Scieur, Vincent Roulet, Francis Bach, and Alexandre d'Aspremont. 2017. Integration methods and accelerated optimization algorithms. In Proceedings of NeurIPS.
Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of EMNLP.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research, 15(1):1929-1958.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS.
Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
Elena Voita, David Talbot, F. Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of ACL.
Sarah Wiegrefe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of EMNLP.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of NAACL.