SlowGuess/ACL_2020_OCR huggingface dataset:
Figure 4: Comparison of Median output difference on randomly permuting the attention weights in the vanilla, Diversity and Orthogonal LSTM models. The Dataset names are mentioned at the top of each figure. Colors indicate the different models as shown legend.
nal LSTMs on different tasks. We observe that the performance of Diversity LSTM is comparable to that of vanilla LSTM with accuracy values within  $-7.7\%$  to  $+6.7\%$  (relative) of the vanilla model's accuracy. However, there is a substantial decrease in the conicity values with a drop between  $70.6\%$  to  $93.2\%$  when compared to the vanilla model's conicity. Similarly, for the Orthogonal LSTM, the predictive performance is mostly comparable ex
cept for an increase in accuracy by  $39.9\%$  on bAbI 2 and a drop of  $-13.91\%$  on CNN. Similar to the Diversity LSTM, the conicity values are much lower than in the vanilla model. We also report the performance of a non-contextual model: Multilayer Perceptron (MLP) + attention in the same table. We observe that both Diversity LSTM and Orthogonal LSTM perform significantly better than the MLP model, especially in difficult tasks such as Question Answering with an average relative increase in accuracy of  $73.73\%$ . Having established that the performance of Diversity and Orthogonal LSTMs is comparable to the vanilla LSTM and significantly better than a Multilayer Perceptron model, we now show that these two models give more faithful explanations for its predictions.
5.3 Importance of Hidden Representation
We examine whether attention weights provide a useful importance ranking of hidden representations. We use the intermediate representation erasure by Serrano and Smith (2019) to evaluate an importance ranking over hidden representations. Specifically, we erase the hidden representations in the descending order of the importance (highest to lowest) until the model's decision changes. In Figure 3, we report the box plots of the fraction of hidden representations erased for a decision flip when following the ranking provided by attention weights. For reference, we also show the same plots when a random ranking is followed. In several datasets, we observe that a large fraction of the
representations have to be erased to obtain a decision flip in the vanilla LSTM model, similar to the observation by Serrano and Smith (2019). This suggests that the hidden representations in the lower end of the attention ranking do play a significant role in the vanilla LSTM model's decision-making process. Hence the usefulness of attention ranking in such models is questionable. In contrast, there is a much quicker decision flip in our Diversity and Orthogonal LSTM models. Thus, in our proposed models, the top elements of the attention ranking are able to concisely describe the model's decisions. This suggests that our attention weights provide a faithful explanation of the model's performance (as higher attention implies higher importance).
In tasks such as paraphrase detection, the model is naturally required to carefully go through the entire sentence to make a decision and thereby resulting in delayed decision flips. In the QA task, the attention ranking in the vanilla LSTM model itself achieves a quick decision flip. On further inspection, we found that this is because these models tend to attend onto answer words which are usually a span in the input passage. So, when the representations corresponding to the answer words are erased, the model can no longer accurately predict the answer resulting in a decision flip.
Following the work by (Jain and Wallace, 2019), we randomly permute the attention weights and observe the difference in the model's output. In Figure 4, we plot the median of Total Variation Distance (TVD) between the output distribution before and after the permutation for different values of maximum attention in the vanilla, Diversity and Orthogonal LSTM models. We observe that randomly permuting the attention weights in the Diversity and Orthogonal LSTM model results in significantly different outputs. However, there is little change in the vanilla LSTM model's output for several datasets suggesting that the attention weights are not so meaningful. The sensitivity of our attention weights to random permutations again suggests that they provide a more faithful explanation for the model's predictions whereas similar outputs raises several questions about the reliability of attention weights in the vanilla LSTM model.
5.4 Comparison with Rationales
For tasks with a single input sentence, we analyze how much attention is given to words in the sentence that are important for the prediction. Specifi
Dataset	Vanilla LSTM	Diversity LSTM
Rationale Attention	Rationale Length	Rationale Attention	Rationale Length
SST	0.348	0.240	0.624	0.175
IMDB	0.472	0.217	0.761	0.169
Yelp	0.438	0.173	0.574	0.160
Amazon	0.346	0.162	0.396	0.240
Anemia	0.611	0.192	0.739	0.237
Diabetes	0.742	0.458	0.825	0.354
20News	0.627	0.215	0.884	0.173
Tweets	0.284	0.225	0.764	0.306
Table 3: Mean Attention given to the generated rationales with their mean lengths (in fraction)
cally, we select a minimum subset of words in the input sentence with which the model can accurately make predictions. We then compute the total attention that is paid to these words. These set of words, also known as rationales, are obtained from an extractive rationale generator (Lei et al., 2016) that is trained using the REINFORCE algorithm (Sutton et al., 1999) to maximize the following reward:
$$
R = p _ {m o d e l} (y | \mathbf {Z}) - \alpha | | \mathbf {Z} | |
$$
where  $y$  is the ground truth class,  $Z$  is the extracted rationale,  $||Z||$  represents the length of the rationale,  $p_{model}(.)$  represents the classification model's output probability,  $\alpha$  is a hyperparameter that penalizes long rationales. With a fixed  $\alpha$ , we trained generators to extract rationales from the vanilla and Diversity LSTM models. We observed that the accuracy of predictions made from the extracted rationales was within 5% of the accuracy made from the entire sentences. In Table 3, we report the mean length (in fraction) of the rationales and the mean attention given to them in the vanilla and Diversity LSTM models. In general, we observe that the Diversity LSTM model provides much higher attention to rationales which are even often shorter than the vanilla LSTM model's rationales. On average, the Diversity LSTM model provides 53.52% (relative) more attention to rationales than the vanilla LSTM across the 8 Text classification datasets. Thus, the attention weights in the Diversity LSTM are able to better indicate words that are important for making predictions.
5.5 Comparison with attribution methods
We now examine how well our attention weights agree with attribution methods such as gradients and integrated gradients (Sundararajan et al., 2017). For every input word, we compute these attributions and normalize them to obtain a distribution over the input words. We then compute the Pearson
Pearson Correlation ↑	JS Divergence ↓
Dataset	Gradients (Mean ± Std.)	Integrated Gradients (Mean ± Std.)	Gradients (Mean ± Std.)	Integrated Gradients (Mean ± Std.)
Vanilla Diversity	Vanilla Diversity	Vanilla Diversity	Vanilla Diversity
Text Classification
SST	0.71 ± 0.21	0.83 ± 0.19	0.62 ± 0.24	0.79 ± 0.22	0.10 ± 0.04	0.08 ± 0.05	0.12 ± 0.05	0.09 ± 0.05
IMDB	0.80 ± 0.07	0.89 ± 0.04	0.68 ± 0.09	0.78 ± 0.07	0.09 ± 0.02	0.09 ± 0.01	0.13 ± 0.02	0.13 ± 0.02
Yelp	0.55 ± 0.16	0.79 ± 0.12	0.40 ± 0.19	0.79 ± 0.14	0.15 ± 0.04	0.13 ± 0.04	0.19 ± 0.05	0.19 ± 0.05
Amazon	0.43 ± 0.19	0.77 ± 0.14	0.43 ± 0.19	0.77 ± 0.14	0.17 ± 0.04	0.12 ± 0.04	0.21 ± 0.06	0.12 ± 0.04
Anemia	0.63 ± 0.12	0.72 ± 0.10	0.43 ± 0.15	0.66 ± 0.11	0.20 ± 0.04	0.19 ± 0.03	0.34 ± 0.05	0.23 ± 0.04
Diabetes	0.65 ± 0.15	0.76 ± 0.13	0.55 ± 0.14	0.69 ± 0.18	0.26 ± 0.05	0.20 ± 0.04	0.36 ± 0.04	0.24 ± 0.06
20News	0.72 ± 0.28	0.96 ± 0.08	0.65 ± 0.32	0.67 ± 0.11	0.15 ± 0.07	0.06 ± 0.04	0.21 ± 0.06	0.07 ± 0.05
Tweets	0.65 ± 0.24	0.80 ± 0.21	0.56 ± 0.25	0.74 ± 0.22	0.08 ± 0.03	0.12 ± 0.07	0.08 ± 0.04	0.15 ± 0.06
Natural Language Inference
SNLI	0.58 ± 0.33	0.51 ± 0.35	0.38 ± 0.40	0.26 ± 0.39	0.11 ± 0.07	0.10 ± 0.06	0.16 ± 0.09	0.13 ± 0.06
Paraphrase Detection
QQP	0.19 ± 0.34	0.58 ± 0.31	-0.06 ± 0.34	0.21 ± 0.36	0.15 ± 0.08	0.10 ± 0.05	0.19 ± 0.10	0.15 ± 0.06
Question Answering
Babi 1	0.56 ± 0.34	0.91 ± 0.10	0.33 ± 0.37	0.91 ± 0.10	0.33 ± 0.12	0.21 ± 0.08	0.43 ± 0.13	0.24 ± 0.08
Babi 2	0.16 ± 0.23	0.70 ± 0.13	0.05 ± 0.22	0.75 ± 0.10	0.53 ± 0.09	0.23 ± 0.06	0.58 ± 0.09	0.19 ± 0.05
Babi 3	0.39 ± 0.24	0.67 ± 0.19	-0.01 ± 0.08	0.47 ± 0.25	0.46 ± 0.08	0.37 ± 0.07	0.64 ± 0.05	0.41 ± 0.08
CNN	0.58 ± 0.25	0.75 ± 0.20	0.45 ± 0.28	0.66 ± 0.23	0.22 ± 0.07	0.17 ± 0.08	0.30 ± 0.10	0.21 ± 0.10
correlation and JS divergence between the attribution distribution and the attention distribution. We note that Kendall  $\tau$  as used by (Jain and Wallace, 2019) often results in misleading correlations because the ranking at the tail end of the distributions contributes to a significant noise. In Table 4, we report the mean and standard deviation of these Pearson correlations and JS divergence in the vanilla and Diversity LSTMs across different datasets. We observe that attention weights in Diversity LSTM better agree with gradients with an average (relative)  $64.84\%$  increase in Pearson correlation and an average (relative)  $17.18\%$  decrease in JS divergence over the vanilla LSTM across the datasets. Similar trends follow for Integrated Gradients.
5.6 Analysis by POS tags
Figure 5 shows the distribution of attention given to different POS tags across different datasets. We observe that the attention given to punctuation marks is significantly reduced from  $28.6\%$ ,  $34.0\%$  and  $23.0\%$  in the vanilla LSTM to  $3.1\%$ ,  $13.8\%$  and  $3.4\%$  in the Diversity LSTM on the Yelp, Amazon and QQP datasets respectively. In the sentiment classification task, Diversity LSTM pays greater attention to the adjectives, which usually play a crucial role in deciding the polarity of a sentence. Across the four sentiment analysis datasets, Diversity LSTM gives an average of  $49.27\%$  (relative) more attention to adjectives than the vanilla LSTM. Similarly, for the other text classification tasks where nouns play an important role, we observe higher attention to nouns.
Attention given (in percentage)
Figure 5: Distribution of cumulative attention given to different part-of-speech tags in the test dataset. Blue and Orange indicate the vanilla and Diversity LSTMs.
Table 4: Mean and standard deviation of Pearson correlation and Jensen-Shannon divergence between Attention weights and Gradients/Integrated Gradients in Vanilla and Diversity LSTM models
Dataset	Overall	Completeness	Correctness
Vanilla/Divers.	Vanilla/Divers.	Vanilla/Divers.
Yelp	27.7% / 72.3%	35.1% / 64.9%	10.5% / 89.5%
SNLI	37.8% / 62.2%	32.3% / 67.7%	38.9% / 61.1%
QQP	11.6% / 88.4%	11.8% / 88.2%	7.9% / 92.1%
bAbI 1	1.0% / 99.0%	4.2% / 95.8%	1.0% / 99.0%
Table 5: Percentage preference given to Vanilla vs Diversity model by human annotators based on 3 criteria
5.7 Human Evaluations
We conducted human evaluations to compare the extent to which attention distributions from the vanilla and Diversity LSTMs provide plausible explanations. We randomly sampled 200 data points each from the test sets of Yelp, SNLI, QQP, and bAbI1. Annotators were shown the input sentence, the attention heatmaps, and predictions made by the vanilla and Diversity LSTMs and were asked to choose the attention heatmap that better explained the model's prediction on 3 criteria 1) Overall - which heatmap is better in explaining the prediction overall 2) Completeness - which heatmap highlights all the words necessary for the prediction. 3) Correctness - which heatmap only highlights the important words and not unnecessary words. Annotators were given the choice to skip a sample in case they were unable to make a clear decision. A total of 15 in-house annotators participated in the human evaluation study. The annotators were Computer Science graduates competent in English. We had 3 annotators for each sample and the final decision was taken based on majority voting. In Table 5, we report the percentage preference given to the vanilla and Diversity LSTM models on the Yelp, SNLI, QQP, and bAbI 1 datasets; the attention distributions from Diversity LSTM significantly outperforms the attention from vanilla LSTM across all the datasets and criteria.
Our work in many ways can be seen as a continuation to the recent studies (Serrano and Smith, 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019) on the subject of interpretability of attention. Several other works (Shao et al., 2019; Martins and Astudillo, 2016; Malaviya et al., 2018; Niculae and Blondel, 2017; Maruf et al., 2019; Peters et al., 2018) focus on improving the interpretability of attention distributions by inducing sparsity. However, the extent to which sparse attention distributions actually offer faithful and plausible explanations haven't been studied in detail. Few works (Bao et al., 2018) map attention distributions to human annotated rationales. Our work on the other hand does not require any additional supervision. Work by (Guo et al., 2019) focus on developing interpretable LSTMs specifically for multivariate time series analysis. Several other works (Clark et al., 2019; Vig and Belinkov, 2019; Tenney et al., 2019; Michel et al., 2019; Jawahar et al., 2019; Tsai
et al., 2019) analyze attention distributions and attention heads learned by transformer language models. The idea of orthogonalizing representations in an LSTM have been used by (Nema et al., 2017) but they use a different diversity model in the context of improving performance of Natural Language Generation models
7 Conclusion & Future work
In this work, we have analyzed why existing attention distributions can neither provide a faithful nor a plausible explanation for the model's predictions. We showed that hidden representations learned by LSTM encoders tend to be highly similar across different timesteps, thereby affecting the interpretability of attention weights. We proposed two techniques to effectively overcome this shortcoming and showed that attention distributions in the resulting models provide more faithful and plausible explanations. As future work, we would like to extend our analysis and proposed techniques to more complex models and downstream tasks.
We would like to thank Department of Computer Science and Engineering, IIT Madras and Robert Bosch Center for Data Sciences and Artificial Intelligence, IIT Madras (RBC-DSAI) for providing us sufficient resources. We acknowledge Google for supporting Preksha Nema's contribution through their Google India Ph.D. fellowship program. We also express our gratitude to the annotators who participated in human evaluations.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.
Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay. 2018. Deriving machine attention from human rationales. In EMNLP.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP.
Chandrahas, Aditya Sharma, and Partha P. Talukdar. 2018. Towards understanding the geometry of knowledge graph embeddings. In ACL.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does bert look at? an analysis of bert's attention. ArXiv, abs/1906.04341.
Tian Guo, Tao Lin, and Nino Antulov-Fantulin. 2019. Exploring interpretable lstm neural networks over multi-variable data. In ICML.
Karl Moritz Hermann, Tomás Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. *Neural Computation*, 9:1735-1780.
Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In NAACL-HLT.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does bert learn about the structure of language? In ACL.
Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Liwei H. Lehman, Mengling Feng, Mohammad M. Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016. Mimic-iii, a freely accessible critical care database. In Scientific data.
Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2016. Rationalizing neural predictions. In EMNLP.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In ACL.
Chaitanya Malaviya, Pedro Ferreira, and André F. T. Martins. 2018. Sparse and constrained attention for neural machine translation. In ACL.
Andre F. T. Martins and Ramón Fernández Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. *ArXiv*, abs/1602.02068.
Sameen Maruf, André F. T. Martins, and Gholamreza Haffari. 2019. Selective attention for context-aware neural machine translation. In *NAACL-HLT*.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? ArXiv, abs/1905.10650.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven attention model for query-based abstractive summarization. In ACL.
Vlad Niculae and Mathieu Blondel. 2017. A regularized framework for sparse and structured neural attention. In NIPS.
Azadeh Nikfarjam, Abeed Sarker, Karen O'Connor, Rachel E. Ginn, and Graciela Gonzalez-Hernandez. 2015. Pharmacovigilance from social media: mining adverse drug reaction mentions using sequence labeling with word embedding cluster features. In JAMIA.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Ben Peters, Vlad Niculae, and Andre F. T. Martins. 2018. Interpretable structure induction via sparse attention. In BlackboxNLP@EMNLP.
Slav Petrov, Dipanjan Das, and Ryan T. McDonald. 2011. A universal part-of-speech tagset. In LREC.
Ananya Sai, Mithun Das Gupta, Mitesh M. Khapra, and Mukundhan Srinivasan. 2019. Re-evaluating adem: A deeper look at scoring dialogue responses. In AAAI.
Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In ACL.
Wenqi Shao, Tianjian Meng, Jingyu Li, Ruimao Zhang, Yudian Li, Xiaogang Wang, and Ping Luo. 2019. Ssn: Learning sparse switchable normalization via sparsestmax. In CVPR.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In ICML.
Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. In NIPS.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bertrediscoverstheclassical nlppipeline.In ACL.
Yao-Hung Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Empirical study of transformer's attention mechanism via the lens of kernel. In IJCNLP 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. ArXiv, abs/1906.04284.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Black-boxNLP@EMNLP.
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. CoRR, abs/1502.05698.
Sarah Wiegrefe and Yuval Pinter. 2019. Attention is not not explanation. ArXiv, abs/1908.04626.