SlowGuess/ACL_2020_OCR huggingface dataset:
Akash Kumar Mohankumar<sup>1</sup> Preksha Nema<sup>1,2</sup> Sharan Narasimhan<sup>1</sup>  
Mitesh M. Khapra<sup>1,2</sup> Balaji Vasan Srinivasan<sup>3</sup> Balaraman Ravindran<sup>1,2</sup>
$^{1}$ Indian Institute of Technology Madras  $^{3}$ Adobe Research  $^{2}$  Robert Bosch Center for Data Science and Artificial Intelligence, IIT Madras makashkumar99@gmail.com {preksha,miteshk,ravi}@cse.iitm.ac.in sharan.n21@gmail.com balsrini@adobe.com
Recent studies on interpretability of attention distributions have led to notions of faithful and plausible explanations for a model's predictions. Attention distributions can be considered a faithful explanation if a higher attention weight implies a greater impact on the model's prediction. They can be considered a plausible explanation if they provide a human-understandable justification for the model's predictions. In this work, we first explain why current attention mechanisms in LSTM based encoders can neither provide a faithful nor a plausible explanation of the model's predictions. We observe that in LSTM based encoders the hidden representations at different time-steps are very similar to each other (high conicity) and attention weights in these situations do not carry much meaning because even a random permutation of the attention weights does not affect the model's predictions. Based on experiments on a wide variety of tasks and datasets, we observe attention distributions often attribute the model's predictions to unimportant words such as punctuation and fail to offer a plausible explanation for the predictions. To make attention mechanisms more faithful and plausible, we propose a modified LSTM cell with a diversity-driven training objective that ensures that the hidden representations learned at different time steps are diverse. We show that the resulting attention distributions offer more transparency as they (i) provide a more precise importance ranking of the hidden states (ii) are better indicative of words important for the model's predictions (iii) correlate better with gradient-based attribution methods. Human evaluations indicate that the attention distributions learned by our model offer a plausible explanation of the model's predictions. Our code has been made publicly available at https://github.com/akashkm99/Interpretable-Attention
Question 1: What is the best way to improve my spoken English soon?
Question 2: How can I improve my English speaking ability?
Is paraphrase (Actual & Predicted): Yes
Attention Distribution
Vanilla LSTM	How can I improve my English speaking ability?
Diversity LSTM	How can I improve my English speaking ability?
Passage: Sandra went to the garden. Daniel went to the garden.
Question: Where is Sandra?
Answer (Actual & Predicted): garden
Attention Distribution:
Vanilla LSTM	Sandra went to the garden.
Daniel went to the garden
Diversity LSTM	Sandra went to the garden.
Daniel went to the garden
Table 1: Samples of Attention distributions from Vanilla and Diversity LSTM models on the Quora Question Paraphrase (QQP) & Babi 1 datasets.
Attention mechanisms (Bahdanau et al., 2014; Vaswani et al., 2017) play a very important role in neural network-based models for various Natural Language Processing (NLP) tasks. They not only improve the performance of the model but are also often used to provide insights into the working of a model. Recently, there is a growing debate on whether attention mechanisms can offer transparency to a model or not. For example, Serrano and Smith (2019) and Jain and Wallace (2019) show that high attention weights need not necessarily correspond to a higher impact on the model's predictions and hence they do not provide a faithful explanation for the model's predictions. On the other hand, Wegreffe and Pinter (2019) argues that there is still a possibility that attention distributions may provide a plausible explanation for the predictions. In other words, they might provide
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4206-4216 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
a plausible reconstruction of the model's decision making which can be understood by a human even if it is not faithful to how the model works.
In this work, we begin by analyzing why attention distributions may not faithfully explain the model's predictions. We argue that when the input representations over which an attention distribution is being computed are very similar to each other, the attention weights are not very meaningful. Since the input representations are very similar, even random permutations of the attention weights could lead to similar final context vectors. As a result, the output predictions will not change much even if the attention weights are permuted. We show that this is indeed the case for LSTM based models where the hidden states occupy a narrow cone in the latent space (i.e., the hidden representations are very close to each other). We further observe that for a wide variety of datasets, attention distributions in these models do not even provide a good plausible explanation as they pay significantly high attention to unimportant tokens such as punctuation. This is perhaps due to hidden states capturing a summary of the entire context instead of being specific to their corresponding words.
Based on these observations, we aim to build more transparent and explainable models where the attention distributions provide faithful and plausible explanations for its predictions. One intuitive way of making the attention distribution more faithful is by ensuring that the hidden representations over which the distribution is being computed are very diverse. Therefore, a random permutation of the attention weights will lead to very different context vectors. To do so, we propose an orthogonalization technique which ensures that the hidden states are farther away from each other in their spatial dimensions. We then propose a more flexible model trained with an additional objective that promotes diversity in the hidden states. Through a series of experiments using 12 datasets spanning 4 tasks, we show that our model is more transparent while achieving comparable performance to models containing vanilla LSTM based encoders. Specifically, we show that in our proposed models, attention weights (i) provide useful importance ranking of hidden states (ii) are better indicative of words that are important for the model's prediction (iii) correlate better with gradient-based feature importance methods and (iv) are sensitive to random permutations (as should indeed be the case).
We further observe that attention weights in our models, in addition to adding transparency to the model, are also more explainable i.e. more human-understandable. In Table 1, we show samples of attention distributions from a Vanilla LSTM and our proposed Diversity LSTM model. We observe that in our models, unimportant tokens such as punctuation marks receive very little attention whereas important words belonging to relevant part-of-speech tags receive greater attention (for example, adjectives in the case of sentiment classification). Human evaluation on the attention from our model shows that humans prefer the attention weights in our Diversity LSTM as providing better explanations than Vanilla LSTM in  $72.3\%$ ,  $62.2\%$ ,  $88.4\%$ ,  $99.0\%$  of the samples in Yelp, SNLI, Quora Question Paraphrase and Babi 1 datasets respectively.
2 Tasks, Dataset and Models
Our first goal is to understand why existing attention mechanisms with LSTM based encoders fail to provide faithful or plausible explanations for the model's predictions. We experiment on a variety of datasets spanning different tasks; here, we introduce these datasets and tasks and provide a brief recap of the standard LSTM+attention model used for these tasks. We consider the tasks of Binary Text classification, Natural Language Inference, Paraphrase Detection, and Question Answering. We use a total of 12 datasets, most of them being the same as the ones used in (Jain and Wallace, 2019). We divide Text classification into Sentiment Analysis and Other Text classification for convenience.
Sentiment Analysis: We use the Stanford Sentiment Treebank (SST) (Socher et al., 2013), IMDB Movie Reviews (Maas et al., 2011), Yelp and Amazon for sentiment analysis. All these datasets use binary target variable (positive /negative).
Other Text Classification: We use the Twitter ADR (Nikfarjam et al., 2015) dataset with 8K tweets where the task is to detect if a tweet describes an adverse drug reaction or not. We use a subset of the 20 Newsgroups dataset (Jain and Wallace, 2019) to classify news articles into baseball vs hockey sports categories. From MIMIC ICD9 (Johnson et al., 2016), we use 2 datasets: Anemia, to determine the type of Anemia (Chronic vs Acute) a patient is diagnosed with and Diabetes, to predict whether a patient is diagnosed with Diabetes or not.
Natural Language Inference: We consider the SNLI dataset (Bowman et al., 2015) for recogniz
ing textual entailment within sentence pairs. The SNLI dataset has three possible classification labels, viz entailment, contradiction and neutral.
Paraphrase Detection: We utilize the Quora Question Paraphrase (QQP) dataset (part of the GLUE benchmark (Wang et al., 2018)) with pairs of questions labeled as paraphrased or not. We split the training set into  $90:10$  training and validation; and use the original dev set as our test set.
Question Answering: We made use of all three QA tasks from the  $bAbI$  dataset (Weston et al., 2015). The tasks consist of answering questions that would require one, two or three supporting statements from the context. The answers are a span in the context. We then use the CNN News Articles dataset (Hermann et al., 2015) consisting of 90k articles with an average of three questions per article along with their corresponding answers.
2.1 LSTM Model with Attention
Of the above tasks, the text classification tasks require making predictions from a single input sequence (of words) whereas the remaining tasks use pairs of sequences as input. For tasks containing two input sequences, we encode both the sequences  $\mathbf{P} = \{w_1^p,\dots ,w_m^p\}$  and  $\mathbf{Q} = \{w_1^q,\dots ,w_n^q\}$  by passing their word embedding through a LSTM encoder (Hochreiter and Schmidhuber, 1997),
$$
\begin{array}{l} \mathbf {h} _ {t} ^ {p} = \operatorname {L S T M} _ {P} \left(e \left(w _ {t} ^ {p}\right), \mathbf {h} _ {t - 1} ^ {p}\right) \forall t \in [ 1, m ], \\ \mathbf {h} _ {t} ^ {q} = \mathrm {L S T M} _ {Q} \big (e (w _ {t} ^ {q}), \mathbf {h} _ {t - 1} ^ {q} \big) \forall t \in [ 1, n ], \\ \end{array}
$$
where  $e(w)$  represents the word embedding for the word  $w$ . We attend to the intermediate representations of  $\mathbf{P}$ ,  $\mathbf{H}^p = \{\mathbf{h}_1^p, \dots, \mathbf{h}_m^p\} \in \mathbb{R}^{m \times d}$  using the last hidden state  $\mathbf{h}_n^q \in \mathbb{R}^d$  as the query, using the attention mechanism (Bahdanau et al., 2014),
$$
\begin{array}{l} \tilde {\alpha} _ {t} = \mathbf {v} ^ {T} \operatorname {t a n h} \left(\mathbf {W} _ {1} \mathbf {h} _ {t} ^ {p} + \mathbf {W} _ {2} \mathbf {h} _ {n} ^ {q} + \mathbf {b}\right) \forall t \in [ 1, m ] \\ \alpha_ {t} = \operatorname {s o f t m a x} \left(\tilde {\alpha} _ {t}\right) \\ \mathbf {c} _ {\alpha} = \sum_ {t = 1} ^ {m} \alpha_ {t} \mathbf {h} ^ {p} _ {t} \\ \end{array}
$$
where  $\mathbf{W}_1\in \mathbb{R}^{d_1\times d}$ $\mathbf{W}_2\in \mathbb{R}^{d_1\times d}$ $\mathbf{b}\in \mathbb{R}^{d_1}$  and  $\mathbf{v}\in \mathbb{R}^{d_1}$  are learnable parameters. Finally, we use the attended context vector  $\mathbf{c}_{\alpha}$  to make a prediction  $\hat{y} = \mathrm{softmax}(\mathbf{W}_o\mathbf{c}_{\alpha})$
For tasks with a single input sequence, we use a single LSTM to encode the sequence, followed by an attention mechanism (without query) and a final output projection layer.
3 Analyzing Attention Mechanisms
Here, we first investigate the question - Why Attention distributions may not provide a faithful explanation for the model's predictions? We later examine whether Attention distributions can provide a plausible explanation for the model's predictions, not necessarily faithful.
3.1 Similarity Measures
We begin with defining similarity measures in a vector space for ease of analysis. We measure the similarity between a set of vectors  $\mathbf{V} = \{\mathbf{v}_1,\dots ,\mathbf{v}_m\}$  using the conicity measure (Chandrahas et al., 2018; Sai et al., 2019) by first computing a vector  $\mathbf{v}_i$  'alignment to mean' (ATM),
$$
\operatorname {A T M} (\mathbf {v} _ {i}, \mathbf {V}) = \operatorname {c o s i n e} (\mathbf {v} _ {i}, \frac {1}{m} \sum_ {j = 1} ^ {m} \mathbf {v} _ {j})
$$
Conicity is defined as the mean of ATM for all vectors  $\mathbf{v}_i\in \mathbf{V}$  ..
$$
\operatorname {c o n i c i t y} (\mathbf {V}) = \frac {1}{m} \sum_ {i = 1} ^ {m} \operatorname {A T M} (\mathbf {v} _ {i}, \mathbf {V})
$$
A high value of conicity indicates that all the vectors are closely aligned with their mean  $i.e$  they lie in a narrow cone centered at origin.
3.2 Attention Mechanisms
As mentioned earlier, attention mechanisms learn a weighting distribution over hidden states  $\mathbf{H} = \{\mathbf{h}_1,\dots ,\mathbf{h}_n\}$  using a scoring function  $f$  such as (Bahdanau et al., 2014) to obtain an attended context vector  $\mathbf{c}_{\alpha}$ .
$$
\mathbf {c} _ {\alpha} = \sum_ {t = 1} ^ {n} \alpha_ {t} \mathbf {h} _ {t}; \alpha_ {t} = \operatorname {s o f t m a x} (f (\mathbf {h} _ {t}, \mathbf {h} _ {\text {q u e r y}}))
$$
The attended context vector is a convex combination of the hidden states which means it will lie within the cone spanned by the hidden states. When the hidden states are highly similar to each other (high conicity), even diverse sets of attention distributions would produce very similar attended context vector  $\mathbf{c}_{\alpha}$  as they will always lie within a narrow cone. This could result in outputs  $\hat{y} = \mathrm{softmax}(\mathbf{W}_o\mathbf{c}_{\alpha})$  with very little difference. In other words, when there is a higher conicity in hidden states, the model could produce the same prediction for several diverse sets of attention weights. In such cases, one cannot reliably say that high
Figure 1: Left: high conicity of hidden states results in similar attended context vectors. Right: low conicity of hidden states results in very different context vectors
attention weights on certain input components led the model to its prediction. Later on, in section 5.3, we show that when using vanilla LSTM encoders where there is higher conicity in hidden states, even when we randomly permute the attention weights, the model output does not change much.
3.3 Conicity of LSTMs Hidden States
We now analyze if the hidden states learned by an LSTM encoder do actually have high conicity. In Table 2, we report the average conicity of hidden states learned by an LSTM encoder for various tasks and datasets. For reference, we also compute the average conicity obtained by vectors that are uniformly distributed with respect to direction (isotropic) in the same hidden space. We observe that across all the datasets the hidden states are consistently aligned with each other with conicity values ranging between 0.43 to 0.77. In contrast, when there was no dependence between the vectors, the conicity values were much lower with the vectors even being almost orthogonal to its mean in several cases ( $\sim 89^{\circ}$  in Diabetes Anemia datasets). The existence of high conicity in the learned hidden states of an LSTM encoder is one of the potential reasons why the attention weights in these models are not always faithful to its predictions (as even random permutations of the attention weights will result in similar context vectors,  $\mathbf{c}_{\alpha}$ ).
3.4 Attention by POS Tags
We now examine whether attention distributions can provide a plausible explanation for the model's predictions even if it is not faithful. Intuitively, a plausible explanation should ignore unimportant tokens such as punctuation marks and focus on words relevant for the specific task. To examine this, we categorize words in the input sentence by its universal part-of-speech (POS) tag (Petrov et al., 2011) and cumulate attention given to each POS tag over the entire test set. Surprisingly, we
Figure 2: Orthogonal LSTM: Hidden state at a timestep is orthogonal to the mean of previous hidden states
find that in several datasets, a significant amount of attention is given to punctuations. On the Yahoo, Amazon and QQP datasets, attention mechanisms pay  $28.6\%$ ,  $34.0\%$  and  $23.0\%$  of its total attention to punctuations. Notably, punctuations only constitute  $11.0\%$ ,  $10.5\%$  and  $11.6\%$  of the total tokens in the respective datasets signifying that learned attention distributions pay substantially greater attention to punctuations than even an uniform distribution. This raises questions on the extent to which attention distributions provide plausible explanations as they attribute model's predictions to tokens that are linguistically insignificant to the context.
One of the potential reasons why the attention distributions are misaligned is that the hidden states might capture a summary of the entire context instead of being specific to their corresponding words as suggested by the high conicity. We later show that attention distributions in our models with low conicity value tend to ignore punctuation marks.
4 Orthogonal and Diversity LSTM
Based on our previous argument that high conicity of hidden states affect the transparency and explainability of attention models, we propose 2 strategies to obtain reduced similarity in hidden states.
4.1 Orthogonalization
Here, we explicitly ensure low conicity exists between hidden states of an LSTM encoder by orthogonalizing the hidden state at time  $t$  with the mean of previous states as illustrated in Figure 2. We use the following set of update equations:
$$
\mathbf {f} _ {t} = \sigma \left(\mathbf {W} _ {f} \mathbf {x} _ {t} + \mathbf {U} _ {f} \mathbf {h} _ {t - 1} + \mathbf {b} _ {f}\right)
$$
$$
\mathbf {i} _ {t} = \sigma \left(\mathbf {W} _ {i} \mathbf {x} _ {t} + \mathbf {U} _ {i} \mathbf {h} _ {t - 1} + \mathbf {b} _ {i}\right)
$$
$$
\mathbf {o} _ {t} = \sigma \left(\mathbf {W} _ {o} \mathbf {x} _ {t} + \mathbf {U} _ {o} \mathbf {h} _ {t - 1} + \mathbf {b} _ {o}\right)
$$
$$
\hat {\mathbf {c}} _ {t} = \tanh  (\mathbf {W} _ {c} \mathbf {x} _ {t} + \mathbf {U} _ {c} \mathbf {h} _ {t - 1} + \mathbf {b} _ {c})
$$
$$
\mathbf {c} _ {t} = \mathbf {f} _ {t} \odot \mathbf {c} _ {t - 1} + \mathbf {i} _ {t} \odot \hat {\mathbf {c}} _ {t}
$$
4209
$$
\hat {\mathbf {h}} _ {t} = \mathbf {o} _ {t} \odot \tanh  (\mathbf {c} _ {t})
$$
$$
\overline {{\mathbf {h}}} _ {t} = \sum_ {i = 1} ^ {t - 1} \mathbf {h} _ {i} \tag {1}
$$
$$
\mathbf {h} _ {\mathbf {t}} = \hat {\mathbf {h}} _ {t} - \frac {\hat {\mathbf {h}} _ {t} ^ {T} \overline {{\mathbf {h}}} _ {t}}{\overline {{\mathbf {h}}} _ {t} ^ {T} \overline {{\mathbf {h}}} _ {t}} \overline {{\mathbf {h}}} _ {t} \tag {2}
$$
where  $\mathbf{W}_f, \mathbf{W}_i, \mathbf{W}_o, \mathbf{W}_c \in \mathbb{R}^{d_2 \times d_1}$ ,  $\mathbf{U}_f, \mathbf{U}_i, \mathbf{U}_o, \mathbf{U}_c \in \mathbb{R}^{d_2 \times d_2}$ ,  $\mathbf{b}_f, \mathbf{b}_i, \mathbf{b}_o, \mathbf{b}_c \in \mathbb{R}^{d_2}$ ,  $d_1$  and  $d_2$  are the input and hidden dimensions respectively. The key difference from a vanilla LSTM is in the last 2 equations where we subtract the hidden state vector's  $\hat{\mathbf{h}}_t$  component along the mean  $\overline{\mathbf{h}}_t$  of the previous states.
4.2 Diversity Driven Training
The above model imposes a hard orthogonality constraint between the hidden states and the previous states' mean. We also propose a more flexible approach where the model is jointly trained to maximize the log-likelihood of the training data and minimize the conicity of hidden states,
$$
L (\theta) = - p _ {m o d e l} (y | \mathbf {P}, \mathbf {Q}, \theta) + \lambda \operatorname {c o n i c i t y} (\mathbf {H} ^ {P})
$$
where  $y$  is the ground truth class,  $\mathbf{P}$  and  $\mathbf{Q}$  are the input sentences,  $\mathbf{H}^P = \{\mathbf{h}_1^p,\dots ,\mathbf{h}_m^p\} \in \mathbb{R}^{m\times d}$  contains all the hidden states of the LSTM,  $\theta$  is a collection of the model parameters and  $p_{model}(.)$  represents the model's output probability.  $\lambda$  is a hyperparameter that controls the weight given to diversity in hidden states during training.
5 Analysis of the model
We now analyse the proposed models by performing experiments using the tasks and datasets described earlier. Through these experiments we establish that (i) the proposed models perform comparably to vanilla LSTMs (Sec. 5.2) (ii) the attention distributions in the proposed models provide a faithful explanation for the model's predictions (Secs. 5.3 to 5.5) and (iii) the attention distributions are more explainable and align better with a human's interpretation of the model's prediction (Secs. 5.6, 5.7). Throughout this section we will compare the following three models:
1. Vanilla LSTM: The model described in section 2.1 which uses the vanilla LSTM.
2. Diversity LSTM: The model described in section 2.1 with the vanilla LSTM but trained with the diversity objective described in section 4.2.
3. Orthogonal LSTM: The model described in
Figure 3: Box plots of fraction of hidden representations removed for a decision flip. Dataset and models are mentioned at the top and bottom of figures. Blue and Yellow indicate the attention and random ranking.
section 2.1 except that the vanilla LSTM is replaced by the orthogonal LSTM described in section 4.1.
For all datasets except bAbi, we either use pretrained Glove (Pennington et al., 2014) or fastText (Mikolov et al., 2018) word embeddings with 300 dimensions. For the bAbi dataset, we learn 50 dimensional word embeddings from scratch during training. We use a 1-layered LSTM as the encoder with hidden size of 128 for bAbi and 256 for the other datasets. For the diversity weight  $\lambda$ , we use a value of 0.1 for SNLI, 0.2 for CNN, and 0.5 for the remaining datasets. We use Adam optimizer with a learning rate of 0.001 and select the best model based on accuracy on the validation split. All the subsequent analysis are performed on the test split.
5.2 Empirical evaluation
Our main goal is to show that our proposed models provide more faithful and plausible explanations for their predictions. However, before we go there we need to show that the predictive performance of our models is comparable to that of a vanilla LSTM model and significantly better than non-contextual models. In other words, we show that we do not compromise on performance to gain transparency and explainability. We report the performance of our model on the tasks and datasets described in section 2. In Table 2, we report the accuracy and conicity values of vanilla, Diversity and Orthogo
Dataset	LSTM	Diversity LSTM	Orthogonal LSTM	Random	MLP
Accuracy	Conicity	Accuracy	Conicity	Accuracy	Conicity	Conicity	Accuracy
Binary Classification
SST	81.79	0.68	79.95	0.20	80.05	0.28	0.25	80.05
IMDB	89.49	0.69	88.54	0.08	88.71	0.18	0.08	88.29
Yelp	95.60	0.53	95.40	0.06	96.00	0.18	0.14	92.85
Amazon	93.73	0.50	92.90	0.05	93.04	0.16	0.13	87.88
Anemia	88.54	0.46	90.09	0.09	90.17	0.12	0.02	88.27
Diabetes	92.31	0.61	91.99	0.08	87.05	0.12	0.02	85.39
20News	93.55	0.77	91.03	0.15	92.15	0.23	0.13	87.68
Tweets	87.02	0.77	87.04	0.24	83.20	0.27	0.24	80.60
Natural Language Inference
SNLI	78.23	0.56	76.96	0.12	76.46	0.27	0.27	75.35
Paraphrase Detection
QQP	78.74	0.59	78.40	0.04	78.61	0.33	0.30	77.78
Question Answering
bAbI 1	99.10	0.56	100.00	0.07	99.90	0.22	0.19	42.00
bAbI 2	40.10	0.48	40.20	0.05	56.10	0.21	0.12	33.20
bAbI 3	47.70	0.43	50.90	0.10	51.20	0.12	0.07	31.60
CNN	63.07	0.45	58.19	0.06	54.30	0.07	0.04	37.40
Table 2: Accuracy and conicity of Vanilla, Diversity and Orthogonal LSTM across different datasets. Accuracy of a Multilayered Perceptron (MLP) model and conicity of vectors uniformly distributed with respect to direction is also reported for reference.