SlowGuess/ACL_2020_OCR huggingface dataset:
Xiaobing Sun and Wei Lu
StatNLP Research Group
xiaobing_sun@mymail.sutd.edu.sg, luwei@sutd.edu.sg
Attention has been proven successful in many natural language processing (NLP) tasks. Recently, many researchers started to investigate the interpretability of attention on NLP tasks. Many existing approaches focused on examining whether the local attention weights could reflect the importance of input representations. In this work, we present a study on understanding the internal mechanism of attention by looking into the gradient update process, checking its behavior when approaching a local minimum during training. We propose to analyze for each word token the following two quantities: its polarity score and its attention score, where the latter is a global assessment on the token's significance. We discuss conditions under which the attention mechanism may become more (or less) interpretable, and show how the interplay between the two quantities may impact the model performance.
Attention mechanism (Bahdanau et al., 2015) has been used as an important component across a wide range of NLP models. Typically, an attention layer produces a distribution over input representations to be attended to. Such a distribution is then used for constructing a weighted combination of the inputs, which will then be employed by certain downstream modules.
Recently, several research efforts on investigating the interpretability of attention on tasks such as text classification, question answering, and natural language inference (Jain and Wallace, 2019; Wegreffe and Pinter, 2019; Arras et al., 2019) have been conducted. One of their important arguments was whether the attention distribution could adequately reflect the significance of inputs. To answer this question, they designed a series of metrics and
conducted corresponding experiments. In their approaches, they were mainly observing how the attention may impact the outputs on the pre-trained models by changing some elements in the inputs. While such approaches have resulted in interesting findings, the attention mechanism itself remains a black box to us – it is still largely unclear what are the underlying factors that may have an impact on the attention mechanism.
When analyzing the results of a typical model with attention on the text classification tasks, we noticed that in some instances, many of the word tokens with large attention weights were adjectives or adverbs which conveyed explicit signals on the underlying class label. On the other hand, in some other instances, we also noticed that such useful words may not always be able to receive significant attention weights, especially under certain configurations of hyperparameters, making the attention mechanism less interpretable.
Such observations lead to several important questions. First, the attention weight for a word token appears to be the relative measurement to its significance, and is largely local and instance specific. Would there be an instance-independent quantity to assess the corpus-level importance of a word token? And if so, what role would such a quantity play in terms of interpreting the overall attention mechanism? Second, when the attention mechanism appears to be less interpretable, how would the underlying model be affected in terms of performance?
In this work, we focus on answering the above questions. We argue that the attention scores (rather than attention weights) are able to capture the global, absolute importance of word tokens within a corpus. We present a study to figure out the underlying factors that may influence such attention scores under a simple neural classification model. Inspired by Qian (1999), we analyzed the gradients as well as the updates of intermediate variables in the process of gradient descent, and
Supplementary material and code at https://github.com/richardsun-voyager/UAFTC
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3418-3428 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
found that there exist some implicit trends on the intermediate variables related to attention: the degree of association between a word token and the class label may impact their attention scores. We argue that when certain hyperparameters are properly set, tokens with strong polarity – high degree of association with specific labels, would likely end up with large attention scores, making them more likely to receive large attention weights in a particular sentence. While in such scenarios, the attention mechanism would appear to be more interpretable, we also discuss scenarios where the attention weights may become less interpretable, and show how the polarity scores, another important token-level quantity, will play their roles in the overall model in terms of contributing towards the model performance.
Research on interpretability of neural models has received significant attention recently. One approach was using visualization to explore patterns that exist in the intermediate representations of neural networks. Simonyan et al. (2013) visualized the image-specific class saliency on image classification tasks using learnt ConvNets, and displayed the features captured by the neural networks. Li et al. (2016a,b) proposed visualization methods to look into the neural representations of the embeddings from the local composition, concessive sentences, clause composition, as well as the saliency of phrases and sentences, and illustrated patterns based on the visualizations. An erasure method was also adopted to validate the importance of different dimensions and words. Vig and Belinkov (2019) analyzed the attention structure on the Transformer (Vaswani et al., 2017) language model as well as GPT-2 (Radford et al., 2019) pre-trained model.
Another approach to understanding neural approaches is to conduct theoretical analysis to investigate the underlying explanations of neural models. One example is the work of Levy and Goldberg (2014), which regarded the word embedding learning task as an optimization problem, and found that the training process of the skip-gram model (Mikolov et al., 2013a,b) can be explained as implicit factorization of a shifted positive PMI (pointwise mutual information) matrix.
Recently, several research efforts have focused on the interpretability of the attention mechanism. Jain and Wallace (2019) raised the question on the explainability of feature importance as captured by the attention mechanism. They found the attention weights may not always be consistent with
Figure 1: Classification architecture with attention
the feature importance from the human perspective in tasks such as text classification and question answering. Serrano and Smith (2019) also carried out an analysis on the interpretability of the attention mechanism, with a focus on the text classification task. They conducted their study in a cautious way with respect to defining interpretability and the research scope. The paper concluded that the attention weights are noisy predictors of importance, but should not be regarded as justification for decisions. Wiegreffe and Pinter (2019) suggested that the notion of explanation needs to be clearly defined, and the study of the explanation requires taking all components of a model into account. Their results indicated that prior work could not disprove the usefulness of attention mechanisms with respect to explainability. Moreover, Michel et al. (2019) and Voita et al. (2019) examined the multi-head self-attention mechanism on Transformer-based models, particularly the roles played by the heads.
Our work and findings are largely consistent with such findings reported in the literature. We believe there are many factors involved when understanding the attention mechanism. Inspired by Qian (1999), which investigated the internal mechanism of gradient descent, in this work we focus on understanding attention's internal mechanism.
3 Classification Model with Attention
We consider the task of text classification, with a specific focus on binary classification. The architecture of the model is depicted in Figure 1.
There are various attention mechanisms introduced in the field (Luong et al., 2015). Two commonly used mechanisms are the additive attention (Bahdanau et al., 2015) and scaled dot-product attention (Vaswani et al., 2017). In this work, we will largely focus our analysis on the latter approach (but we will also touch the former approach later).
$^{2}$ Extending to multi-class classification is possible. See the supplementary material for detailed analysis and discussion.
Consider an input token sequence of length  $n$ :  $x = e_1, e_2, \ldots, e_n$ , where  $e_j$  is the  $j$ -th input token whose representation before the attention layer is  $h_j \in \mathbb{R}^d$ . The attention score for the  $j$ -th token is:
$$
a _ {j} = \frac {\boldsymbol {h} _ {j} ^ {\top} \boldsymbol {V}}{\lambda}, \tag {1}
$$
where the hyperparameter  $\lambda$  is the scaling factor (typically set to a large value, e.g.,  $\sqrt{d}$  is often used in the literature (Vaswani et al., 2017)), and  $V\in \mathbb{R}^d$  is the context vector that can be viewed as a fixed query asking for the "most informative word" from the input sequence (Yang et al., 2016). The token representation  $h_j$  can be the word embedding, or the output of an encoder.
The corresponding attention weight would be:
$$
\alpha_ {j} = \frac {\exp (a _ {j})}{\sum_ {j ^ {\prime}} \exp (a _ {j ^ {\prime}})} . \tag {2}
$$
The complete input sequence is represented as:
$$
\boldsymbol {h} = \sum_ {j} \alpha_ {j} \boldsymbol {h} _ {j}, \tag {3}
$$
and the output of the linear layer is:
$$
s = \boldsymbol {h} ^ {\top} \boldsymbol {W}, \tag {4}
$$
which we call instance-level polarity score of the input sequence. Here,  $\mathbf{W} \in \mathbb{R}^d$  is the weight vector for the linear layer.
When we make predictions, if the resulting polarity score  $s$  is positive, the corresponding input sequence will be classified as positive (i.e.,  $y = +1$ , where  $y$  is the output label). Otherwise, it will be classified as negative (i.e.,  $y = -1$ ).
During training, assume we have a training set  $\mathcal{D} = \{(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),\ldots ,(x^{(m)},y^{(m)})\}$  with  $m$  labeled instances. Our overall loss is:
$$
\ell = \frac {1}{m} \sum_ {t = 1} ^ {m} \ell^ {(t)} = - \frac {1}{m} \sum_ {t = 1} ^ {m} \log \left(\sigma \left(y ^ {(t)} s ^ {(t)}\right)\right). \tag {5}
$$
where  $y^{(t)}$  and  $s^{(t)}$  are the gold output label and the instance-level polarity score for the  $t$ -th instance respectively, and  $\sigma$  is the sigmoid function.
The instance-level polarity score  $s$  can also be written as:
$$
s = \sum_ {j} \alpha_ {j} \boldsymbol {h} _ {j} ^ {\top} \boldsymbol {W} = \sum_ {j} \alpha_ {j} s _ {j}. \tag {6}
$$
Here, we have introduced the token-level polarity score  $s_j$  for the input token representation  $h_j$ :
$$
s _ {j} = \boldsymbol {h} _ {j} ^ {\top} \boldsymbol {W}. \tag {7}
$$
From here we can observe that the instance-level polarity score of the input sequence can be interpreted as the weighted sum of the token-level polarity scores, where the weights are given by the attention weights  $(\alpha_{j}$  for  $h_j)$ . Such attention weights measure the relative importance of the token within a specific input sequence.
On the other hand, the attention score  $a_{j}$  captures the absolute importance of the token. We believe such absolute measurements to the significance of words may be playing a more crucial role (than attention weights) when understanding the attention mechanism. Thus, unlike many previous research efforts, we will instead focus on the understanding of attention scores in this work.
In this paper, we will mainly investigate a simple neural model where  $h_j = e_j$ . Here  $e_j$  is the word embedding for the  $j$ -th input token. In other words, we assume the word embeddings are used as the inputs to the attention layer. Detailed discussions on other assumptions on  $h_j$  can be found in the supplementary material.
We conduct some analysis in this section to understand how the attention mechanism works for the task of text classification. First, let us consider the following 3 different types of tokens:
- positive tokens: tokens that frequently appear in positive training instances only,
- negative tokens: tokens that frequently appear in negative training instances only, and
- neutral tokens: tokens that appear evenly across both positive and negative training instances.
We also call the first two types of tokens polarity tokens. For the ease of analysis and discussion, we assume each token belongs to either of these 3 types, and we assume the dataset is balanced and symmetric<sup>3</sup>. While some of these assumptions may seem strong, having them would significantly simplify our analysis. As we will see later in experiments, even though some of the above assumptions do not hold in some real datasets, our findings are still valid in practice.
The gradient descent algorithm that minimizes a loss  $\ell$  could be interpreted as the integration of
3In other words, if we flip the signs of the  $y$  labels for all documents in the training set, we arrive at exactly the same training set (under a particular mapping between tokens).
the gradient flow equation using Euler's Method (Scieur et al., 2017; Qian, 1999), written as:
$$
\frac {d \boldsymbol {z} (\tau)}{d \tau} = - \nabla \ell (\boldsymbol {z} (\tau)), \boldsymbol {z} (0) = \boldsymbol {z} _ {0}, \tag {8}
$$
where  $z$  is the parameter vector, and  $z_0$  is its initialization, and  $\tau$  is the time step. We assume that all parameters have initializations, and will omit such initializations in the subsequent differential equations. We will not seek to solve the differential equations directly but to find out whether there exist some trends and patterns for certain variables during training.
4.1 Polarity Score
Consider the token  $e$  in the vocabulary whose vector representation is  $e$ . Let us have an analysis on the polarity score  $s_e$  for the token  $e$ . This token may appear somewhere in the training set. We write  $e_j^{(t)} \equiv e$  if and only if this token  $e$  appears as the  $j$ -th token in the  $t$ -th instance.
Gradient update iteration will be represented as:
$$
\frac {d s _ {e} (\tau)}{d \tau} = \left(\frac {d \boldsymbol {e} (\tau)}{d \tau}\right) ^ {\top} \boldsymbol {W} (\tau) + \boldsymbol {e} ^ {\top} (\tau) \frac {d \boldsymbol {W} (\tau)}{d \tau}, \tag {9}
$$
where  $W(\tau)$  is the linear layer weight vector at the time  $\tau$ . Its update can be represented by another ordinary differential equation:
$$
\frac {d \boldsymbol {W} (\tau)}{d \tau} = - \frac {\partial \ell}{\partial \boldsymbol {W}} (\tau), \tag {10}
$$
$$
\frac {d \boldsymbol {e} (\tau)}{d \tau} = - \frac {\partial \ell}{\partial \boldsymbol {e}} (\tau). \tag {11}
$$
For simplicity, we will omit the time step  $\tau$  in the equations. The derivative of the token level polarity score will be written as:
$$
\frac {d s _ {e}}{d \tau} = \underbrace {- \left(\frac {\partial \ell}{\partial e}\right) ^ {\top} \boldsymbol {W}} _ {\Delta s _ {e} ^ {\prime}} + \underbrace {\left(- \boldsymbol {e} ^ {\top} \frac {\partial \ell}{\partial \boldsymbol {W}}\right)} _ {\Delta s _ {e} ^ {\prime \prime}}. \tag {12}
$$
The two partial derivatives can be calculated as:
$$
\frac {\partial \ell}{\partial \boldsymbol {e}} = - \frac {1}{m} \sum_ {(t, j): e _ {j} ^ {(t)} \equiv e} y ^ {(t)} \beta^ {(t)} \alpha_ {j} ^ {(t)} \left[ \frac {\boldsymbol {V} (\boldsymbol {e} - \boldsymbol {h} ^ {(t)}) ^ {\top}}{\lambda} + \mathbf {I} \right] \boldsymbol {W}, \tag {13}
$$
$$
\frac {\partial \ell}{\partial \boldsymbol {W}} = - \frac {1}{m} \sum_ {t = 1} ^ {m} y ^ {(t)} \beta^ {(t)} \boldsymbol {h} ^ {(t)}, \tag {14}
$$
where  $(t,j):e_j^{(t)}\equiv e$  means we are selecting such tokens from the  $t$  -th instance at the  $j$  -th position that are exactly  $e$  , and  $\alpha_{j}^{(t)}$  is the attention weight for that  $j$  -th token in the selected  $t$  -th instance. The vector  $\pmb{h}^{(t)}$  is the representation of the  $t$  -th instance, and  $\beta^{(t)}$  is defined as  $\beta (t) = 1 - \sigma (y^{(t)}s^{(t)})$
The first term in Equation 12 can be written as:
$$
\begin{array}{l} \Delta s _ {e} ^ {\prime} = \frac {1}{m} \sum_ {(t, j): e _ {j} ^ {(t)} \equiv e} y ^ {(t)} \beta^ {(t)} \alpha_ {j} ^ {(t)} \frac {\left(s _ {e} - s ^ {(t)}\right)}{\lambda} \boldsymbol {V} ^ {\top} \boldsymbol {W} \\ + \frac {1}{m} \| \boldsymbol {W} \| _ {2} ^ {2} \sum_ {(t, j): e _ {j} ^ {(t)} \equiv e} y ^ {(t)} \beta^ {(t)} \alpha_ {j} ^ {(t)}. \tag {15} \\ \end{array}
$$
The sign of the second term above depends on:
$$
\pi (e) = \sum_ {(t, j): e _ {j} ^ {(t)} \equiv e} y ^ {(t)} \beta^ {(t)} \alpha_ {j} ^ {(t)}. \tag {16}
$$
This term has the following property: it is positive if  $e$  is a positive token, negative if  $e$  is negative, and close to 0 if  $e$  is neutral.
The second term in Equation 12 is:
$$
\begin{array}{l} \Delta s _ {e} ^ {\prime \prime} = \frac {1}{m} \sum_ {t = 1} ^ {m} y ^ {(t)} \beta^ {(t)} e ^ {\top} h ^ {(t)} \\ = \frac {1}{m} \sum_ {t = 1} ^ {m} y ^ {(t)} \beta^ {(t)} \sum_ {j} \alpha_ {j} ^ {(t)} e ^ {\top} e _ {j} ^ {(t)} \\ = \frac {1}{m} \sum_ {(t, j)} y ^ {(t)} \beta^ {(t)} \alpha_ {j} ^ {(t)} e ^ {\top} e _ {j} ^ {(t)}. \tag {17} \\ \end{array}
$$
Equation 17 involves dot-products between embeddings. During training, certain trends and patterns will be developed for such dot-products. Near a local minimum, we can show that it is desirable to have  $e_i^\top e_j > 0$  when  $e_i$  and  $e_j$  are both positive tokens or both negative tokens, and  $e_i^\top e_j < 0$  when one is a positive token and the other is a negative token. More details and analysis on the desirability of these properties can be found in the supplementary material.
Now let us look at the last term in Equation 17. This term can be re-written as:
$$
\begin{array}{l} \frac {1}{m} \sum_ {(t, j): y ^ {(t)} = + 1} \beta^ {(t)} \alpha_ {j} ^ {(t)} \left(\boldsymbol {e} ^ {\top} \boldsymbol {e} _ {j} ^ {(t)}\right) \\ + \frac {1}{m} \sum_ {(t, j): y ^ {(t)} = - 1} \beta^ {(t)} \alpha_ {j} ^ {(t)} \left(- e ^ {\top} e _ {j} ^ {(t)}\right). \tag {18} \\ \end{array}
$$
where we split the term into two based on the polarity of the training instances.
4See the supplementary material for details.
In the first term, each  $e_j$  token would be either a positive or a neutral token; in the second term, each  $e_j$  would be either a negative or a neutral token, and again under the assumption on the dataset, all the terms involving neutral  $e_j$  tokens would roughly sum to a value close to 0 (regardless of  $e$ ). So we may assume there are no neutral  $e_j$  tokens. Now, if  $e$  is a positive token, we can see it is desirable for both terms to be positive. If  $e$  is negative, it is desirable for both terms to be negative. If  $e$  is neutral, likely this term is close to 0.
Overall, the update of  $s_e$  is:
$$
\begin{array}{l} \frac {d s _ {e}}{d \tau} = \underbrace {\frac {1}{m} \left(\boldsymbol {V} ^ {\top} \boldsymbol {W} / \lambda\right) \rho (e)} _ {(A)} \\ + \underbrace {\frac {1}{m} | | \boldsymbol {W} | | _ {2} ^ {2} \pi (e)} _ {(B)} \\ + \underbrace {\frac {1}{m} \sum_ {(t , j)} y ^ {(t)} \beta^ {(t)} \alpha_ {j} ^ {(t)} e ^ {\top} e _ {j} ^ {(t)}} _ {(C)}, \tag {19} \\ \end{array}
$$
$$
\rho (e) = \sum_ {(t, j): e _ {j} ^ {(t)} \equiv e} y ^ {(t)} \beta^ {(t)} \alpha_ {j} ^ {(t)} \left(s _ {e} - s ^ {(t)}\right). \tag {20}
$$
Under the assumption that  $\mathbf{V}^{\top}\mathbf{W} / \lambda$  is reasonably small (for example, we may set  $\lambda$  to an appropriate value, which is reasonably large), we have  $A\approx 0$ . We then have the following results:
- For positive tokens, we have  $B > 0$  and  $C > 0$ . The corresponding polarity scores will likely increase after each update when approaching the local minimum, and may end up with relatively large positive polarity scores eventually.
- For negative tokens, we have  $B < 0$  and  $C < 0$ . The corresponding polarity scores will likely decrease after each update when approaching the local minimum, and may end up with relatively large negative polarity scores eventually.
- For neutral tokens, we have  $B \approx 0$  and  $C \approx 0$ . Their polarity scores will likely not change significantly after each update when approaching the local minimum, and may end up with polarity scores that are neither significantly positive nor significantly negative eventually.
Based on the above results, we can also quickly note that  $\rho(e)$  has the following property: it is positive if  $e$  is a polarity token, and close to zero if  $e$  is neutral.
These results are desirable as the token-level polarity scores will be used for defining the instance-level polarity scores, which are in term useful for prediction of the final polarity of the sentence containing such tokens.
However, we note that the above results depend on the assumption that term  $A$  is small. As we mentioned above, we may assume  $\lambda$  is large to achieve this. When  $V^{\top}W / \lambda$  is not small enough, the term  $A$  may lead to a gap in the polarity scores between the positive and negative tokens, depending on the sign of  $V^{\top}W$  - a term that will appear again in the next section when examining the attention scores.
4.2 Attention Score
Now let us have an analysis on the attention score for each token. Again given a token  $e$ , the corresponding attention score is  $a_{e} = \frac{e^{\top}V}{\lambda}$ . Note that this is a global score that is independent of any instance. The update of  $a_{e}$  is:
$$
\frac {d a _ {e} (\tau)}{d \tau} = \frac {1}{\lambda} \left(\frac {d \boldsymbol {e} (\tau)}{d \tau}\right) ^ {\top} \boldsymbol {V} (\tau) + \frac {1}{\lambda} \boldsymbol {e} ^ {\top} (\tau) \frac {d \boldsymbol {V} (\tau)}{d \tau}. \tag {21}
$$
Similarly, let us rewrite the equation as:
$$
\frac {d a _ {e}}{d \tau} = \underbrace {- \frac {1}{\lambda} \left(\frac {\partial \ell}{\partial \boldsymbol {e}}\right) ^ {\top} \boldsymbol {V}} _ {\Delta a _ {e} ^ {\prime}} + \underbrace {\left(- \frac {1}{\lambda} \boldsymbol {e} ^ {\top} \frac {\partial \ell}{\partial \boldsymbol {V}}\right)} _ {\Delta a _ {e} ^ {\prime \prime}}. \tag {22}
$$
$$
\frac {\partial \ell}{\partial \boldsymbol {V}} = - \frac {1}{m \lambda} \sum_ {(t, j)} y ^ {(t)} \beta^ {(t)} \alpha_ {j} ^ {(t)} \boldsymbol {e} _ {j} ^ {(t)} \left(s _ {j} ^ {(t)} - s ^ {(t)}\right). \tag {23}
$$
The first term can be calculated as:
$$
\begin{array}{l} \Delta a_{e}^{\prime} = \frac{1}{m\lambda^{2}} ||\boldsymbol {V}||_{2}^{2}\sum_{(t,j):e_{j}^{(t)}\equiv e}y^{(t)}\beta^{(t)}\alpha_{j}^{(t)}\Big(s_{e} - s^{(t)}\Big) \\ + \frac {1}{m \lambda} \sum_ {(t, j): e _ {j} ^ {(t)} \equiv e} y ^ {(t)} \beta^ {(t)} \alpha_ {j} ^ {(t)} \boldsymbol {W} ^ {\top} \boldsymbol {V}. \tag {24} \\ \end{array}
$$
The second term is:
$$
\Delta a _ {e} ^ {\prime \prime} = \frac {1}{m \lambda^ {2}} \sum_ {(t, j)} y ^ {(t)} \beta^ {(t)} \alpha_ {j} ^ {(t)} e ^ {\top} e _ {j} ^ {(t)} \left(s _ {j} ^ {(t)} - s ^ {(t)}\right). \tag {25}
$$
Similarly, this can be re-written as:
$$
\begin{array}{l} \frac{1}{m\lambda^{2}}\sum_{(t,j):y^{(t)} = +1}\beta^{(t)}\alpha_{j}^{(t)}\Big(s_{j}^{(t)} - s^{(t)}\Big)e^{\top}e_{j}^{(t)} \\ + \frac {1}{m \lambda^ {2}} \sum_ {(t, j): y ^ {(t)} = - 1} \beta^ {(t)} \alpha_ {j} ^ {(t)} \left(s ^ {(t)} - s _ {j} ^ {(t)}\right) e ^ {\top} e _ {j} ^ {(t)}. \tag {26} \\ \end{array}
$$
This term shall be close to zero initially, regardless of  $e$ . However, this term may become positive for a polarity token  $e$  as learning progresses.
The update of  $a_{e}$  is (note that  $\bar{\pmb{W}}^{\top}\bar{\pmb{V}} = \pmb{V}^{\top}\pmb{W}$ ):