Venue: NAACL 2022

GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformers

Ali Modarressi\({}^{1*}\) Mohsen Fayyaz\({}^{2*}\)

**Yadollah Yaghoobzadeh\({}^{2}\) Mohammad Taher Pilehvar\({}^{3}\) \({}^{1}\) Iran University of Science and Technology, Iran \({}^{2}\) University of Tehran, Iran \({}^{3}\) Tehran Institute for Advanced Studies, Khatam University, Iran m_modarressi@comp.iust.ac.ir {mohsen.fayya277, y.yaghoobzadeh}@ut.ac.ir mp792@cam.ac.uk**

###### Abstract

There has been a growing interest in interpreting the underlying dynamics of Transformers. While self-attention patterns were initially deemed as the primary option, recent studies have shown that integrating other components can yield more accurate explanations. This paper introduces a novel token attribution analysis method that incorporates all the components in the encoder block and aggregates this throughout layers. Through extensive quantitative and qualitative experiments, we demonstrate that our method can produce faithful and meaningful global token attributions. Our experiments reveal that incorporating almost every encoder component results in increasingly more accurate analysis in both local (single layer) and global (the whole model) settings. Our global attribution analysis significantly outperforms previous methods on various tasks regarding correlation with gradient-based saliency scores. Our code is freely available at [https://github.com/mohsenfrayyaz/GlobeEnc](https://github.com/mohsenfrayyaz/GlobeEnc).

+
Footnote †: Equal contribution.

+
Footnote †: Equal contribution.

## 1 Introduction

The stellar performance of Transformers Vaswani et al. (2017) has garnered a lot of attention to analyzing the reasons behind their effectiveness. The self-attention mechanism has been one of the main areas of focus Clark et al. (2019); Kovaleva et al. (2019); Reif et al. (2019); Htut et al. (2019). However, there have been debates on whether raw attention weights are reliable anchors for explaining model's behavior or not Wiegreffe and Pinter (2019); Serrano and Smith (2019); Jain and Wallace (2019). Recently, it was shown that incorporating vector norms should be an indispensable part of any attention-based analysis1 Kobayashi et al. (2020, 2021). However, these norm-based studies incorporate only the attention block into their analysis, whereas Transformer encoder layer is composed of more components.

Footnote 1: We also have shown the unreliability of weights due to norm disparities in probing studies Fayyaz et al. (2021).

Another limitation of the existing analysis techniques is that they are usually constrained to the analysis of single layer attributions. In order to expand the analysis to multi-layered encoder-based models in their entirety, an aggregation technique has to be employed. Abnar and Zuidema (2020) proposed two aggregation methods, _rollout_ and _max-flow_, which combine raw attention weights across layers. Despite showing the outcome of their method to be faithful to a model's inner workings in specific cases, the final results are still unsatisfactory on a wide range of fine-tuned models.

Additionally, gradient-based alternatives Simonyan et al. (2014); Kindermans et al. (2016); Li et al. (2016) have been argued to provide a more robust basis for token attribution analysis Atanasova et al. (2020); Brunner et al. (2020); Pascual et al. (2021). Nonetheless, the gradient-based alternatives have not been able to fully replace attention-based counterparts, mainly due to their high computa

Figure 1: Aggregated attribution maps (\(\mathcal{N}_{\mathrm{ENC}}\)) for the [CLS] token for fine-tuned BERT on SST2 dataset (sentiment analysis). Our method (GlobEnc) is able to accurately quantify the global token attribution of the model.

 

tional intensity.

In this paper, we propose a new global token attribution analysis method (GlobEnc) which is based on the encoder layer's output. In GlobEnc, the second layer normalization is also included in the norm-based analysis of each encoder layer. To aggregate attributions over all layers, we applied a modified attention rollout technique, returning global scores.

Through extensive experiments and comparing the global attribution with the input token attributions obtained by gradient-based saliency scores, we show that our method produces faithful and meaningful results (Figure 1). Our evaluations on models with distinct pre-training objectives and sizes Devlin et al. (2019); Clark et al. (2020) show high correlations with gradient-based methods in global settings. Furthermore, with comparative studies on each aspect of GlobEnc , we find that: (i) norm-based methods achieve higher correlations than weight-based methods; (ii) incorporating residual connections plays an essential role in token attribution; (iii) considering the two layer normalizations improve our analysis only if coupled together; and (iv) aggregation across layers is crucial for an accurate whole-model attribution analysis.

In summary, our main contributions are:

* We expand the scope of analysis from attention block in Transformers to the whole encoder.
* Our method significantly improves over existing techniques for quantifying global token attributions.
* We qualitatively demonstrate that the attributions obtained by our method are plausibly interpretable.

## 2 Background

In encoder-based language models (such as BERT), a Transformer encoder layer is composed of several components (Figure 2). The core component of the encoder is the self-attention mechanism Vaswani et al. (2017), which is responsible for the information mixture of a sequence of token representations \(({\mathbf{x}_{1},...,{\mathbf{x}_{n}}})\). Each self-attention head computes a set of attention weights \({\mathbf{A}^{h}=\{\alpha_{i,j}^{h}|1\leq i,j\leq n\}}\), where \(\alpha_{i,j}^{h}\) is the raw attention weight from the \(i^{\text{th}}\) token to the \(j^{\text{th}}\) token in head \(h\in\{1,...,H\}\). Therefore, the output representation (\({\mathbf{z}_{i}\in\mathbb{R}^{d}}\)) for the \(i^{\text{th}}\) token of a multi-head (with \(H\) heads) self-attention module is computed by concatenating the heads' outputs followed by a head-mixing \({\mathbf{W}_{O}}\) projection:

\[{\mathbf{z}_{i}=\textsc{Concat}({\mathbf{z}_{i}^{1},...,{\mathbf{z}_{i}^{H}}}){\mathbf{W}_{O}}} \tag{1}\]

where each head's output vector is generated by performing a weighted sum over the transformed value vectors \({\mathbf{v}({\mathbf{x}_{j}})\in\mathbb{R}^{d_{v}}}\):

\[{\mathbf{z}_{i}^{h}=\sum_{j=1}^{n}\alpha_{i,j}^{h}{\mathbf{v}^{h}({\mathbf{x}_{j}})} \tag{2}\]

**Norm-based attention.** While one may interpret the attention mechanism using the attention weights \({\mathbf{A}}\), Kobayashi et al. (2020) argued that doing so would ignore the norm of the transformed vectors multiplied by the weights, elucidating that the weights are insufficient for interpretation. Their solution enhanced the interpretability of attention weights by incorporating the value vectors \({\mathbf{v}({\mathbf{x}_{j}})}\) and the following projection \({\mathbf{W}_{O}}\). By reformulating Equation 1, we can consider \({\mathbf{z}_{i}}\) as a summation over the attentions heads:

\[{\mathbf{z}_{i}=\sum_{h=1}^{H}\sum_{j=1}^{n}\alpha_{i,j}^{h}\underbrace{{\mathbf{v}^{ h}({\mathbf{x}_{j}}){\mathbf{W}_{O}^{h}}}}_{f^{h}({\mathbf{x}_{j}})}} \tag{3}\]

Figure 2: The internal structure of a Transformer encoder layer. We show on the diagram the components that are incorporated by each token attribution analysis method. Our method incorporates the whole encoder (\(\mathcal{N}_{\textsc{ENC}}\)) except for the direct effect of the fully connected feed-forward module. Diagram inspired by Alammar (2018).

 

Using this reformulation2, Kobayashi et al. proposed a _norm-based_ token attribution analysis method, \(\mathcal{N}:=(||\mathbf{z}_{i\gets j}|j)|\in\mathbb{R}^{n\times n}~{}\), to measure each token's contribution in a self-attention module:

Footnote 2: \(W_{O}^{h}\) is a head-specific slice of the original \(\mathbf{W}_{O}\) projection. For more information about the reformulation process, see Appendix C in Kobayashi et al. (2021)

\[\mathbf{z}_{i\gets j}=\sum_{h=1}^{H}\alpha_{i,j}^{h}f^{h}(\mathbf{x}_{j}) \tag{4}\]

They showed that incorporating the magnitude of the transformation function (\(f^{h}(\mathbf{x})\)) is crucial in assessing the input tokens' contribution to the self-attention output.

**Residual connections & Layer Normalizations.**Kobayashi et al. (2021) added the attention block's Layer Normalization (**LN#1**) and Residual connection (**RES#1**) to its prior norm-based analysis to assess the impact of residual connections and layer normalization inside an attention block. \(\mathcal{N}_{\text{RES}}:=(||\mathbf{z}_{i\gets j}^{+}||)\!\in\!\mathbb{R}^{n \times n}\) is the analysis method which incorporates the attention block's residual connection. The input vector \(\mathbf{x}\) is added to the attribution of each token to itself to incorporate the influence of RES#1:

\[\mathbf{z}_{i\gets j}^{+}=\sum_{h=1}^{H}\alpha_{i,j}^{h}f^{h}(\mathbf{x}_{i})+ \mathbf{1}[i=j]\mathbf{x}_{i} \tag{5}\]

They proposed a method for decomposing LN3 into a summation of normalizations:

Footnote 3: \(\mathbf{Y}\in\mathbb{R}^{d}\) and \(\mathbf{\beta}\in\mathbb{R}^{d}\) are the trainable weights of LN. Similar to Kobayashi et al. (2021) we ignore \(\mathbf{\beta}\).

\[\begin{split}\text{LN}(\mathbf{z}_{i}^{*})&=\sum_{j=1} ^{n}g_{\mathbf{z}_{i}^{*}}(\mathbf{z}_{i\gets j}^{+})+\mathbf{\beta}\\ g_{\mathbf{z}_{i}^{*}}(\mathbf{z}_{i\leftarrow j}^{+})&:= \frac{\mathbf{z}_{i\gets j}^{+}-m(\mathbf{z}_{i\gets j}^{+})}{s(\mathbf{z}_{i}^{* })}\odot\mathbf{\gamma}\end{split} \tag{6}\]

where \(m(.)\) and \(s(.)\) are the element-wise mean and standard deviation of the input vector (cf. SSA.1). The decomposition can be applied to the contribution vectors:

\[\tilde{\mathbf{z}}_{i\gets j}=g_{\mathbf{z}_{i}^{*}}(\sum_{h=1}^{H}\alpha_{i,j}^ {h}f^{h}(\mathbf{x}_{i})+\mathbf{1}[i=j]\mathbf{x}_{i}) \tag{7}\]

Accordingly, we can compute the magnitude \(\mathcal{N}_{\text{RESLN}}:=(||\widetilde{\mathbf{z}}_{i\gets j}||)\in\mathbb{ R}^{n\times n}\), which represents the amount of influence of an encoder layer's input token \(j\) on its output token \(i\). Based on this formulation, a context-mixing ratio could be defined as:

\[r_{i}=\frac{||\sum_{j=1,j\neq i}^{n}\tilde{\mathbf{z}}_{i\gets j}||}{||\sum_{ j=1,j\neq i}^{n}\tilde{\mathbf{z}}_{i\gets j}||+||\tilde{\mathbf{z}}_{i \gets i}||} \tag{8}\]

Experiments by Kobayashi et al. (2021) revealed considerably low \(r\) values which indicate the huge impact of the residual connections. In other words, the model tends to preserve token representations more than mixing them with each other.

## 3 Methodology

Our method for input token attribution analysis has a holistic view and takes into account almost every component within the encoder layer. To this end, we first extend the norm-based analysis of Kobayashi et al. (2021) by incorporating the encoder's output LN#2. We then apply an aggregation technique to combine the information flow throughout all layers.

**Encoder layer output \(\neq\) Attention block output.** While the RES#1 and the LN#1 from the attention block are included in the analysis of Kobayashi et al. (2021), the subsequent FFN, RES#2, and output LN#2 are ignored (see Fig. 2). Hence, \(\mathcal{N}_{\text{RESLN}}\) might not be indicative of the entire encoder layer's function. To address this issue, we additionally include the encoder layer components from the attention block outputs (\(\tilde{z}_{i}\)) to the output representations (\(\tilde{x}_{i}\)). The output of each encoder (\(\tilde{x}_{i}\)) is computed as follows:

\[\begin{split}\tilde{\mathbf{z}}_{i}^{+}&=\text{FFN}( \tilde{\mathbf{z}}_{i})+\tilde{\mathbf{z}}_{i}\\ \tilde{\mathbf{x}}_{i}&=\text{LN}(\tilde{\mathbf{z}}_{i}^{ +})\end{split} \tag{9}\]

We apply the LN decomposition rule in Eq. 7 to separate the impacts of residual and FFN output:

\[\tilde{\mathbf{x}}_{i}=\sum_{j=1}^{n}\left(g_{\tilde{\mathbf{z}}_{i}^{*}}(\text{FFN}( \tilde{\mathbf{z}}_{i\gets j}))+g_{\tilde{\mathbf{z}}_{i}^{*}}(\tilde{\mathbf{z}}_{i \gets j})\right)+\mathbf{\beta} \tag{10}\]

Given that the activation function between the two fully connected layers in the FFN component is non-linear Vaswani et al. (2017), a linear decomposition similar to Eq. 7 cannot be derived. As a result, we omit FFN's influence on the contribution of each token and instead consider RES#2, approximating \(\tilde{\mathbf{z}}_{i\gets j}\) as \(g_{\tilde{\mathbf{z}}_{i}^{*}}(\tilde{\mathbf{z}}_{i\gets j})\). Nevertheless, it should be noted that the FFN _still preserves some influence on this new setting due to the presence of 

\(s(\mathbf{z}_{i}^{*})\) in \(g_{\mathbf{\hat{z}}_{i}^{*}}(\mathbf{\tilde{z}}_{i-j})\). Similarly to Eq. 7, we can introduce a more inclusive layerwise analysis method \(\mathcal{N}_{\text{ENC}}:=(||\mathbf{\tilde{x}}_{i\gets j}||)\in\mathbb{R}^{n \times n}\) from input token \(j\) to output token \(i\) using:

\[\begin{split}&\mathbf{\tilde{x}}_{i\gets j}\approx g_{\mathbf{\hat{ z}}_{i}^{*}}(\mathbf{\tilde{z}}_{i\gets j})=\frac{\mathbf{\tilde{z}}_{i\gets j }-m(\mathbf{\tilde{z}}_{i\gets j})}{s(\mathbf{\tilde{z}}_{i}^{*})}\odot\mathbf{\gamma} \end{split} \tag{11}\]

Aggregating multi-layer attention.To create an aggregated attribution score, Abnar and Zuidema (2020) proposed describing the model's attentions via modelling the information flow with a directed graph. They introduced _attention rollout_ method, which linearly combines raw attention weights along all available paths in the pairwise attention graph. The attention rollout of layer \(\ell\) w.r.t. the inputs is computed recursively as follows:

\[\begin{split}\tilde{\mathbf{A}}_{\ell}=\begin{cases}\hat{\mathbf{A}}_{ \ell}\tilde{\mathbf{A}}_{\ell-1}&\ell>1\\ \hat{\mathbf{A}}_{\ell}&\ell=1\end{cases}\end{split} \tag{12}\]

\[\begin{split}\hat{\mathbf{A}}_{\ell}=0.5\bar{\mathbf{A}}_{\ell}+0.5\mathbf{I }\end{split} \tag{13}\]

\(\bar{\mathbf{A}}_{\ell}\) is the raw attention map averaged across all heads in layer \(\ell\). This method assumes equal contribution from the residual connection and multi-head attention (See Fig. 2). Hence, an identity matrix is summed and renormalized, giving \(\hat{\mathbf{A}}_{\ell}\).

For aggregating the layerwise analysis methods, we use the rollout technique with minor modifications. As many of the methods already include residual connections, we only use Eq. 12 (replacing \(\hat{\mathbf{A}}_{\ell}\) with the desired method's attribution matrix in layer \(\ell\)) to calculate the rollout of a given method. However, for methods that do not assume the residual connection, we define a corresponding "Fixed" variation using Eq. 13 that incorporates a fixed residual effect (\(r_{i}\approx 0.5\)). **We refer to our proposed global method--aggregating the \(\mathcal{N}_{\text{ENC}}\) across all layers by the rollout method--as GlobEnc.** In what follows we report our experiments, comparing GlobEnc with several other settings.

## 4 Experiments

In this section, we introduce the datasets and the token attribution analysis methods used in our evaluations, followed by the experimental setup and results.

### Datasets

All analysis methods are evaluated on three different classification tasks. To cover sentiment detection tasks we use SST2 Socher et al. (2013), MNLI Williams et al. (2018) for Natural Language Inference and Hateexplain Mathew et al. (2021) in hate speech detection.

### Analysis Methods

We use two categories of explainability approaches in our work: _Weight-based_ and _Norm-based_.4 The _Weight-based_ approaches employed in our experiments are as follows:

Footnote 4: Note that in most of our experiments, we use all these methods within the rollout aggregation technique.

* \(\mathcal{W}\): The raw attention maps averaged across all heads (See \(\overline{\mathbf{A}}_{\ell}\) in SS2).
* \(\mathcal{W}_{\text{FIXEDRes}}\) : Abnar and Zuidema's assumption; add an identity matrix as a fixed residual to \(\overline{\mathbf{A}}_{\ell}\) (see \(\hat{\mathbf{A}}_{\ell}\) in Eq. 13).
* \(\mathcal{W}_{\text{RES}}\) : The corrected version of \(\mathcal{W}\) in which accurate residuals are added based on the context-mixing ratios of \(\mathcal{N}_{\text{ENC}}\): \[\begin{split}\hat{r}_{i}=\dfrac{\left\|\sum_{j=1,j\neq i}^{n} \tilde{\mathbf{x}}_{i\gets j}\right\|}{\left\|\sum_{j=1,j\neq i}^{n}\tilde{\bm {x}}_{i\leftarrow j}\right\|+\left\|\tilde{\mathbf{x}}_{i\gets i}\right\|}\end{split}\] (14) In order to enforce \(\mathcal{W}_{\text{RES}}\) to have a context-mixing ratio equal to \(\hat{r}_{i}\), it is essential to zero-out the diagonal elements (the tokens' attentions to themselves) of \(\bar{\mathbf{A}}_{\ell}\) and renormalize it: \[\begin{split}\mathbf{A}_{\ell}^{\prime}=(\mathbf{I}-\textbf{diag}\,( \mathbf{\bar{A}}_{\ell}))^{-1}(\mathbf{\bar{A}}_{\ell}-\textbf{diag}\,(\mathbf{\bar{A}}_{ \ell}))\\ \mathcal{W}_{\text{RES}}:=&\textbf{diag}\,(\hat{r}_{ 1},\cdots,\hat{r}_{n})\,\mathbf{A}_{\ell}^{\prime}\\ &+\textbf{diag}\,(1-\hat{r}_{1},\ldots,1-\hat{r}_{n})\,\mathbf{I} \end{split}\] (15) The _Norm-based_ analysis methods, namely \(\mathcal{N}\), \(\mathcal{N}_{\text{RES}}\) and \(\mathcal{N}_{\text{RESLN}}\) were discussed in detail in SS2. Our proposed norm-based method \(\mathcal{N}_{\text{ENC}}\) was explained in SS3. For an ablation study, we introduce \(\mathcal{N}_{\text{FIXEDRes}}\) which is \(\mathcal{N}\), corrected with a fixed residual similar to \(\mathcal{W}_{\text{FIXEDRes}}\)5.

Footnote 5: The only difference is that we need to normalize \(\mathcal{N}\) before adding an identity matrix.

\[\begin{split}\check{\mathcal{N}}=\Bigg{(}\dfrac{||\mathbf{z}_{i \gets j}||}{\sum_{j}||\mathbf{z}_{i\gets j}||}\Bigg{)}\in\mathbb{R}^{n \times n}\\ \mathcal{N}_{\text{FIXEDRes}}:=0.5\,\check{\mathcal{N}}+0.5\,\mathbf{ I}\end{split}\] (16) 

In SS4.5, we will demonstrate our comparative studies between the aforementioned methods and GlobEnc.

### Gradient-based Methods for Faithfulness Analysis

Gradient-based methods are widely used as alternatives for attention-based counterparts for quantifying the importance of a specific input feature in making the right prediction (Li et al., 2016; Atanasova et al., 2020). In this section we discuss the specific gradient-based methods we use, namely saliency, HTA, and our adjusted HTA.

#### 4.3.1 Saliency

Gradient-based saliency is based on the gradient of the output (\(y_{c}\)) w.r.t. the input embeddings (\(\mathbf{e}^{0}_{i}\)). One of the most accurate variations of the saliency family is the _gradient\(\times\)input_ method (Kindermans et al., 2016) where the input embeddings is multiplied by the gradients. Thus, the contribution score of input token \(i\) is determined by first computing the element-wise product of the input embeddings (\(\mathbf{e}^{0}_{i}\)) and the gradients of the true class output score (\(y_{c}\)) w.r.t. the input embeddings. Then, the L2 norm of the scaled gradients is computed to derive the final score:

\[Saliency_{i}=\left\|\frac{\partial y_{c}}{\partial\mathbf{e}^{0}_{i}}\odot\mathbf{e}^ {0}_{i}\right\|_{2} \tag{17}\]

#### 4.3.2 Hta x Inputs

To determine an upper bound on the information mixing within each layer, we use a modified version of _Hidden Token Attribution_(Brunner et al., 2020, HTA). In the original version, HTA is the sensitivity between any two vectors in the model's computational graph. However, inspired by the _gradient\(\times\)input_ method (Kindermans et al., 2016), which has shown more faithful results (Atanasova et al., 2020; Wu and Ong, 2021), we multiply the input vectors by the gradients and then apply a Frobenius norm. We compute the attribution from hidden embedding \(j\) (\(\mathbf{e}^{\ell-1}_{j}\)) to hidden embedding \(i\) (\(\mathbf{e}^{\ell}_{i}\)) in layer \(\ell\) as:

\[\mathbf{c}^{\ell}_{i\gets j}=\left\|\frac{\partial\mathbf{e}^{\ell}_{i}}{\partial \mathbf{e}^{\ell-1}_{j}}\odot\mathbf{e}^{\ell-1}_{j}\right\|_{F} \tag{18}\]

Computing HTA-based attribution matrices is an extremely computation-intensive task (especially for long texts) due to the high dimensionality of hidden embeddings. Hence, we only use this method for 256 examples from the SST-2 task's validation set. It is worth noting that extracting the HTA-based contribution maps for the aforementioned data took approximately 2 hours, whereas computing the maps for the entire analysis methods stated in SS4.2 took only 5 seconds.6

Footnote 6: Conducted on a 3070 GPU machine.

### Setup

We employ HuggingFace's Transformers library7(Wolf et al., 2020) and the BERT-base-uncased model. For fine-tuning BERT, epochs vary from 3 to 5, and the batch size and learning rate are 32 and 3e-5, respectively.8 We also carried out the main experiment on BERT-large and ELECTRA (Devlin et al., 2019; Clark et al., 2020) where the results are reported at SSA.2.

Footnote 7: [https://github.com/huggingface/transformers](https://github.com/huggingface/transformers)

Footnote 8: Recommended by Devlin et al. (2019).

After rollout aggregation of each analysis method, we obtain an accumulated attribution matrix for every layer (\(\ell\)) of BERT. These matrices indicate the overall contribution of each input token to all token representations in layer \(\ell\). Since the classifier in a fine-tuned model is attached to the final layer representation of the [CLS] token, we consider the first row (corresponding to [CLS] attributions) of the last layer attribution matrix. This vector represents the contribution of each input token to the model's final decision. As a measure of faithfulness of the resulting vector with the saliency scores, we report the _Spearman's rank correlation_ between the two vectors.

### Results

Table 1 shows the Spearman correlation of saliency scores with the aggregated attribution scores from [CLS] to input tokens at the final layer. In order to determine the contribution of each component of encoder layer to the overall performance, we report the results for attribution analysis methods discussed in SS4.2. Our results demonstrate that incorporating the vector norms, residual connection, and both layer normalizations yields the highest correlation (\(N_{\text{ENC}}\)). In what follows, we discuss the impact of incorporating various parts in the analysis.

#### 4.5.1 On the role of vector norms

As also suggested by Kobayashi et al. (2020), vector norms play an important role in determining 

attention outputs. This is highlighted by the significant gap between weight-based and norm-based settings across all datasets in Table 1.

We also show the correlation of the aggregated attention for all layers in Figure 3. The norm-based settings (\(\mathcal{N}\) and \(\mathcal{N}_{\text{RES}}\)) attain higher correlation than the weight-based counterparts (\(\mathcal{W}\) and \(\mathcal{W}_{\text{RES}}\)) almost in all layers, confirming the importance of incorporating vector norms.

#### 4.5.2 On the role of residual connections

Kobayashi et al. (2021) showed that in the encoder layer, the output representations of each token is mainly determined by its own representation, and the contextualization from other tokens' plays a marginal role. This is in contrary to the simplifying assumption made by Abnar and Zuidema (2020) who used a fixed context-mixing ratio of \(0.5\) (assuming that BERT equally preserves and mixes the representations). This setting is shown as weighted with fixed residual (\(\mathcal{W}_{\text{FIxEDRES}}\)) in Table 1. We compare this setting against \(\mathcal{W}_{\text{RES}}\) (see SS4.2). \(\mathcal{W}_{\text{RES}}\) is similar to \(\mathcal{W}_{\text{FIxEDRes}}\) (in that it does not take into account vector norms) but differs in that it considers a dynamic mixing ratio (the one from \(\mathcal{N}_{\text{ENC}}\)). The huge performance gap between the two settings in Table 1 clearly highlights the importance of considering accurate context-mixing ratios. Therefore, it is crucial to consider the residual connection in the attention block for input token attribution analysis.

To further demonstrate the role of residual connections, we utilize the introduced method in SS4.2, where we modified the norm-based attentions with fixed residual (\(r\approx 0.5\)). The comparison of normal-based without any residual (\(\mathcal{N}\)) and with a fixed residual (\(\mathcal{N}_{\text{FIxEDRES}}\)) shows a consistent improvement for the latter across all the datasets. This provides evidence on that having a fixed uniform context-mixing ratio is better than neglecting the residual connection altogether.

Finally, when we aggregate the norm-based analysis with an accurate dynamic context-mixing ratio (\(\mathcal{N}_{\text{RES}}\)), we observe the highest correlation up to

Figure 3: Spearman’s rank correlation of aggregated attribution scores with saliency scores across layers. The 99% confidence intervals are shown as (narrow) shaded areas around each line. \(\mathcal{N}_{\text{ENC}}\) achieves the highest correlation in almost every layer.

 

this point, without layer normalization.

#### 4.5.3 On the role of layer normalization

In Table 1 we see a sudden drop in correlations for \({\cal N}_{\rm ResLN}\). Although this method considers vector norms and residuals, incorporating LN#1 in the encoder seems to have deteriorated the accuracy for token attribution analysis. To determine whether this deterioration of correlation in aggregated attributions is also present in individual single layers, we compare the HTA maps as a baseline with the attribution matrices extracted from different analysis methods. Figure 4 shows the correlation of HTA attribution maps with the maps obtained by \({\cal N}_{\rm RES}\), \({\cal N}_{\rm ResLN}\), and \({\cal N}_{\rm ENC}\) methods. The results indicate that \({\cal N}_{\rm ResLN}\) exhibits a significantly lower association.

The question that arises here is that how incorporating an additional component of the encoder (LN#1) in \({\cal N}_{\rm ResLN}\) degrades the results (compared to \({\cal N}_{\rm REs}\)). To answer this question, we investigated the learned weights of LN#1 and LN#2. The outlier weights10 in specific dimensions of LNs are shown to be significantly influential on the model's performance Kovaleva et al. (2021); Luo et al. (2021). It is interesting to note that based on our observations, the outlier weights of the two layer norms seem to be the opposite of each other. Figure 5 demonstrates the weight values in layer 11 and also the correlation of the outlier weights across layers. The large negative correlations confirm that the outlier weights work contrary to each other. We speculate that the effect of outliers in the two layer norms is partly cancelled out when both are considered.

Footnote 10: We identify the dimensions where the weights are at least \(3\sigma\) from the mean as outliers Kovaleva et al. (2021).

As shown in Figure 2, the FFN and the second layer normalization are on top of the attention block. However, \({\cal N}_{\rm ResLN}\) does not incorporate the components outside of the attention block. As described in SS3, in our local analysis method \({\cal N}_{\rm ENC}\) we incorporate the second layer normalization in the transformer's encoder (Figure 2), thus considering the whole encoder block (except FFN). Overall, our global method, GlobEnc, yields the best results among all the methods evaluated in our experiments. In general, Table 1 suggests that incorporating each component of the encoder will increase the correlation; however, the two layer normalizations should be considered together.

#### 4.5.4 On the role of aggregation

We carried out an additional analysis to verify if incorporating vector norms, residual connection and layer normalizations in individual layers is adequate for achieving high correlations, or if it is also necessary to aggregate them via rollout. Table 2

Figure 4: Single layer Pearson correlation of HTA maps with attribution maps. The 99% confidence intervals are shown as shaded areas around each line. \({\cal N}_{\rm ResLN}\) shows considerably less association with HTA.

Figure 5: The Pearson correlation between outlier weights of LN#1 and LN#2 across layers. The weight values for layer 11 are shown as well.

 

shows the correlation results in different layers for raw attributions (without aggregation) and for the aggregated attributions using the rollout method. Applying rollout method on attribution maps up to each layer results in higher correlations with the saliency scores than the raw single layer attribution maps, especially in deeper layers. Therefore, attention aggregation is essential for global input token attribution analysis.

An interesting point in Figure 3, which shows the correlation of the aggregated methods throughout the layers, is that the correlation curves flatten out after only a few layers.11 This indicates that BERT identifies decisive tokens only after the first few layers. The final layers only make minor adjustments to this order. Nevertheless, it is worth noting that the order of attribution does not necessarily imply the model's final decision and the final result may still change for the better or worse Zhou et al. (2020).

Footnote 11: \(\mathcal{N}_{\text{Resk}}\) is the only exception with a constant increase; this method is gradually and artificially corrected by \(\mathcal{N}_{\text{Enc}}\) context mixing ratios.

#### 4.5.5 Qualitative analysis

To qualitatively answer if the aggregated attribution maps provide plausible and meaningful interpretations, we take a closer look at the attribution maps generated by GlobEnc. Figure 1 shows the GlobEnc attribution of the model trained on SST-2. Each layer demonstrates the [CLS] token's aggregated attribution to input tokens up to the corresponding layer. The example inputs are "a deep and meaningful film." and "big fat waste of time.", both correctly classified by the model. In both cases, GlobEnc focuses on the relevant words for sentiment classification, i.e., "meaningful" and "waste". An interesting observation in Figure 1 is that in the first few layers, the [CLS] token mostly attends to itself while other tokens have marginal impact. As the representations get more contextualized in deeper layers, the attribution correctly shifts to the words which indicate the sentiment of the sentence.12 More examples from MNLI and SST2 datasets, including misclassified examples are available at SSA.3. Our qualitative analysis suggests that GlobEnc can be useful for a reasonable interpretation of attention mechanism in BERT, ELECTRA, and possibly any other transformer-based model.

Footnote 12: Complete attention maps in Figure A.3 show that, similarly to [CLS], other tokens also focus on sentiment tokens.

## 5 Related Work

While numerous studies have used attention weights to analyze and interpret the self-attention mechanism Clark et al. (2019); Kovaleva et al. (2019); Reif et al. (2019); Hutt et al. (2019), the use of mere attention weights to explain a model's inner workings has been an active topic of debate Serrano and Smith (2019); Jain and Wallace (2019); Wiegreffe and Pinter (2019). Several solutions have been proposed to address this issue, usually through converting raw attention weights to scores that provide better explanations. Brunner et al. (2020) used the transformation function \(f^{h}(\mathbf{x}_{j})\) to introduce _effective attentions_--the orthogonal component of the attention matrix in \(f^{h}(\mathbf{x}_{j})\) null space--to explain the inner workings of each layer. However, this technique ignores other components in the encoder and is computationally expensive due to the SVD required to compute the effective attentions. Kobayashi et al. (2020) incorporated the modified vector and introduced a vector norms-based analysis. This was later extended by integrating residual connections and layer normalization components to enhance the accuracy of explanations Kobayashi et al. (2021). But, as discussed in SS4.5, relying solely on LN#1 does not produce accurate results.

While these methods can be employed for single-layer (local) analysis, multi-layer attributions are not necessarily correlated with single-layer attributions due to the significant degree of information combination through multi-layer language models Pascual et al. (2021); Brunner et al. (2020). Various saliency methods exist for explaining the model's decision based on the input Li et al. (2016); Bastings and Filippova (2020); Atanasova et al. (2020); Wu and Ong (2021); Mohebbi et al. (2021). However, these approaches are not primarily designed for computing inter-token attributions. To fill this gap, Brunner et al. (2020) proposed HTA, which is based on the gradient of each hidden embedding in relation to the input embeddings. In SS4.3.2, we extend HTA to incorporate the impact of the input vectors. However, HTA is extremely computationally intensive. Attention rollout (see SS3) and attention flow--which involve solving a max-flow problem on the attention graph--are two aggregation approaches introduced by Abnar and Zuidema (2020), in which raw attention weights (with equally weighted residual weights) are aggregated within multiple layers. We showed that attention rollout does not perform well on the raw 

attention maps of language models fine-tuned on downstream tasks and that this problem can be resolved by utilizing attribution norms.

## 6 Conclusions

In this work, we proposed a novel method for single layer token attribution analysis which incorporates the whole encoder layer, i.e., the attention block and the output layer normalization. When aggregated across layers using the rollout method, our technique achieves quantitatively and qualitatively plausible results. Our evaluation of different analysis methods provided evidence on roles played by individual components of the encoder layer, i.e., the vector norms, the residual connections, and the layer normalizations. Furthermore, our in-depth analysis suggested that the two layer normalizations in the encoder layer counteract each other; hence, it is important to couple them for an accurate analysis.

Additionally, using a newly proposed and improved version of Hidden Token Attribution, we demonstrated that encoder-based attribution analysis is more accurate when compared to other partial solutions in a single layer (local-level). This is consistent with our global observations. Quantifying global input token attribution based on our work can provide a meaningful explanation of the whole model's behavior. In future work, we plan to apply our global analysis method on various datasets and models, to provide valuable insights into model decisions and interpretability.

## References

* Abar and Zuidema (2020) Samira Abnar and Willem Zuidema. 2020. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 4190-4197. External Links: Link Cited by: SS3.1.
* J. Alammar (2018)The illustrated transformer [blog post]. Cited by: SS1.
* P. Atanasova, J. G. Simonsen, C. Lioma, and I. Augenstein (2020)A diagnostic study of explainability techniques for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 3256-3274. External Links: Link Cited by: SS1.
* J. Bastings and K. Filippova (2020)The elephant in the interpretability room: why use attention as explanation when we have saliency methods?. In Proceedings of the Third BlackboxNLP Workshop NLP, Online, pp. 149-155. External Links: Link Cited by: SS1.
* G. Brunner, Y. Liu, D. Pascual, O. Richter, M. Ciaramita, and R. Wattenhofer (2020)On identifiability in transformers. In International Conference on Learning Representations, External Links: Link Cited by: SS1.
* K. Clark, U. Khandelwal, O. Levy, and C. D. Manning (2019)What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Florence, Italy, pp. 276-286. External Links: Link Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link Cited by: SS1.
* M. Fayyaz, E. Aghazadeh, A. Modaressi, H. Mohebbi, and M. Taher Pilehvar (2021)Not all models localize linguistic knowledge in the same place: a layer-wise probing on BERToids' representations. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, Punta Cana, Dominican Republic, pp. 375-388. External Links: Link Cited by: SS1.
* Ph. M. Hutt, J. Phang, S. Bordia, and S. R. Bowman (2019)Do attention heads in BERT track syntactic dependencies?. CoRRabs/1911.12246. External Links: Link Cited by: SS1.
* S. Jain and B. C. Wallace (2019)Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 3543-3556. External Links: Link Cited by: SS1.
* P. Kindermans, K. Schutt, K. Muller, and S. Dahne (2016)Investigating the influence of noise and distractors on the interpretation of neural networks. Cited by: SS1.
* G. Kobayashi, T. Kuribayashi, S. Yokoi, and K. Inui (2020)Attention is not only a weight: analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) 

pages 7057-7075, Online. Association for Computational Linguistics.
* Kobayashi et al. (2021) Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2021. Incorporating Residual and Normalization Layers into Analysis of Masked Language Models. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 4547-4568, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Kovaleva et al. (2021) Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. 2021. BERT busters: Outlier dimensions that disrupt transformers. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 3392-3405, Online. Association for Computational Linguistics.
* Kovaleva et al. (2019) Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4365-4374, Hong Kong, China.
* Li et al. (2016) Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 681-691, San Diego, California. Association for Computational Linguistics.
* Luo et al. (2021) Ziyang Luo, Artur Kulmizev, and Xiaoxi Mao. 2021. Positional artefacts propagate through masked language model embeddings. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 5312-5327, Online. Association for Computational Linguistics.
* Mathew et al. (2021) Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animeesh Mukherjee. 2021. Hateexplain: A benchmark dataset for explainable hate speech detection. _Proceedings of the AAAI Conference on Artificial Intelligence_, 35(17):14867-14875.
* Mohebbi et al. (2021) Hossein Mohebbi, Ali Modarressi, and Mohammad Taher Pilchvar. 2021. Exploring the role of BERT token representations to explain sentence probing results. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 792-806, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Pascual et al. (2021) Damian Pascual, Gino Brunner, and Roger Wattenhofer. 2021. Telling BERT's full story: from local attention to global aggregation. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 105-124, Online. Association for Computational Linguistics.
* Reif et al. (2019) Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry ofbert. In _Advances in Neural Information Processing Systems_, pages 8594-8603.
* Sann et al. (2019) Victor Sann, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. In _NeurIPS EMC\({}^{2}\) Workshop_.
* Serrano and Smith (2019) Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2931-2951, Florence, Italy.
* Simonyan et al. (2014) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. _CoRR_, abs/1312.6034.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Wiegreffe and Pitter (2019) Sarah Wiegreffe and Yuval Pitter. 2019. Attention is not explanation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 11-20, Hong Kong, China. Association for Computational Linguistics.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julian Phu, Cannwen Xu, Teven Le Scao, Sylvian Gugger, Mariama Drame, Quentin Liboset, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics 

Zhengxuan Wu and Desmond C. Ong. 2021. On explaining your explanations of BERT: an empirical study with sequence classification. _CoRR_, abs/2101.00196.
* Zhou et al. (2020) Wangchunshu Zhuou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020. Bert loses patience: Fast and robust inference with early exit. In _Advances in Neural Information Processing Systems_, volume 33, pages 18330-18341. Curran Associates, Inc.

## Appendix A Appendix

### LN Formulation

\(m({\mathbf{a}}):=\frac{1}{d}\sum_{k}{\mathbf{a}}^{(k)}\),

\(s({\mathbf{a}}):=\sqrt{\frac{1}{d}\sum_{k}(m({\mathbf{a}})-{\mathbf{a}}^{(k)}+\epsilon)^{2}}\) where \(\epsilon\) is a small constant

### More Models

In this section we provide the results for BERT-large and ELECTRA-base. For both models, our method outperforms the previous analysis methods. The results are reported in Tables A.1 and A.2.

### More Examples

Aggregated attributions by different methods throughout layers is shown in Figure A.2. Our proposed method shows more plausible results.

Aggregated attribution map for layer 12 is shown in Figure A.3. In this figure, the effect of each token can be seen on all other tokens and not just the [CLS] token.

More examples for MNLI dataset are shown for BERT-base in Figure A.4, for BERT-large in Figure A.6, and for ELECTRA in Figure A.5. Moreover, misclassified examples of SST2 dataset are shown in Figure A.1.

 



\begin{table}
\begin{tabular}{l c c c} \hline \hline BERT-large & \multicolumn{4}{c}{**Attention Rollout**} \\ \cline{2-3}  & SST2 & MNLI & HateXplain \\ \hline Weight-based (\(\mathcal{W}\)) & \(-0.38\pm 0.16\) & \(-0.61\pm 0.14\) & \(-0.41\pm 0.25\) \\ w/ Fixed Residual (\(\mathcal{W}_{\textsc{fixedRes}}\)) & \(-0.25\pm 0.19\) & \(-0.48\pm 0.19\) & \(-0.21\pm 0.30\) \\ w/ Residual (\(\mathcal{V}_{\textsc{RES}}\)) & \(-0.10\pm 0.21\) & \(0.33\pm 0.23\) & \(0.09\pm 0.30\) \\ \hline Norm-based (\(\mathcal{N}\)) & \(0.44\pm 0.24\) & \(0.13\pm 0.27\) & \(0.48\pm 0.25\) \\ w/ Fixed Residual (\(\mathcal{N}_{\textsc{FixedRes}}\)) & \(0.49\pm 0.24\) & \(0.26\pm 0.25\) & \(0.49\pm 0.30\) \\ w/ Residual (\(\mathcal{N}_{\textsc{RES}}\)) & \(0.77\pm 0.11\) & \(0.66\pm 0.12\) & \(0.73\pm 0.16\) \\ w/ Residual + Layer Norm 1 (\(\mathcal{N}_{\textsc{resLN}}\)) & \(-0.07\pm 0.23\) & \(-0.35\pm 0.24\) & \(0.06\pm 0.32\) \\ w/ **GlobEnc**: [Residual + Layer Norm 1, 2] (\(\mathcal{N}_{\textsc{Exc}}\)) & \(\mathbf{0.83\pm 0.08}\) & \(\mathbf{0.77\pm 0.09}\) & \(\mathbf{0.76\pm 0.17}\) \\ \hline \hline \end{tabular}
\end{table}
Table A.1: Spearman’s rank correlation of attribution based importance (aggregated by rollout) with saliency scores for the validation set for the BERT-large model fine-tuned on SST-2, MNLI, and HateXplain. The numbers are the average on all the validation set examples (1024 examples for MNLI dataset due to resource limitations) \(\pm\) the standard deviation.

Figure A.1: Aggregated \(\mathcal{N}_{\textsc{Esc}}\) attribution maps (GlobEnc) for the [CLS] token for fine-tuned BERT on SST2 dataset (sentiment analysis). These examples were misclassified by the model.



Figure A.2: Aggregated attributions via rollout with different methods across layers. The model is fine-tuned on SST2 dataset and the attention of the CLS token is shown in each layer.

Figure A.3: Aggregated attributions via rollout with different methods in layer 12. The model is fine-tuned on SST2 dataset. Each row indicates how much other tokens impact the token written on the row.

 

Figure A.4: Aggregated \({\cal N}_{\rm ENC}\) attribution maps (GlobEnc) for the [CLS] token for fine-tuned BERT on MNLI dataset.

Figure A.5: Aggregated \({\cal N}_{\rm ENC}\) attribution maps (GlobEnc) for the [CLS] token for fine-tuned ELECTRA on MNLI dataset.