SlowGuess/NAACL_2022_OCR huggingface dataset:
Ali Modarressi $^{1\star}$  Mohsen Fayyaz $^{2\star}$   
Yadollah Yaghoobzadeh $^{2}$  Mohammad Taher Pilehvar $^{3}$
$^{1}$  Iran University of Science and Technology, Iran  $^{2}$  University of Tehran, Iran
$^{3}$  Tehran Institute for Advanced Studies, Khatam University, Iran
m_modarressi@comp.iust.ac.ir
{mohsen.fayyaz77，y.yaghoobzadeh}@ut.ac.ir
mp792@cam.ac.uk
There has been a growing interest in interpreting the underlying dynamics of Transformers. While self-attention patterns were initially deemed as the primary option, recent studies have shown that integrating other components can yield more accurate explanations. This paper introduces a novel token attribution analysis method that incorporates all the components in the encoder block and aggregates this throughout layers. Through extensive quantitative and qualitative experiments, we demonstrate that our method can produce faithful and meaningful global token attributions. Our experiments reveal that incorporating almost every encoder component results in increasingly more accurate analysis in both local (single layer) and global (the whole model) settings. Our global attribution analysis significantly outperforms previous methods on various tasks regarding correlation with gradient-based saliency scores. Our code is freely available at https://github.com/mohsenfayyaz/GlobEnc.
The stellar performance of Transformers (Vaswani et al., 2017) has garnered a lot of attention to analyzing the reasons behind their effectiveness. The self-attention mechanism has been one of the main areas of focus (Clark et al., 2019; Kovaleva et al., 2019; Reif et al., 2019; Htut et al., 2019). However, there have been debates on whether raw attention weights are reliable anchors for explaining model's behavior or not (Wiegreffe and Pinter, 2019; Serrano and Smith, 2019; Jain and Wallace, 2019). Recently, it was shown that incorporating vector norms should be an indispensable part of any attention-based analysis<sup>1</sup> (Kobayashi et al., 2020,
Figure 1: Aggregated attribution maps  $(\mathcal{N}_{\mathrm{ENC}})$  for the [CLS] token for fine-tuned BERT on SST2 dataset (sentiment analysis). Our method (GlobEnc) is able to accurately quantify the global token attribution of the model.
2021). However, these norm-based studies incorporate only the attention block into their analysis, whereas Transformer encoder layer is composed of more components.
Another limitation of the existing analysis techniques is that they are usually constrained to the analysis of single layer attributions. In order to expand the analysis to multi-layered encoder-based models in their entirety, an aggregation technique has to be employed. Abnar and Zuidema (2020) proposed two aggregation methods, rollout and max-flow, which combine raw attention weights across layers. Despite showing the outcome of their method to be faithful to a model's inner workings in specific cases, the final results are still unsatisfactory on a wide range of fine-tuned models.
Additionally, gradient-based alternatives (Simonyan et al., 2014; Kindermans et al., 2016; Li et al., 2016) have been argued to provide a more robust basis for token attribution analysis (Atanasova et al., 2020; Brunner et al., 2020; Pascual et al., 2021). Nonetheless, the gradient-based alternatives have not been able to fully replace attention-based counterparts, mainly due to their high computa
\* Equal contribution.
<sup>1</sup>We also have shown the unreliability of weights due to norm disparities in probing studies (Fayyaz et al., 2021).
Human Language Technologies, pages 258 - 271
tional intensity.
In this paper, we propose a new global token attribution analysis method (GlobEnc) which is based on the encoder layer's output. In GlobEnc, the second layer normalization is also included in the norm-based analysis of each encoder layer. To aggregate attributions over all layers, we applied a modified attention rollout technique, returning global scores.
Through extensive experiments and comparing the global attribution with the input token attributions obtained by gradient-based saliency scores, we show that our method produces faithful and meaningful results (Figure 1). Our evaluations on models with distinct pre-training objectives and sizes (Devlin et al., 2019; Clark et al., 2020) show high correlations with gradient-based methods in global settings. Furthermore, with comparative studies on each aspect of GlobEnc, we find that: (i) norm-based methods achieve higher correlations than weight-based methods; (ii) incorporating residual connections plays an essential role in token attribution; (iii) considering the two layer normalizations improve our analysis only if coupled together; and (iv) aggregation across layers is crucial for an accurate whole-model attribution analysis.
- We expand the scope of analysis from attention block in Transformers to the whole encoder.
- Our method significantly improves over existing techniques for quantifying global token attributions.
- We qualitatively demonstrate that the attributions obtained by our method are plausibly interpretable.
In encoder-based language models (such as BERT), a Transformer encoder layer is composed of several components (Figure 2). The core component of the encoder is the self-attention mechanism (Vaswani et al., 2017), which is responsible for the information mixture of a sequence of token representations  $(\pmb{x}_1,\dots,\pmb{x}_n)$ . Each self-attention head computes a set of attention weights  $\pmb{A}^{h} = \{\alpha_{i,j}^{h}|1\leq i,j\leq n\}$  where  $\alpha_{i,j}^{h}$  is the raw attention weight from the  $i^{\mathrm{th}}$  token to the  $j^{\mathrm{th}}$  token in head  $h\in \{1,\ldots ,H\}$ . Therefore, the output representation  $(z_{i}\in \mathbb{R}^{d})$  for
Figure 2: The internal structure of a Transformer encoder layer. We show on the diagram the components that are incorporated by each token attribution analysis method. Our method incorporates the whole encoder  $(\mathcal{N}_{\mathrm{ENC}})$  except for the direct effect of the fully connected feed-forward module. Diagram inspired by Alammar (2018).
the  $i^{\mathrm{th}}$  token of a multi-head (with  $H$  heads) self-attention module is computed by concatenating the heads' outputs followed by a head-mixing  $W_{O}$  projection:
$$
\boldsymbol {z} _ {i} = \operatorname {C O N C A T} \left(\boldsymbol {z} _ {i} ^ {1}, \dots , \boldsymbol {z} _ {i} ^ {H}\right) \boldsymbol {W} _ {O} \tag {1}
$$
where each head's output vector is generated by performing a weighted sum over the transformed value vectors  $\pmb{v}(\pmb{x}_j) \in \mathbb{R}^{d_v}$ :
$$
\boldsymbol {z} _ {i} ^ {h} = \sum_ {j = 1} ^ {n} \alpha_ {i, j} ^ {h} \boldsymbol {v} ^ {h} \left(\boldsymbol {x} _ {j}\right) \tag {2}
$$
Norm-based attention. While one may interpret the attention mechanism using the attention weights  $\mathbf{A}$ , Kobayashi et al. (2020) argued that doing so would ignore the norm of the transformed vectors multiplied by the weights, elucidating that the weights are insufficient for interpretation. Their solution enhanced the interpretability of attention weights by incorporating the value vectors  $\boldsymbol{v}(\boldsymbol{x}_j)$  and the following projection  $W_O$ . By reformulating Equation 1, we can consider  $\boldsymbol{z}_i$  as a summation over the attentions heads:
$$
\boldsymbol {z} _ {i} = \sum_ {h = 1} ^ {H} \sum_ {j = 1} ^ {n} \alpha_ {i, j} ^ {h} \underbrace {\boldsymbol {v} ^ {h} (\boldsymbol {x} _ {j}) \boldsymbol {W} _ {O} ^ {h}} _ {f ^ {h} (\boldsymbol {x} _ {j})} \tag {3}
$$
Using this reformulation $^2$ , Kobayashi et al. proposed a norm-based token attribution analysis method,  $\mathcal{N} \coloneqq (||z_{i \leftarrow j}||) \in \mathbb{R}^{n \times n}$ , to measure each token's contribution in a self-attention module:
$$
\boldsymbol {z} _ {i \leftarrow j} = \sum_ {h = 1} ^ {H} \alpha_ {i, j} ^ {h} f ^ {h} (\boldsymbol {x} _ {j}) \tag {4}
$$
They showed that incorporating the magnitude of the transformation function  $(f^{h}(\pmb{x}))$  is crucial in assessing the input tokens' contribution to the self-attention output.
Residual connections & Layer Normalizations. Kobayashi et al. (2021) added the attention block's Layer Normalization (LN#1) and Residual connection (RES#1) to its prior norm-based analysis to assess the impact of residual connections and layer normalization inside an attention block.  $\mathcal{N}_{\mathrm{RES}} := (||z_{i\leftarrow j}^{+}||) \in \mathbb{R}^{n\times n}$  is the analysis method which incorporates the attention block's residual connection. The input vector  $\pmb{x}$  is added to the attribution of each token to itself to incorporate the influence of RES#1:
$$
\boldsymbol {z} _ {i \leftarrow j} ^ {+} = \sum_ {h = 1} ^ {H} \alpha_ {i, j} ^ {h} f ^ {h} (\boldsymbol {x} _ {i}) + \mathbf {1} [ i = j ] \boldsymbol {x} _ {i} \quad (5)
$$
They proposed a method for decomposing  $\mathrm{LN}^3$  into a summation of normalizations:
$$
\mathrm {L N} \left(\boldsymbol {z} _ {i} ^ {+}\right) = \sum_ {j = 1} ^ {n} g _ {\boldsymbol {z} _ {i} ^ {+}} \left(\boldsymbol {z} _ {i \leftarrow j} ^ {+}\right) + \boldsymbol {\beta} \tag {6}
$$
$$
g _ {\boldsymbol {z} _ {i} ^ {+}} (\boldsymbol {z} _ {i \leftarrow j} ^ {+}) := \frac {\boldsymbol {z} _ {i \leftarrow j} ^ {+} - m (\boldsymbol {z} _ {i \leftarrow j} ^ {+})}{s (\boldsymbol {z} _ {i} ^ {+})} \odot \boldsymbol {\gamma}
$$
where  $m(.)$  and  $s(.)$  are the element-wise mean and standard deviation of the input vector (cf.  $\S \mathrm{A}.1$ ). The decomposition can be applied to the contribution vectors:
$$
\tilde {\boldsymbol {z}} _ {i \leftarrow j} = g _ {\boldsymbol {z} _ {i} ^ {+}} \left(\sum_ {h = 1} ^ {H} \alpha_ {i, j} ^ {h} f ^ {h} (\boldsymbol {x} _ {i}) + \mathbf {1} [ i = j ] \boldsymbol {x} _ {i}\right) \tag {7}
$$
Accordingly, we can compute the magnitude  $\mathcal{N}_{\mathrm{RESLN}} := (||\tilde{z}_{i \leftarrow j}||) \in \mathbb{R}^{n \times n}$ , which represents the amount of influence of an encoder layer's input
token  $j$  on its output token  $i$ . Based on this formulation, a context-mixing ratio could be defined as:
$$
r _ {i} = \frac {\left| \left| \sum_ {j = 1 , j \neq i} ^ {n} \tilde {z} _ {i \leftarrow j} \right| \right|}{\left| \left| \sum_ {j = 1 , j \neq i} ^ {n} \tilde {z} _ {i \leftarrow j} \right| \right| + \left| \left| \tilde {z} _ {i \leftarrow i} \right| \right|} \tag {8}
$$
Experiments by Kobayashi et al. (2021) revealed considerably low  $r$  values which indicate the huge impact of the residual connections. In other words, the model tends to preserve token representations more than mixing them with each other.
Our method for input token attribution analysis has a holistic view and takes into account almost every component within the encoder layer. To this end, we first extend the norm-based analysis of Kobayashi et al. (2021) by incorporating the encoder's output LN#2. We then apply an aggregation technique to combine the information flow throughout all layers.
Encoder layer output  $\neq$  Attention block output. While the RES#1 and the LN#1 from the attention block are included in the analysis of Kobayashi et al. (2021), the subsequent FFN, RES#2, and output LN#2 are ignored (see Fig. 2). Hence,  $\mathcal{N}_{\mathrm{RESLN}}$  might not be indicative of the entire encoder layer's function. To address this issue, we additionally include the encoder layer components from the attention block outputs  $(\tilde{z}_i)$  to the output representations  $(\tilde{x}_i)$ . The output of each encoder  $(\tilde{x}_i)$  is computed as follows:
$$
\tilde {z} _ {i} ^ {+} = \operatorname {F F N} \left(\tilde {z} _ {i}\right) + \tilde {z} _ {i} \tag {9}
$$
$$
\tilde {\boldsymbol {x}} _ {i} = \operatorname {L N} \left(\tilde {\boldsymbol {z}} _ {i} ^ {+}\right)
$$
We apply the LN decomposition rule in Eq. 7 to separate the impacts of residual and FFN output:
$$
\tilde {\boldsymbol {x}} _ {i} = \sum_ {j = 1} ^ {n} \left(g _ {\tilde {\boldsymbol {z}} _ {i} ^ {+}} \left(\operatorname {F F N} \left(\tilde {\boldsymbol {z}} _ {i \leftarrow j}\right)\right) + g _ {\tilde {\boldsymbol {z}} _ {i} ^ {+}} \left(\tilde {\boldsymbol {z}} _ {i \leftarrow j}\right)\right) + \beta \left. \right. \tag {10}
$$
Given that the activation function between the two fully connected layers in the FFN component is non-linear (Vaswani et al., 2017), a linear decomposition similar to Eq. 7 cannot be derived. As a result, we omit FFN's influence on the contribution of each token and instead consider RES#2, approximating  $\tilde{\pmb{x}}_{i\leftarrow j}$  as  $g_{\tilde{z}_i^+}(\tilde{z}_{i\leftarrow j})$ . Nevertheless, it should be noted that the FFN still preserves some influence on this new setting due to the presence of
$\mathbf{W}_O^h$  is a head-specific slice of the original  $W_{O}$  projection. For more information about the reformulation process, see Appendix C in Kobayashi et al. (2021)  $\gamma \in \mathbb{R}^d$  and  $\beta \in \mathbb{R}^d$  are the trainable weights of LN. Similar to Kobayashi et al. (2021) we ignore  $\beta$ .
$s(\tilde{z}_i^+)$  in  $g_{\tilde{z}_i^+}(\tilde{z}_{i\leftarrow j})$  . Similarly to Eq. 7, we can introduce a more inclusive layerwise analysis method  $\mathcal{N}_{\mathrm{ENC}}\coloneqq (||\tilde{\boldsymbol{x}}_{i\leftarrow j}||)\in \mathbb{R}^{n\times n}$  from input token  $j$  to output token  $i$  using:
$$
\tilde {\boldsymbol {x}} _ {i \leftarrow j} \approx g _ {\tilde {\boldsymbol {z}} _ {i} ^ {+}} (\tilde {\boldsymbol {z}} _ {i \leftarrow j}) = \frac {\tilde {\boldsymbol {z}} _ {i \leftarrow j} - m (\tilde {\boldsymbol {z}} _ {i \leftarrow j})}{s (\tilde {\boldsymbol {z}} _ {i} ^ {+})} \odot \gamma \tag {11}
$$
Aggregating multi-layer attention. To create an aggregated attribution score, Abnar and Zuidema (2020) proposed describing the model's attentions via modelling the information flow with a directed graph. They introduced attention rollout method, which linearly combines raw attention weights along all available paths in the pairwise attention graph. The attention rollout of layer  $\ell$  w.r.t. the inputs is computed recursively as follows:
$$
\tilde {\boldsymbol {A}} _ {\ell} = \left\{ \begin{array}{l l} \hat {\boldsymbol {A}} _ {\ell} \tilde {\boldsymbol {A}} _ {\ell - 1} & \ell > 1 \\ \hat {\boldsymbol {A}} _ {\ell} & \ell = 1 \end{array} \right. \tag {12}
$$
$$
\hat {\boldsymbol {A}} _ {\ell} = 0. 5 \bar {\boldsymbol {A}} _ {\ell} + 0. 5 \boldsymbol {I} \tag {13}
$$
$\bar{A}_{\ell}$  is the raw attention map averaged across all heads in layer  $\ell$ . This method assumes equal contribution from the residual connection and multi-head attention (See Fig. 2). Hence, an identity matrix is summed and renormalized, giving  $\hat{A}_{\ell}$ .
For aggregating the layerwise analysis methods, we use the rollout technique with minor modifications. As many of the methods already include residual connections, we only use Eq. 12 (replacing  $\hat{A}_{\ell}$  with the desired method's attribution matrix in layer  $\ell$ ) to calculate the rollout of a given method. However, for methods that do not assume the residual connection, we define a corresponding "Fixed" variation using Eq. 13 that incorporates a fixed residual effect ( $r_i \approx 0.5$ ). We refer to our proposed global method—aggregating the  $\mathcal{N}_{\mathrm{ENC}}$  across all layers by the rollout method—as GlobEnc. In what follows we report our experiments, comparing GlobEnc with several other settings.
In this section, we introduce the datasets and the token attribution analysis methods used in our evaluations, followed by the experimental setup and results.
All analysis methods are evaluated on three different classification tasks. To cover sentiment detection tasks we use SST2 (Socher et al., 2013), MNLI (Williams et al., 2018) for Natural Language Inference and Hatexplain (Mathew et al., 2021) in hate speech detection.
4.2 Analysis Methods
We use two categories of explainability approaches in our work: Weight-based and Norm-based. The Weight-based approaches employed in our experiments are as follows:
-  $\mathcal{W}$ : The raw attention maps averaged across all heads (See  $\overline{A}_{\ell}$  in §2).
-  $\mathcal{W}_{\mathrm{FIXEDRES}}$ : Abnar and Zuidema's assumption; add an identity matrix as a fixed residual to  $\overline{\boldsymbol{A}}_{\ell}$  (see  $\hat{\boldsymbol{A}}_{\ell}$  in Eq. 13).
-  $\mathcal{W}_{\mathrm{RES}}$ : The corrected version of  $\mathcal{W}$  in which accurate residuals are added based on the context-mixing ratios of  $\mathcal{N}_{\mathrm{ENC}}$ :
$$
\hat {r} _ {i} = \frac {\left\| \sum_ {j = 1 , j \neq i} ^ {n} \tilde {\boldsymbol {x}} _ {i \leftarrow j} \right\|}{\left\| \sum_ {j = 1 , j \neq i} ^ {n} \tilde {\boldsymbol {x}} _ {i \leftarrow j} \right\| + \| \tilde {\boldsymbol {x}} _ {i \leftarrow i} \|} \tag {14}
$$
In order to enforce  $\mathcal{W}_{\mathrm{RES}}$  to have a context-mixing ratio equal to  $\hat{r}_i$ , it is essential to zero-out the diagonal elements (the tokens' attentions to themselves) of  $\bar{\boldsymbol{A}}_{\ell}$  and renormalize it:
$$
\boldsymbol {A} _ {\ell} ^ {\prime} = (\boldsymbol {I} - \operatorname {d i a g} (\bar {\boldsymbol {A}} _ {\ell})) ^ {- 1} (\bar {\boldsymbol {A}} _ {\ell} - \operatorname {d i a g} (\bar {\boldsymbol {A}} _ {\ell}))
$$
$$
\begin{array}{l} \mathcal {W} _ {\text {R E S}} := \mathbf {d i a g} \left(\hat {r} _ {1}, \dots , \hat {r} _ {n}\right) A _ {\ell} ^ {\prime} \\ + \operatorname {d i a g} (1 - \hat {r} _ {1}, \dots , 1 - \hat {r} _ {n}) I \tag {15} \\ \end{array}
$$
The Norm-based analysis methods, namely  $\mathcal{N}$ ,  $\mathcal{N}_{\mathrm{RES}}$  and  $\mathcal{N}_{\mathrm{RESLN}}$  were discussed in detail in §2. Our proposed norm-based method  $\mathcal{N}_{\mathrm{ENC}}$  was explained in §3. For an ablation study, we introduce  $\mathcal{N}_{\mathrm{FIXEDRES}}$  which is  $\mathcal{N}$ , corrected with a fixed residual similar to  $\mathcal{W}_{\mathrm{FIXEDRES}}^5$ .
$$
\hat {\mathcal {N}} = \left(\frac {| | \boldsymbol {z} _ {i \leftarrow j} | |}{\sum_ {j} | | \boldsymbol {z} _ {i \leftarrow j} | |}\right) \in \mathbb {R} ^ {n \times n} \tag {16}
$$
$$
\mathcal {N} _ {\text {F I X E D R E S}} := 0. 5 \hat {\mathcal {N}} + 0. 5 \boldsymbol {I}
$$
4Note that in most of our experiments, we use all these methods within the rollout aggregation technique.
5The only difference is that we need to normalize  $\mathcal{N}$  before adding an identity matrix.
In §4.5, we will demonstrate our comparative studies between the aforementioned methods and GlobEnc.
4.3 Gradient-based Methods for Faithfulness Analysis
Gradient-based methods are widely used as alternatives for attention-based counterparts for quantifying the importance of a specific input feature in making the right prediction (Li et al., 2016; Atanasova et al., 2020). In this section we discuss the specific gradient-based methods we use, namely saliency, HTA, and our adjusted HTA.
4.3.1 Saliency
Gradient-based saliency is based on the gradient of the output  $(y_{c})$  w.r.t. the input embeddings  $(e_i^0)$ . One of the most accurate variations of the saliency family is the gradient  $\times$  input method (Kindermans et al., 2016) where the input embeddings are multiplied by the gradients. Thus, the contribution score of input token  $i$  is determined by first computing the element-wise product of the input embeddings  $(e_i^0)$  and the gradients of the true class output score  $(y_{c})$  w.r.t. the input embeddings. Then, the L2 norm of the scaled gradients is computed to derive the final score:
$$
\text {S a l i e n c y} _ {i} = \left\| \frac {\partial y _ {c}}{\partial \boldsymbol {e} _ {i} ^ {0}} \odot \boldsymbol {e} _ {i} ^ {0} \right\| _ {2} \tag {17}
$$
4.3.2 HTA x Inputs
To determine an upper bound on the information mixing within each layer, we use a modified version of Hidden Token Attribution (Brunner et al., 2020, HTA). In the original version, HTA is the sensitivity between any two vectors in the model's computational graph. However, inspired by the gradient  $\times$  input method (Kindermans et al., 2016), which has shown more faithful results (Atanasova et al., 2020; Wu and Ong, 2021), we multiply the input vectors by the gradients and then apply a Frobenius norm. We compute the attribution from hidden embedding  $j(e_j^{\ell -1})$  to hidden embedding  $i(e_i^\ell)$  in layer  $\ell$  as:
$$
\boldsymbol {c} _ {i \leftarrow j} ^ {\ell} = \left\| \frac {\partial \boldsymbol {e} _ {i} ^ {\ell}}{\partial \boldsymbol {e} _ {j} ^ {\ell - 1}} \odot \boldsymbol {e} _ {j} ^ {\ell - 1} \right\| _ {F} \tag {18}
$$
Computing HTA-based attribution matrices is an extremely computation-intensive task (especially for long texts) due to the high dimensionality of hidden embeddings. Hence, we only use this method
for 256 examples from the SST-2 task's validation set. It is worth noting that extracting the HTA-based contribution maps for the aforementioned data took approximately 2 hours, whereas computing the maps for the entire analysis methods stated in §4.2 took only 5 seconds. $^{6}$
4.4 Setup
We employ HuggingFace's Transformers library<sup>7</sup> (Wolf et al., 2020) and the BERT-base-uncased model. For fine-tuning BERT, epochs vary from 3 to 5, and the batch size and learning rate are 32 and 3e-5, respectively.<sup>8</sup> We also carried out the main experiment on BERT-large and ELECTRA (Devlin et al., 2019; Clark et al., 2020) where the results are reported at §A.2.
After rollout aggregation of each analysis method, we obtain an accumulated attribution matrix for every layer  $(\ell)$  of BERT. These matrices indicate the overall contribution of each input token to all token representations in layer  $\ell$ . Since the classifier in a fine-tuned model is attached to the final layer representation of the [CLS] token, we consider the first row (corresponding to [CLS] attributions) of the last layer attribution matrix. This vector represents the contribution of each input token to the model's final decision. As a measure of faithfulness of the resulting vector with the saliency scores, we report the Spearman's rank correlation between the two vectors.
Table 1 shows the Spearman correlation of saliency scores with the aggregated attribution scores from [CLS] to input tokens at the final layer. In order to determine the contribution of each component of encoder layer to the overall performance, we report the results for attribution analysis methods discussed in §4.2. Our results demonstrate that incorporating the vector norms, residual connection, and both layer normalizations yields the highest correlation  $(\mathcal{N}_{\mathrm{ENC}})$ . In what follows, we discuss the impact of incorporating various parts in the analysis.
4.5.1 On the role of vector norms
As also suggested by Kobayashi et al. (2020), vector norms play an important role in determining
Conducted on a 3070 GPU machine.
<sup>7</sup>https://github.com/huggingface/transformers
8Recommended by Devlin et al. (2019).
Attention Rollout
SST2	MNLI	HATEXPLAIN
Weight-based (W)	-0.11 ± 0.26	-0.06 ± 0.22	0.12 ± 0.26
w/ Fixed Residual (\( \mathcal{W}_{FIXEDRES} \))9	-0.24 ± 0.26	-0.05 ± 0.26	0.13 ± 0.28
w/ Residual (\( \mathcal{W}_{RES} \))	0.19 ± 0.26	0.27 ± 0.25	0.53 ± 0.24
Norm-based (N)	0.44 ± 0.20	0.47 ± 0.16	0.43 ± 0.22
w/ Fixed Residual (\( \mathcal{N}_{FIXEDRES} \))	0.48 ± 0.20	0.55 ± 0.16	0.48 ± 0.22
w/ Residual (\( \mathcal{N}_{RES} \))	0.73 ± 0.13	0.75 ± 0.10	0.66 ± 0.17
w/ Residual + Layer Norm 1 (\( \mathcal{N}_{RESLN} \))	-0.21 ± 0.26	-0.06 ± 0.26	0.08 ± 0.28
w/ GlobEnc: [Residual + Layer Norm 1, 2] (\( \mathcal{N}_{ENC} \))	0.77 ± 0.12	0.78 ± 0.09	0.72 ± 0.17
Table 1: Spearman's rank correlation of attribution based importance (aggregated by rollout) with saliency scores for the validation set for the BERT model fine-tuned on SST-2, MNLI, and HateXplain. In fixed residual cases, the context-mixing ratio is roughly 0.5, and in weight-based w/ residual  $(\mathcal{W}_{\mathrm{RES}})$ , it is corrected with context-mixing ratio of  $(\mathcal{N}_{\mathrm{ENC}})$ . The numbers are the average on all the validation set examples  $\pm$  the standard deviation.