SlowGuess/NAACL_2022_OCR huggingface dataset:
Figure 3: Spearman's rank correlation of aggregated attribution scores with saliency scores across layers. The  $99\%$  confidence intervals are shown as (narrow) shaded areas around each line.  $\mathcal{N}_{\mathrm{ENC}}$  achieves the highest correlation in almost every layer.
attention outputs. This is highlighted by the significant gap between weight-based and norm-based settings across all datasets in Table 1.
We also show the correlation of the aggregated attention for all layers in Figure 3. The norm-based settings  $(\mathcal{N}$  and  $\mathcal{N}_{\mathrm{RES}})$  attain higher correlation than the weight-based counterparts  $(\mathcal{W}$  and  $\mathcal{W}_{\mathrm{RES}})$  almost in all layers, confirming the importance of incorporating vector norms.
4.5.2 On the role of residual connections
Kobayashi et al. (2021) showed that in the encoder layer, the output representations of each token is mainly determined by its own representation, and the contextualization from other tokens' plays a marginal role. This is in contrary to the simplifying assumption made by Abnar and Zuidema (2020) who used a fixed context-mixing ratio of 0.5 (assuming that BERT equally preserves and mixes the representations). This setting is shown as weight-based with fixed residual  $(\mathcal{W}_{\mathrm{FIXEDRES}})$  in Table 1. We compare this setting against  $\mathcal{W}_{\mathrm{RES}}$  (see §4.2).  $\mathcal{W}_{\mathrm{RES}}$  is similar to  $\mathcal{W}_{\mathrm{FIXEDRES}}$  (in that it does not take into account vector norms) but differs in that it considers a dynamic mixing ratio (the one from  $\mathcal{N}_{\mathrm{ENC}}$ ). The huge performance gap between the two settings in Table 1 clearly highlights the importance of considering accurate context-mixing ratios. Therefore, it is crucial to consider the residual connection in the attention block for input token attribution analysis.
To further demonstrate the role of residual connections, we utilize the introduced method in §4.2, where we modified the norm-based attentions with fixed residual  $(r\approx 0.5)$ . The comparison of norm-based without any residual  $(\mathcal{N})$  and with a fixed residual  $(\mathcal{N}_{\mathrm{FIXEDRES}})$  shows a consistent improvement for the latter across all the datasets. This provides evidence on that having a fixed uniform context-mixing ratio is better than neglecting the residual connection altogether.
Finally, when we aggregate the norm-based analysis with an accurate dynamic context-mixing ratio  $(\mathcal{N}_{\mathrm{RES}})$ , we observe the highest correlation up to
As mentioned in §4.2, this analysis method is based on the original experiment by Abnar and Zuidema (2020). Our experiments on SST2 differ from theirs in two aspects: (i) we opted for gradient  $\times$  input saliencies, while they used the sum of gradients (sensitivity) (ii) instead of BERT, they used a DistillBERT fine-tuned model (Sanh et al., 2019). However, their reported results in their sepcific setup (Spearman Corr. = 0.14) still yields significantly lower results than GlobEnc.
Figure 4: Single layer Pearson correlation of HTA maps with attribution maps. The  $99\%$  confidence intervals are shown as shaded areas around each line.  $\mathcal{N}_{\mathrm{RESLN}}$  shows considerably less association with HTA.
this point, without layer normalization.
4.5.3 On the role of layer normalization
In Table 1 we see a sudden drop in correlations for  $\mathcal{N}_{\mathrm{RESLN}}$ . Although this method considers vector norms and residuals, incorporating LN#1 in the encoder seems to have deteriorated the accuracy for token attribution analysis. To determine whether this deterioration of correlation in aggregated attributions is also present in individual single layers, we compare the HTA maps as a baseline with the attribution matrices extracted from different analysis methods. Figure 4 shows the correlation of HTA attribution maps with the maps obtained by  $\mathcal{N}_{\mathrm{RES}}$ ,  $\mathcal{N}_{\mathrm{RESLN}}$ , and  $\mathcal{N}_{\mathrm{ENC}}$  methods. The results indicate that  $\mathcal{N}_{\mathrm{RESLN}}$  exhibits a significantly lower association.
The question that arises here is that how incorporating an additional component of the encoder (LN#1) in  $\mathcal{N}_{\mathrm{RESLN}}$  degrades the results (compared to  $\mathcal{N}_{\mathrm{RES}}$ ). To answer this question, we investigated the learned weights of LN#1 and LN#2. The outlier weights[10] in specific dimensions of LNs are shown to be significantly influential on the model's performance (Kovaleva et al., 2021; Luo et al., 2021). It is interesting to note that based on our observations, the outlier weights of the two layer norms seem to be the opposite of each other. Figure 5 demonstrates the weight values in layer 11 and also the correlation of the outlier weights across layers. The large negative correlations confirm that the outlier weights work contrary to each other. We speculate that the effect of outliers in the two layer norms is
Figure 5: The Pearson correlation between outlier weights of LN#1 and LN#2 across layers. The weight values for layer 11 are shown as well.
L1	L6	L12	MAX
Indiv.	N	-.50 ± .18	+.28 ± .23	+.40 ± .21	+.41 ± .21
NRES	-.48 ± .18	+.29 ± .24	+.41 ± .19	+.41 ± .19
NENC	-.47 ± .18	+.29 ± .24	+.41 ± .19	+.41 ± .19
Rollout	N	-.50 ± .18	+.44 ± .20	+.44 ± .20	+.44 ± .20
NRES	-.48 ± .18	+.70 ± .14	+.73 ± .13	+.73 ± .13
NENC	-.47 ± .18	+.74 ± .14	+.77 ± .12	+.78 ± .12
Table 2: Spearman's rank correlation of attribution-based scores (individual and aggregated by rollout) with saliency scores for the validation set for the BERT model fine-tuned on SST-2. The results are reported for layers 1, 6, 12, and the maximum of all layers. Utilizing rollout aggregation achieves higher correlations than individual layers.
partly cancelled out when both are considered.
As shown in Figure 2, the FFN and the second layer normalization are on top of the attention block. However,  $\mathcal{N}_{\mathrm{RESLN}}$  does not incorporate the components outside of the attention block. As described in §3, in our local analysis method  $\mathcal{N}_{\mathrm{ENC}}$  we incorporate the second layer normalization in the transformer's encoder (Figure 2), thus considering the whole encoder block (except FFN). Overall, our global method, GlobEnc, yields the best results among all the methods evaluated in our experiments. In general, Table 1 suggests that incorporating each component of the encoder will increase the correlation; however, the two layer normalizations should be considered together.
4.5.4 On the role of aggregation
We carried out an additional analysis to verify if incorporating vector norms, residual connection and layer normalizations in individual layers is adequate for achieving high correlations, or if it is also necessary to aggregate them via rollout. Table 2
<sup>10</sup>We identify the dimensions where the weights are at least  $3\sigma$  from the mean as outliers (Kovaleva et al., 2021).
shows the correlation results in different layers for raw attributions (without aggregation) and for the aggregated attributions using the rollout method. Applying rollout method on attribution maps up to each layer results in higher correlations with the saliency scores than the raw single layer attribution maps, especially in deeper layers. Therefore, attention aggregation is essential for global input token attribution analysis.
An interesting point in Figure 3, which shows the correlation of the aggregated methods throughout the layers, is that the correlation curves flatten out after only a few layers. This indicates that BERT identifies decisive tokens only after the first few layers. The final layers only make minor adjustments to this order. Nevertheless, it is worth noting that the order of attribution does not necessarily imply the model's final decision and the final result may still change for the better or worse (Zhou et al., 2020).
4.5.5 Qualitative analysis
To qualitatively answer if the aggregated attribution maps provide plausible and meaningful interpretations, we take a closer look at the attribution maps generated by GlobEnc. Figure 1 shows the GlobEnc attribution of the model trained on SST-2. Each layer demonstrates the [CLS] token's aggregated attribution to input tokens up to the corresponding layer. The example inputs are "a deep and meaningful film." and "big fat waste of time.", both correctly classified by the model. In both cases, GlobEnc focuses on the relevant words for sentiment classification, i.e., "meaningful" and "waste". An interesting observation in Figure 1 is that in the first few layers, the [CLS] token mostly attends to itself while other tokens have marginal impact. As the representations get more contextualized in deeper layers, the attribution correctly shifts to the words which indicate the sentiment of the sentence. $^{12}$  More examples from MNLI and SST2 datasets, including misclassified examples are available at  $\S A.3$ . Our qualitative analysis suggests that GlobEnc can be useful for a reasonable interpretation of attention mechanism in BERT, ELECTRA, and possibly any other transformer-based model.
While numerous studies have used attention weights to analyze and interpret the self-attention mechanism (Clark et al., 2019; Kovaleva et al., 2019; Reif et al., 2019; Htut et al., 2019), the use of mere attention weights to explain a model's inner workings has been an active topic of debate (Serrano and Smith, 2019; Jain and Wallace, 2019; Wegreffe and Pinter, 2019). Several solutions have been proposed to address this issue, usually through converting raw attention weights to scores that provide better explanations. Brunner et al. (2020) used the transformation function  $f^{h}(\boldsymbol{x}_{j})$  to introduce effective attentions—the orthogonal component of the attention matrix in  $f^{h}(\boldsymbol{x}_{j})$  null space—to explain the inner workings of each layer. However, this technique ignores other components in the encoder and is computationally expensive due to the SVD required to compute the effective attentions. Kobayashi et al. (2020) incorporated the modified vector and introduced a vector norms-based analysis. This was later extended by integrating residual connections and layer normalization components to enhance the accuracy of explanations (Kobayashi et al., 2021). But, as discussed in §4.5, relying solely on LN#1 does not produce accurate results.
While these methods can be employed for single-layer (local) analysis, multi-layer attributions are not necessarily correlated with single-layer attributions due to the significant degree of information combination through multi-layer language models (Pascual et al., 2021; Brunner et al., 2020). Various saliency methods exist for explaining the model's decision based on the input (Li et al., 2016; Bastings and Filippova, 2020; Atanasova et al., 2020; Wu and Ong, 2021; Moebbi et al., 2021). However, these approaches are not primarily designed for computing inter-token attributions. To fill this gap, Brunner et al. (2020) proposed HTA, which is based on the gradient of each hidden embedding in relation to the input embeddings. In §4.3.2, we extend HTA to incorporate the impact of the input vectors. However, HTA is extremely computationally intensive. Attention rollout (see §3) and attention flow—which involve solving a max-flow problem on the attention graph—are two aggregation approaches introduced by Abnar and Zuidema (2020), in which raw attention weights (with equally weighted residual weights) are aggregated within multiple layers. We showed that attention rollout does not perform well on the raw
$^{11}\mathcal{W}_{\mathrm{RES}}$  is the only exception with a constant increase; this method is gradually and artificially corrected by  $\mathcal{N}_{\mathrm{ENC}}$  context mixing ratios.
12 Complete attention maps in Figure A.3 show that, similarly to [CLS], other tokens also focus on sentiment tokens.
attention maps of language models fine-tuned on downstream tasks and that this problem can be resolved by utilizing attribution norms.
In this work, we proposed a novel method for single layer token attribution analysis which incorporates the whole encoder layer, i.e., the attention block and the output layer normalization. When aggregated across layers using the rollout method, our technique achieves quantitatively and qualitatively plausible results. Our evaluation of different analysis methods provided evidence on roles played by individual components of the encoder layer, i.e., the vector norms, the residual connections, and the layer normalizations. Furthermore, our in-depth analysis suggested that the two layer normalizations in the encoder layer counteract each other; hence, it is important to couple them for an accurate analysis.
Additionally, using a newly proposed and improved version of Hidden Token Attribution, we demonstrated that encoder-based attribution analysis is more accurate when compared to other partial solutions in a single layer (local-level). This is consistent with our global observations. Quantifying global input token attribution based on our work can provide a meaningful explanation of the whole model's behavior. In future work, we plan to apply our global analysis method on various datasets and models, to provide valuable insights into model decisions and interpretability.
Samira Abnar and Willem Zuidema. 2020. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190-4197, Online. Association for Computational Linguistics.
Jay Alammar. 2018. The illustrated transformer [blog post].
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. 2020. A diagnostic study of explainability techniques for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3256-3274, Online. Association for Computational Linguistics.
Jasmijn Bastings and Katja Filippova. 2020. The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? In Proceedings of the Third BlackboxNLP Workshop
on Analyzing and Interpreting Neural Networks for NLP, pages 149-155, Online. Association for Computational Linguistics.
Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. 2020. On identifiability in transformers. In International Conference on Learning Representations.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pretraining text encoders as discriminators rather than generators. In ICLR.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Mohsen Fayyaz, Ehsan Aghazadeh, Ali Modarressi, Ho-sein Mohebbi, and Mohammad Taher Pilehvar. 2021. Not all models localize linguistic knowledge in the same place: A layer-wise probing on BERToids' representations. In Proceedings of the Fourth Black-boxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 375-388, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Phu Mon Htut, Jason Phang, Shikha Bordia, and Samuel R. Bowman. 2019. Do attention heads in BERT track syntactic dependencies? CoRR, abs/1911.12246.
Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543-3556, Minneapolis, Minnesota.
Pieter-Jan Kindermans, Kristof Schütt, Klaus-Robert Müller, and Sven Dähne. 2016. Investigating the influence of noise and distractors on the interpretation of neural networks.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
pages 7057-7075, Online. Association for Computational Linguistics.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2021. Incorporating Residual and Normalization Layers into Analysis of Masked Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4547-4568, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. 2021. BERT busters: Outlier dimensions that disrupt transformers. In *Findings of the Association for Computational Linguistics: ACLIJCNLP* 2021, pages 3392-3405, Online. Association for Computational Linguistics.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365-4374, Hong Kong, China.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 681-691, San Diego, California. Association for Computational Linguistics.
Ziyang Luo, Artur Kulmizev, and Xiaoxi Mao. 2021. Positional artefacts propagate through masked language model embeddings. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5312-5327, Online. Association for Computational Linguistics.
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. Hatexplain: A benchmark dataset for explainable hate speech detection. Proceedings of the AAAI Conference on Artificial Intelligence, 35(17):14867-14875.
Hosein Mohebbi, Ali Modarressi, and Mohammad Taher Pilehvar. 2021. Exploring the role of BERT token representations to explain sentence probing results. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 792-806, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Damian Pascual, Gino Brunner, and Roger Wattenhofer. 2021. Telling BERT's full story: from local attention to global aggregation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 105-124, Online. Association for Computational Linguistics.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of bert. In Advances in Neural Information Processing Systems, pages 8594-8603.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In NeurIPS EMC² Workshop.
Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931-2951, Florence, Italy.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Sarah Wiegrefe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11-20, Hong Kong, China. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Zhengxuan Wu and Desmond C. Ong. 2021. On explaining your explanations of BERT: an empirical study with sequence classification. CoRR, abs/2101.00196.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020. Bert loses patience: Fast and robust inference with early exit. In Advances in Neural Information Processing Systems, volume 33, pages 18330-18341. Curran Associates, Inc.
A.1 LN Formulation
$$
m (\boldsymbol {a}) := \frac {1}{d} \sum_ {k} \boldsymbol {a} ^ {(k)},
$$
$$
s (\boldsymbol {a}) := \sqrt {\frac {1}{d} \sum_ {k} (m (\boldsymbol {a}) - \boldsymbol {a} ^ {(k)} + \epsilon) ^ {2}}
$$
where  $\epsilon$  is a small constant
A.2 More Models
In this section we provide the results for BERTlarge and ELECTRA-base. For both models, our method outperforms the previous analysis methods. The results are reported in Tables A.1 and A.2.
A.3 More Examples
Aggregated attributions by different methods throughout layers is shown in Figure A.2. Our proposed method shows more plausible results.
Aggregated attribution map for layer 12 is shown in Figure A.3. In this figure, the effect of each token can be seen on all other tokens and not just the [CLS] token.
More examples for MNLI dataset are shown for BERT-base in Figure A.4, for BERT-large in Figure A.6, and for ELECTRA in Figure A.5. Moreover, misclassified examples of SST2 dataset are shown in Figure A.1.
BERT-large	Attention Rollout
SST2	MNLI	HATEXPLAIN
Weight-based (W)	-0.38 ± 0.16	-0.61 ± 0.14	-0.41 ± 0.25
w/ Fixed Residual (\( \mathcal{W}_{FIXEDRES} \))	-0.25 ± 0.19	-0.48 ± 0.19	-0.21 ± 0.30
w/ Residual (\( \mathcal{W}_{RES} \))	-0.10 ± 0.21	0.33 ± 0.23	0.09 ± 0.30
Norm-based (N)	0.44 ± 0.24	0.13 ± 0.27	0.48 ± 0.25
w/ Fixed Residual (\( \mathcal{N}_{FIXEDRES} \))	0.49 ± 0.24	0.26 ± 0.25	0.49 ± 0.30
w/ Residual (\( \mathcal{N}_{RES} \))	0.77 ± 0.11	0.66 ± 0.12	0.73 ± 0.16
w/ Residual + Layer Norm 1 (\( \mathcal{N}_{RESLN} \))	-0.07 ± 0.23	-0.35 ± 0.24	0.06 ± 0.32
w/ GlobEnc: [Residual + Layer Norm 1, 2] (\( \mathcal{N}_{ENC} \))	0.83 ± 0.08	0.77 ± 0.09	0.76 ± 0.17