SlowGuess/NAACL_2021_OCR huggingface dataset:
Figure 6: Correlations of redundancy matrices of task pairs (using pre-trained BERT-base). Redundancy patterns are task-agnostic. They are very similar across different tasks, even with random inputs.
Figure 7: Performances of pruned BERT-base model for GLUE tasks.  $x$ -axis is the pruning ratio. Performances can be preserved after pruning even  $75 - 85\%$  attention heads for most tasks.
duce the computational load in the inference phase. However, our proposed pruning strategy is before fine-tuning and zero-shot (i.e., without knowing any data in fine-tuning tasks).
Comparing and evaluating existing pruning works are beyond the scope of this paper. We leave the pruning topic as a future work. However, we believe that attention redundancy analyses in this paper benefit developing efficient pruning methods.
7.1.1 Pruning Method
The whole procedure is as the following. First, we feed randomly generated token sequences as input data in BERT-base model and obtain the redundancy matrices like Figure 2. Second, a clustering algorithm is applied on one redundancy matrix or the averaged redundancy matrices. If a pruning ratio  $p$  is given and the clustering method needs a given number of cluster, then we set the cluster number to be  $\lceil 144 \times (1 - p) \rceil$ . Otherwise, a cluster-number-free clustering method is preferred. Third, a clustering goodness metric is used for each object (i.e., attention head) to obtain the cluster representative head which is kept during pruning. We can simply prune trainable parameters corresponding to other heads and relevant parameters in subsequent feed-forward layers, then fine-tune as usual for downstream tasks.
7.1.2 Pruning Performance
In this section, we applied the well-known spectral clustering $^{6}$  and Silhouette Score $^{7}$  as the clustering goodness metric to obtain cluster representative heads. We prune attention heads using the same obtained pruning strategy (since the pruning is zero-shot and task-agnostic) and fine-tune the pruned BERT-base model for each GLUE task with suggested hyper-parameters in Section 3. The fine-tuned performances on development sets are shown in Figure 7. Each sub-figure corresponds to one task.  $x$ -axis is the pruning ratio between  $5\%$  and  $95\%$  with an interval of  $5\%$ . We conduct 10 trials for each fine-tuning task. Red dashed line reflects the average performance without pruning $^{8}$ . Blue line plots averaged performances under different pruning ratios. Boundaries of shadow areas cover the best and worst results of the 10 trials.
In Figure 7, not surprisingly, performances drop as the pruning ratio increases. However, in 4 (SST-2, MRPC, QQP and RTE) out of these 8 tasks, we could use a pruning ratio as big as  $85\%$ , without significant performance loss  $(< 5\%)$  against an unpruned model. In MNLI and QNLI, the performance loss is bigger. But still, the pruning ratio
$^{6}$ https://scikit-learn.org/stable/modules/generated/sklearn.cluster.SpectralClustering.html
<sup>7</sup>https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html
<sup>8</sup>We obtain comparable or even better performances as the reported scores (https://github.com/huggingface/transformers/tree/master/ examples/text-classification) averaged on 10 trials.
Figure 8: Chances of a head being pruned at various pruning ratios when using our redundancy clustering based zero-shot pruning strategy (BERT-base model with random inputs). Results are averaged over 10 clustering trials. In each heatmap, the title shows the pruning ratio (larger pruning ratios reflect that more heads are pruned). The lighter the entry color is, the more often the head gets pruned. We observe that heads in earlier and deeper layers are pruned with high chances under larger pruning ratios. Please refer to Section 7.1.3 for more details.
can be as big as  $75\%$  to guarantee a small performance loss  $(< 5\%)$ . The only two outliers are CoLA and STS-B. We think their smaller validation sets  $(\leq 1000)$  may result in fluctuations and prevent us from making a strong conclusion, and a further study is deferred to future work.
7.1.3 Pruning Heads Visualization
In Figure 8, we visualize some pruning results averaged over 10 clustering trials. Lighter colors represent that one head is pruned more often. When pruning ratios are small (e.g.,  $< 15\%$ ), heads in the first four layers are pruned with higher chances than those in further layers. When pruning ratio increases  $(25\% \sim 75\%)$ , heads in earlier and deeper layers are pruned more often. However, some heads are always kept, for example, Head-2,6,10,12 in Layer-11 and Head-1,3,12 in Layer-12. In extreme cases (85% and 95% pruning ratios), pruning these heads hurts fine-tuning performances (Figure 7). Interestingly, along all pruning ratios (including cases of 85% and 95%), some heads in the middle layers are kept with high chances.
We observe that heads in the first four layers are always very likely to be pruned. This verifies the cluster structure in redundancy matrices (e.g., Figure 2). This may due to the fact that heads in earlier layers capture more superficial linguistic features (Kovaleva et al., 2019) which might be less informative in fine-tuning than other heads. As pruning ratio increases, heads in deeper layers are also more likely to be pruned, than those in the middle layers. We conjecture that after the middle layers, the contextualized embeddings are already very "strong" for down-stream tasks. Therefore the deeper layers do not require too many heads (though a few are left) to handle the task.
To summarize the case study of pruning, we emphasize that the proposed simple but robust redundancy clustering based pruning method is task
Figure 9: The effects of dropout ratio on the attention redundancy. "N"-shape is shown on the left (token-based JS distance). Almost monotonic effects exist on the right (sentence-based dCor distance): higher dropout results in more redundancy. Redundancy is more sensitive to hidden-dropout-ratio than attention-head-dropout-ratio.
agnostic and zero-shot. Compared to other pruning methods, (i) it requires no data from downstream tasks; (ii) one pruning strategy obtained from the pre-trained model can robustly prune less informative heads and preserve comparable finetuning performances for all GLUE downstream tasks. The powerful strength of the simple pruning strategy results from the phase-independent and task-independent attention redundancy patterns existing in BERT-base model. As a future work, we would check if similar attention redundancy patterns exist in other multi-layer multi-head self-attention based models and develop corresponding pruning mechanisms.
8 “Why”: Effects of Pre-Training Dropouts
It is hard to answer the fundamental question why attention redundancy exists. However, in this section, we examine one factor that may be the cause.
Dropout in the pre-training phase is suspected to be one reason resulting in redundant attentions (Clark et al., 2019). But Clark et al. (2019) didn't evaluate that. In this section, we investigate effects of various dropout ratios in the pre-training
process on the attention redundancy. In BERT pretraining, there are two dropout ratios, attention-head-dropout-ratio and hidden-dropout-ratio. The former randomly deactivates a ratio of trainable weights in the attention head (key, query, and value transformation matrices). The latter deactivates some weights in the linear transformation matrices which integrate all attention heads after each attention layer. Their default values are 0.1.
We manually set various values (from 0.0 to 0.9) for those two dropout ratios and train a BERT masked language model on Wiki103 training dataset from scratch. We train 100 epochs or until training loss converges. Then we randomly select 1000 sentences from the test dataset as objects. For each sentence, we calculate the mean value of attention redundancy matrix using JS and dCor distance. The final reported values are averaged over the 1000 sentences.
Results of three settings are shown in Figure 9. "Only attn" means that we only change the attention-head-dropout-ratio and keep the hidden-dropout-ratio as the default value 0.1. "Only hidden" is changing hidden-dropout-ratio and keeping attention-head-dropout-ratio as default. "Attn & hidden" modifies both dropout ratios. In both figures, smaller distance values reflect heavier attention redundancy.
We can see that the token-based measure JS and sentence-based measure dCor show reversed trends in the middle range ([0,2 0.7]). For JS on all three settings, when dropout ratio increases (i.e., more inactive weights updating in pre-training) the distances first drop down to the lowest values (when dropout ratio is 0.2) and increase and then drop again heavily. On the other hand, for dCor, the distance decreases (heavier attention redundancy) along the increased dropout ratio.
We conclude that dropout ratio does not play a simple effect on the attention redundancy. The "N"-shape effects shown in the left figure (token-based distance JS) is demonstrated from the view of attention vectors' similarity. On the other hand, the sentence-based distance (dCor) figure shows that higher dropout ratios result in more redundancy<sup>10</sup>.
In both figures, we observe that slopes of "Attn &
hidden" and "Only hidden" are steeper than that of "Only attn". This means that dropout in the hidden transformations affects the attention redundancy more than the attention dropout.
There is no doubt that more factors must affect the attention redundancy. We leave the "why" in future study.
Using BERT-base model as an example, we comprehensively investigated the attention redundancy in multi-layer multi-head self-attention based language models. The redundancy was measured by distance functions at token level and sentence level. At both levels, we found that many heads are not distinct from each other, and clear clustering effects were observed. We discovered that the attention redundancy is phase-independent and task-agnostic. Specifically, compared to a pre-trained model, the redundancy patterns do not change much after fine-tuning on multiple downstream tasks. We also shown complex influences on redundancy of dropout ratios in hidden transformations and self-attention. Based on these discoveries, we design a zero-shot strategy to prune attention heads. Compared to existing methods, the zero-shot pruning is simple and robust (task-agnostic). In the near future, we are interested in experimenting this method over more self-attention based pre-trained language models and more downstream tasks.
Anil Bhattacharyya. 1946. On a measure of divergence between two multinomial populations. Sankhya: the indian journal of statistics, pages 401-406.
Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. 2021. Isotropy in the contextual embedding space: Clusters and manifolds. In ICLR.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.04341.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL.
Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. arXiv preprint arXiv:1909.00512.
Allyson Ettinger. 2020. What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48.
$^{9}$ https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/  
 $^{10}$ Note that we use 1 - original distance correlation for consistency among distance functions. Original distance correlation=0 means that two sets of random variables are independent.
Angela Fan, Edouard Grave, and Armand Joulin. 2019. Reducing transformer depth on demand with structured dropout. In ICLR.
Yoav Goldberg. 2019. Assessing bert's syntactic abilities. arXiv preprint arXiv:1901.05287.
JC Gower. 1971. Statistical methods of comparing different multivariate analyses of the same data. Mathematics in the archaeological and historical sciences, pages 138-149.
Harold Hotelling. 1992. Relations between two sets of variates. In *Breakthroughs in statistics*, pages 162-190. Springer.
Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. In *NAACL*.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does bert learn about the structure of language? In ACL.
Xiaqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351.
Julie Josse and Susan Holmes. 2016. Measuring multivariate association and beyond. Statistics surveys, 10:132.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of bert. arXiv preprint arXiv:1908.08593.
Pierre Legendre and MARIE-JOSEE FORTIN. 2010. Comparison of the mantel test and alternative approaches for detecting complex multivariate relationships in the spatial analysis of genetic data. Molecular ecology resources, 10(5):831-844.
Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. Open sesame: Getting inside bert's linguistic knowledge. arXiv preprint arXiv:1906.01698.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
J Scott McCarley. 2019. Pruning a bert-based question answering model. arXiv preprint arXiv:1910.06360.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In NeurIPS.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assetss/researchcovers/languageeunsupervised/language understanding paper.pdf.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in bertology: What we know about how bert works. arXiv preprint arXiv:2002.12327.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2019. Ernie 2.0: A continual pre-training framework for language understanding. arXiv preprint arXiv:1907.12412.
Gábor J Székely, Maria L Rizzo, Nail K Bakirov, et al. 2007. Measuring and testing dependence by correlation of distances. The annals of statistics, 35(6):2769-2794.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel Bowman, Dipanjan Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In ICLR.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. arXiv preprint arXiv:1906.04284.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In ACL.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do nlp models know numbers? probing numeracy in embeddings. In EMNLP.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multitask benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. 2019. Structured pruning of large language models. arXiv preprint arXiv:1910.04732.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierrick Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
This is the appendix for NAACL-HLT 2021 paper: Yuchen Bian, Jiaji Huang, Xingyu Cai, Jiahong Yuan, and Kenneth Church. On Attention Redundancy: A Comprehensive Study.
A Distance Functions
We provide detailed descriptions of distance functions and our modifications in this section. Most of them exist in python scipy and sklearn packages. For others, we also provide implementation references or implement by ourselves. But more mathematical details are beyond this paper's scope. Please refer to original papers or (Josse and Holmes, 2016) for discussions.
A.1 Token-Based Distances
For an input sentence, each token corresponds to 144 attention vectors.
Let  $\mathbf{p},\mathbf{q}$  be two attention vectors or attention distributions (since the sum of each attention vector is 1). Four modified distance functions used in this paper are:
- cosine similarity ( $\cos$ ): Since  $\cos(\mathbf{p}, \mathbf{q})$  for a pair of distribution vectors is bounded in  $[0, 1]$ , we modify it with  $1 - \cos(\mathbf{p}, \mathbf{q})$  to keep the distance properties in Section 4.
- Pearson correlation coefficient (corr): We normalize as  $(1 - corr(\mathbf{p},\mathbf{q})) / 2$  due to its range  $[-1,1]$ .
- Jensen-Shannon distance (JS): Its range is [0,1] and it's consistent with the distance properties. It is also used in the literature (Clark et al., 2019; Jain and Wallace, 2019) to measure the distance of two attention distributions.
- Bhattacharyya coefficient (BC) (Bhattacharyya, 1946): It measures the amount of overlap between two statistical populations. It can be used to determine the relative closeness of the two attention vectors.  $BC(\mathbf{p}, \mathbf{q}) = \sum_{x} \mathbf{p}(x) \mathbf{q}(x)$ . Its range is [0, 1], and we modify it by  $1 - BC(\mathbf{p}, \mathbf{q})$ .
A.2 Sentence-Based Distances
For an input sentence, let  $\mathbf{A}_i$  and  $\mathbf{A}_j$  be arbitrary two attention matrices among the 144 attention matrices extracted from the BERT-base model.
- Distance correlation<sup>11</sup> ( $dCor$ ) (Székely et al., 2007): It is introduced to address the deficiency of Pearson's correlation which is sensitive to a linear relationship between two variables. It's widely used in statistical community. It's based on nonlinear operation and the range is  $[0,1]$  where  $0$  is got when two sets of random variables are independent. We modify it as  $1 - dCor(\mathbf{A}_i,\mathbf{A}_j)$ .
- Procrustes coefficient (PC) (Gower, 1971): It can measure the closeness of two data matrices. It's also known as Lingoes and Schonemann (RLS) coefficient (Legendre and FORTIN, 2010). It varies from 0 to 1 and can be used as a distance measure.
- Canonical correlation coefficient  $(CC)$  (Hotelling, 1992): It's a famous method to study the relationship between two sets of variables. It's defined as the trace of a matrix combining the covariance of two input data matrices. We modify it as  $1 - CC(\mathbf{A}_i, \mathbf{A}_j)$ .
B Attention Redundancy Matrices
In this section, we provide attention redundancy matrices visualization results for GLUE datasets and randomly generated token sequences (Figure 10 to Figure 18).
C Consistency of Redundancy Patterns in GLUE Tasks
In this section, we provide more consistency results of attention redundancy patterns measured based on token-based and sentence-based distances for GLUE tasks in Figure 19 including both pre-trained and fine-tuned BERT-base model.
D Cross-Task Correlations of Redundancy Patterns in GLUE Tasks
In this section, we show the cross-task correlation results of attention redundancy patterns measured based on different distances for GLUE tasks and random inputs in fine-tuned BERT-base model in Figure 20.
11https://github.com/vnmabus/dcor
(a) Redundancy matrices in pre-trained BERT-base for CoLA
(b) Redundancy matrices in fine-tuned BERT-base for CoLA
(c) Difference of redundancy matrices between pre-trained and fine-tuned BERT-base for CoLA
Figure 10: Redundancy matrices in BERT-base for CoLA
(a) Redundancy matrices in pre-trained BERT-base for SST-2
(b) Redundancy matrices in fine-tuned BERT-base for SST-2
(c) Difference of redundancy matrices between pre-trained and fine-tuned BERT-base for SST-2
Figure 11: Redundancy matrices in BERT-base for SST-2
(a) Redundancy matrices in pre-trained BERT-base for MRPC
(b) Redundancy matrices in fine-tuned BERT-base for MRPC
(c) Difference of redundancy matrices between pre-trained and fine-tuned BERT-base for MRPC
Figure 12: Redundancy matrices in BERT-base for MRPC
(a) Redundancy matrices in pre-trained BERT-base for QQP
(b) Redundancy matrices in fine-tuned BERT-base for QQP
(c) Difference of redundancy matrices between pre-trained and fine-tuned BERT-base for QQP
Figure 13: Redundancy matrices in BERT-base for QQP
(a) Redundancy matrices in pre-trained BERT-base for STS-B
(b) Redundancy matrices in fine-tuned BERT-base for STS-B
(c) Difference of redundancy matrices between pre-trained and fine-tuned BERT-base for STS-B
Figure 14: Redundancy matrices in BERT-base for STS-B
(a) Redundancy matrices in pre-trained BERT-base for MNLI
(b) Redundancy matrices in fine-tuned BERT-base for MNLI
(c) Difference of redundancy matrices between pre-trained and fine-tuned BERT-base for MNLI
Figure 15: Redundancy matrices in BERT-base for MNLI
(a) Redundancy matrices in pre-trained BERT-base for QNLI
(b) Redundancy matrices in fine-tuned BERT-base for QNLI
(c) Difference of redundancy matrices between pre-trained and fine-tuned BERT-base for QNLI
Figure 16: Redundancy matrices in BERT-base for QNLI
(a) Redundancy matrices in pre-trained BERT-base for RTE
(b) Redundancy matrices in fine-tuned BERT-base for RTE
(c) Difference of redundancy matrices between pre-trained and fine-tuned BERT-base for RTE
Figure 17: Redundancy matrices in BERT-base for RTE
Figure 18: Redundancy matrices in pre-trained BERT-base for randomly generated token sequences
(a) Pre-trained BERT-base model
(b) Fine-tuned BERT-base model
Figure 19: Correlation of redundancy matrices in BERT-base for GLUE tasks and random inputs
Figure 20: Correlations of redundancy matrices between task pairs (with fine-tuned BERT-base)