SlowGuess/NAACL_2021_OCR huggingface dataset:
Yuchen Bian, Jiaji Huang, Xingyu Cai, Jiahong Yuan, Kenneth Church Baidu Research, Sunnyvale, CA, USA
{yuchenbian,huangjiaji,xingyucai,jiahongyuan,kennethchurch}@baidu.com
Multi-layer multi-head self-attention mechanism is widely applied in modern neural language models. Attention redundancy has been observed among attention heads but has not been deeply studied in the literature. Using BERT-base model as an example, this paper provides a comprehensive study on attention redundancy which is helpful for model interpretation and model compression. We analyze the attention redundancy with Five-Ws and How. (What) We define and focus the study on redundancy matrices generated from pre-trained and fine-tuned BERT-base model for GLUE datasets. (How) We use both token-based and sentence-based distance functions to measure the redundancy. (Where) Clear and similar redundancy patterns (cluster structure) are observed among attention heads. (When) Redundancy patterns are similar in both pre-training and fine-tuning phases. (Who) We discover that redundancy patterns are task-agnostic. Similar redundancy patterns even exist for randomly generated token sequences. ("Why") We also evaluate influences of the pre-training dropout ratios on attention redundancy. Based on the phase-independent and task-agnostic attention redundancy patterns, we propose a simple zero-shot pruning method as a case study. Experiments on fine-tuning GLUE tasks verify its effectiveness. The comprehensive analyses on attention redundancy make model understanding and zero-shot model pruning promising.
Multi-layer multi-head self-attention architectures (Transformer (Vaswani et al., 2017)) are widely applied in modern language models, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), OpenAI GPT (Radford et al., 2018), GPT-2 (Radford et al., 2019) and ERNIE2.0 (Sun et al., 2019), to name a few.
Redundancy phenomenon is discovered among
Figure 1: Pair-wise Jensen-Shannon distance of attention heads for the pre-trained BERT-base model (12-layer-12-head self-attention). Attention redundancy (cluster with small distances) exists in adjacent attention heads and layers.
attention heads. It demonstrates that many attention heads generate very similar attention matrices (Clark et al., 2019; Kovaleva et al., 2019). We take the pre-trained BERT-base model as an example. It learns 12-layer-12-head self-attention matrices describing dependencies between each pair of tokens in a sentence. Then for each token, there are 144 attention vectors. We use Jensen-Shannon distance to measure the relationship between each pair of vectors. Then for one sentence (consisting of a sequence of tokens), the token-averaged distance is utilized to imply the redundancy between each pair of attention matrices. Smaller distance values reflect more redundancy. Figure 1 shows the redundancy (distance) among  $144 \times 144$  pairs of attention matrices averaged over 1000 randomly sampled sentences. We can see clear redundancy patterns (clusters with smaller distance areas) in consecutive attention layers.
Analyzing the attention redundancy helps to interpret the multi-layer multi-head self-attention architecture. Various studies have attempted to re
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 930-945 June 6-11, 2021. ©2021 Association for Computational Linguistics
veal the relationship among attention heads. Examples are attention visualization (Vig and Belinkov, 2019), common attention patterns (Kovaleva et al., 2019), attention head pruning (Voita et al., 2019), and probing test (Clark et al., 2019). Existing works either focus on the  $12 \times 12$  attention matrices and their effects on (pre-training or/and finetuning) performances or focus on linguistic features extracted by latent token vectors and attention matrices.
Though the redundancy phenomenon was discovered, no existing work studies the attention redundancy pattern itself (i.e., the  $144 \times 144$  distance matrix in Figure 1) deeply. This motivates us to conduct a comprehensive and complementary study on the attention redundancy phenomenon.
In this paper, we take the BERT-base model as a representative model to analyze the attention redundancy with Five Ws and How. As far as we know, many of the following discoveries are new to the research community.
What is attention redundancy?
Given a distance function, we define the pairwise distance matrix  $(\in \mathbb{R}^{144\times 144})$  of the  $12\times 12$  attention matrices of BERT-base model as attention redundancy matrix. In this paper, we obtain redundancy matrices from both pre-trained and finetuned BERT-base model for GLUE tasks as the research objects.
How to measure attention redundancy?
Except for the two token-based measures, Jensen-Shannon distance (Clark et al., 2019) and cosine similarity (Kovaleva et al., 2019) used in literature, we employ two more token-based distance function and three sentence-based ones to measure attention redundancy and analyze their similar redundancy patterns (please refer to Section 4.1 for more details). The purpose is to alleviate the measuring bias of just using one distance function. Sentence-based distances directly measure the relationship between two attention matrices without averaging over tokens. We visualize the redundancy patterns using various distance functions.
Where does attention redundancy exist?
We find common hierarchical cluster structures in the set of token-based redundancy matrices and the set of sentence-based redundancy matrices, respectively. Attention heads of earlier, middle, and deeper attention layers are clearly clustered in the redundancy matrices. We also demonstrate that highly correlated similar redundancy patterns exist
in redundancy matrices generated based on different type of distances.
When does attention redundancy occur?
The redundancy is phase-independent. Common redundancy patterns are discovered in both the pre-trained phase and fine-tuned phases. For any downstream task with any distance function, we notice highly correlated attention redundancy patterns between two phases.
Who (which task) has attention redundancy?
We surprisingly realize that the redundancy is task-agnostic. The redundancy patterns are highly correlated across different tasks. We even randomly generate token sequences as input in the pre-trained BERT-base model. Very similar attention redundancy patterns occur as well.
Based on this astonishing discovery, as a case study application, we propose a simple zero-shot head-pruning strategy based on clustering results using redundancy matrices. Compared to other complex pruning strategies, e.g., (Tang et al., 2019; Jiao et al., 2019; Fan et al., 2019; Wang et al., 2019; McCarley, 2019), the most important is that without knowing any data of fine-tuning tasks, this pruning can be effectively and efficiently conducted just based on some randomly generated token sequences with the pre-trained BERT-base model. The only effort is to compute one or several redundancy matrices. Results reflect that for most GLUE tasks, the proposed pruning strategy based on redundancy matrices can prune up to  $75\%$  to  $85\%$  of attention heads while keeping comparable fine-tuning performances.
"Why" does the phase-independent and task-agnostic attention redundancy happen?
It's hard to tell the reason of the redundancy patterns (that's why we use the quoted "Why"). However, we conduct experiments to evaluate the effects on attention redundancy of dropout ratios in the pre-training phase which are suspected as one of reasons (Clark et al., 2019).
When we use sentence-based distance, a monotonic trend is found. Attention heads tend to be more redundant when increasing dropout ratios. When we use token-based distances, a complex "N"-shape effect exists. We also notice that the redundancy is more sensitive to dropouts in hidden linear transformations than to dropouts in the self-attention mechanism.
We believe that above-mentioned new findings in this paper make the redundancy analyses a promis
ing research direction in model interpretation and model compression, and so on.
Existing literature analyzes multi-layer multi-head self-attention architectures based language models (e.g., BERT) from different aspects including word-token embedding latent space, linguistic knowledge interpretability, attention mechanism, and so on (Rogers et al., 2020).
The output of BERT attention layers are token embedding vectors. One output vector for one token aggregates contextual information from the whole sentence. In the vector space, attention can produce strong representations for syntactic phenomena and phrasal information (Jawahar et al., 2019), but small improvements on semantic tasks (Tenney et al., 2019). Ethayarajh (2019) showed that contextualized representations of all words are not isotropic in any layer. Cai et al. (2021) revealed isotropy in the clustered contextual embedding space, and found low-dimensional manifolds. In the fine-tuning process, Kovaleva et al. (2019) showed that the last two attention layers encode task-specific features while earlier layers capture more fundamental information.
Another set of works focus on the ability of extracting linguistic knowledge. BERT can obtain syntactic dependencies, parts of speech tags, word disambiguation, and so forth (Ethayarajh, 2019; Vig and Belinkov, 2019; Clark et al., 2019; Goldberg, 2019). It showed that the same layer learns similar knowledge (Clark et al., 2019). Positional information is encoded in BERT lower layers (Lin et al., 2019). Vig and Belinkov (2019) argued that middle and last attention layers can extract dependency relations and distant information, respectively. Even with BERT's success, it still struggles handling some linguistic information and tasks. BERT does not excel at numbers, negation, inferences and role-based event prediction (Wallace et al., 2019; Ettinger, 2020). It's questionable to provide transparency or meaningful explanations for model predictions on downstream tasks (Jain and Wallace, 2019).
Since self-attention is the fundamental mechanism in BERT, existing works also investigate extracted attention vectors and/or matrices. These are most relevant to our study. Clark et al. (2019) and Kovaleva et al. (2019) found some common attention patterns, such as patterns on delimiter tokens,
block, and heterogeneous patterns. Also, redundancy and overparameterization is discovered in attention heads. By using attention-based probing classifiers, Clark et al. (2019) showed that heads in the same layer often exhibits similar behaviors. Attention heads can be pruned with different strategies but keep comparable performance in downstream tasks (Voita et al., 2019; Clark et al., 2019). Some layers can even be reduced to a single head (Michel et al., 2019).
Understanding attention redundancy can help interpret pre-trained/fine-tuned language models and guide model compression. But no systematic study on attention redundancy exists. This paper provides a deep study (Five-Ws and How) on the attention redundancy.
3 What: Redundancy Matrices
In this paper, we use BERT-base model as a representative to study the attention redundancy existing in multi-layer multi-head self-attention mechanisms. As a pre-trained language model, BERT has been verified with outstanding fine-tuning performances on many downstream language understanding tasks.
In BERT-base model there are 12 self-attention layers each of which consists of 12 self-attention heads. For one input (sentence or sentence pair with special tokens [CLS] and [SEP]), we extract  $12 \times 12$  attention matrices. The size of each matrix is  $n \times n$  where  $n$  is the number of tokens after BERT tokenization. Given a metric (e.g., distance function) and extracted attention matrices, we can measure the relationship between each pair of the 144 attention matrices, for instance the  $144 \times 144$  matrix in Figure 1. We define the pair-wise relationship among the 144 heads (i.e., attention matrices) as redundancy matrix. We consider that the smaller the distances among some attention heads are, the more attention redundancy exists in them.
As for the studying objects, we use the well-known natural language understanding benchmark GLUE task $^{1}$  (Wang et al., 2018) as evaluation objects for the attention redundancy analysis. Among GLUE, CoLA is for English sentence acceptability judgments. SST-2 and STS-B are sentiment analyses tasks. MRPC and QQP are for sentence-pair similarity classification. MNLI, QNLI, and
<sup>1</sup>We eliminate WNLI due to its very small size. The same elimination was conducted in the literature (Kovaleva et al., 2019).
RTE are natural language inference tasks. CoLA and SST-2 are single sentence tasks, while others are predicting relationship between a pair of sentences. Except that STS-B is a regression task, in the BERT framework, others are formed as classification tasks.
In experiments, we randomly select 1000 data samples from each development set $^2$  as data instances and report the averaged results. We feed them to (pre-trained and fine-tuned) BERT-base model to generate attention matrices.
We use PyTorch BERT-base model as the pretrained model $^3$  (Wolf et al., 2019). For finetuning, we train each task with suggested hyperparameters max-length=128, num-epoch=3, batch-size-per-GPU=32 on 8 GPUs. All results in this paper are averaged over 5 trials.
4 How: Distance Functions
A fundamental question about attention redundancy is how to measure the similarity or distance between two attention matrices. The smaller the distance is, the more redundancy should exist. In this section, we introduce two levels of distance functions, token-based and sentence-based distances. The aim is to inspect their interpretabilities and consistency on quantifying the attention redundancy and to alleviate measuring bias of just using a certain distance function.
Let  $dist(\mathbf{A}_i, \mathbf{A}_j)$  be a distance function to measure the relationship between the  $i$ th and  $j$ th attention matrices  $\mathbf{A}_i$  and  $\mathbf{A}_j$ . Note that we reshape the 12 layers  $\times 12$  heads into 144 attention matrices because the inputs of  $dist$  are two matrices. Intuitively, there should be three properties in a defined distance function. (i)  $dist(\mathbf{A}_i, \mathbf{A}_i) = 0$ ; (ii) the distance function should be symmetric, i.e.,  $dist(\mathbf{A}_i, \mathbf{A}_j) = dist(\mathbf{A}_j, \mathbf{A}_i)$ ; (iii) If  $\mathbf{A}_i$  is more similar to  $\mathbf{A}_j$  than  $\mathbf{A}_k$ , then  $dist(\mathbf{A}_i, \mathbf{A}_j) < dist(\mathbf{A}_i, \mathbf{A}_k)$ . So for the following distances, we modify them according to these requirements and normalize them into range [0,1] for easy comparisons.
4.1 Two Levels of Distances
Token-Based Distance For one input sentence, we can extract 144 attention weight matrices. Each matrix is  $\in [0,1]^{n\times n}$  where  $n$  is the number of
tokens in the sentence. Then for each token in a sentence, we get 144 attention vectors  $(\in [0,1]^n)$ . We can compute the pairwise distance of them and average on the  $n$  tokens to obtain a final scalar value for a pair of attention matrices.
Four token-based distance functions are generated from the following: cosine similarity (cos), Pearson correlation coefficient (corr), Jensen-Shannon distance (JS), and Bhattacharyya coefficient (BC). Note that for readability, please refer to Appendix A for detailed descriptions, implementations, and our modifications.
Sentence-Based Distance Unlike the token-based distances, here we directly measure the distance between two  $n \times n$  attention matrices corresponding to the whole input sentence. We modified three measures: distance correlation (dCor) (Székely et al., 2007), Procrustes coefficient (PC) (Gower, 1971), and Canonical correlation coefficient (CC) (Hotelling, 1992). Please refer to Appendix A for more details.
In general, sentence-based measures care about covariance and/or linear/nonlinear dependency for the given two attention matrices while token-based ones only focus on two single attention vectors.
4.2 Visualization of Redundancy Matrices
We take CoLA with pre-trained BERT-base as an example to visualize the redundancy matrices based on the two sets of distance functions. We also conduct visualization for other GLUE tasks in Appendix B. Shown in Figure 2, each heatmap shows the normalized distances (averaged over the 1000 selected CoLA development data) using one distance function. Each entry reflects the distance between a pair of attention heads. We can see clear common redundancy patterns existing in the redundancy matrices based on token-based distances. Similar observations are shown for sentence-based distances.
5 Where: Redundancy in Layers
In this section, we discuss the redundancy patterns in details and show that similar redundancy patterns exist in redundancy matrices when using the four token-based distance functions. So do for the three sentence-based distance functions.
5.1 Redundancy Patterns
There are two observations in Figure 2. First, attention heads in the same layer and consecutive layers
2If the total number of development data is less than 1000, we select all data.
Figure 2: Redundancy matrices in the pre-trained BERT-base model for CoLA development data using various distance functions. The left four and right three are for token-based and sentence-based distances, respectively. Similar attention redundancy patterns (cluster with small distances) exist in each set of distance functions.
tend to be more redundant than heads in far-apart layers. Second, redundancy hierarchical cluster structure exists among layers. However, different cluster structures exist as well when comparing results of the two sets of distance measures. The left four sub-figures are for token-based distances. They cluster the first 4 layers, the middle 6 layers and the last 2 layers into three clusters and the last 8 layers into a bigger cluster. The last three sub-figures are for sentence-based measures. The cluster structures are not clearer than that of token-based measures. But we can still observe that the middle 10 layers have smaller distances. Heads in the same layer have closer distances, especially the first and last layer. From the view of attention redundancy, these observations can be explained by conclusions in Kovaleva et al. (2019); Clark et al. (2019) that heads within the same layer or nearby attention layers are extracting similar features (e.g., task-specific features).
5.2 Similar Patterns
Next, we show that similar redundancy patterns occur when the four token-based distances are employed. So do the three sentence-based distances. In Figure 3, we compute the correlation between each pair of the redundancy matrices (with CoLA and pre-trained BERT-base) in Figure 2. High correlations are observed in redundancy matrices of each type of distances. Note that we also provide experimental results for other tasks with both pretrained and fine-tuned BERT-base model in Appendix C.
As for the different cluster structures for token-based and sentence-based distance functions, the reason is related to the information captured by different distance functions. Sentence-based measures consider the covariance or nonlinear relation
Figure 3: Correlation in two types of redundancy matrices (with pre-trained BERT-base for CoLA). There are high correlations among token-based and sentence-based distances, respectively.
ship between two sets of attention matrices. But token-based ones target on two single attention vectors. Also, different distance functions measure relationships in different ways. For example, The sentence-based dCor distance is based on nonlinear operations and 1 is obtained if and only if the two sets of input random variables are independent<sup>5</sup>. The token-based Pearson correlation (corr) is a measure for linear correlations.
In summary, by utilizing token-based and sentence-based distances to measure the attention redundancy, highly correlated similar redundancy patterns exist in each type, respectively.
6 When: Redundancy Is Phase-Independent
As a pre-trained language model, BERT is finetuned for downstream tasks. In this section, we check the redundancy patterns in both pre-trained and fine-tuned BERT-base model.
Figure 4 shows the attention redundancy matrices of fine-tuned BERT-base for CoLA using the two levels of distances (results for other tasks are shown in Appendix B). Very similar redundancy patterns to those in Figure 2 can be observed.
$^5$  dCor distance  $= 1$  is equivalent to that the original dCor correlation  $= 0$  since we modify it with being subtracted by 1.
Similar normalization and modification in Section 4.1 are conducted without being subtracted by 1.
Figure 4: Redundancy matrices in fine-tuned BERT-base for CoLA. They are very similar to redundancy patterns in pre-trained BERT-base shown in Figure 2.
Figure 5: Correlation of redundancy matrices between pre-trained and fine-tuned BERT-base model for GLUE tasks. Highly correlated attention redundancy patterns occur in the two phases.
To quantify the similarity, we simply compute the absolute values of differences between two corresponding redundancy matrices in Figure 2 and Figure 4. The mean difference value is just 0.035. In addition, we compute correlations between pretrained and fine-tuned BERT-base for each GLUE task under each distance function. As shown in Figure 5, high correlations are shown in pre-trained and fine-tuned phases for all GLUE tasks and all distance functions.
To summarize, attention redundancy patterns in pre-trained and fine-tuned phases are very similar and highly correlated.
7 Who: Redundancy Is Task-Agnostic
In this section, we analyze the attention redundancy from the task-oriented perspective. Specifically, for each distance function and each phase, we compute the correlation between redundancy matrices of each pair of tasks. The correlation results for pre-trained BERT-base model under different distances are shown in Figure 6 (results for fine-tuned BERT-base are presented in Appendix D). We find that the redundancy patterns across tasks are very similar with each other. Based on the task categories provided in Wang et al. (2018), we can also
notice that relatively higher correlations occur in the two single-sentence tasks (CoLA and SST-2), the three similarity and paraphrase tasks (MRPC, QQP, and STS-B), and the three inference tasks (MNLI, QNLI, and RTE), respectively.
To further check the task-agnostic observation, we even randomly generate 1000 token sequences with various lengths (uniformly distributed) as input for the pre-trained BERT-base ("RANDOM" in Figure 6). We can observe high correlations among redundancy patterns of GLUE datasets and random inputs.
We conclude that attention redundancy patterns are task-agnostic. We think that it may be caused by the input formatting in BERT. In fine-tuning, the special token [CLS] and [SEP] are inserted in the beginning and at the end of one input sentence, respectively. If there are two sentences as input, they are also delimited by the special token [SEP]. It is found that [CLS] and [SEP] may dominate the attention distribution in some attention heads (Clark et al., 2019; Kovaleva et al., 2019) such that high similarities are observed through different tasks.
Based on this surprising discovery, we can apply this task-agnostic observation for pruning redundant attention heads. Without knowing any data of downstream task, we conduct a simple but effective clustering-based zero-shot pruning strategy based on attention redundancy in the following.
7.1 Case Study Application: Zero-Shot Attention Head Pruning
In this section, we introduce a simple zero-shot pruning strategy based on the task-agnostic cluster structure of attention redundancy matrices. Note that there are some complicated pruning strategies in the literature, such as (Tang et al., 2019; Jiao et al., 2019; Fan et al., 2019; Wang et al., 2019; McCarley, 2019). Most of them compress pretrained models during or after fine-tuning to re