Finding the Pillars of Strength for Multi-Head Attention
Year: 2023

Abstract:
Recent studies have revealed some issues of Multi-Head Attention (MHA), e.g., redundancy and over-parameterization. Specifically, the heads of MHA were originally designed to attend to information from different representation subspaces, whereas prior studies found that some attention heads likely learn similar features and can be pruned without harming performance. Inspired by the minimumredundancy feature selection, we assume that focusing on the most representative and distinctive features with minimum resources can mitigate the above issues and lead to more effective and efficient MHAs. In particular, we propose Grouped Head Attention, trained with a self-supervised group constraint that group attention heads, where each group focuses on an essential but distinctive feature subset. We additionally propose a Voting-to-Stay procedure to remove redundant heads, thus achieving a transformer with lighter weights. Moreover, our method achieves significant performance gains on three well-established tasks while considerably compressing parameters.

heading: 1 Introduction
text: Transformers have shown promising performance across various tasks . However, they have some issues, e.g., redundancy and over-parameterization, which is mainly caused by Multi-Head Attention (MHA) (Michel et al., 2019; Voita et al., 2019) and Feed-Forward Network (FFN) (Sukhbaatar et al., 2019; Wu et al., 2019, 2020) of transformer. We aim to mitigate the redundancy and over-parameterization issues by optimizing the MHA module. The multi-heads were originally designed to attend to different representation subspaces of input (Vaswani et al., 2017). However, prior works (Michel et al., 2019; Voita et al., 2019) have shown that the attention heads are highly redundant and over-parameterized after training because some heads can be switched off with a negligible performance drop.
Such an issue is probably caused by their parallel design: the heads naturally work in the same way and likely attend to similar features (Cordonnier et al., 2020). The existing redundancy optimization methods are mainly based on homogenization, diversification, and head significance. However, they all have some limits. (1) The homogenization-based methods mitigate redundancy and over-parameterization by making heads similar and removing unnecessary parameters. Cordonnier et al. (2020) homogenized attention heads by sharing most weights between all heads, which reduced the redundant parameters but sacrificed the performance somewhat because of the lack of diversity. (2) The diversificationbased methods diversify the heads to enrich features and reduce the inter-head redundancy. Li et al. (2018) found that diversifying attention heads by adding a regularization could force MHA to reduce inter-head information redundancy, yielding performance gains in Machine Translation. However, such strategy that retains all feature subsets is suboptimal, because it does not address the issue that MHA is over-parameterized. (3) The significancebased methods (Michel et al., 2019; Voita et al., 2019; Li et al., 2021) learn significance scores for the heads to prune unimportant ones. However, the retained important heads still remain inter-head redundancy without diversifying them.
Considering the issues of the above-mentioned methods, we hypothesize that attending to the most representative and distinctive feature subsets with minimum resources leads to more effective and efficient MHAs, which is inspired by the minimum-redundancy feature selection (Cordonnier et al., 2020). Accordingly, we propose a divide-and-conquer strategy, including GroupConstrained Training (GCT) and Voting-to-Stay (V2S), to achieve the setting of our assumption and mitigate the above-mentioned issues. We illustrate them below.
14526
We first propose a strategy to group and distinguish attention heads, where a Grouped Head Attention (GHA) is obtained via the self-supervised GCT. By encouraging homogenization within a group and diversification between groups, the MHA is forced to divide its heads to work in several separate groups, where each group focuses on an essential but unique feature subset, being in line with the setting of our assumption. Note that the redundancy exists when the resources deployed by the model are more than enough to process current information (Cordonnier et al., 2020). GHA reduces the redundancy in two aspects:
• The intra-group homogenization reduces redundancy by encouraging similar intra-group heads and only retaining the most representative one later to lower the resource deployment.
• The inter-group diversification reduces redundancy by forcing heads to attend to more diversified features (with less overlap between heads) so that the unique information to process increases and matches the resources deployed.
Next, we show that GHA-PS (GHA with the Pillar of Strength), a lighter-weight GHA, can be achieved by excluding the redundant heads of GHA via the V2S procedure. V2S culls the redundant heads that share similar patterns with the most representative head (PS head) of a group, which is selected by voting on different training batches. Note that upon the convergence of the GCT, the heads are highly homogenized within a group, thus being redundant because they process similar information. As a result, once the redundant heads are culled, the PS heads can still achieve the essential utility of the original attention layer and yield comparable performance to the unculled model. The Lottery Ticket hypothesis (Frankle and Carbin, 2019) argues that subnetworks in an over-parameterized neural network can converge faster and achieve comparable or better performance than the original network. Our GHA-PS achieving better results is also in line with this hypothesis.
Such a divide-and-conquer combination resolves the issues of previous redundancy optimization methods: (1) Our model achieves better parameter efficiency, resolving the issue of previous diversification-based methods; (2) The feature diversity is guaranteed and the inter-head redundancy is reduced, resolving the problems of previous homogenization- and significance-based methods.
We evaluate our method on three benchmarking tasks. We denote the corresponding transformer architectures of GHA and GHA-PS as Grouped Head Transformers (GHT) and Grouped Head Transformers with the Pillars of Strength (GHT-PS), respectively. GHT and GHT-PS achieve significant improvements over the strong baselines in Machine Translation (MT) BLEU scores (+3.8% and +4.4% averaged on 7 datasets), Language Modeling (LM) perplexity (-2.8% and -2.9%), and Abstractive summarization (AS) F1-Rouge (+6.7% and +7.0% on average). GHT-PS exhibits higher efficiency in model size, inference speed, and floating-point operations (FLOPs). The light architecture of GHTPS reduces 63.6% parameters of the vanilla transformer and yields comparable performance. The key contributions of our work1 are threefold:
• We find that, in a certain range, higher compactness of attention heads (i.e., the intra-group heads become closer to each other and the intergroup ones become farther) improves MHA’s performance, forcing MHA to focus on the most representative and distinctive features. It provides guidance for future architectural designs of MHA.
• We propose a divide-and-conquer strategy that consists of GCT and V2S. It mitigates the redundancy and over-parameterization issues of MHA. Our method uses fewer parameters and achieves better performance, outperforming the existing MHA redundancy/parameter reduction methods.
• We verify our methods on three well-established NLP tasks. The superior results on datasets with multiple languages, domains, and data sizes demonstrate the effectiveness of our method.

heading: 2 Related Work
text: Parameter efficiency. Different methods were proposed to achieve lightweight transformers: (1) replacing attention with lightweight modules, e.g., convolution modules, such as Dynamic Conv (Wu et al., 2019) and Lite Transformer (Wu et al., 2020); (2) removing or replacing the feed-forward layers, such as Sukhbaatar et al. (2019) and Wu et al. (2020); (3) pruning the model, such as Michel et al. (2019), Voita et al. (2019), and Li et al. (2021).
1https://github.com/SenticNet/GHA
Modified multi-head mechanism. Ahmed et al. (2017) learned to weight the projected output of different heads, performing weighted sum over them. Li et al. (2019) aggregated the output of different heads by dynamic routing; Cui et al. (2019) used different attention mechanisms, e.g., global/local and forward/backward attention for different heads; Shazeer et al. (2020) mixed different heads before and after the softmax operation in an attention function to achieve communication between heads.
Head redundancy optimization. Michel et al. (2019) and Voita et al. (2019) found that only a subset of the attention heads have significant utilities in transformer, where the important heads could be identified by Expected Sensitivity and Layer-wise Relevance Propagation (LRP) (Ding et al., 2017). Upon this, Li et al. (2021) learned per-head importance scores and pruned the heads. Cordonnier et al. (2020) homogenized the attention heads by sharing a part of the weights between heads, which lowered the number of parameters but sacrificed performance. Li et al. (2018) found that diversifying attention heads by adding a regularization can force MHA to reduce inter-head redundancy, yielding performance gains for Machine Translation. However, previous methods either traded performance for efficiency or retained extra parameters.

heading: 3 Methodology
text: There are two core components in our method, namely the Group-Constrained Training (GCT) and the Voting-to-Stay (V2S) procedure. GHA (Figure 1) is developed with GCT that removes head redundancy; GHA-PS is developed by removing the redundant parameters of GHA in V2S. In this section, we detail the process of developing the GHA and finding its Pillars of Strength (PS).

heading: 3.1 Grouped Head Attention with Hidden Units
text: First, we detail the core module of GHT, the GHA with hidden units, which is built based on MHA via the GCT. The GCT divides the attention heads of MHA into several groups and makes heads within a group become more similar, whereas heads between groups become more different. Thus, MHA is forced to divide its heads to work in several separate groups, where each group focuses on an essential but unique feature subset to reduce head redundancy. We will show the effectiveness in § 5.
Given a transformer model f(x; θ) with n attention layers, the set of heads at attention layer l is denoted as Hl = {h1,l, ...,hk,l}, where k is the number of heads. The outputs of the attention heads are concatenated and projected with W out, where the i-th head output oi,l in layer l results from the computation of the projection matrices WQi,l , W K i,l , and W V i,l of this head:
MHAl(Q,K,V) = Concate(o1,l, ...,ok,l)W out
(1)
oi,l = softmax( (QWQi,l )(KW K i,l ) T
√ dk
)(VWVi,l ).
(2) Three feature maps (FMs) of GHA are extracted for the self-supervised GCT: (1) the result of VWVl , denoted as V̂l = {v1,l, ...,vk,l} (the value FM); (2) the attention weights of the l-th layer, denoted as Al= {a1,l, ...,ak,l} (the attention FM); (3) the output of the l-th layer before the output projection W out, denoted as Ol= {o1,l, ...,ok,l} (the head output FM). Moreover, V̂ = {V̂1, ..., V̂l}, A = {A1, ...,Al}, O = {O1, ...,Ol}. Given the FMs, a Hidden Unit Discovery System (HUDS) Ω assigns a hidden unit zji,l for each head to represent its group property, where i denotes the i-th head and j denotes the j-th group hidden unit. zji,l ∈ Ẑl, where Ẑl = {z1l , ..., zCl } represents the hidden unit candidates, and the hidden units assigned to the heads are denoted as Zl = {zi,l, ..., zi,l}.
Zl is discovered by the HUDS Ω: Zl = Ω(El), where El denotes either one of the V̂l, Al, or Ol. Here Ω(·) is an unsupervised algorithm that divides the heads into C groups given their FMs, such as K-means2:
Ω(El) = argmin Zl
C∑
i=1
∑
x∈Êil
||x− µi||2, (3)
where Êil is the set of feature maps of the i-th head group in the l-th attention layer. Then, the feature map groups of the l-th attention layer are denoted as Êl = {Ê1l , ..., Êil, ..., ÊCl }. µi is the mean of the feature map vectors in Êil . The hidden units Z = {Z1, ...,Zl} are C-class categorical variables (Eq.4(A)) or continuous vectors (Eq.4(B)) to supervise the GCT. The objective of the self-supervised GCT is termed as:
Lz(f ;A, V̂,O,Z) =   − 1 kn α n∑ l=1 k∑ i=1 log pz(zi,l|vi,l,ai,l,oi,l) + 1 (C − 1)knβ n∑ l=1 k∑ i=1 ∑ j2 ̸=j1 log pz(z j2 l |vj1i,l,aj1i,l,oj1i,l) (A) 1 kn α n∑ l=1 k∑ i=1 φ(vi,l,ai,l,oi,l; zi,l) −
1( C 2 ) n β
n∑
l=1
C−1∑
j1=1
C∑
j2=j1+1
φ(zj1l ; z j2 l ) (B)
(4)
Either when Z are categorical variables (Eq.4(A)) or continuous vectors (Eq.4(B)), the objective is composed of a homogenization term and a diversification term3. vji,l, a j i,l, and o j i,l denote the feature maps of the i-th head belonging to the j-th group. pz(zi,l|vi,l,ai,l,oi,l) denotes the predicted probability of the assigned group hidden variable zi,l, given vi,l, ai,l, and oi,l. φ(x;y) denotes a cosine similarity measurement between x and y (following Li et al. (2018)). φ(vi,l,ai,l,oi,l; zi,l) = τ1φ(vi,l; zi,l) + τ2φ(ai,l; zi,l) + τ3φ(oi,l; zi,l), where τ is a coefficient, determined by the specific settings for each dataset & task. When Z are categorical variables, the grouping is a classification task whose classification heads project the output into C classes; when Z are continuous vectors, the grouping process is a metric learning task whose
2K-means fixes the group numbers for fair comparisons in §5. Other clustering algorithms may also be applicable.
3The coefficients α and β of Eq.4 respectively control the intra-group homology and inter-group diversity degrees to achieve different group intensities in different tasks/datasets.
similarity computations are conducted between Z and the projected FM representations. In both conditions, GHA is supervised by Z to make the heads in the same group yield similar patterns, whereas those in different groups repulse from each other. The overall objective is given by L = Lt + Lz , where Lt is the task-specific objective.

heading: 3.2 The Pillars of Strength
text: Being consistent with Lottery Ticket hypothesis (Frankle and Carbin, 2019), we establish the GHT-PS from GHT as its subnetwork by removing redundant heads from GHA to achieve higher parameter efficiency. We propose the V2S procedure to find the Pillars of Strength (PS) heads that constitute the core of the GHA and remove other heads. We first describe the V2S roughly. In GHA, the heads within each group exhibit similar patterns upon the convergence of the Group-Constrained Training (GCT). Then, we only keep the heads with the most explicit group patterns (the PS heads), and switch off the other ones within the same group via V2S. The main idea of V2S is to vote on all heads of the GHA, and only retain one head for each group – the head receiving the most votes. Specifically, it takes an entire epoch to collect the layer-wise votes mbl ∈ {0, 1}k from the whole training set (each data batch b creates one layer-wise vote mbl per attention layer), where k denotes the head number; 0 indicates that the corresponding head should be switched off and 1 indicates that a head is retained.
We assume that there are B mini-batches in the training set. Then, each attention layer receives B layer-wise votes within which each head-wise vote is denoted by either 0 or 1. For each group, the head receiving the most ‘1’s are assigned a ‘1’ in the final head mask ml ∈ {0, 1}k for attention layer l, indicating that this head will be retained. Following Michel et al. (2019) and Voita et al. (2019), we mask out the output of heads as the equivalent operation of head removal4. The V2S procedure is outlined in Algorithm 1. We detail some of its definitions below. (1) ρ indicates the full convergence of GHT, i.e., the hidden units found by Ω have a center shift less than a threshold. (2) In Step 7-9, given feature maps V̂l, Al, and Ol of the lth attention layer, the vote vectors mbl,v, m b l,a, and mbl,o ∈ {0, 1}k are determined by the group pattern scores ηi,l of each head, indicating the explicitness of group patterns.
4We perform real head removal when test inference speed.
Algorithm 1 The Voting-to-Stay (V2S) algorithm
1: Procedure Voting-to-Stay(f , V̂,A,O,Z) 2: if satisfy ρ, and m is none then 3: Start voting epoch; Freeze f . 4: Γl ← [ ] ▷ Creat Γl to store votes 5: for batch b in B training batches do 6: for layer l in L layers do 7: for El in {V̂l,Al,Ol} do 8: Based on ηl = {η1,l, ..., η1,k}, 9: create mbl,v,m b l,a,m b l,o.
10: Add mbl,v, m b l,a, m b l,o to Γl.
11: for l in n do ▷ Vote at each attn layer 12: ml ← V OTE(Γl) 13: m← [m1, ...,mn] ▷ Stack layer votes 14: Unfreeze f ; end voting epoch. 15: f = f ⊙m ▷ Mask GHT attn outputs with m
We set the corresponding digit in the vote vectors as 1 for the head achieving the highest ηi,l in its group, indicating the most representative head of the group. Here ηi,l = pz(zi,l|ei,l) if z is categorical; otherwise ηi,l = −φ(ei,l; zi,l). ei,l denotes the i-th head feature map (either one of the vi,l, ai,l, or oi,l). (3) V OTE means counting the ‘1’s for each head based on the 0-1 votes in Γl and only keeping the heads with the highest counts5. After V2S, a finetuning is applied to adapt the pruned network.
GHT-PS compresses considerable parameters. In the case of two head groups, GHT-PS reduces 75% parameters for an attention layer and 32.1% for the entire model6. We will show that V2S removing non-PS heads does not sacrifice model performance. Instead, it brings accuracy gains in some cases and improves inference speed.

heading: 4 Experimental Setup
text: In this section, we detail the key architectural configurations. Further training, model, dataset & evaluation setups are detailed in A.1, A.2, & A.3. We follow the transformer of Vaswani et al. (2017) as a backbone architecture for all datasets and tasks in our experiments. Following Wu et al. (2019, 2020), for Machine Translation and Abstractive Summarization, we adopt the same 8-head encoderdecoder architecture with 6 layers for both encoder
5Besides voting, there is an alternative way to create the mask. Instead of using 0-1 number as a discrete voting unit, the group pattern scores can be added up to rank the head pattern explicitness. We find that the two ways perform similarly.
6The encoder-decoder arch in Vaswani et al. (2017).
and decoder, where the model dimension dmodel = 512 and feed-forward dimension df = 2048. For LM, we adopt the 16-head decoder-only architecture with 16 layers, where the model dimension dmodel = 1024 and feed-forward dimension df = 4096. The layer normalization is applied before the residual connection of each layer. The parameters of decoder input and output projections are shared. Our models are based on fairseq (Ott et al., 2019) implementations.
We perform the GCT as a metric learning task because it does not introduce additional projection layers when the shapes of similarity inputs are identical (Eq.4(B)), which makes GHT weight-lighter. In addition, it performs better in our experiments compared to the classification-based grouping.

heading: 5 Results and Analysis

heading: 5.1 Machine Translation
text: Ours vs. vanilla transformer. We first report results by comparing GHT and GHT-PS with the vanilla transformer (Vaswani et al., 2017) which is the backbone of our model. As shown in Table 1, the models are compared at different parameter levels7. GHT does not have weight reduction, keeping the same parameter size as the vanilla transformer (44M, the same setting as transformer base (Vaswani et al., 2017)). In contrast, GHT-PS is compressed to 30M parameters via V2S. For a fair comparison, we first compare GHT-PS with two lite architectures, Transformer-Lite1 and TransformerLite2, whose parameter numbers are 30M as well. Keeping other settings unchanged, the encoder and decoder of Transformer-Lite1 are reduced to 4 layers, respectively. Transformer-Lite2 reduces the model dimension dmodel to 424, and df to 1696.
GHT and GHT-PS consistently and significantly outperform their backbone models at the same parameter level across all datasets. On average, the GHT surpasses 44M vanilla transformer by 3.8% in BLEU (Papineni et al., 2002); GHT-PS surpasses Lite1 and Lite2 by 4.9% and 4.4%, respectively. Although GHT-PS reduces 32.1% parameters, it significantly outperforms both 44M and 30M vanilla transformers, which is comparable to GHT on all datasets. It shows that V2S reduces the parameter size of the original transformer without sacrificing accuracy on MT. Efficiency is analyzed later.
7The parameters analyzed in this paper exclude the embedding layer since they vary a lot between different datasets when the vocabulary sizes are different.
Ours vs. efficient attention models. We compare GHT with two state-of-the-art (SOTA) MHA redundancy optimization baselines. Cordonnier et al. (2020) and Li et al. (2018) are respectively homogenization- and diversification-based methods. In addition, we compare GHT-PS with four SOTA baselines that made major contributions to attention parameter compression and redundancy optimization8. Voita et al. (2019) and Li et al. (2021) are significance-based pruning methods. Dynamic Conv (Wu et al., 2019) and Lite Transformer (Wu et al., 2020) modify the MHA arch to reduce parameters.
Table 2 shows that GHT outperforms all its baselines on all datasets, exceeding the strongest baseline by 2.9% in averaged BLEU scores. GHT-PS outperforms all its baselines on 6 out of 7 datasets, exceeding the strongest baseline by 4.4% on average. Model compression of the baselines may sacrifice performance (especially on large datasets, e.g., WMT en-de and en-fr), while GHT-PS is al-
8Works optimizing parameters of transformer modules rather than the MHA are not compared. In addition, we do not compare to Michel et al. (2019) (post-pruning), because their method performs extremely bad when the parameter level is low, e.g., 30M (Li et al., 2021).
most not affected by the parameter reduction, even surpassing GHT’s baselines with 44M parameters.
Ablation Study. We evaluate the impacts of the features we choose for GHT and GHT-PS (Table 3). We first ablate the diversification/homogenization term of GCT (see Eq.4), which lowers the BLEU scores. Next, we show the importance of GCT for V2S. w/o GCT denotes that we directly perform V2S at the very beginning without GCT. w/o GC
denotes that V2S is employed after normal training without Group Constrain (GC). Both ablation models yield lower BLEU, because they do not homogenize unnecessary heads and prepare them for pruning. Next, we validate the power of Pillars of Strength. w/o HUDS denotes we replace HUDS with randomly switching off heads after GCT; w/o PS stay denotes we keep random group members instead of the Pillars of Strength after GCT. We observe lower BLEU in w/o HUDS and w/o PS stay. Finally, we find that GC only needs to be added before V2S. We denote the training stages before and after V2S as stages 1 and 2. We compare the proposed Stage 1-based GHT-PS with models that perform GCT at Stage 2 (w/ stage 2 GC) and at both stages (w/ stage 1& 2 GC). BLEU scores of both ablation models decrease.
Effect of group compactness. We hypothesize that more compact group patterns bring performance gains to the GHT. Figure 2 shows the correlation between the compactness of the final group patterns and the final BLEU scores GHT achieved on 5 IWSLT’14 development sets when the GHT is fully converged in GCT. One data point corresponds to an independent run. We choose Silhouette Coefficient (SC) (Rousseeuw, 1987) and Dunn’s Index (DI) (Bezdek and Pal, 1995) as the measurements of group pattern compactness, both of which increase as the intra-group samples become more similar and the inter-group ones become more separated. The SC and DI are computed with the FMs of GHA (§ 3.1) and controlled by tuning the α and β (Eq.4).
Figure 2 shows that, within the normal range, the BLEU scores rise with higher SC/DI scores, which is in line with our assumption. The BLEUs start to drop after the peak as the SC/DI scores increase, because the very heavy group constraint prohibits the model from learning useful task-specific knowledge.
Effect of group number. Figure 3 shows the performance trends of 16-head GHT and GHT-PS by different numbers of group hidden units. For GHT, different datasets have different optimal hidden unit quantities, while a similar trend is observed. The optimal group number is between 2 and 8, which is in line with the claim that our group strategy is superior to sole homogenization (1 group) or diversification (16 groups) strategies. For GHT-PS, when the group number is larger than 1, it shows comparable performance to GHT on most datasets. This also verifies that non-PS heads can be switched off without sacrificing performance.
Group pattern trends. Figure 4 shows the trends of intra-group homogeneity (given by the 1st term of Eq.4(B)) and inter-group diversity (given by the 2nd term of Eq.4(B)) of GHT and vanilla transformer in the training process on five IWSLT datasets. By training, GHT yields higher intragroup homogeneity and inter-group diversity absolute values, leading to more compact groups, while the vanilla transformer shows flattened trends. It shows that GCT can effectively homogenize intragroup heads and diversify inter-group heads.
Model BLEU↑ Param↓ Infer Speed↑ FLOPs↓ Transformer base 25.8 44M 1016.4 sent/s 1996M Transformer big 26.4 176M 707.1 sent/s 6635M Lite Conv 26.6 166M 722.1 sent/s 6184M Dynamic Conv 26.9 176M 710.0 sent/s 6315M
GHT-PS-LITE (ours) 26.6 16M 1170.2 sent/s 1181M
Table 5: Efficiency comparison by parameter, inference speed (averaged on five runs), and FLOPs. All results are generated with beam size 5, batch size 256, max decoding length 10 on a NVIDIA Quadro RTX A6000.
Efficiency analysis. In Tables 1 and 2, the efficiency metrics are controlled to be identical. Our models with higher inference speed and lower FLOPs show efficiency by culling redundant parameters. We also compare the efficiency metrics by controlling BLEU scores. In Table 5, we select models from the works in Table 1 and 2 that are reported to achieve close BLEU scores on newstest2013 as the baselines. The GHT-PS-LITE is a light version of GHT-PS that has a df of 1024. Given BLEU ranges from 25.8 to 26.9, GHT-PSLITE is much more efficient than the baselines. Noticeably, GHT-PS-LITE achieves 90.36% fewer parameters, 62.05% faster inference speed, and 80.90% fewer FLOPs against Lite Conv which yields the same BLEU as it.

heading: 5.2 Abstractive Summarization
text: We evaluate the ability of our model to process longer inputs via Abstractive Summarization on the CNN-DailyMail dataset. We take vanilla transformer as the backbone. Table 4 shows that both GHT and GHT-PS achieve higher F1-Rouge scores (Lin, 2004a) on this task. GHT-PS achieves 4.1% higher Rouge-1, 18.6% higher Rouge-2, and 4.4% higher Rouge-L against vanilla transformer. It also achieves 0.4% higher Rouge-1, 31.1% higher Rouge-2 and 2.4% higher Rouge-L against the best-performing baseline (Dynamic Conv). Meanwhile, GHT-PS only takes 68.18% parameters of the vanilla transformer and exhibits higher inference speed and fewer FLOPs.

heading: 5.3 Language Modeling
text: We evaluate LM performance on WIKITTEXT103 dataset. The backbone is a decoder-only transformer with 16 layers and adaptive inputs (Baevski and Auli, 2019). We compare with the backbone model, as well as comparable SOTA LM models, including S4 (Gu et al., 2022), BERT-LargeCAS (Wang et al., 2019), and GPT-2 Large (Radford et al., 2018).
Table 6 shows that both GHT and GHT-PS achieve lower perplexity (Vajapeyam, 2014) than the baselines on both validation and test sets (2.9% and 9.0% less perplexity against the backbone and the best performing LM baseline, respectively). Meanwhile, GHT-PS achieves 16.92% parameter reduction, 2 times faster inference speed, and 75% FLOPs compared with the backbone.

heading: 6 Conclusion
text: In this paper, we assume that only focusing on the most representative and distinctive features with minimum resources may mitigate the redundancy and over-parameterization issues of MHA. Accordingly, we propose a divide-and-conquer strategy, including GCT and V2S to mitigate the issues. The improvements on three tasks and the extensive analysis verify our hypothesis and the effectiveness of our redundancy optimization methods. Our study may inspire future MHA design and training to achieve higher accuracy and efficiency.

heading: Limitations
text: In this work, we evaluate the proposed models for NLP tasks only. However, tasks in other fields such as Computer Vision may present a very different input inductive bias, thus affecting the performance. Moreover, our models are trained from scratch, hence it is unknown whether the same divide-andconquer strategy works for pre-trained models. We will study these limitations in the future to give a more extensive exploration.

heading: Ethics Statement
text: This article follows the ACL Code of Ethics. The annotations are based on public datasets that do not contain private data. The algorithm we developed is an architectural optimization technique for improving model performance. To our best knowledge, there are no foreseeable potential risks to using this technique.

heading: Acknowledgments
text: This research is supported by the Agency for Science, Technology and Research (A*STAR) under its AME Programmatic Funding Scheme (Project #A18A2b0046).

heading: A Appendix

heading: A.1 Trainig Settings

heading: A.1.1 Machine Translation
text: We use Adam to optimize the MT models and set the β1 = 0.9, β2 = 0.98. We use the Inverse Square Root Schedule (Vaswani et al., 2017) where it first warms up for 4K steps until the learning rate reaches 5 × 10−4, and then it exponentially decays the learning rate. We apply early stop as a termination condition. We apply a 0.3 dropout rate for all Machine Translation models. A weight decay of 10−4 is used for all IWSLT 2014 models, whereas for WMT models we use a weight decay of 0. We apply a 0.1 label smoothing (Szegedy et al., 2016; Pereyra et al., 2017) for the uniform prior distribution over the vocabulary.

heading: A.1.2 Language Modeling
text: Following Baevski and Auli (2019), we use Nesterov’s accelerated gradient method (Sutskever et al., 2013) with a momentum of 0.99. We clip the gradient norm if it exceeds 0.1 (Pascanu et al., 2013). The learning rate is linearly warmed up from 10−7 to 1 for 16K steps and then annealed using a cosine learning rate schedule (Loshchilov and Hutter, 2017) with multiple cycles. Each cycle doubles the number of updates than the previous cycle and we shrink the maximum and minimum learning rates by 0.75 compared to the previous cycle. The initial minimum learning rate is 10−4 and the maximum is 1. We apply 0.2 adaptive softmax dropout rate, 0.1 attention dropout rate, and 0.1 activation dropout rate.

heading: A.1.3 Abstractive Summarization
text: We use the same training setup with IWSLT 2014 models. We apply 0.1 clip norm and 0.2 attention dropout. The model is warmed up for 10K updates.

heading: A.2 Further Model Settings
text: Different α, β, and head feature maps (V̂, A, and O) are preferred for different datasets to achieve optimal performance. The Machine Translation configurations are detailed in Table 7; Language Modeling and Abstractive Summarization configurations are detailed in Table 8.
Note that φ(vi,l,ai,l,oi,l; zi,l) = τ1φ(vi,l; zi,l) + τ2φ(ai,l; zi,l) + τ3φ(oi,l; zi,l), we set one of the {τ1, τ2, τ3} to be 1, the other to be 0.
A.3 Datasets and Evaluation9

heading: A.3.1 Efficiency Metrics settings
text: Inference speed. All inference speed results are generated with beam size 5, batch size 256, maximum decoding length 10 on a single NVIDIA Quadro RTX A6000.
FLOPs. We use the fvcore10 to calculate the FLOPs, with a fixed input length of 30.

heading: A.3.2 Machine Translation
text: To fully evaluate the effectiveness of our methods, we evaluate seven MT datasets of IWSLT’14 and WMT 2014 benchmarks. We closely follow the setup of Vaswani et al. (2017) for data preparation. WMT 2014 English-German dataset consists of about 4.5M sentence pairs. It is encoded with byte-pair encoding (Britz et al., 2017), having a shared source-target vocabulary of about 40K tokens. Following the standard setting (Vaswani et al., 2017), we validate on newstest2013 and test on newstest2014 for experiments on this dataset. The WMT 2014 English-French dataset consists of 36M sentence pairs and is encoded with a joint source-target BPE of about 43K vocabularies. Following the standard split, we validate on a concatenation of newstest2012 and newstest2013 and test on newstest2014. For IWSLT’14 German to English, IWSLT’14 English to German, IWSLT’14 English to French, IWSLT’14 English to Italian and IWSLT’14 Italian to English, we encode the sentence pairs with joint source-target BPE. Following Edunov et al. (2018), the validation set is randomly splited from the training set with a ratio of 1:23. The testset consists TED.tst2010, TED.tst2011, TED.tst2012 and TED.dev2010, TEDX.dev2012 for IWSLT’14 German to English, IWSLT’14 English to German, and IWSLT’14 English to French; the TEDX.dev2012 is replaced by TEDX.dev2014 for IWSLT’14 English to Italian and IWSLT’14 Italian to English.
For all Machine Translation datasets, we use detokenized BLEU. WMT 2014 English-German and WMT 2014 English-French are evaluated with a beam size 4 and length penalty 0.6; IWSLT’14 datasets are evaluated with a beam size 5 and without length penalty.
9For all three tasks, we follow the data pipeline of fairseq: https://github.com/facebookresearch/ fairseq/blob/main/examples
10https://github.com/facebookresearch/ fvcore

heading: Model α/β/FM

heading: A.3.3 Language Modeling
text: We evaluate LM on WIKITEXT-103 (Merity et al., 2017) which has about 100M tokens and a 260K BPE vocabulary. Following Baevski and Auli (2019), we use perplexity as an evaluation metric and a context window of 2047 at the inference stage.

heading: A.3.4 Abstractive Summarization
text: We also evaluate on CNN-DailyMail (Hermann et al., 2015) for AS to test the ability of GHT in hard tasks with long inputs. The dataset comprises over 280K news articles paired with multi-sentence summaries. Following Wu et al. (2019), articles are truncated to 512 tokens and encoded with 50K BPE. We use F1-Rouge (Lin, 2004b) to evaluate the performance, including Rouge-1, Rouge-2 and Rouge-L.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7
7 A2. Did you discuss any potential risks of your work? The study is theoretical. It does not involve human in the experiment or involve ethic issues. It will not result in any potential risks.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4; Appendix A.3
3 B1. Did you cite the creators of artifacts you used? Section 4; Appendix A.3
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The tools are for public use.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No specific intended use is specified by the artifact creators.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? They does not contain any information that names or uniquely identifies individual people or offensive content.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 4; Appendix A.3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A.3
C 3 Did you run computational experiments? Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5; Appendix A.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4, 5; Appendix A.1, 2, 3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A.1, 2, 3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We will study these limitations in the future to give a more extensive exploration