Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention
Authors: Huiyin Xue
Nikolaos Aletras
Venue: EMNLP 2023 Findings (2023)
Keywords: multi-head attention; memory efficiency

Abstract: Scaling pre-trained language models has resulted in large performance gains in various natural language processing tasks but comes with a large cost in memory requirements. Inspired by the position embeddings in transformers, we aim to simplify and reduce the memory footprint of the multi-head attention (MHA) mechanism. We propose an alternative module that uses only a single shared projection matrix and multiple head embeddings (MHE), i.e. one per head. We empirically demonstrate that our MHE attention is substantially more memory efficient compared to alternative attention mechanisms while achieving high predictive performance retention ratio to vanilla MHA on several downstream tasks. MHE attention only requires a negligible fraction of additional parameters ($3nd$, where $n$ is the number of attention heads and $d$ the size of the head embeddings) compared to a single-head attention, while MHA requires $(3n^2-3n)d^2-3nd$ additional parameters.

Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-effcient Multi-head Attention
Huiyin Xue and Nikolaos Aletras
Department of Computer Science, University of Sheffield United Kingdom
(hxue12, n. alleraasesseeeffeaeea ac. uk
Abstract
Number of Paremetorr (M)
Scaling pre-trained language models has re- sulted in large performance gains in various natural language processing tasks but comes with a large cost memory requirements. In- spired by the position embeddings in transform- ers, we aim to simplify and reduce the memory footprint the multi-head attention (MHA) mechanism. We propose alternative module that uses only a single shared projection ma- trix and multiple head embeddings (MHE), i.c. one per head. We empirically demonstrate that our MHE attention is substantially more mem- ory efficient compared to alternative attention mechanisms while achieving high predictive performance retention ratio to vanilla MHA several downstream tasks. MHE attention only requires a negligible fraction of additional parameters (3nd, where is the number of attention heads and d the size of the head em- beddings) compared to a single-head attention, while MHA requires (3n² 3n)d? 2 3nd ad- ditional parameters.'
MHE
30
20
40
60
80
100
120
Number of Attendion Heads
Figure 1: Number of parameters for an attention sub- layer and different number attention heads using multi-head attention MHA and our multi-head embed- ding attention MHE. We fix the dimension of atten- tion to 64, only counting the parameters for projecting queries, keys, and values.
with the principles of Green artificial intelligence development (Schwartz et al., 2020). Moreover, scaling can hinder researchers with limited access to computing resources to participate in advane- ing the field (Schwartz et al., 2020). This results in inequalities, where only privileged few actively contribute, potentially impeding diversity and inclusivity (Weidinger et al., 2022).
1 Introduction
Scaling pre-trained language models (PLMs) aims to enhance performance by increasing their size and capacity, leading to models with an unprecedented number of parameters (Kaplan et al., 2020; Chowd- hery et al., 2022; Hoffmann et al., 2022). Just by in- creasing the size of PLMs and the pre-training data has yielded state-of-the the--tterrt performance various natural language processing (NLP) tasks (Devlin al., 2019; Liu et al., 2019; Clark al., 2020; Raffel et al., 2020; Brown et al., 2020; Clark et al., 2022a; Ouyang et al., 2022; Touvron et al., 2023). However, the pursuit of developing larger PLMs comes with large computational requirements. This has direct environmental implications such as large carbon emissions (Lacoste et al., 2019; Strubell al.. 2019; Weidinger al., 2022), conflicting
The backbone of transformers (Vaswani et al., 2017) is the multi-head attention (MHA) mod- ule that extends the standard single-head attention (SHA) proposed by Cho et al. (2014). MHA ap- plies attention mechanism (i.e. head) multiple times for the same set of queries, keys and values by using different set of parameters (i.e. pro- jection matrices) for each of them. This results in MHA modules with large memory footprint that increases with the number of layers and atten- tion heads per layer in PLMs (Devlin et al., 2019; Brown et al., 2020; Ouyang et al., 2022; Touvron et al., 2023). Figure/ 1 shows how the number of parameters of single attention sublayer increases with its number of attention heads.
Previous work has attempted to address this is- sue by proposing to share projection matrices or
'Code: https: http::///iitbbb..oo//I/II/////// www.//III......EEEE /ttp://wwIW.EE//mme//EeeE

---

eliminating them entirely to improve the parameter efficiency of MHA. Lan et al. (2020a) proposed sharing projection parameters for keys, queries and values across layers, while Kitaev et al. (2020) in- troduced a method for sharing the projection ma- trix between keys and values within each trans- former layer. Additionally, similar approaches use multi-query attention approach that uses pair of global projection matrices for keys and values in each layer (Shazeer, 2019; Chowdhery al., 2022; Ainslie al., 2023). Furthermore, Yan et (2021) climinate the projection matrices entirely and di- rectly treat the input hidden states as both keys and values. In different direction, Lee-Thorp et al. (2022) propose models that replace the attention blocks with token-mixture blocks (i.e. using linear Fourier transformations) that contain fewer or no parameters compared to MHA.
Hubara et al. (2017) proposed rep- resenting weights using fewer bits to reduce mem- ory requirements. Zadeh et (2020) introduced a method for identifying the outliers in weights and excluded them during quantization. Another direc- tion involves additional training steps to adjust the quantized weights. i.e. quantization-aware training (Zafrir et al., 2019; Boo and Sung, 2020; Stock et al., 2020; Shen et al., 2020; Tambe al., 2021; Tao et al., 2022). Bai al. (2022) developed a more efficient post-training quantization approach that minimizes the reconstruction error incurred by quantization.
Quantization
These compression approaches remove entirely parts of the network such as weights close to zero (Gordon et al., 2020; Mao et al., 2020; Chen et al., 2020) and weights that move towards zero during fine-tuning (Sanh al., 2020; Tambe et al., 2021). Different to operating on individual weights, previous work attempted to remove struc- tured blocks of weights or even architectural com- ponents such as attention heads and encoder lay- ers (Fan et al., 2019; Prasanna et al., 2020; Khetan and Karnin, 2020; Li et al., 2020a; Lin et al., 2020; Tay et al., 2021).
Pruning
Inspired by the position embeddings in trans- formers (Vaswani et al., 2017; Devlin al., 2019), we aim to simplify and reduce the memory foot- print of the MHA mechanism. We achieve this using only a single projection matrix for each of the keys, queries and values respectively shared across all attention heads, and one embedding per head (MHE).
Our contributions are as follows:
Knowledge Distillation This set of techniques typically train light student model mimic the outputs of a larger teacher PLM (Sun et al., 2019; et al., 2020b; Jiao et al., 2020; Sun et al., 2020; Li et al., 2021; Tahaei et al., 2022). In a similar direction, smaller PLMs have been recently fine-tuned on text generated by larger PLMs (Chi- ang et al., 2023; Taori et al., 2023).
We propose MHE, novel attention module that uses shared projection matrices across heads that are modified by corresponding em- bedding heads. Our method generates mul- tiple attention heads requiring only a small fraction of additional parameters compared to single-head attention.
We empirically demonstrate that our MHE attention is substantially more parameter effi- cient compared to alternative attention mech- anisms while achieving high predictive per- formance retention ratio (i.e. 92.9~98.77 (%)) to MHA on several downstream tasks, MHE is (3n dd 3nd smaller than MHA for single attention sublayer with n attention heads and hidden dimension of d per head.
Weight Matrix Decomposition Previous work also proposed replacing large weight matrices by the product of two smaller ones for reducing model size and runtime memory. Weight matrix decompo- sition has been applied to linear layers (Mao et al., 2020; Ben Noach and Goldberg, 2020), the embed- ding matrix (Lan et al., 2020b; Tambe et al., 2021; Wang et al., 2022), and attention blocks (Hu et al.. 2022; Wang et al., 2022).
2 Related Work
2.1 Model Compression
Embedding Matrix Compression Finally, vari- ous attempts have been introduced for compressing the embedding matrix during pre-training and fine- tuning (Xue et al., 2022; Clark et al., 2022b; Xue and Aletras, 2022).
To make PLMs memory efficient, previous work has focused on the following post-hoe model com- pression approaches (Ganesh et al., 2021; Tay et al., 2022).

---

2.2 Improving Attention Efficiency
in different contexts, we propose Multiple Head Embeddings (MHE) attention. MHE uses a shared 'seed' projection matrix that is subsequently combined with distinct head embeddings to generate multiple attention heads.
Previous work on making attention more efficient includes efforts towards (I) speeding-up pairwise computations between token representations; and (2) parameter efficiency.
Computational Efficiency While improving computational efficiency of attention is out of the scope of our paper, we provide a brief overview of previous work since it is complementary to pa- rameter efficiency. One approach to speed up at- tention computation is by reducing the number of similarity computations between representations in different positions using predefined local win- dows, fixed or dynamic strides (Child et al., 2019; Zaheer et al., 2020; Beltagy et al., 2020; Kitaev et al., 2020). Other methods leverage the approxi- mation of SoftMax to change the order of matrix multiplications, resulting in lower computational complexity (Katharopoulos al., 2020; Choroman- ski et al., 2021; Schlag et al., 2021; Qin et al., 2022). Related approaches along this direction proposed kemnel functions that require additional parameters (Choromanski et al., 2021; Wang et al., 2020). Fi- nally, Dao et al. (2022) proposed improvements in GPU memory access to optimize and accelerate the MHA computation.
3.1 Multi-head Attention (MHA)
We first begin by formally defining MHA. MHA consists of different projection matrices (W W w. E 1gdmxdy 1....... 1, where dm is the dimension of the input representation and dę is the dimension of attention heads) for queries (Q), keys (K) and values (V) per head, X n in total. It is computed as follows:
Q K, XWO.
H. Au(Q C K;, V:)
SoftMax Q.K. VV,
(3)
Vdn
Note that we use scale-dot attention, but our method can be used with any other attention mech- anism.
3.2 Seed Projection Matrix
Unlike MHA that uses different projection matri- per head, MHE attention employs only single projection matrix for each of the queries, keys and values, W W W E Rdn xdz, These matri- ces are shared across all attention heads.
Memory Efficiency Lan et al. (2020a) intro- duced a method for sharing the projection parame- ters for queries, keys and values across transformer layers. Furthermore. Kitaev et al. (2020) proposed sharing the projection matrix between keys and val- ues within each layer. Additionally, other methods multi-query attention approach that shares projection weights for keys and values across differ- ent heads (Shazeer, 2019; Chowdhery et al., 2022; Ainslie et al., 2023), while Yan al. (2021) directly treat the input hidden states as both keys and values. In a different direction, Lee-Thorp et al. (2022) proposed replacing the attention blocks with faster token-mixture blocks consisting of few parame- ters or no parameters at all. This includes methods such as linear or Fourier transformations in the loken-mixtur block. However, these approaches tend to yield lower predictive performance com- pared to MHA.
We obtain query, key and values projections of the input sequence X as follows:
Q K, V XWO.VV
(4)
3.3 Attention Head Embeddings
Using seed projection matrix for Q.K, K, V is equivalent single-head attention (SHA) mod- ule. Therefore, we need a mechanism to trans- form the seed projection matrices to obtain differ- ent attention head. For this purpose, we represent each attention head by specific head embeddings e,
Q K ee E IRd i = 1, 1,... n for queries, key and values. These embeddings have a substantially smaller memory footprint compared using differ- ent projection matrices per head. The contextual- ized representation H, of the entire input sequence X for head is computed as follows:
2 K. V. = (Q K; V, C.K.V) H. Au(Q. K. V.)
(5)
3 Multiple Head Embeddings Attention
Inspired by the absolute position embed- dings (Vaswani et al., 2017; Devlin et al., 2019) for distinguishing the representation of the same token
where (-) function that modifies the query, key and value matrices with a corresponding head embedding e,,

---

Multi-head Attention (MHA): This is the original multi-head attention mecha- nism (Vaswani et al., 2017; Devlin et al., 2019).
Qu K. V.
Q K,
erabedang
imsogration oger,
Single-head Attention (SHA): Similar to MHA but using only attention head.
EL-ATT:
Introduced by Yan et al. (2021). this attention variant completely eliminates the projection matrices for all keys and values,
pr0j
proj
W K.V
Wikr
MQA: Introduced by Shazeer (2019), this approach uses shared projection matrices for keys and values across all attention heads. Note that different projection matrices are used for queries across heads.
X
X
MHA
MHE
Figure 2: Multi-head attention (left) requires 3 x n pro- jection matrices queries, keys and values V where is the number of attention heads. Multi-head embedding attention (right) uses only three projection matrices and 3 head embeddings.
SKV: Introduced by Kitaev et al. (2020), this attention variant enforces keys and values share the same projection matrix within each attention module.
3.4 Modifying Queries, Keys and Values with Head Embeddings
4.2 Data
We experiment with a diverse range of tasks includ- ing: (1) two standard natural language understand- ing benchmarks in English, GLUE (Wang et al.. 2018) and SUPERGLUE (Wang et al., 2019); (2) two question and answering benchmarks in En- glish, SQUAD v1.1 (Rajpurkar et al., 2016) and SQUAD v2.0 (Rajpurkar et al., 2018); ; (3) WMT- 14 English-to- Beriss-tt-Genre -o--t-m-aGnnnnn machine translation (Bo- jar et al., 2014); and (4) two language modelling datasets in English WIKITEXT- 10---03 03 (Merity et al.. 2017) and PENN TREEBANK (Marcus et al., 1993).
We propose two MHE variants, one adds and the other multiplies the head embeddings with the seed projection matrices.
MHE-A :D:: Motivated by the absolute posi- tion embedding (Devlin et al., 2019), we use the addition operation in Equation 5, represented as ((A,b) = A + b, where A E {Q and b E _ eK e respectively.
Likewise, motivated by the rotary position embedding (Su et al., 2021), MHE-MUL employs multiplication as the integrating operation in Equation 5 as r¹( (A,Ab)
MHE-MUL:
4.3 Models
We test all different attention variants on two architectures: (1) encoder-only transformer (De- vlin et al., 2019) and (2) encoder-decoder trans- former (Vaswani et al., 2017).
A @ (b 1), where @ represents the Hadamard product.?
Figure shows an overview of the MHE mecha- nism compared to MHA.
Encoder-only For GLUE, SUPERGLUE, SQUAD v1.1 and SQUAD v2.0, we use a BERT-basc architecture. This consists 12 transformer layers, embedding size of 768, hidden states dimension of 768, 12 attention heads and a maximum sequence length of 512.
4 Experimental Setup
4.1 Attention Mechanisms
We compare our MHE attention with the following attention mechanisms:"
Decoder-only We also test decoder-only model using the GPT2-base architecture on WIKITEXT- 103, PENN TREEBANK and GLUE. GPT2-base consists of transformer layers, embedding size of 768, hidden states dimension of 768, attention heads and a maximum sequence length of 512.
We add avoid elements queries, keys values become too small during initialization.
*W have also experimented with Linear and Fourier token- mixture madels Therr et al., 2022) yielding substantially lower performance. For full results of these methods, see Ap- pendix A.

---

Encoder-decoder For WMT-14, we train an encoder-decoder transformer from scratch. It con- sists of 12 layers (6 for the encoder and decoder respectively). an embedding size of 512, hidden states dimension of 512 and 8 attention-heads and a maximum sequence length of 100.
and SQUAD v2.0. We use BLEU report perfor- mance in WMT-14 English-to-German machine translation task. We use perplexity (PPL) to report generative performance on WIKITEXT- 103 and PENN TREEBANK by fixing the stride length to 256.
We set the number of attention heads to for all SHA models. Experimenting with larger models and different number of attention heads is of the scope of our paper and left for future work due to limited access to computing resources.
4.6 Memory Efficiency Evaluation
Furthermore, we use the following metrics to mea- and compare the memory efficiency of MHE and the baselines.
4.4 Implementation Details
Performance Retention Ratio: We compute the ratio between the predictive performance of each attention mechanism compared to MHA upper-bound baseline performance (the higher the better).
Pre-training We pre-train all models the En- glish Wikipedia and BookCorpus (Zhu al., 2015) from HuggingFacce (Lhoest al., 2021) for up to IM steps with a batch size of 128. We choose masked language modelling as the pre-training ob- jective. For all models, we use a 30K WordPiece vocabulary (Devlin et al., 2019).
For direct indicator (e.g. accuracy etc.):
scorCmudel
PRR
Sсотиыин
Fine-tuning and Training For GLUE, SUPER- GLUE, SQUAD v1.1 and SQUAD v2.0, we fine- tune all pre-trained models up 20 epochs with early stopping fixing the batch size to 32. For each task, we use five different seeds and report the av- crage.
For inverse indicator (e.g. perplexity etc.):
SCOrCmedel SCOrEмHA
PRR
SCOTEMIA
Performance Elasticity of Parameters: In- spired by the concept of elasticity in eco- nomics (Bittermann, 1934), which measures the responsiveness of an economic variable (c.g. investment demand) to change in an- other (e.g. interest rate), we extend it to measure the parameter utilization rate of a target model compared to the SHA lower- bound. The performance elasticity of pa- rameters (PEoP) indicates how effectively pa- rameters contribute to predictive performance, compared to SHA. is computed as follows: For direct indicator (e.g. accuracy etc.):
We train the encoder-decoder model from scratch the training set of WMT-14 English- to- to-German machine translation dataset up to 100K steps with batch size of 256. WMT- contains 4.5M sentence pairs and evaluate on its test set. We train the tokenizer using byte-pair-encoding (Sennrich et al., 2016) with 37K merging steps on the training set. We enable both source language and target language to share the vocabulary. We use one random seed and report the average on the last five epochs. We optimize all models using AdamW (Loshchilov and Hutter, 2019).
PFoP (Score /scorosAA)
Hy perraaarssesess Hyperparameter selection de- tails are Appendix B.
[params modes params
For inverse indicator (e.g. perplexity etc.):
Hardware For pre- eve-training, we use four NVIDIA Tesla A100 GPUs and one for fine-tuning on down- stream tasks.
(score (scoresHs) -
PEoP
(params paramsssa)
PEoP quantifies the extent which model's performance can be boosted with 1% ad- ditional parameters compared a baseline model (the higher the better)."
4.5 Predictive Performance Evaluation
For GLUE, SUPERGLUE, SQUAD v1.1 and SQUAD v2.0, we use the official metric of each task (see Appendix A for details on metrics for each task). We report score for SQUAD v1.1
"We subtract in hoth nominator and denominator, follow- ing the original definition of clasticity.

---

GLUR
SUPERGLUE PRR
SQUAD LL.1
SQUAD V2.0
Attention
#params
Acc
PRR
PEoP
Acc
PEoP
Acc
PRR
PEoP
Acc
PRR
PEoP
SHA
8.85M
79.2
96.7
67.1
95.1
82.5
93.1
67.6
87.8
MHA
28.32M
81.9
100.0
0.02
70.5
100.0
0.02
88.6
100.0
0.03
77.0
100.0
0.06
AT-ATT
14.16M
80.3
98.0
0.02
98.5
0.06
86.5
97.6
72.2
93.8
0.11
MQA
15.34M
81.3
99.2
0.04
69.3
0.04
86.7
0.07
74.8
97.1
0.15
SKV
21.23M
81.4
99,4
0.02
69.9
99.1
0.03
88.1
99.4
0.05
75.9
98.6
0.09
MHE-ADD
88M
80.4
98.2
4,92
69.
97.9
9.44
83.7
94.5
3.65
71.8
93.2
19783
MHE-ME MUL
8.88M
80.6
98.3
5.53
69.6
98.7
12.07
85.9
97.0
13.19
72.3
93.9
22.25
Table I: Results of the encoder-only architecture GLUE, SUPERGLUE, SQUAD vI.1 and SQUAD v2.0 dev sets with performance retention ratio (PRR) and performance elasticity of parameters (PEoP) over five runs. Bold values denote best performing method each benchmark.
GLUE PRR
WIKITEXT-103
PENN TREEBANK
Attention
#params
Acc
PEoP
PPL
PRR
PEoP
PPL
PRR
PEoP
SHA
8.85M
75.3
97.2
62.0
55.8
68.1
46.3
MHA
28.32M
77.5
100.0
0.01
43.0
100.0
0.14
44.3
100.0
0.16
EL-ATT
14,16M
76.6
98.9
0.03
57.1
67.2
0.13
56.1
73.4
0.29
MQA
15,34M
76.9
99.2
0.03
49.7
84.4
0.27
49.3
88.7
0,38
SKV
21 21,23M
77.1
99.5
0.02
46,2
92.6
0.18
45.5
97.3
0.24
MHEEADD
8.8M
75.8
97.8
2.18
54.0
74.4
41.29
55.3
75.2
60.13
MHE-MUI 8.88M
76.7
99.0
5.92
53.8
74.9
42.32
50.7
85.6
81.76
Table Results of only architecture on GLUE dev sets and WIKITEXT-103. PENN TREEBANK sets with performance retention ratio (PRR) and performance elasticity of parameters (PEoP) over five runs. Bold values denote best performing method in each benchmark.
5 Results
rics for evaluation, MHE-MUL leads to the perplex-
ities of 53.8 and 50.7 compared to 43.0 and 44.3 for MHA on WIKITEXT- 103 and PENN TREEBANK respectively, indicating stable PRR higher than 74.9%.
5.1
Predictive Performance Comparison
Table presents results on GLUE, SUPERGLUE. SQUAD v1.1 and SQuAD v2.0 for our MHE variants and all baselines. We first observe that both the performance of our MHE-ADD and MHE- MUT. are comparable to the vanilla MHA on two text classification benchmarks (80.4, 80.6 81.9 on average GLUE and 69.1, 69.6 70.5 on age SUPERGLUE) with high performance retention ratios (PRR) between 97.9% and 98.7%. On ques- tion answering tasks SQUAD v v1.1 and SQUAD v2.0, both MHE variants are also competitive, with PRRs higher than 93%.
In all tasks, MHE consistently outperforms SHA by large margin with only 0.03M extra param- eters, i.e. 0.6-17.4. For example, 69.6 67.1 in SUPERGLUE, 72.3 vs. 67.6 SQUAD v2.0. 23.6 vs. 22.5 WMT- 14 and 62.0 vs. 53.8
WIKITEXT-10 103 for the MHE-MUL variant. We also note that MQA and SKV attention mecha- nisms generally perform better than MHE, how- ever they are 1.7 and 2.4 times larger than MHE, i.e. 15.34M and 21.23M vs. 8.88M parameters. It is worth noting that MHE-MUL outperforms EL-ATT three out of five benchmarks, despite having nearly half the parameters in the attention module.
Similar results observed the WMT-14 English-to-German machine translation task for the encoder-decoder transformer. According Table 3, MHE-ADD and MHE-MUL achieve BLEU scores 23.0 and 23.6, respectively. The performance of MHE-MUL is negligibly lower than that of MHA (24.8) while being substantially smaller.
5.2 Memory Efficiency Comparison
Consistent results for the decoder-only trans- former are shown in Table 2. The PRRs for MHE-
Our results so far indicate that performance in- creases with the number of attention mechanism parameters, which expected. Next, we inspect how efficiently different attention mechanisms uti-
ADD and MHE-MUL on GLUE are still high (i.c. 97.8% and 99.0%). While using the intrinsic met-

---

lize their parameters Tables and 3 show how parameter efficient our MHE attention variants and all baselines are, measured PEoP. Note that
Attention
#params
BLEU
PRR
PEoP
SHA
6.49M
22.5
90.8
MHA
18.87M
24.8
100.0
0.1
PEoP scores for SHA cannot be computed as is used as the point for reference model. We also report PRR using MHA baseline for complete- ness, however this metric does not take the model size into account.
EL-ATT
9.44M
23.9
96.6
0.1
MQA
10.62M
24.2
97.6
0.1
SKV
14.16M
24.7
99.5
0.1
MHE-ADD
6.52M
23.0
92.9
3.5
MHE-MUL 6.52M
23.6
95.0
11.0
We first observe in Table that both our MHE- ADD and MHE-MUl achieve the highest PłoP scores on the two natural language understanding benchmarks (4.92, 5.53 on GLUE, and 9.44, 12.07 SUPERGIUEE) and two question answering tasks (4.65, 13.19on SQUAD v1.1, and 19.88, 22.25 on SQUAD v2.0). In contrast, vanilla MHA results in the lowest PEoP score among all models ex- pected, ranging from 0.02 to 0.06. It indicates the memory inefficiency of MHA.
Table 3: BLEU scores on WMT- 14 English to German machine translation task with performance retention ratio (PRR) and performance elasticity of parameters (PEoP), Bold values denote best performing method each benchmark.
TREEBANK) and GLUE (2.18 and 5.92). At the same time, MHA fail perform well on GLUE and PENN TREEBANK with PEoP of 0.01 and 0.16 respectively. MHE-ADD and MHE-MUL also con- sistently outperform other officient-attention vari- ants (i.e. EL-ATT, MQA and SKV) by 72-340 times on PEoP across the three benchmarks.
The PEoPs of more light-weight EL-ArT and SKV are similar to that of MHA (0.02) on aver- age GLUE, barely %cof that of MHE, indicating they are far more memory-inefficien compared to MHE.
In all tasks, MHE consistently outperforms MHA by orders of magnitude in parameter effi- ciency. We also note that EL-ATT, MQA and SKV V only lead to PEoP scores with the same magnitude as MHA. This highlights the more superior parame- ter utilization of MHE attention variants, achieving memory-efficiency.
Similar findings are observed in WMT- for the encoder-decoder models depicted Table 3. MHE-ADD and MHE-MUL achieve PEoP scores of 20.0 and 27.9, respectively. PEoP scores of MHA, EL-ATT MQA and SKV are close to zero (barely 0.1). This means that investing more parameters into their attention mod- ules would not bring proportional benefits in predic- tive performance. Even for the SKV which is half the size of MHA and achieves high PRR, when the number of parameters increase by 1,, the BLEU score increases a negligible 0. 0.1% while evolving from SHA. However, with the same number of parameters, our most memory-inefffiient MHE- MUL is able to improve the BLEU score by 1 11.0% Such rate of return is 110 times larger than that of SKV. Leveraging the head embeddings by adding only negligible number of parameters efficiently improves the predictive performance.
contrast, the
5.3 Theoretical Memory Complexity
Table 4 presents the theoretical memory complex- ity and the total number of parameters of our two MHE and baseline attention mechanisms in sin- gle transformer sublayer. First, we see that the theoretical memory complexity of MHA and other efficient parameters (EL-ATT, MQA and SKV) are quadratic with the number of attention heads, while our MHE are the only two variants having the com- plexity linear with the attention heads similar to SHA.
We further observe that MHE-ADD and MHE- ceatteectuur-anossttt obtaining similar memory efficiency for the decoder-only model in Table 2. Both our MHE-ADD and MHE-MUL achieve the highest PEoP scores on the two lan- guage modelling benchmarks (41.29, 42.32 on WIKITExT-103 and 60. and 81.76 on PENN
MUL
Taking a closer look at the rightmost column in Table 4, we observe that the number of extra pa- rameters of all attention variants compared to SHA have quadratic relationship both the number n and the dimension of attention heads d, except our two MHE variants. MHE only requires relatively small fraction additional parameters compared to SHA.
For a detailed report on the memory usage of different attention mechanisms, see Appendix C.

---

Attention
Complexity
#Params
#Paarams (+)
SHA
O(n)
3dn nd
Number of Farametes (14)
MHA
O(n²)
(3n² - 3r) 3n)dd
MQA
EL-ATT
O(n²)
d
(n² 3n)d
SKV
MHE
MQA
O(n²)
d'n² + n
(n² - n)dd
100
SHA
SKV
O(n²)
nd
(2n² 3rd
50
MHE (ours)
-ADD
O(n)
3d n 3dm
3nd
-MUL
O(n)
3d 3dn
3nd
20
6O Number Altention Heads
40
80
100
120
Table Memory complexity regarding the number of parameters in each attention sublayer, while fixing the dimension of attention heads to d. n denotes the number attention heads. simplify, dimension hidden states dm is set to nd. The last projection for pooling attention heads is excluded.
Figure 3: Number of parameters per attention sublayer, while scaling the number of attention heads in different attention variants. We fix the dimension of attention to 64.
heads cutting down the number of layers to quarter when working under tight memory budget. However, MHE does not suffer by such issues. Further, we project these estimates to the pop- ular GPT-3 model (Brown et al., 2020). is a decoder-only model with 96 decoder layers, 96 attention heads per layer, and head dimension of 128. The vanilla multi-head attention module requires massive 43.48B parameters. However, using MHE attention, this number can be signif- icantly reduced 0.46B parameters, i.e. approx- imately reduction by 98.9%. Comparing this to other parameter-efficien attention variants such as EL-ATT (14.50B parameters), MQA attention (14.80B parameters), and SKV attention (28.99 (28.99B parameters), it becomes evident that our MHE of- fers better memory efficiency. This makes it com- pelling alternative for memory-constrainee scenar- ios. See Appendix for detailed study on the robustness of MHE model size changes (i.e. scal- ing).
5.4 Scaling the Number of Attention Parameters
Delving deeper the effect of scaling memory footprint, we show in Figure the total number of parameters needed for a single attention mod- ule (c.g. in an encoder layer). We fix the dimen- sion of attention heads to 64 commonly used by BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), GPT-2 (Radford et al., 2019), BART (Lewis et al., 2020) and T5 (Raffel et al., 2020). In general, note that the number of parameters in MHA could reach more than 200M if employing 128 at- tention heads. At the same time, SKV, MQA and EL-ATT would require 2/3, 1/3 and 1/3 of that num- ber respectively. In contrast, MHE only accounts for 1% of the MHA parameters.
Moreover, we also present Figure 4 the to- tal number of parameters required across attention variants when stacking 12, 24 and 48 layers along with 32 and 64 attention heads respectively. We also fix the dimension of attention heads to 64. We can observe, when the number of attention head reaches 64, MHA with 24 layers already occupies more than 1B parameters, while EL-ATT and MQA reach 0.8B parameters with 48 layers. SKV takes 24 layers to reach 0.8B parameters. However, the total number of parameters in MHE attention does not exceed B even when scaling to 48 layers with 64 attention heads. It is also clear that scal- ing the attention module 48 layers, 32 attention heads and 12 layers needs comparable number of parameters for MHA, EL-ATT, MQA or SKV. This indicates, that LLM developers have to make a choice whether doubling the number of attention
Discussion
MHA enables the model attend to information from different representation subspaces at differ- ent positions (Vaswani et al., 2017). uses dis- tinct projection matrices for each attention head and integrates the information from these differ- ent representation subspaces. However, Vaswani et al. (2017) did not explore different methods for performing space transformations per head. Previous work has pointed out that over- parameterized models might have a low intrinsic dimension. Therefore, transforming the projection
'It would have been great to report results by pre-training our own MHE GPT-3 model, however this is prohibitive with the modest compute we have available.

---

matrices smaller low-rank ones usually does not severely harm model predictive performance (Li et al., 2018; Aghajanyan al., 2020). Meanwhile, the classic MHA approach also does not impose any constraints on the orthogonality of these sub- spaces during pre-training and fine-tuning. The col- umn vectors in those projection matrices could be highly collinear, i.e. the projection matrices could be rank-deficient. As result, its inner-working mechanism could be simply understood as intro- ducing levels of variation to the encoded represen- tation of the same token at the same position across different heads.
Number
Heeds 64
Number Parsmnattre 86
2.0
1.0
Figure 4: Total number of parameters in attention sub- layers, while scaling the number of attention layers 12, 24 and 48 with 32 attention heads and 64 attention heads respectively. We fix the dimension attention to 64.
Our MHE approach is possible achieve mem- ory efficiency (similar to SHA) together with high PRR compared to MHA by mimicking the posi- tion embeddings for representing different attention heads.
mental results have demonstrated that MHE at- tention exhibits superior memory efficiency com- pared to other memory-efficiett attention variants, while achieving high predictive performance ra- tio to MHA on various downstream tasks. Com- pared to a single-head attention, MHA requires (3n² - -3n)d² parameters for n attention heads and head dimensionality d, while MHE barely requires negligible 3nd. For future research, investigate scaling up MHE models and explore its linguistic capabilities (Vulié al., 2020; Koto et al., 2021).
On one hand, the addition operation in MHE- ADD is used for transforming the keys, queries and values. This can be subspace obtained through projection, followed by rotation. For an input representation, the difference between the projected and injected (i.e. through head embedding addition) queries, keys and values is a constant vector across any pair of heads. On the other hand, the MHE-MUL approach employs a multiplication operation, which more aggressively distorts and reshapes the keys, queries and values subspaces. Head embeddings in MHE-MUL, play role the scaling factors, respectively stretching each dimension of the input representation. Thus, the difference between the keys, queries, and values generated by different heads for the same input representation, vector parallel to the projected input. This vector is dependent on the specific input, unlike the constant vector in MHE-ADD.
small distortion of the
plan
Limitations
We experiment only using 'base' size models with- out experimenting with larger architectures, due to limited access to computational resources. Sim- ilarly, we did not experiment with decoder only architectures (Brown et al., 2020) which we leave for future work. We have not combined our MHE method with computationally efficient at- tention methods with linear complexity, such Linformer (Wang al., 2020). We expect that it would speed up computation of MHE, but it is out of the scope of our paper.
Interestingly, our experimental results consis- tently show that the multiplication operation outper- forms addition in the majority of benchmarks. This corroborates findings of a previous empirical study by Su et al. (2021) that compared rotary position embeddings (somehow analogous to MHE-MUL) with absolute position embeddings (analogous to MHE--AD)..
Acknowledgments
We would like to thank Constantinos Karouzos, Miles Williams and the anonymous reviewers for their invaluable feedback.
7 Conclusions
We have proposed MHE attention that employs single shared projection matrix along with mul- tiple head embeddings, to simplify and reduce the memory footprint the MHA. Our experi-
References
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2020. Intrinsic dimensionality explains the

---

effectiveness of language model fine-tuning. arXiv preprint 2rX2::2012.13255.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long se- quences with sparse transformers. arXiv preprint arXiv:1904. 10509.
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago On- tañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Teopph Yi Tay, Yun- Hsuan Sung, and Sumit Sanghai. 2023. Colt5: Faster long-range transformers with conditional computa- tion.
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bah- danau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder ap- proaches. In Proceedings of SSST-8, -8, Eighth Work- shop on Syntax, Semantics and Structure in Statistical Translation. pages 103-111, Doha, Qatar. Associa- tion for Computational Linguistics.
Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Irwin King, and Michael R Lyu. 2022. Towards efficient post- training quantization of pre-trained language models. Advances in Neural Information Processing Systems. 35:1405-144 1418.
Krzysztof Marcin Choromanski, Valerii Likhasherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Be- langer. Lucy Colwell, and Adrian Weller. 2021. Rethinking attention with performers. In Interna- rional Conference on Learning Represenations.
Iz Beltagy, Matthew E Peters, and Arman Cohan, 2020. Longformer: The doccmeetnn transformer. arXiv preprint arXiv:2004.05/50
Matan Ben Noach and Yoav Goldberg. 2020. Compress- ing pre-trained language models by matrix decompo- sition. In Proceedings of the 1si Conference of the Asia-Pacific Chapter of the Association for Compu- tational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 884-889, Suzhou, China. Association for Computa- tional Linguistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sehastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Henry Bittermann. 1934. Elasticity of supply. The American Economic Review, pages 417-429.
Jonathan H. Clark, Dan Garrette, Iulia Turc. and John Wieting. 2022a. Canine: Pre-training an Efficient Tokenization- Encoder for Language Represen- tation. Transactions of the Association for Computa- tional Linguistics, 10:73-91.
Ondiej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina. Matt Post, Herve Saint- Amand, Radu Soricut, Lucia Specia, and Ales Tam- chyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore, Maryland, USA. Associa- tion for Computational Linguistics.
Jonathan H. Clark, Dan Garrette, Iulia Ture, and John Wieting. 2022b. Canine: Pre-training an efficient tokenization-free encoder for language representa- tion. Transactions of the Association for Computa- rional Linguistics, 10:73-91.
Kevin Clark, Minh-Thang Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre- training text encoders as discriminators rather than generators. In ICLR.
Yoonho Boo and Wonyong Sung. 2020. Fixed-point op- timization of transformer neural network. In ICASSP 2020-2023 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 1753-1757. IEEE.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1999..
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, Volume (Long and Short Papers), pages Minneapolis, Minnesota. Association for Computational Linguistics.
Tianlong Chen. Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020. The lottery ticket hypothesis for pre- trained bert networks. Advances in neural informa- tion processing systems, 5834-15846
Win Chiang. Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See http::/vicuna. Imsys. org (accessed 14 April 2023).
Angela Fan, Edouard Grave, and Armand Joulin. 2019. Reducing transformer depth on demand with struc- tured dropout. In International Conference on Learn- ing Representations.

---

Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjd, Preslay Nakov, Dem- ing Chen, and Marianne Winslett. 2021, Compress- ing large-scale transformer-baeed models: A case study on BERT. Transactions of the Association for Computational Linguistics, 9: 9:1061-1080.
American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 3849-3866 Online. Association for Computa- tional Linguistics.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700
Mitchell Gordon, Kevin Dub, and Nicholas Andrews. 2020. Compressing BERT: Studying the effects of weight pruning on transfer learning. In Proceedings of the 5th Workshop on Representation Learning for NLP,, pages 143-155, Online. Association for Com- putational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020a. Albert: A lite bert for self-supervised learn- ing of language representations. In International Conference on Learning Representations.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Train- ing compute- compute-optimal large language models. arXiv preprint arXiv:2203. /5556.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020b. Albert: A lite bert for self-supervised learn- ing of language representations. In International Conference on Learning Representations.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
James Lee-T Thorp. Joshua Ainslic, Ilya Eckstein, and Santiago Ontanon. 2022. FNet: Mixing tokens with Fourier transforms. In Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguisties; Human Language Technologies, pages 4296-4313, Seattle, United States. Association for Computational Lin- guistics.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran Yaniv, and Yoshua Bengio. 2017. Quantized neu- ral networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1) (88666666-
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7888 Online. Association for Computa- tional Linguistics.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang. Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.
TinyBERT: Distilling BERT for natural language un- derstanding. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 4163- 4174, Online. Association for Computational Lin- guistics.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško. Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac, Thibault Gochringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scout Gray. Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint 2rXiv:2001.088 003611.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pap- pas, and François Fleuret. 2020. Transformers mns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156-5165. PMLR.
Ashish Khetan and Zohar Karnin. 2020. SchuBERT: Optimizing elements of BERT. Proceedings of the 58th Annual Meeting of the Association for Compu- lational Linguistics. pages 2807-2811 Online. Asso- ciation for Computational Linguistics.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Inter- national Conference on Learning Representations.
Bingbing Li, Zhenglun Kong, Tianyun Zhang, Ji Li, Zhengang Li, Hang Liu, and Caiwen Ding. 2020a. Efficient transformer-baat based large scale language rep- resentations using hardware-friendly block structured pruning. In Findings of the Association for Computa- tional Linguisties: EMNLP 2020, pages 3187-3199. Online. Association for Computational Linguistics.
Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. Discourse probing of pretrained language models. In Proceedings of the 2021 Conference of the North

---

Chunyuan Li. Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. 2018. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations.
human feedback. Advances in Neural Information Processing Systems.
Sai Prasanna, Anna Rogers, and Anna Rumshisky. 2020. When BERT Plays the Lottery, All Tickets Are Win- ning. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages Computational Linguistics.
Jianquan Li, Xiaokang Liu, Honghong Zhao, Ruifeng Xu, Min Yang, and Yaohong Jin. 2020b. BERT- EMD: Many-to-many layer mapping for BERT com- pression with earth mover's s distance. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3009-3018, Online. Association for Computational Linguistics.
Online. Association for
Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yun- shen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. 2022. cosformer: Rethinking soft- max in attention. In International Conference on Learning Representations.
Lei Li. Yankai Lin, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. 2021. Dynamic knowledge distillation for pre-trained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 379-389, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188):
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research. 21(1):5485555)
Zi Lin, Jeremiah Liu, Zi Yang. Nan Hua, and Dan Roth. 2020. Pruning redundant mappings in transformer models via ppectral-normalicn
identity prior. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 719-730, Online. As- sociation for Computational Linguistics.
Pranav Rajpurkar, Robin Jía, and Percy Liang, 2018. Know what you don't 't know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach, arXiv preprint arXiv: 11692.
Tlya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Quanlu Zhang, Yaming Yang, Yunhai Tong, and Jing Bai. 2020. LadaBERT: Lightweight adaptation of BERT through hybrid model compres- sion. In Proceedings of the 28th International Con- ference on Computational Linguisties. pages 3225- 3234, Barcelona, Spain (Online). International Com- mittee on Computational Linguistics.
Victor Sanh, Thomas Wolf, and Alexander Rush. 2020. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Neural Information Processing Systems. 33:20378-20389.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993, Building a large annotated cor- pus of English: The Penn Treehank. Computational Linguistics, (9(2):3333330
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. 2021. Linear transformers are secretly fast weight programmers. In International Conference on Ma- chine Learning, pages 9355-936660 136559936..666 PMLR.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In International Conference on Learning Repre- sentations.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 2020, Green AL. Communications of the ACM, 63(12):54 63.
Long Ouyang, Jeffrey WU, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.

---

Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:191.002150.
Thiemy Tambe, Coleman Hooper, Lillian Pentecost, Tianyu Jia, En- Yu Yang. Marco Donato, Victor Sanh, Paul Whatmough, Alexander M Rush, David Brooks, et al. 2021. Edgebert: Sentence-level energy opti- mizations for latency-aware multi-task nlp inference. In MICRO-54: 54th Annual IEEF/ACM International Symposium on Microarchteecture, pages 830-844.
Sheng Shen, Zhen Dong, Jiay Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2020. Q-bert: Hessian based ultra low precision quantization of hert. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 8815 8821.
Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2022. Compression of generative pre trained language mod- els via quantization. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 4821- 4836, Dublin, Ireland. Association for Computational Linguistics.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deep- speed and megatron to train megatron-turing nlg 530b, large-scall generative language model. arXiv preprint UrXiv22201.1199)
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin. Percy Liang. and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An cntructiion-ollowinng llama model. https: // gi thub. cot/
Pierre Stock, Angela Fan, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. 2020. Training with quantization noise for extreme model compression. In International Con- ference Learning Representations.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe 7hao, and Che Zheng. 2021. Synthesizer: Re- thinking self-attention for transformer models. In International conference on machine learning, pages 10183-10192. PMLR.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy, Asso- ciation for Computational Linguistics.
Yi Tay, Mostafa Dehghami, Dara Bahri, and Donald Met- zler. 2022. Efficient transformers: A survey, ACM Computing Surveys, 55(6):1-28.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2021. Roformer: En- hanced transformer with rotary position embedding. arXiv preprint arXiv:2104.0980
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397/
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model com- pression. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing (EMNLP-IJCNLP),, pages 4332, Hong Kong, China. Association for Com- putational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information pro- cessing systems, pages 5998 6008.
Ivan Vulić, Edoardo Maria Ponti, Robert Litschko. Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNLP), pages 7222-7240, Online, Association for Computa- tional Linguistics.
Zhiqing Sun, Hongkun Yu, Xiaodan Song. Renjie Liu, Yiming Yang, and Denny Zhou. 2020. MobileBERT: a compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meet- ing of the Association for Computationall Linguistics, pages 2158-2170, Online. Association for Computa- tional Linguisties.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stick- ier benchmark for general- purpose language under- standing systems. Advances in neural information processing systems, 32.
Marzich Tahaci, Ella Charlaix, Vahid Nia, Ali Ghodsi, and Mehdi Rezagholizadeh. 2022. KroneckerBERT: Significant compression of pre-trained language mod- els through kronecker decomposition and knowledge distillation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2116-2127, Seattle, United States. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, 2018. GLUB: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the

---

2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
and reading books, In Proceedings of the IEEE ternational conference on computer vision, pages 19-27.
Benyou Wang, Yuxin Ren, Lifeng Shang. Xin Jiang, and Qun Liu. 2022. Exploring extreme parameter compression for pre-trained language models. In In- ternational Conference on Learning Representations.
Sinong Wang. Belinda Z Li, Madian Khabsa, Han Fang. and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04 04768.
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang. John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. 2022. Taxonomy of risks posed by language models. In 2022 ACM Conference on Fairness, Ac- countability, and Transparency, pages 214-229.
Huiyin Xue and Nikolaos Aletras. 2022. HashFormers: Towards
independent pre-trained trans- formers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7862-7874, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Linting Xue, Aditya Barua, Noah Constant, Rami AI- Rfou. Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards foken-free future with pre-trained byte-to-byte models. Transac- tions of the Association for Computational Linguis- tics, 10:291-306.
Yu Yan, Jiusheng Chen, Weizhen Qi, Nikhil Bhen- dawade, Yeyun Gong, Nan Duan, and Ruofei Zhang, 2021. El-attention: Memory efficient lossless atten- tion for generation. International Conference on Machine Learning, pages 1148-1 1658. PMLR.
Ali Hadi Zadeh, Isak Edo,, Omar Mohamed Awad, and Andreas Moshovos. 2020. Gobo: Quantiz- ing attention-basse nlp models for low latency and 2020 53rd Annual IEEE/ACM International Symposium on Microarchi- tecture (MICRO), pages 81 1 824. IEEE.
energy efficient inference.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. QSbert: Quantized bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NuurPP? Edition (EMC2-NIPS), pages 36-39, IEEE
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-17297.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies

---

A Reported Metrics for Each Task
1.37 times better than EL-ATT, which indicates
SotA memory saving capabilities compared all other parameter-effiiiet attention variants.
We evaluate all models on GLUE (Wang et al., 2018), SUPERGLUE (Wang et al., 2019), SQUAD v1.1 (Rajpurkar et al., 2016) and SQUAD v2.0 (Rajpurkar et al., 2018). We report matched accuracy for MNLI, Matthews correlation for CoLA, Spearman correlation for STS, Fl score for QQP, CB, MultiRC and SQUAD and accuracy for all other tasks. Table and Table present results on GLUE and SUPERGLUE respectively for our MHE-FORMERS models and all baselines with the encoder-only architecture. Table 7 and 8 present results of the scores and performance elasticity of parameters (PEoP) across all models over each task in GLUE and SLPERGLUE. Table 10 presents re- models
D Robustness to Scaling
We also conduct experiments observe the effec- tiveness and the robustness of our best MHE-MUL while scaling the model size.
Table 14 presents average accuracy on two text classification benchmarks (GLUE and SUPER- GLUE), perplexities on two language modelling benchmarks (WIKITEXT- 103 and PENN TREE- BANK) with their corresponding performance re- tention ratio (PRR) for MHA and MHE-MUL in both encoder-only and decoder-only architecture across different model sizes.? 7 For the encoder-only models, observe that the PRR of MHE-MUS re- mains stable on GLUE (from 98.4% to 98.7%) and SUPERGLUE (from 98.7% to 96.2%) while scaling the number of parameters in the attention blocks to 3.5 times larger. For the decoder-only models, the PRR on GLUE for MHE-MUl stabilizes at 97.9% (i.e. 1.1% % lower) after scaling, Surprisingly,
sults on GLUE for our and all baselines with the decoder-only architec- ture. Table 10 presents results the scores and performance elasticity of parameters (PEoP) across all models over each task in GLUE.
B Hyperparameters
The hyperparameters used in pre-training are listed in Table 11. The hyperparameters used in fine- tuming are listed in Table 12.
the PRR of MHE-MUL increases WIKITEXT- 103 (from 74.9% to 95.2%) and PENN TREEBANK (from 85.6% to 88.5%) while scaling to MEDIUM size.
C Memory Usage
Similar results are observed for the encoder- decoder architecture on WMT14 machine trans- lation task. According to Table 15, we first ob- serve the PRR of MHE-MUL remains stable (i.e. between 91.5% and 96.0%) across all different sizes, where the number of parameters in the cor- responding MHA ranges from 19.87M to 75.50M. Meanwhile, we also observe that making the model deeper by stacking more encoder and decoder lay- ers results in steady increment on PRR for MHE- MUL (e.g. 93.6%, 95.0% and 96.0% respectively, for 8 layers, 12 layers and 16 layers in total). More- over, for the same number of parameters in the attention, wider attention heads consistently leads to better PRR for MHE-MUL. i.e. 91.5%, 95.0% and 95.3% for the dimensionality of 32, 64 and 128 of attention heads respectively.
To further illustrate the memory-efficiency of our MHE models compared to the baselines, we take the BERT-base architccture (12 attention heads, each with dimension of 64) as example, and measure the memory usage per attention block as in Section 2.1.1 from Smith al. (2022) and re- port the memory usage saving ratio (%) during the attention calculation in Table 13:
The calculation is based on inputs with batch size of 32, hidden dimension of 768, sequence length of 512 and fp16 mixture precision training using the following formula:
Memoryowegghhss-staaraan+++++)) bytes;
Menooyggaaiinntsstpprrmmm+C+ bytes;
Memory(Adam states) (ttpspdmm bytes;
These results indicate MHE consistently achieves good performance retention ratios and is robust to model size change.
Memory(activatiooss)> baich-size**sseuueeee dendddhhiiddennddmenninnn""" bytes.
From Table 13, we observe the memory usage saving ratio of our proposed MHE is 2.75 times better than SKV, 1.50 times better than MQA and
"BSSE: 12 encoder/dccoder layers, each containing 12 attention heads; LARGE/MEDIUM 24 encoder/decnder layers, each containing 16 attention heads.

---

ATTENTION
MNLI
QNLI
QQP
RTE
SST
MRPC
CoLA
STS
GLUE Avg.
SHA
80.5(0.3)
87.5(0.2)
86.7(0.1)
63.6(0.9)
90.7(0.3)
85.1(0.7)
53.8(1.1)
85.8(0.4)
79.2(0.1)
MHA
83.4(0.1)
89.8(0.3)
87.8(0.1)
67.6(1.5)
92.0(0.3)
86.8(0.4)
59.6(1.3)
88.5(0.3)
81.9(0.3)
EL-ATT
81.7(0.1)
88.4(0.2)
87.3(0.2)
67.6(1.0)
91.7(006)
85.9(0.7)
52.4(1.7)
87.7(0.2)
80.3(0.3)
MQA
82.6(0.1)
88.8(0.2)
87.3(0.1)
66.5(0.9)
91.4(0.5)
87.3(0.2)
58.4(1.3)
87.9(0.2)
81.3(0.2)
SKV
82.6(0.1)
89.4(0.3)
87.7(0.1)
68.2(1.7)
91.6(0.3)
87.4(0.6)
56.2(1.2)
88.6(0.2)
81.4(0.2)
FNET
76.3(0.1)
83.8(0.1)
84.((0.))
63.2(2.0)
88.(0.7)
78.00.44
43.2(2.5)
83.7(0.3)
75.2(0..)
LINEAR
75.4(0.1)
81,((0.))
85,5(0.2)
$4.7(2.3)
90.4(0.4)
72.2(0.))
50.3(1.0)
70.9(0.5)
72.6(1.))
MHE-ADI
81.5(02))
87.8().2)
87.2(0.1)
66.9(2.0)
90.5(0.4)
87.2(0.3)
54.7(0.9)
87.7(0.))
80.4(002)
MHE-MUL
81.1(0.))
87.9(0.1)
87.4(0.1)
67.1(1.5)
91.((0.5)
85.4(0.5)
56.6(1.7)
87.3(0.2)
80.6(0.2)
МНА(M)
84.4(0.2)
91.1(0.4)
84.0(0.6)
70.5(1.0)
92.0(0.2)
87.2(0.8)
62.5(1.0)
88.8(0.2)
82.6(0.4)
MHE-MOT, (M)
82.7(0.2)
89.2(0.4)
87.2(0.2)
67.9(0.4)
90.7(0.3)
86.3(1.0)
59.8(1.)
88.0(0.2)
81.5(0.3)
Table Results for encoder-only models on GLUE dev sets with standard deviations over five runs parentheses. Bold values denote best performing method in each task.
ATTENTION
BoolQ
CB
RTE
WiC
MultiRC
COPA
WSC
SUPERGLUE Avg.
SHA
72.3(0.7)
88.7 88.7(2.5)
62.5(1.0)
63.6(0.5)
59.7(15.7)
59.2(2.8)
63.8(1.5)
67.1(2.4)
MHA
76.6(0.6)
89.4(1.8)
67.9(1.3)
65.4(0.8)
69.0(1.2)
64.0(3.1)
61.5(3.3)
70.5(0.5)
EL-ATT
73.5(1.0)
85.7(4.8)
69.5(1.4)
63.8(0.8)
67.9(0.3)
62.2(118)
63.8(0.5)
69.5(1.0)
MQA
74.6(0.6)
86.7(1.6)
65.4(0.8)
64.0(1.2)
68.8(0.5)
62.2(2.0)
63.3(2.7)
69.3(0.7)
SKV
75.2(0.3)
84.5(3.7)
67.5(0.9)
65.2(1.1)
68.7(0.2)
64.0(1.0)
64.4(1.2)
69.9(0.4)
FNET
68.4(0.5)
51.8(4.3)
60.7(0.9)
63.((.1))
62.3(0.6)
58.2(3.7)
60.0(1.6)
60.7(0.6)
LINEAR
70.4(0.2)
50.6(2.1)
55.2(1.8)
62.9(0.7)
57.8(0.5)
60.0(2.8)
61.0(1.1)
59.7(0.9)
MHE-ADD
73.3(0.2)
88.8(1.7)
67.5(1.5)
64.2(0.5)
67.1(0.2)
60.2(2.8)
62.3(1.4)
69.1(0.5)
MHE-MUL
74.9(0.6)
89.4(1.0)
67.8(1.3)
64.7(0.6)
68.0(0.3)
61.6(1.5)
61.2(2.9)
69.6(0.3)
MHA(M)
70.3)
88. (6.1()
70.3(1.3)
67.8(0.8)
72.9(0.6)
68.2(4.1)
64.6(2.9)
72.9(0.6)
MHE-MUL (M)
75.2(0.5)
84.6(2.4)
68.6(1.8)
66.3(0.9)
69.8(0.4)
61.6(3.8
64.6(0.8)
70.1(1.1)
Table 6: Results for encoder-only models on SUPERGLUE dev with standard deviations over five runs in parentheses. Bold values denote best performing method in each task.
ATTEN
GLUE
-TION
QNLI
QQP
RTE
MRPC
CoLA
STS
SHA
80.5
87.5-
86.7-
63.6-
90.7-
85.1-
53.8
85.8-
MHA
83.4 (0.02)
89.8 (0.01)
87.8 (0.01)
67.6 (0.03)
92.0 (0.01)
86.8 (0.01)
59.6 (0.05)
88.5 (0.01)
EL-ATT
81.7 (0.02)
(0.02)
(0.01)
(0.10)
91.7 (0.02)
85.9 (0.02)
52.4 (-0.04)
87.7 (0.04)
MQA
82.6 (0.04)
88.8 (0.02)
87.3 (0.01)
66.5 (0.06)
91.4 (0.01)
87.3 (0.04)
58.4 (0.12)
87.9 (0.03)
82.6 (0.02)
89.4 (0.02)
87.7 (0.01)
68.2 (0.05)
91.6 (0.01)
87.4 (0.02)
56.2 (0.03)
88.6 (0.02)
FNET
76.3
83.8
(-)
84,8
(
63.2
())
88.4
())
78.0
())
43.2
83.7
(
LINEAR
75.4
(-)
81.4
(-)
(-)
54.7
(-)
90.4
(4)
72.2
(-) 50.3
(-)
70.9 (-)
MHE-ADD
81.5
(3.88)
87.8
(1.34)
87.2
(1.86)
(169()
90.5
(-0.81)
87.2
(7.93)
54.7 (5.22)
87.7 (7.05)
MHE-MUl.
81.9 (5.41) 87.9 (1.51)
87.4 (2.54)
67.1 (17.80)
91.1
(1.29)
85.4 (1.29) 56.6
87.3 (5.60)
Table 7: Detailed average scores and performance elasticity of parameters (in parentheses) on GLUB for MHE models and the baselines with encoder-only architecture using MLM pre-training objectives. Underlined values denote the best performing method and bold values denote the method with best PEoP in each task.

---

ATTEN
SuperGlue
-TION
BoolQ
CB
RTE
WIC
MultiRC
COPA
WSC
SHA
72.3-
88.7
62.5
63.6-
59.7-
59.2-
63.8
MHA
76.6 (0.03)
89.4 (0.00)
67.9 (0.04)
65.4 (0.01)
69.0 (0.07)
64.0 (0.04)
61.5
EL-ATT
73.5 (0.03)
85.7 (-0.06)
69.5 (0.19)
63.8 (0.00)
67.9 (0.23)
62.2 (0.08)
63.8 (0.00)
MQA
74.6 (0.04)
86.7 (-0.03)
65,4 (0.06)
64.0 (0.01)
68.8 (0.21)
62.2 (0.07)
(-.00))
SKV
75.2 (0.03)
84.5 (-0.03)
67.5 (0.06)
65.2 (0.02)
68.7 (0.11)
64.0 (0.06)
64.4 (0.01)
FNET
68.4
(-)
51.8
(-)
60.7
(-)
63.8
(-)
62.3
(-)
58.2
(-) 60.0
())
LINEAR
70.4
(-)
50.6
(-) 55.2
(-)
62.9
(
57.8
(
60.0
( 61.0 (-)
MHE-ADD
73.3 (4.58)
88.8
(0.54)
625.525.50
64.2
(3.00)
67.1 (39.90)
60.2 (5.41)
62.5 (-6.75)
MHE-MUL
74.9 (11.78)
89.4
(2.52)
67.8 (26.97)
64.7
(5.36)
68.0 (44.63) 61.6
61.2(-13.49)
Table 8: Detailed average
and performance elasticity of parameters (in parentheses) SUPERGLUE for MHE models and the baselines with encoder-only architecture using MLM as pre-training objectives. Underlined values denote the best performing method and bold values denote the method with best PEoP in each task.
ATTENTION
MNL.
QNLI
QQP
RTF
SST
MRPC
CoLAA
STS
GLUE Avg.
SHA
78.7(0.1)
86.0(0.2)
85.0(0.1)
66.5(0.9)
89.8(0.2)
76.8(0.4)
38.0(1.3)
81.5(0.4)
75.3(0.3)
MHA
80.6(0.1)
87.9(0.2)
86.3(0.1)
66.9(1.1)
90.2(0.3)
79.0(0.7)
42.9(1.3)
86.0(0.2)
77.5(0.2)
EL-ATT
79.5(0.2)
86.8(0.3)
85.7(0.1)
65.7(1.4)
90.0(0.4)
79.2(1.4)
41.5(2.2)
84.3(0.2)
76.6(0.4)
MQA
80.0(0.1)
86.3(0.1)
85.9(0.1)
66.2(0.7)
90.3(0.3)
80.7(0.6)
41.3(0.8)
84.3(0.4)
76.9(0.2)
SKV
80.3(0.1)
87.5(0.3)
85.9(0.1)
(6.1(L.))
90.6(0.5)
79.6(0.5)
41.9((.8)
84.9(0.2)
77.1(0.4)
MHE-ADD
1)
85.6(0.2)
85.4(0.1)
66.4(2.5)
89.6(0.4)
78.7(0.6)
38.((12)
83.5(0.3)
75.8(0.3)
MHE-MUL
79.0(0.2)
85.5(0.1)
85.6(0.1)
70.2(2.5)
90.9(0.2)
78.9(0.8)
39.4(1.3
84.0(0.3)
76.7(0.2)
Table 9: Results for decoder-only models on GLUE dev sets with standard deviations over five runs in parentheses. Bold values denote best performing method in each task.
ATTEN
GLUE
-TION
MNLI
QNLI
QQP
RTE
SST
MRPC
CoLA
STS
SHA
78.7-
86.0-
85.0-
66.5
89.8-
76.8
38.0-
81.5
MHА
80.6 (0.01)
87.9 (0.01)
86.3
(0.01)
66.9 (0.00)
90.2 (0.00)
79.0 (0.01)
42.9 (0.06)
86.0 (0.03)
EL-ATT
79.5 (0.02)
86.8 (0.02)
85.7
(0.01)
65.7 (-0.02)
90.0 (0.01)
79.2 (0.05)
41.5 (0.16)
84.3 (0.06)
MQA
80.0 (0.02)
86.3 (0.01)
85.9 (0.01)
66.2 (-0.01)
90.3 (0.01)
80.7 (0.07)
41.3 (0.12)
84.3 (0.05)
SKV
80.3 (0.01)
87.5 (0.01)
85.9 (0.01)
66.1 (-0.00)
90.6 (0.01)
79.6 (0.03)
41.9 (0.07)
84.9 (0.03)
MHE-ADO
78.7 (-0.02)
85.6 (-T.50)
85.4
([.60)
66.7 (-0.69)
89.6 (-0.57)
78.7 (7.96)
38.4 (3.75
83.5 (7.98)
MHE-MUL
79.0 (0.98)
85.5 (-1.88)
85.6 (2.05)
(17.72)
(4.01)
78.9
(8.58)
(12.32
84.0 (9.97)
Table 10: Detailed average scores and performance elasticity of parameters (in parentheses) GLUE for MHE
models and the baselines with decoder-only architecture using MLM as pre-training objectives. Underlined values denote the best performing method and bold values denote the method with best PEoP in each task.

---

Hyperparameter
Pretraining
Maximum train steps
1000000 steps
Batch size (per GPU)
32 instances
Adam
Ie-8
Adam &1
0.9
Adam 322
0.9999
Sequence length
512
Peak leamiing rate
Ic-4 for ML.M
Learning rate schedule
linear
Warmup steps
10000
Weight decay
0.01
Attention Dropout
0.1
Dropout
0.1
Table Details of hyperparameters used in pre- training.
Hyporparamcter
Fine-taning
Maximum trin opechs size (per GPU)
20 epochs for GLUE, SUPERGLUE and SQUAD
Adam
le-6
Adam 5:
0.9
Adam
0.999
SQUAD;
Posk loamiing ra8e
Se-5 for SUPERGLUE
Learning schedule
cosine with hard
steps
Warmup steps
5327 for SQUAD v1.1:
4950 for SQUAD v2.0
Weight decay
Attention Dropout
0.1
Derpont
0.1
Evalustion steps
2455
2275 for QQP: RTE.
421 for SST, 69 for MRPC.
CaL
for BoalQ. for
47 for RTE, 102 for WiC,
512 for MultiRC, for COPA,
11 for WSC,
518 for SQUAD v1.1.
815 for SQUAD v2.0
Table 12: Details of hyperparameters used in fine- tuning.

---

ATTENTION
weights
gradients
Adam states
activations
Total
Memory Saving Ratios (%)
SHA
4423680
4423680
5898240
25165824
39911424
44.84
MHA
14155776
14155776
18874368
25165824
72351744
0.00
EL.ATT
7077888
7077888
9437184
25165824
48758784
32.61
MQA
7667712
7667712
10223616
25165824
50724864
29.89
SKV
10616832
10616832
14155776
25165824
60555264
16.30
MHE-ADD
4437504
4437504
5916672
25165824
39957504
44.77
MHE-MUP
4437504
4437504
5916672
25165824
39957504
44.77
Table 13: Memory usage (in bytes) and memory saving ratios (compared to MHA) per attention block for our MHE and other baselines. MHA denotes BERT-hase here.
#Params(M) M)
GLUE
SUPERGLUE
WIKITEXT- 103
PENN TREEBANK
MHE
MHE
MHE
MHE -MUL
MHE
MHA
MHA
-MUL
-MLL
-MUL
-MUL
Encoder
28.32
8.88
81.9
80.6
98.4
70.5
69.6
-only
Decoder
BASE
28.32
8.88
77.5
76.7
99.0
43.0
53.8
74.9
44.3
50.7
85.6
-only
MEDIUM
100.66
29.96
79.4
77.7
97.9
35.5
37.2
95.2
37.5
41.6
88.5
Table 14: Results of evaluation metrics on two text classification benchmarks (GLUE, SUPERGLUUE) and two language modelling benchmarks (WIKITEXT- 103 and PENN TREEBANK) with performance retention ratio (PRR) for MHA and MHE-MUL across different model sizes.
N
dm
dy
Pdrop
#Steps
#Params(M)
BLEU
PRR
MHA
MHE-MUL
MHA
BASE
512
8
100K
18.87
6.52
24.8
23.6
95.0
12
512
16
32
0.1
100K
18.87
5.63
25.1
22.9
91.5
12
512
4
128
0.1
100K
18.87
8.29
24.7
23.6
95.3
4L
512
g
0.1
1ook
12.58
4.34
23.9
22.4
93.6
8I.
16
512
8
64
0.1
100K
25.17
8.69
25.3
24.3
96.0
12H
12
768
12
64
0.15
100K
42.47
13.31
25.7
24.2
94.2
BIG
12
1024
0.3
300K
75.50
22.47
26.5
24.8
93.6
Table 15; Results of BLEU scores WMT- 14 English to German machine translation task with performance retention ratio (PRR) for MHA and MHE-MUL across different model sizes.


Reviews:
[{"review_id": "kzSGzeT9NF", "reviewer": "Reviewer_oMfC", "summary": "", "questions": "A. How is the performance changing when scaling up the number of parameters?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your insightful feedback.\n\n**Additional Results towards Scaling**: It's important to mention that our experimentation was conducted under the constraints of limited computational resources, which naturally hindered our ability to pre-train models on a very large scale. Despite these inherent limitations, we have performed supplementary experiments that involve comparing the vanilla Multi-Head Attention and our best performing MHE-Mul approach on different scales by adjusting the number of attention heads. \n\nSpecifically, we experimented with 12 and 16 heads per encoder and decoder layer (in the paper we only report results using 8 heads) on the WMT14 machine translation task. We observed comparable performance retention ratios for MHE-Mul when scaling to 12 and 16 heads of 94.2% and 93.6% respectively (similar to the results obtained with 8 heads per layer reported in the paper). Moreover, we extended our experiments by varying the numbers of layers in the encoder-decoder model, i.e. 8 and 16 layers in total (4 and 8 layers for both the encoder and the decoder respectively) from 12 that we have reported in the paper. Similarly, the performance retention ratios are 93.6% for 8 layers and 96.0% for 16 layers with MHE-Mul. \n\nWe have also pre-trained a BERT-large model using MHE-Mul (16 heads and 24 layers). MHE-Mul yielded a SuperGlue Avg. score of 70.1, in comparison to the 72.9 achieved by using MHA in BERT-large, resulting in a performance retention ratio of 96.2%. \n\nCollectively, these additional results demonstrate the high and robust performance retention ratio of MHE-Mul across different model sizes by varying the number of attention heads and layers. We hope that these additional experiments and results address your concerns regarding the generalizability of our parameter efficient attention given the constraints in computing resources. We will include all these extra results in the camera ready version given the extra space.\n\n|                | #params(att) | SuperGlue Avg. | PRR   |\n| -------------- | ------------:| --------------:| -----:|\n| MHA(base)      | 28.32M       | 70.5           | 100.0 |\n| MHE-Mul(base)  | 8.88M        | 69.6           | 98.7  |\n| MHA(large)     | 100.66M      | 72.9           | 100.0 |\n| MHE-Mul(large) | 29.96M       | 70.1           | 96.2  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The approach is simple and intuitive, and improves memory efficiency in Transformer attention.\n2. Comparing to models with similar number of params, their approach gives better performance.", "reasons_to_reject": "1. The paper claims good memory efficiency when scaling up, but does not show how performance is affected when scaling up. So it is unclear that its performance will not degrade too much when scaling up.", "questions_to_authors": "A. How is the performance changing when scaling up the number of parameters?", "paper_topic_and_main_contributions": "This paper proposes to use one matrix for all query, key, value projections for all layers in Transformer and add/multiply with head embeddings. They show that their model achieves better performance than single headed attention (which has similar number of parameters) and is comparable to other memory saving approaches while saving more memory. They show that their approach does not increase a lot of memory when scaling up the number of attention parameters. They also show that their model has better parameter utilization than previous works.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zUna71ubQY", "reviewer": "Reviewer_ymVs", "summary": "", "questions": "In my understanding, the proposed method can be combined with the previous approaches such as MQA and SKV to increase the expressiveness of these methods. Did the authors conduct such experiments?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your constructive feedback.\n\n**Balance between Predictive Performance and Memory**: We greatly appreciate the reviewer ymVs's attention to the pursuit of high predictive performance. It's our paramount goal to balance the model parameter efficiency with its predictive capabilities. While it's true that preceding parameter-sharing techniques such as SKV and MQA displayed marginally better outcomes, it's noteworthy that these methods demand an exponential increase in parameters relative to the number of heads employed. This equilibrium between parameter efficiency and performance is encapsulated by what we term \"performance elasticity of parameters\" (PEoP) which indicates  how effectively the model parameters contribute to predictive performance. By measuring PEoP, our proposed method achieves state-of-the-art performance, indicating that it obtains substantially higher predictive performance per parameter (i.e. more parameter efficient). It's worth highlighting that our attention models can be further integrated with SKV and MQA, and we plan to explore this in future work. Our methods can be used under limited memory budgets, achieving high performance retention ratios across a diverse set of tasks. \n\n**Main Contributions**: We would like to emphasise that the novelty of our work lies in leveraging multiple head-embeddings with a single shared projection matrix to support multiple attention heads. Our method (MHE) only requires a very small fraction of additional parameters compared to a single-head attention mechanism but achieves comparable predictive performance on a variety of downstream tasks to full multi-head attention (MHA).\n\n**Why not combine MHE with other efficient attention methods?**: Applying MHE with other parameter-efficient attention methods (e.g. MHQ/SKV, parameter-efficient embeddings etc.) is left for future work. The main focus of our paper is to demonstrate the parameter efficiency and robustness in performance retention ratio of vanilla MHE compared to vanilla MHA.\n\n**Are the parameters of multi-head attention dominant in the parameters of the Transformer?**: The parameters of MHA occupy 1/3 of the whole parameters in the encoder and 1/2 in the decoder Transformer layers respectively. On the other hand, the Feed-forward sublayers (FFNs) occupy 2/3 in the encoder and 1/2 in the decoder layers respectively. For example, in a  BERT-base model, our approach could substantially save approximately 18% of the whole model parameters (given that the parameters in the embedding matrix account for the 20%). We admit that the parameter size of the FFN in the Transformer is also significant. Reducing the FFN size does not conflict with our parameter-efficient MHE attention. We opted to use the conventional default settings (intermediate_size = 4 * hidden_size) for all models tested for simplicity and also keeping the number of experiments manageable given our limited compute. Our intuition is that reducing the size of the FFN would reduce model performance but it is something that could be tested in future work. \n\n**Parameter-efficiency ~= Memory-efficiency**: Our understanding is that parameter-efficiency mirrors memory efficiency. For example, when leveraging mixed-precision training with float16 and the Adam optimizer, each parameter needs 20 bytes of memory: 2 bytes for the weights, 4 bytes for the gradients, and an additional 4 bytes for Adam states, summing up to 20 bytes per parameter. Notably, this alignment between parameter-efficiency and memory consumption has been highlighted in Section 2.1.1 of the work titled \"Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model\" (Smith et al., 2022).\n\n**BLEU Calculation Details**: We deeply appreciate the reviewer's valid concerns regarding the computation of BLEU scores which we have updated below. We noticed an issue was present within the GitHub repository we initially referenced, specifically in the file /microsoft/DirectML/blob/master/PyTorch/1.13/attention_is_all_you_need/transformer/Models.py#173/blob/master/PyTorch/1.13/attention_is_all_you_need/transformer/Models.py#173. Subsequently, we conducted a comprehensive reevaluation of our experiments using an alternative repository, namely /sgrvinod/a-PyTorch-Tutorial-to-Machine-Translation, employing the default hyperparameters settings. We employ the official WMT-calculated BLEU score (sacreBLEU signature: BLEU+case.mixed+lang.en-de+numrefs.1+smooth.exp+test.wmt14/full+tok.13a+version.1.4.3) as our benchmark for evaluation. Our findings were subsequently updated and are presented in the revised table. The BLEU score reported in the original paper stands at 27. We acknowledge that this figure might not align with calculations performed using different methods, as evidenced by discussions in Issues comments 317#issuecomment-377580270 and 317#issuecomment-380970191 within the official tensor2tensor repository by TensorFlow. Throughout our experimentation, we have maintained uniformity in hyperparameter settings across all models, thereby illustrating their relative BLEU score differences. Please note that the different calculation of BLEU did not change the relative differences in performance across methods.\n\n|         | #params(att) | Avg. | PRR   |\n| ------- | ------------:| ----:| -----:|\n| SHA     | 6.49M        | 22.5 | 90.8  |\n| MHA     | 18.87M       | 24.8 | 100.0 |\n| EL-Att  | 9.44M        | 23.9 | 96.6  |\n| MQA     | 10.62M       | 24.2 | 97.6  |\n| SKV     | 14.16M       | 24.7 | 99.5  |\n| MHE-Add | 6.52M        | 23.0 | 92.9  |\n| MHE-Mul | 6.52M        | 23.6 | 95.0  |\n\nWe will update Table 2 with the new results in the camera ready version upon acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method is simple but effective. It consistently achieved better performance than the single-head attention.\n\nIn my understanding, the proposed method can be combined with previous parameter-efficient method. As in the question, the authors did not investigate the effectiveness but the proposed method might improve the performance.", "reasons_to_reject": "The proposed method achieved better performance the single-head attention, but I cannot agree that the proposed method is more useful than the existing parameter-efficient attention methods such as multi-query attention. In fact, MQA and SKV consistently achieved better performance than the proposed method.\nIn addition, I wonder about the baseline performance in the machine translation experiments. In WMT English-to-German 2014 dataset, we can achieve 27 BLEU score with the standard (i.e., MHA) Transformer. If the authors did not use SacreBLEU to compute the BLEU score, the standard Transformer achieve around 28 BLEU score as in [Takase and Kiyono, 2023]. In short, I suspect that the baseline is weak in comparison to the widely used configuration.\n\nIn my understanding, the proposed method is parameter-efficient but not memory-efficient because the space complexity of the proposed method is almost the same as the one of the multi-head attention during the attention calculation. I recommend the authors indicate the saving rate of the memory. In addition, I wonder if the parameters of multi-head attention are dominant in parameters of the Transformer. The authors show the parameter size on attention mechanisms in the experiments but I'm not sure the proposed method has a significant effect on the whole parameter size.\n\n\nTakase and Kiyono, 2023: Lessons on Parameter Sharing across Layers in Transformers.", "questions_to_authors": "In my understanding, the proposed method can be combined with the previous approaches such as MQA and SKV to increase the expressiveness of these methods. Did the authors conduct such experiments?", "paper_topic_and_main_contributions": "This paper proposes a parameter-efficient method for multi-head attention. The standard multi-head attention provides weight matrices corresponding to the number of attention heads. In contrast, the proposed method provides only one weight matrix for key, value, and query in the attention. To modify each key, value, and query, the proposed method uses weight vectors corresponding to the number of heads. In the modification, the authors provides two approaches, addition and multiplication. The proposed method outperformed the single-head attention method though the number of parameters is almost the same.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LNd7REfHue", "reviewer": "Reviewer_M1Z9", "summary": "", "questions": "1. It seems that MHE performs worse in translation task (92%~95% in PRR) than the other natural language understanding tasks tested (most of them are greater than 95%). Are there any more results in other natural language generation tasks, e.g., Text Summarization and Question-Answering tasks, to support the effectiveness of your method?\n\n2. Why not conduct additional experiments to further verify the effectiveness of MHE in parameter utilization? For example, under the same parameters used settings, the parameters saved in MHA sub-layer are re-invested to the other sub-layers, e.g., the following feed-forward network, to compare with the MHA baseline.\n\n3. What are the results of models in Table 1 on the corresponding test set?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to “Focusing the memory-efficient attention sub-layer, has less significance compared to the one in feed-forward network”**: Same as the response to the comment by Reviewer ymVs:  The parameters of MHA occupy 1/3 of the whole parameters in the encoder and 1/2 in the decoder Transformer layers respectively. On the other hand, the Feed-forward sublayers (FFNs) occupy 2/3 in the encoder and 1/2 in the decoder layers respectively. For example, in a  BERT-base model, our approach could substantially save approximately 18% of the whole model parameters (given that the parameters in the embedding matrix account for the 20%). We admit that the parameter size of the FFN in the Transformer is also significant. Reducing the FFN size does not conflict with our parameter-efficient MHE attention. We opted to use the conventional default settings (intermediate_size = 4 * hidden_size) for all models tested for simplicity and also keeping the number of experiments manageable given our limited compute. Our intuition is that reducing the size of the FFN would reduce model performance but it is something that could be tested in future work.\n\nWe also provide responses to each of your questions as follows:\n\n1. **Experiments on Decoder-only Architecture**: We extended experiments by pretraining GPT2 models, both in base and large sizes, using a 512-token context. The results of our evaluation on general text classification tasks are as follows:\n- Base-sized MHE-Mul (12 heads, 12 decoder layers) achieved a Performance Retention Ratio (PRR) of 99.0% on the GLUE Avg. score.\n\n- Medium-sized MHE-Mul (16 heads, 24 decoder layers) attained a PRR of 97.9%.\n\nWe will add this in the camera-ready upon acceptance given the extra space.\n\nWe also report model perplexities:\n\n|            |         | perplexity   |               |\n| ---------- | -------:| ------------:| -------------:|\n| Model      | #params | WikiText-103 | Penn Treebank |\n| SHA        | 8.85M   | 62.0         | 68.1          |\n| MHA        | 28.32M  | 43.0         | 44.3          |\n| EL-Att     | 14.16M  | 57.1         | 56.1          |\n| MQA        | 15.34M  | 49.7         | 49.3          |\n| SKV        | 21.23M  | 46.2         | 45.5          |\n| MHE-Add    | 8.88M   | 54.0         | 55.3          |\n| MHE-Mul    | 8.88M   | 53.8         | 50.7          |\n| MHA(M)     | 100.66M | 35.5         | 37.5          |\n| MHE-Mul(M) | 29.96M  | 37.2         | 41.6          |\n\nAlthough we have not conducted experiments on text summarization tasks due to the resource-intensive nature of pre-training with longer contexts, we think that this can be addressed in future work. We believe that our new experiments together with those included in the original submission demonstrate that our proposed MHE exhibits great parameter efficiency (PEoP scores)  and consistently achieves good performance retention ratios.\n\n2. **Scope of the Study and Feed-forward Layers**: The scope of our present study centres on enhancing the parameter-efficiency of the multi-head attention mechanism by introducing attention-head embeddings. While we acknowledge the potential for parameter savings within feed-forward layers, our focus remained on the design and efficacy of the attention mechanism. The interplay between our approach and parameter-efficient feed-forward sublayers falls within the scope of future research, as we anticipate that these strategies can seamlessly complement each other. See our response above.\n\n3. **Ground Truth Labels and Benchmark Protocols**: We operated under the same protocol as established by previous work such as the experimental design of RoBERTa and DistilBERT papers. Given that the ground truth labels in the test datasets are inaccessible, we adhered to these established protocols to ensure consistency and comparability with prior research.\n\nWe trust that our responses provide comprehensive insights into our methodology, scope, and results in the context of the questions raised.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method, Multiple Head Embedding module (MHE), reduces one order of memory needed in the attention sub-layer of Transformers, while preserving more than 90% performance of vanilla multi-head attention module in the downstream tasks tested.\n- According to the results in this paper, it is interesting to be investigated further under the large language model settings.", "reasons_to_reject": "- Experiments of decoder-only models and many natural language generation tasks are missing while this paper aims to mitigate the computational problems in the scaling language models, which are mainly decoder only models (e.g., GPT-3, LLaMA, and GLM).\n\nMinor:\n- Test set results are more compelling and able to show the actual performance of models than the dev set results in Table 1.\n\n- In transformer layer, most of parameters are contained in the feed-forward networks (≈2/3) rather than multi-head attention sub-layers (≈1/3). Thus the method in this paper, focusing the memory-efficient attention sub-layer, has less significance compared to the one in feed-forward network.", "questions_to_authors": "1. It seems that MHE performs worse in translation task (92%~95% in PRR) than the other natural language understanding tasks tested (most of them are greater than 95%). Are there any more results in other natural language generation tasks, e.g., Text Summarization and Question-Answering tasks, to support the effectiveness of your method?\n\n2. Why not conduct additional experiments to further verify the effectiveness of MHE in parameter utilization? For example, under the same parameters used settings, the parameters saved in MHA sub-layer are re-invested to the other sub-layers, e.g., the following feed-forward network, to compare with the MHA baseline.\n\n3. What are the results of models in Table 1 on the corresponding test set?", "paper_topic_and_main_contributions": "This paper proposes a memory efficient method to build the attention module in Transformers, which shows a high performance retention in many natural language understanding tasks and one translation task compared to the vanilla multi-head attention module of encoder only and encoder-decoder models. The contributions of this paper lie in the approaches for compute efficiency and NLP engineering experiments.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 21 in the abstract: (3n^2-3n)d^2 --> (3n^2-3n)d^2-3nd (-3nd is missing)"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to save space in transformers by using a single matrix for all query, key, value projections for all layers. They compare their approach to various baselines, showing memory savings. Reviewers found the approach simple and intuitive, and liked that it can be combined with other methods. They also generally found the results strong, but  complained about missing details in the reporting, and about the quality of the baselines. The rebuttal helped alleviate some of these concerns, though the baseline concern still remains.
Decision: Accept-Findings