Multi-Head Attention: Collaborate Instead of Concatenate
Authors: Jean-Baptiste Cordonnier
Andreas Loukas
Martin Jaggi
Venue:  (2021)
Keywords: attention
self-attention
bert
multi-head
tensor factorization

Abstract: Attention layers are widely used in natural language processing (NLP) and are beginning to influence computer vision architectures. However, they suffer from over-parameterization. For instance, it was shown that the majority of attention heads could be pruned without impacting accuracy. This work aims to enhance current understanding on how multiple heads interact. Motivated by the observation that trained attention heads share common key/query projections, we propose a collaborative multi-head attention layer that enables heads to learn shared projections. Our scheme decreases the number of parameters in an attention layer and can be used as a drop-in replacement in any transformer architecture.For instance, by allowing heads to collaborate on a neural machine translation task, we can reduce the key dimension by 4× without any loss in performance. We also show that it is possible to re-parametrize a pre-trained multi-head attention layer into our collaborative attention layer. Even without retraining, collaborative multi-head attention manages to reduce the size of the key and query projections by half without sacrificing accuracy. Our code is public.

Under review as a conference paper at ICLR 2021
MULTI-HEAD ATTENTION: COLLABORATE INSTEAD OF CONCATENATE
Anonymous authors Paper under double-blind review
ABSTRACT
Attention layers are widely used in natural language processing (NLP) and are be- ginning influence computer vision architectures. Training very large transformer models allowed significan improvement in both fields, but once trained, these networks show symptoms wver-aarameterization many attention heads can be pruned without impacting accuracy. This work aims to enhance current understanding on how multiple heads interact. Motivated by the observation that trained attention heads share common key/query projections, we propose a collaborative multi-head attention layer that enables heads to learn shared projections. Our scheme decreases the number of parameters in attention layer and can be used as a drop-in replacement in any transformer architecture, For instance, by allowing heads to collaborate we can reduce the key dimension by 4x without any loss in performance. We also show that it is possible re-parametrize pre-trained multi-head attention layer into our collaborative attention layer. Even without retraining, collaborative multi-head attention manages reduce the size of the key and query projections by half without sacrificing accuracy. Our code is public "
For instance, it known that
neural machine translation task,
INTRODUCTION
Since the invention of attention (Bahdanau et al., (0144) and its popularization in the transformer architecture (Vaswani et al. 2017), multi-head attention (MHA) has become the de facto architecture for natural language understanding tasks (Devlin et al. 2019) and neural machine translation. Atten- tion mechanisms have also gained traction in computer vision following the work of Ramachandran ctt al. (2019) and Bello et al. (2019). Nevertheless, despite their wide adoption, we currently lack solid thcoretical understanding of how transformers operate. In fact, many of their modules and hyperparameters are derived from empirical evidences that are possibly circumstantial.
The uncertainty is amplified in multi-head attention, where both the roles and interactions between heads are still poorly understood. Empirically, is well known that using multiple heads can improve model accuracy. However, not all heads are equally informative, and it has been shown that certain heads can be pruned without impacting model performance. For instance, Voita et al. (2019) present method quantify head utility and prune redundant members. Michel al. (2019 go further to question the utility of multiple heads by testing the effect of heavy pruning in several settings. On the other hand, Cordonnier et al. (2020) prove that multiple heads are needed for self-attention to perform convolution, specifically requiring one head per pixel in the filter's 's receptive field. Beyond the number of heads, finding the adequate head dimension is also an open question. [Bhojanapalli] et al. (220)) finds that the division of the key/query projection between heads gives rise to low-rank bottleneck for each attention head expressivity that can be fixed by increasing the head sizes. In contrast, approach increases heads expressivity by leveraging the low-rankness accross heads to share common query/key dimensions.
This work aims to better detect and quantify head redundancy by asking whether independent heads learn overlapping or distinct concepts. This relates to the work on CNN compression that factorizes common filters trained convolutional network (Kim et al., 2016) using Tucker decomposition. In attention models, we discover that some key/query projected dimensions are redundant, as trained
https:: http//gtttttut coub.ccmm...

---

Under review as a conference paper at ICLR 2021
concatenated heads tend to compute their attention patterns common features. Our finding implies that MHA can be re-pen re-parametrized with better weight sharing for these common projections and a lower number of parameters. This differs from concurrent work (Shazeer et al. 2020 that orchestrate collaboration between heads on top of the dot product attention scores.
Contribution I: Introducing the collaborative multi-head aftention layer. Section describes collaborative attention layer that allows heads to learn shared key and query features. The proposed e--paammtrizzation significantly decreases the number of parameters of the attention layer without sacrificing performance. Our Neural Machine Translation experiments in Section 4 show that the number of FLOPS and parameters to compute the attention scores can be divided by without affecting the BLEU score on the WMT14 Sogllishto-German task.
Contribution 2: Re-parametrizing pre-trained models into collaborative form renders them more efficient. Pre-training large language models has been central to the latest NLP developments. But pre-training transformers from scratch remains daunting for its computational cost even when using more efficient training tasks such as (Clark et al. 22000). Interestingly, our changes to the MHA layers can applied post-hoc on pre-trained transformers, as drop-in replacement of classic attention layers. To achieve this, we compute the weights of the re-parametrized layer using canonical tensor decomposition of the query and key matrices in the original layer. Our experiments in Section 4 show that the key/query dimensions can be divided by 3 without any degradation in performance.
As a side contribution, we identify a discrepancy between the theory and some implementations of attention layers and show that by correctly modeling the biases of key and query layers, clearly differentiate between context and
attention.
MULTI-HEAD ATTENTION
We first review standard multi-head attention introduced by Vaswani et al. 2017
2.1 ATTENTION
Let X E R''xD, and Y Rm be two input matrices consisting of respectively T and T
tokens of Di, dimensions each. An attention layer maps each of the T query token from Di, to Dout dimensions as follows:
QK ddn
Attention Q, K, V) = softmax
V, with Q = XW,, K = YYW. V = =YW, (1)
The layer is parametrized by query matrix We RD.xDs key matrix WR RD.xDs and value matrix Wy E RR D Using attention on the same sequence (i.e. X Y) is known as self-attention and is the basic building block of the transformer architecture.
2.2 CONTENT VS. CONTEXT
Some re-implementations of the original transformer architecture use biases in the linear layers. This differs from the attention operator defined in eq. (1) where the biases by and by E R Ds are ommited. Key and query projections are computed as K XW + 17x bx and Q YW₂ + 17x bo- respectively, where Laxb is an all one matrix of dimension
b. The exact computation of the
(unscaled) attention scores can be decomposed as follows:
QK = (XWo + -rx1b (Y - (IxIb
(2)
XWO T + 11x bb W T + +XWobklIxT + l IbxTbabk
(3)
context
content
As the last two terms of eq. (3) have a constant contribution over all entries of the same row, they do not
contribute the computed attention probabilities (softmax is shift invariant and softmax(22 x + c) softmax(x), Ve). On the other hand, the first two terms have a clear meaning: X W Y
For instance: the BERT orignal implementation, its HuggingFace re-implementation and FairSeq encoder- decoder transformer.
2

---

Under review as a conference paper at ICLR 2021
1.0
1.0
0.9
captaural variance
ciphirel
0.5
W
0.5
W
Wx
Wo
0.0
0.0
16
32
61
61
128
256
512
768
PCA components (heads seperstely)
PCA components (heads conesteusted)
Figure 1: Cumulative captured variance of the key query matrices per head separately (left) and per layer with concatenated heads (right). Matrices are taken from a pre-trained BERT-base model with N₁ 12 heads of dimension dę 64. Bold lines show the means. Even though, by themselves, heads are not low rank (left,, the product of their concatenation W6 WK is low rank (right, in red). Hence, the heads are sharing common projections their column-space.
considers the relation between keys and query pairs, whereas 1xx bg W Y computes attention solely based on key content.
The above findings suggest that the bias by of the key layer can be always be disabled without any consequence. Moreover, the query biases bQ play additional role: they allow for attention scores that are content-buse based, rather than solely depending on key-query interactions. This could provide explanation for the recent success of the Rense-SYNTHESIZER (Tay et al. 22020), method that ignores context and computes attention scores solely as a function of individual tokens. That is, perhaps context always crucial for attention scores, and content can suffice.
2.3 MULTI-HEAD ATTENTION
Traditionally, the attention mechanism is replicated by concatenation to obtain multi-head attention defined for N₁, heads as:
MultiHead (x, Y)
concat
H (8) W
sE[N,]
( Attention X YW
(6) YW
(5)
K E Duxe and W E Daxar are learned for each head i [Nr] and the extra parameter matrix Wo NA Da.. projects the concatenation of the N₁ head outputs (each in (Ro to the output space RrD In the multi-head setting, we call dg the dimension of each head and D Nxd4 the total dimension of the query/key space.
where distinct parameter matrices W
IMPROVING THE MULTI-HEAD MECHANISM
Head concatenation is a simple and remarkably practical setup that gives empirical improvements. However, we show that another path could have been taken instead concatenation. As the multiple heads are inherently solving similar tasks, they can collaborate instead of being independent.
3.1 How MUCH DO HEADS HAVE IN COMMON?
We hypothesize that some heads might attend on similar features in the input space, for example computing high attention on the verb of sentence or extracting some dimensions of the positional encoding. To verify this hypothesis, does suffice look at the similarity between query (or key) matrices (W( )E(N)) of different heads. To illustrate this issue, consider the case where two heads computing the same key/query representations up unitary matrix R JRdx such that
W
( and w(() (2) =w()) R.

---

Under review as a conference paper at ICLR 2021
Even though the two heads computing identical attention scores, i.e. l w W they can have orthogonal column-spaces and the concatenation (W()) w
RR wk (I)
R PDPuxdd 2dx can be full rank.
To disregard artificial differences due common rotations or scaling of the kev/query spaces, study the similarity of the product We w E &R across heads. Figure shows the captured energy by the principal components of the key, query matrices and their product. can be seen on the left that single head key/query matrices W K" (i) are not low rank on average. However, as seen on the right, even if parameter matrices taken separately are not low rank, their concatenation is indeed low rank. This means that heads, though acting independently, learn focus on the same subspaces. The phenomenon is quite pronounced: one third of the dimensions suffices to capture almost all the energy of Wq WK which suggests that there is inefficiency in the way multi-hes attention currently operate.
(T)
3.2 COLLABORATIVE MULTI-HEAD ATTENTION
Following the observation that heads' key/query projections learn redundant projections, we propose to Iearn key/query projections for all heads at once and to let each head use re-weighting of these projections. Our collaborative head attention is defined as follows:
CollabHead (X, Y)= concat <E[NA]
H [
Wo
(6)
H Attention X W (m.), YWK. YW
The main difference with standard multi-head attention defined in eq. (5) is that we do not duplicate
the key and query matrices for each head. Instead, each head learns mixing vector m, E IR Px that defines a custom dot product over the D& projected dimensions of the shared matrices Wq and WK of dimension D x D This approach leads to:
adaptive head expressiveness, with heads being able to use more or fewer dimensions according to attention pattern complexity;
(ii) parameter efficient representation, learned projections are shared between heads, hence stored and learned only once.
is instructive to observe how standard multi-head attention (where heads are simply concatenated) can be seen as a special case of our collaborative framework (with D. = d2) The left of Figure displays the standard attention computed between v,, and ym input vectors with the mixing matrix
M : concat [m:] E Nox xc[Na]
laying the mixing vectors m. rows. In the concatenated MHA, the mixing vector m, for the i-th head is a vector with ones aligned with the dę dimensions allocated to the i-th head among the D& N2, total dimensions.
Some alternative collaborative schema can be seen on the right side of Figure 2 By learning the mixing vectors (m;) [E(N,) instead of fixing them to this "blocks-of-1" structure, we increase the expressive power of each head for negligible increase in the number of parameters. The size dę of each head, arbitrarily set to 64 in most implementations, is now adaptive and the heads can attend to a smaller or bigger subspace if needed.
3.3 HEAD COLLABORATION AS TENSOR DECOMPOSITION
As we show next, there is a simple way to convert any standard attention layer to collaborative attention without retraining. To this end, we must extract the common dimensions between query/key (1)
matrices (W Q w(i) E DRDx Je[N, across the different heads. This can be solved using the Tucker tensor decomposition [Tucker] 1966) of the 3rd-order tensor
Wok := stack W (x)
W
(+) K
I NA
(9)
<E[Na]
4

---

Under review as a conference paper at ICLR 2021
W
W (2)
W
(3)
Other possible mixing matrices M
x
(a)
11111
Zn
Q
Na
(b)
1.1
larger lead
M
D
attention scotes
Um
K
(c)
x
W (1)
W
(d)
De
De Ng
Figure 2: Left: computation of the attention scores between tokens and using standard concatenated multi-head attention with N₂, independent heads. The block structure of the mixing matrix M enforces that each head dot products non overlapping dimensions. Right: we propose to use more general mixing matrices M than (a) heads concatenation, such as (b) allowing heads to have different sizes; (c) sharing heads projections by learning the full matrix; (d) compressing the number of projections from D3 to Dr heads can share redundant projections.
Following the notation of Kolda & Bader 2009). the Tucker decomposition of tensor E R IXJXK written
P
Q
R
B B
Ppgr ap o bb o =: [G: A, A, C.
(10)
with A E ETX., E JJQ and C' E Kx being factor matrices, whereas G E PxQxR R the core tensor. Intuitively, the core entry Oper G.a.r quantifies the level of interaction between the components ap, by. and C,.
In the of attention, suffices consider the dot product of the aligned key/query components of the Q and K matrices, which means that the core tensor is super-diagonal (i.c. Jper # 0 only q r). We further simplify the Tucker decomposition by setting the factors dimensions P, Q and R Dg, a single interpretable hyperparameter equal the dimension of the shared key/query space that controls the amount of compression of the decomposition into collaborative heads. These changes lead
special case of Tucker decomposition called the canonical decomposition, also known CP or PARAFAC (Harshman) 1970) in the literature (Kolda & Bader 2009). Fix any
positive rank The decomposition yields:
R
T =
Σ a, bb |A
(11)
with A E R.RRR B BERR and C KRRR
What is remarkable is that the above can be used to express any (trained) attention layer parametrized by (w( ()) (
(i) b [eE[N,) as collaborative layer. In particular, if we apply the decomposition to the stacked heads Wokk we obtain the three matrices [M, We, Wx] that define a collaborative
attention layer: the mixing matrix M E N. well the key and query projection matrices Wo. WK D.
represents the vector outer product
5

---

Under review as a conference paper at ICLR 2021
On the other hand, biases can be easily dealt with based on the content/context decomposition of eq. (3). by storing for each head the vector
8:
R.P...
(12)
With this in place, the computation of the (unscaled) attention score for the i-th head is given by:
@ ( + 1Txiba) (Y W + +1TxIbk)
T
X diag(m.) Y 1TXIU; Y (13)
where m, is the i-th row of M. If Die 2 Dj: the decomposition is exact (eq. (11) is equality) and our collaborative heads layer can express any concatenation- note that the proposed re- ce-parametrization can be applied to the attention layers of many transformer
attention layer. We also
architectures, such as the ones proposed by Devlin al. (2019); [anh et al. (2019) Lan et al. 2020).
3.4 PARAMETER AND COMPUTATION EFFICIENCY
Table Comparison of layer of con- catenate ys. collaborative MHA with chosen Dy to give negligible perfor- mance difference. T=128.
Collaborative MHA introduces weight sharing across the key/query projections and decreases the number of parame- ters and FLOPS. While the size of the heads the standard attention layer is set to dę 64 and the key/query layers project into a space of dimension D = N,dg, the shared keylquery dimension D2 of collaborative MHA can be set freely. According to our experiments in Section (summa- rized in Table []. a good rule of thumb when transforming a trained MHA layer to collaborative is to set Dr to half or one third of Dg. When training from scratch, Dy can even be set to 1/4-th of D
train FairSoq 44.1
re-param.
HuggingFace 442
concat. collab.
concat.
collab.
De +
512
128
768 + 256
Purams (x10%)
1.05
0.66
2.36
1.58
PLOPS (x 10%)
1.51
1.09
3,27
2.65
inference (mss)
0.99
0.81
1.71
1.65
Parameters. Collaborative heads use (2D + N,) parameters, as compared to Dn in the standard case (ignoring biases). Hence, the compression ratio is Rs D&/ Dr. controlled by the shared key dimension Dj. The collaborative factorization introduces a new matrix M of dimension No. X DR. Nevertheless, the number of heads is small compared to the hidden dimension (in BERT-base N₂, = 12 whereas Die = 768), the extra parameter matrix yields negligible increase as compared to the size of the query/key/values matrices of dimension Di DE-
Computational cost. Our layer decomposes two matrices into three, of modulable dimensions. To compute the attention scores between T tokens for all the N₁, heads, collaborative MHA requires 2T(D., + +N,) T D2 FLOPS, while the concatenation-eet FLOPS. Assuming that Din >> N₁ O(1) (as is common in most implementations), we obtain theoretical speedup of O Dx D₂)) However in practice, having matrix multiplications instead of a larger one makes our implementation slightly slower, if larger multiplications are supported by the hardware.
MHA uses 27 Dr, Dy + +7 Dx
EXPERIMENTS
The goal of our experimental section is two-fold. First, show that concatenation-basel MHA is a drop-in replacement for collaborative MHA in transformer architectures. We obtain significant reduction in the number of parameters and number of FLOPS without sacrificing performance on Neural Machine Translation (NMT) task with an encoder-decoder transformer. Secondly, we verify that our tensor decomposition allows one to reparametrize pre-trained transformers, such as BERT (Devlin et al. 2019) and its variants. To this end, show that collaborative MHA performs par with its concatenation based counter-part on the GLUE benchmark (Wang et al. 2018)) for Natural Language Understanding (NLU) tasks, even without retraining.
The NMT experiments are based on the FairSeq (Ott et al. 2019) implementation of transformer base by Vaswani et al. (2017). For the NLU experiments, implemented the collaborative MHA layer as an extension of the Transformers library (Wolf et al. 2019). The flexibility our layer allows it to be applied to most of the existing transformer architectures, either pre-training after fine-tuning

---

Under review as a conference paper at ICLR 2021
+512
27.6
512
params (x1If)
+.256
X612
27.4
125
236
11
concar:
callab.
collab.
cosccat
oxllab.
27.2
27.2
27.58
61.0
21.0
X236
258x
256
27.10
27.41
56.2
56.2
17.3
19.0
128
26.89
27.40
53.8
53.8
17.3
18.4
collab
X
colab.
64
26.77
21.31
52.6
52.7
16.9
17.9
268
2688
56
58
60
18
19
Number of merametters
K10
Trairing Sma AKVI0D (hours)
Figure 3: Comparison the BLEU score on WMT14 EN-DE translation task for an encoder-decoder transformer-base (Vaswani et al., 2017) using collaborate vs. concatenate heads with key/query dimension Da- We visualize performence as a function of number of parameters (middle) and training time (right). Collaborative attention consistently improves BLEU score, Dy can be decreased by a factor of 4 without drop in performance.
using tensor decomposition. We use the tensor decomposition library Tensorly (Kossifi et al. 2019) with the PyTTrrch backend (Paszke al. 2017) to reparameterize pre-trained attention layers. Our code and datasets are publicly available and all hyperparameters are specified in the Appendix.
4.1 COLLABORATIVE MHA FOR NEURAL MACHINE TRANSLATION
We replace the concatenation-base MHA layers of an encoder-decoder transformer by our collabora- tive MHA and evaluate it the WMT14 Anglish-to-Germann translation task. Following (Vaswani itt al., 2017). we train on the WMT16 train corpus, apply checkpoint averaging and report compound split tokenized BLEU. We use the same hyperparameters as the baseline for all our runs. Results are shown in Figure 3 Our run of the original base transformer with N₁ 8 heads and D₁ 512 key/query total dimensions achieves 40 BLUE (instead 27.30).
As observed in the original paper by Vaswani et al. (2017), decreasing the key/query head size dy degrades the performance (x Figure However, with collaborative heads (+ Figure 3) the shared key/query dimension can be reduced by 4x without decreasing the BLEU score. As 10% decrease number
feed-forward layers and embeddings are left untouched, this translates of parameters for a slight increase in training time. When setting total key/query dimension of D& = 64, corresponding to dę = 8 dimensions per head, the classic MHA model suffers a drop of 0.6 BLEU points, meanwhile the collaborative MHA stays within 0.1 point of the transformer-bas model using concatenation.
We conclude that sharing key/query projections across heads allows attention features to be learned and stored only once. This weight sharing enables decreasing D₁ without sacrificing expressiveness.
4.2 RE-PARAMETRIZE A PRE-TRAINED MHA INTO COLLABORATIVE MHA
We turn to experiments on Natural Language Understanding (NLU) tasks, where transformers have been decisive in im- proving the state-of-the-arr. As pre-training on large text cor- pora remains an expensive task, we leverage the post-hoc re- parametrization introduced in Section 3.3 to cast already pre- trained models into their collaborative form. We proceed in 3 steps for each GLUE task (Wang et al. 2018). First, we take pre-trained transformer and fine-tune on each task individually. Secondly, we replace all the attention layers by collaborative MHA using tensor decomposition
7.5
time [min)]
2.5
0.0
16
128
256
364
512
610
768
pute We. WK and M and re-parametrize the biases into This step only takes few minutes as shown in Figure 4 Fi- nally, fine-tune the compressed model again and evaluate its performance.
Figure 4: Time decompose
BERT-base from Dj 768 Dr.
We experiment with pre-trained BERT-base model (Devlin et variants of BERT designed to be more parameter efficient: ALBERT (Lan et al. 2020), an improved
2019
We also repurpose two
https::/ http://ggtttbbbb/ thub.ccm/.......
7

---

Under review as a conference paper at ICLR 2021
Table 2: Performance of collaborative MHA on the GLUE benchmark (Wang al., 2018 We
report the median of 3 runs for BERT (Devlin et al. 2019). DistilBERT (Sanh et al. 2019 and ALBERT (Lan et al. 2020) with collaborative heads and different compression controlled by DR- Comparing the original models (D,: 768) with their compressed counter part shows that the number of parameters can be decreased with less than 1.5% performance drop (gray rows).
Model
D&
purams
CoLA
SST-2
MRPC
STS-B
QQP
MNLI
QNLI
RTE
Avg.
BERT-base
108.3M
54.7
91.7
88.8/83.8
88.8/88.7
87.6/90.8
84.1
90.9
63.2
83.0
768
108.5M
56.8
90.1
89.6/85.1
89.2/88.9
86.8/90.2
83.4
90.2
65.3
83.2
384
101.4M
56.3
90.7
87.7/82.4
88.3/88.0
86.3/90.0
83.0
90.1
65.3
82.5
256
99.0M
52.6
90.1
88.1/82.6
87,5/87.2
85.9/89.6
82.7
89.5
62.5
81.7
128
96.6M
43.5
89.5
83,4/75.2
84.5/84.3
81.1/85.8
79.4
86.7
60.7
77.6
DistilBERT
66.4M
46.6
89.8
87.0/82.1
84.0/83.7
86.2/89.8
80.0
384
62.9M
45.6
89.2
86.6/80.9
81.7/81.9
86.1/89.6
79.1
ALBERT
11.7M
58.3
90.7
90.8/87.5
91 222/9.88
87.5/90.7
85.2
91.7
73.7
85,3
512
11.3M
51.1
86.0
91 91.4/88.0
88.6/88
87.2/90.4
84.2
90.2
69.0
83.1
384
11.1M
40.7
89.6
82.3/71.1
86.0/85.6
87,2/90.5
84.4
90.0
49.5
77.9
transformer with a single layer unrolled, and DistiBBERT (Sanh al., 2019) a smaller version of BERT trained with distillation. We report in Table the median performance of 3 independent runs of the models on the GLUE benchmark (Wang et al., 2018).
We first verify that tensor decomposition without compression (Ō₃ D 768) does alter performance. As shown in Table 2 both BERT-base and its decomposition performs similarly with an average score of 83.0% and 83.2% respectively. We should clarify that, for consistency, we opted to re-finetune the model in all cases (even when D; = Dz), and that the slight score variation disappears without re-finetuning. Nevertheless, even with re- re-finetuning, reparametrizing the attention layers into collaborative form is beneficial out of the 8 tasks, as well as in terms the average score.
We then experiment with compressed decomposition using smaller De. Comparing the original models with their well well-performing compressed counterpart (gray rows) shows that the key/query dimension of BERT and DistilBERT can be reduced by 2x and 3x respectively without sacrificing more than 1.5% of performance. This is especially remarkable given that DistilBERT was designed to be a parameter-effin efffcccitt version of BERT. It seems that ALBERT suffers more from compression, but the dimension can be reduced by factor 1.5x with minor performance degradation. We suspect that unrolling the same attention layer over the depth of the transformer forces the heads to use different projections and decreases their overlap, decreasing the opportunity for weight-sharing. Our hypothesis is that better performance may be obtained by pre-training the whole BERT architecture variants from scratch.
$311
der arttes
SUS0PO 4ry SUCLERICC
05.0
575-1 de/
00.
00.0
31.1
20.00
255
0.22
01
2.0
354
(420
20 04
125
254
640
D1
02
1%
Figure 5: Performance MNLI, MRPC and STS-B datasets of a wx fine-tuned BERT-base model, decomposed with collaborative heads of compressed dimension Dy (horizontal axis). Repeating fine-tuning after compression can make the model recover the original performance when compression was drastic. The .... GLUE baseline gives reference for catastrophic failure.
Recovering from compression with fine-tuning. We further investigate the necessity of the sec- ond fine-tuning-sted 3 of our experimental protocol-after the model compression. Figure 5 shows the performance of BERT-base 3 GLUE tasks for different compression parameters 7₃ with
8

---

Under review as a conference paper at ICLR 2021
and without the second fine-tuning. We find that for compression up to 1.5x (from D, 768 to D2 512), the re e-parametrization is accurate and performance is maintained without fine-tuning again. Further compressing the model starts to affect performance. Nevertheless, for compression by up to 3x (to D 256), this loss can readily be recovered by second fine-tuning (in orange).
CONCLUSION
This work showed that trained concatenated heads in multi-head attention models can extract redun- dant query/key representations. To mitigate this issue, we propose to replace concatenation- -based MHA by collaborative MHA. When our layer used as a replacement for standard MHA en- coder/decoder transformers for Neural Machine Translation, it enables the decrease of effective individual head size from dN- = 64 to 8 without impacting performance. Further, without pre-training from seratch, switching MHA layer collaborative halves the number of FLOPS and parameters needed to compute the attentions score affecting the GLUE score by less than 1.5%.
Our model can impact every transformer architecture and our code (publicly available) provides post-hoc compression of already trained networks. We believe that using collaborative MHA in models pre-trained from scratch could force heads extract meaningful shared query/key features. We are curious if this would translate to faster pre-training. better performance on downstream tasks and improved interpretability of the attention mechanism.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2014. URL http::tt//axx...rrrgrt/
1t1./4409000000 04733
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V. Le. Attention augmented convolutional networks. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Low-rank bottleneck multi-head attention models, 2020.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https::///// www. wancb com/ Software available from wandb.com.
Kevin Clark, Minh- Than-T Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2020.
wtwp://mieteoooooieee.mmm wtr///orrmmiii/// XMHbBBVE
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. In International Conference Learning Representations, 2020.
URL ht htttpss//oppereee//
net forum? 2 wew.c-HHhllhae. rKPB
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10. 10.18653/v1/n13 1/11//1/111111.... URL
Richard A. Harshman. Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-modal factor analysis. UCIA Working Pupers in Phonetics, 16:1-84, 1970.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compres- sion of deep convolutional neural networks for fast and low power mobile applications, 2016.
Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. SIAM Review, 51 (3):4555500, 2009, ISSN 00361445. doi: 10.1137/0707011XX URL http::ttt/...... dww.//dddooii.ooooooooooo/...o.. www.doroi1.1000000 1137/07070111X
Jean Kossaifi, Yannis Panagakis, Anima Anandkumar, and Maja Pantic. Tensorly: Tensor learning in python. Journal of Machine Learning Research, 20(26):1-6, 2019. URL http:/ /tttp://jml... jmmr..mrr..//.. org/
Papers/v20/13
html
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self self-supervised learning of language representations. In International Conference on Learning Representations, 2020. URL
forumm/d
AleA7.a #leA7AktvS]
Paul Michel, Omer Levy. and Graham Neubig. Are sixteen heads really better than one? In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' A'Alché Buc, E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems 32, pp. 14014-14024. Curran Associates, Inc., 2019. URL latt::/paapers cttipss.ceppper// wwwteeen-eade--eeeeeeeeeee heads readd--realyyy-ettee than one pdf
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. Proceedings of NAACI-HLT 2019 Demonstrations, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito,o Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
10

---

Under review as a conference paper at ICLR 2021
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self- attentitoni in vision models. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' A'ccé--Bue Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Informa- fion Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 14 December 2019, Vancouver, BC, Canada, pp. 68 80, 2019. URL Cnntteeetiionnieee -nr
papers cc ottp::cc..aaaar//00---aaaaaaaaa.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01 108, 2019. URL htRthhttp://axiiii arg/abs/1910.00
Noam Shazeer, Zhenzhong Lan, Youlong Cheng, Nan Ding, and Le Hou. Talking- heads attention, 2020.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention in transformer models, 2020.
Ledyard Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, (1):2277 311, 1966. httpthttt////t/oeee 3:p:279-311
repec. wttp://ete//..r..pprprr...... 2ttp://ee......p..pp....o.......
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 5998-6008, 2017. URL
http://papers nips. wttpssspppppppppptteteeteee..eet ist.://iiaaatteiii----ss-ss all--sss-ll-----eeeee------
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5797-5808, Florence, Italy, July 2019. Association for Computational Linguistics. URL httph:tt///// //ww ac www........... org/ anthology/PI9-1542
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNIPP. Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 11t/:////11115544 URL http:::tt/st..... www. ac wwwb. org/ hntooooyyyyoooooo/ooo/ooootooooh /wtpa////////-- -44-5446
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers: State-of-the-aut natural language processing, ArXiv, abs/1910.03777/ 2019.

---

Under review as a conference paper at ICLR 2021
Supplementary Material
A HYPERPARAMETERS FOR NEURAL MACHINE TRANSLATION EXPERIMENTS
Our implementation is based Fairseq implementation Ott al. 2019). We report in the following tables the specification of the architecture. We used the detault hyperparameters they are not specified below.
Transformer architecture parameters
dataset
architecture
transformer__wmt_e
layers
6
heads
8
hidden-dim
512
collaborative-heal
"encoder
percoder_erross_... cross decoder" or "none"
key-dim
64, 128, 256, 512
share-all -emeedillsmb
True
optimizer
adam
udam-betar
(0.9, 0.98)
clip-norm
0,0
0.0007
min-Ir
1e-09
Ir-scheduler
inverse_sqrt sqrt
warmup-updates
4000
warmup-init-Ir
le-07
dropout
0.1
weight-decay
0.0
criterion
label abbe_ssmotthdd_coos___ttttt
label-smoothing
0.1
max-tokens
3584
update-freq
2
fp16
True
Table 3: Hyperparameters for the NMT experiment.
B HYPERPARAMETERS FOR NATURAL LANGUAGE UNDERSTANDING EXPERIMENTS
We use standard models downloadable from HuggingFace repository along with their configuration.
Models
BERT-base
Devlin et al. 2019
bert hitt-baae-csed
DistilBERT
Sanh et al. 2019
dist -base-cased albert-base-v2
ALBERT
Lan et (2020)
We use HuggingFace default hyperparameters for GLUE fine-tuning in all our runs. We train with a learning rate 2 10-5 for 3 epochs for all datasets except SST-2 and RTE where we train for 10 epochs. In preliminary experiments, hyperparameter among {10 1,10- 10 10110-881 88 but did not see significant improvement and kept the default 10 6 for all our experiments.
tried to tune the tensor decomposition tolerance
12

---

Under review as a conference paper at ICLR 2021
GLUE fine-tuning hyperparameters
Number of epochs
for all tasks but 10 for SST-2 and RTE
Batch size
32
Learning rate
2e-5
Adam
le-8
Max gradient norm
Weight decay
0
Decomposition tolerance
le-6
13


Reviews:
[{"review_id": "hMa-yylY1q-", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzes the multi-head attention in transformers and suggests to use collaboration instead of concatenation of multiple heads. Empirical results on WMT’16 English-German demonstrates that the proposed approach reduces the of parameters without sacrificing performance. Further experiments on pre-trained BERT models also demonstrate its efficiency.  Overall, the paper is well motivated and provides a deep analysis of redundancy of the multi-head attention. \n\nHere are a few detailed comments:\n\n-- In practice, it seems the proposed approach didn’t speed up the training, even though it gives an improvement in terms of FLOPS. This may limit to the large scale of BERT pre-training.\n\n-- From Figure 3 and Table 2, even reducing the $\\hat{D_k}$ from 768 to 128, the total number of parameters of pretrained models (e.g. BERT-base) only reduces from 108.3M to 96.6M. However, the performance has dropped significantly (83.0 -> 77.6 GLUE score) in the BERT-base setting. If we take a closer look, in case of $\\hat{D_k}$=768, the performance on the large tasks (e.g. MNLI) dropped significantly from 84.1 to 83.4 in terms of accuracy, and its improvement is from small tasks (e.g. RTE). It is better to have a deeper analysis and explanation. \n \n===== Update after author response ===== \n\nIf you look a deeper look into Transformers, in case of BERT, the attention block only takes around 25% of total parameters. It is that suspicious that collaborative MHA takes 18% less time in practical since it requires many factors e.g., GPU kernel fusion. \n\nRegarding the performance on MNLI, from Fig 5, it shows that when D_k is larger than 512, MHA reaches to the baseline in terms of accuracy. Additional information from Tab 2, these models have more than 101.4M (in case of D_k = 384) which is almost the same as the original baseline. However, the performance on GLUE is dropped largely, thus it is hard to support papers' claims.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NC8EHArn2nJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates the over-parameterization of attention heads in Transformer’s multi-head attention. The authors show that query-key projections are redundant because trained concatenated heads tend to compute their attention patterns on common features. They propose a reparameterization of multi-head attention allowing the parameters of queries and keys to be shared between heads: this is called “collaborative attention”. This attention can be applied either instead of the standard attention during training, or as a drop-in replacement for an already trained model. To use as a drop-in replacement, the method requires to use tensor decomposition and subsequent model fine-tuning.\n\n------------------------------------------------------------------------------------------------------\nStrengths\n\n1) a nice analysis of PCA components showing that individual heads are not low-rank, but their concatenation is.\n2) the paper is overall clear and the method is explained well.\n\n------------------------------------------------------------------------------------------------------\nWeaknesses\n\n(main) While the main contribution is a more efficient attention layer without a significant drop in performance, this claim is not supported empirically. Since the method operated only within attention layers (reduces only dimensions of queries and keys), in terms of efficiency/quality trade-off it should be compared to other methods, e.g. simple head pruning. While the paper does not provide such comparison, it is clear from the results that the simple head pruning is likely to be superior (and is simpler implementation-wise). \n\nNamely, \n1) when used as a drop-in replacement, the method is more complicated than a simple head pruning, but not more effective. E.g., the proposed method reduces query/key dimension by a factor of 1.5/2/3 and keeps everything else intact. With simple head pruning, about 50% of the heads can be removed without sacrificing quality: in addition to queries and keys, this also halves values and the output projection matrix. \n2) when used in training (see the MT experiments), the results are also not better than post-hoc head pruning. The argument to support this new approach could be that the model is smaller in training, but since the parameter reduction is only in queries and keys, the decrease in the number of parameters is negligible for the whole model. For example, keeping the same quality it reduces the number of parameters from 6.1*10^6 to 5.4*10^6, which is not going to make a large difference.\n\n(minor) As a side contribution, the authors claim to report the discrepancy between the theory and implementation of attention layers. Namely, while the original definition of attention layers does not have biases in linear projections for q/k/v in attention, the authors claim that implementations contain the bias terms, and spend some time showing how to model the biases in key and query layers properly.\n\nHowever, the original Transformer implementation (tensor2tensor) does *not* have biases by default. This means that the authors are probably referring to some specific implementation, different from the original one provided by the Transformer’s authors. Therefore, I can not consider this as a contribution and think that this part is misleading for a reader.\n\nP.S. Here is the tensor2tensor code I was referring to: https://github.com/tensorflow/tensor2tensor/blob/5f9dd2db6d7797162e53adf152310ed13e9fc711/tensor2tensor/layers/common_attention.py#L4416\n\n------------------------------------------------------------------------------------------------------\nOverall recommendation\n\nOverall, I can not recommend accepting this paper. While there are some parts of the paper that I like, the main claim is not supported empirically: both in terms of baselines and the overall decrease in the number of parameters. Additionally, the part with the biases in attention implementation is misleading.\n\n------------------------------------------------------------------------------------------------------\nUpdate after author response\n------------------------------------------------------------------------------------------------------\n\n1) Context and content attention\n\nThank you for updating and saying that only some of the implementations include bias! I think now this part is not misleading and can be of interest. I still have some concerns that this part does not fit the whole story very well - but this is the matter of taste. In the current state, I think it is ok :)\n\n2) On the comparison with head pruning and on the paper going beyond practical realm.\n\nI agree with your comments, but I do think you should make it very clear in the paper. In the current state, the paper tried to make practical contributions and, since they mostly do not hold (e.g., head pruning is simpler in practice), it's hard to appreciate the paper's value. I think you need to modify the things you highlight, and with proper discussion it would be much better. For example, if you state explicitly that in practice pruning may be simpler, but your results say/illustrate something other than practical applications. You won't lose because of it; in fact, I think the opposite.\n\nOverall, I think the paper has improved during the discussion period. In a hope that the authors address my later comments and discuss the pruning in the text, I'm raising the score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LtQDXO9FCN5", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an interesting collaborative MHA to enable heads to share projections, which can be easily applied to most existing transformer-based models, including NMT and pre-training models. With using the new collaborative MHA, the number of parameters and FLOPs can be decreased. \n\nThe PCA-based analysis of the query/key matrices is interesting and impressive, which motivates the newly proposed MHA. The authors also propose a Tensor Decomposition based method to easily convert MHA to its collaborative version without retraining.    \n\nThe paper is well-written and organized, the experiments are thorough. However, I have several concerns:\n1)\tIn the Table 1, it will be more convincing, if the run time on CPU and GPU can be provided. \n2)\tAll the pre-trained models (BERT/DistilBERT/ALBERT) are evaluated on the GLUE dataset, the experiments on more challenging tasks like QA (e.g. SQuAD 1.1/2.0) should be added. \n3)\tThe proposed method seems to be not effective for pre-trained models, e.g. when the number of parameters is decreased from 108.5M to 96.6M, this reduction of model size is not that big, while the average score decreases from 83.2 to 77.6. \n4)\tIt would be interesting to add more analysis on the patterns of learned mixing matrix M for different tasks.\n5)\tIn the Figure 3, the training time of the standard MHA with D_k = 256 is 0.0?", "extra_scores": {}, "extra_text": {}}, {"review_id": "K4mGNXyqoGD", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "========================\n\nPaper Summary:\n\nThis paper proposes a new form of multi-head attention. It can reduce parameters and FLOPs of Transformer models without performance loss on En-De translation. Moreover, for pre-trained language models, no large-scale pretraining is required to convert the attention. A tensor decomposition based method is proposed for the conversion.\n\n==========================\n\nOverall review\n\nThis paper challenges the widely adopted multi-head attention, asking the question whether the concatenation of multiple heads is the best way to fuse heads. The proposed method is well-motivated (Fig. 1), but the empirical performance does not show a clear advantage over the conventional way. More rigorous experiment is needed to justify this new model.\n\nPros\n\n- the method is novel and well-motivated\n- works for both original transformer and pre-trained language models\n\nCons\n\n- improvements (in terms of evaluation metrics) is marginal to original MHA\n- more translation / generation tasks should be evaluated\n- does not measure empirical speed up at inference time\n\n==========================\n\nQuestions / Suggestions\n\n- In the abstract, the authors argued for over-parameterization. In fact, over-parameterized Transformers such as GPT-3 in fact achieves very strong performance. This is still an open question so might not be appropriate to say transformers suffers over-parameterization.\n- In the end of introduction, Synthesizer model is mentioned but this part is not clearly explained in the rest of the paper.\n- Figure 2 is somewhat confusing. I couldn't understand the relationship between (c) & (d) to the rest of the figure.\n\n===========================\n\nMinor Issues\n\n- Figure 4 x-axis: 767 -> 768", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an interesting collaborative multi-head attention (MHA) method to enable heads to share projections, which can reduce parameters and FLOPs of transformer-based models without hurting performance on En-De translation tasks. For pre-trained language models, a tensor decomposition method is used to easily covert the original MHA to its collaborative version without retraining. 

This paper receives 3 weak reject and 1 weak accept recommendations. On one hand, all the reviewers agree that the paper is well motivated and the proposed idea is interesting. On the other hand, all the reviewers also commented that the current empirical results and comparisons are weak, which are not enough to support the paper's main claim. From the current results, it is difficult to draw a conclusion that collaborative MHA is better. 

Specifically, (i) From Table 2, it can be seen that the proposed method is not effective for pre-trained models, i.e., even if the model size is not reduced much, the performance can be dropped significantly. (ii) More experiments, such as QA, more translation/generated tasks will make this paper more convincing. (iii) More rigorous experiments are needed to justify the practical value of the proposed method. If the authors try to emphasize that they go beyond practical realm, then probably a careful re-positioning of the paper is needed, which may not be a trivial task. 

The rebuttal unfortunately did not fully address the reviewers' main concerns. Therefore, the AC regrets that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject