Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ========================

Paper Summary:

This paper proposes a new form of multi-head attention. It can reduce parameters and FLOPs of Transformer models without performance loss on En-De translation. Moreover, for pre-trained language models, no large-scale pretraining is required to convert the attention. A tensor decomposition based method is proposed for the conversion.

==========================

Overall review

This paper challenges the widely adopted multi-head attention, asking the question whether the concatenation of multiple heads is the best way to fuse heads. The proposed method is well-motivated (Fig. 1), but the empirical performance does not show a clear advantage over the conventional way. More rigorous experiment is needed to justify this new model.

Pros

- the method is novel and well-motivated
- works for both original transformer and pre-trained language models

Cons

- improvements (in terms of evaluation metrics) is marginal to original MHA
- more translation / generation tasks should be evaluated
- does not measure empirical speed up at inference time

==========================

Questions / Suggestions

- In the abstract, the authors argued for over-parameterization. In fact, over-parameterized Transformers such as GPT-3 in fact achieves very strong performance. This is still an open question so might not be appropriate to say transformers suffers over-parameterization.
- In the end of introduction, Synthesizer model is mentioned but this part is not clearly explained in the rest of the paper.
- Figure 2 is somewhat confusing. I couldn't understand the relationship between (c) & (d) to the rest of the figure.

===========================

Minor Issues

- Figure 4 x-axis: 767 -> 768
review_title: Official Review by R5

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper analyzes the multi-head attention in transformers and suggests to use collaboration instead of concatenation of multiple heads. Empirical results on WMT’16 English-German demonstrates that the proposed approach reduces the of parameters without sacrificing performance. Further experiments on pre-trained BERT models also demonstrate its efficiency.  Overall, the paper is well motivated and provides a deep analysis of redundancy of the multi-head attention. 

Here are a few detailed comments:

-- In practice, it seems the proposed approach didn’t speed up the training, even though it gives an improvement in terms of FLOPS. This may limit to the large scale of BERT pre-training.

-- From Figure 3 and Table 2, even reducing the $\hat{D_k}$ from 768 to 128, the total number of parameters of pretrained models (e.g. BERT-base) only reduces from 108.3M to 96.6M. However, the performance has dropped significantly (83.0 -> 77.6 GLUE score) in the BERT-base setting. If we take a closer look, in case of $\hat{D_k}$=768, the performance on the large tasks (e.g. MNLI) dropped significantly from 84.1 to 83.4 in terms of accuracy, and its improvement is from small tasks (e.g. RTE). It is better to have a deeper analysis and explanation. 
 
===== Update after author response ===== 

If you look a deeper look into Transformers, in case of BERT, the attention block only takes around 25% of total parameters. It is that suspicious that collaborative MHA takes 18% less time in practical since it requires many factors e.g., GPU kernel fusion. 

Regarding the performance on MNLI, from Fig 5, it shows that when D_k is larger than 512, MHA reaches to the baseline in terms of accuracy. Additional information from Tab 2, these models have more than 101.4M (in case of D_k = 384) which is almost the same as the original baseline. However, the performance on GLUE is dropped largely, thus it is hard to support papers' claims.


review_title: Well motivated but weak results

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents an interesting collaborative MHA to enable heads to share projections, which can be easily applied to most existing transformer-based models, including NMT and pre-training models. With using the new collaborative MHA, the number of parameters and FLOPs can be decreased. 

The PCA-based analysis of the query/key matrices is interesting and impressive, which motivates the newly proposed MHA. The authors also propose a Tensor Decomposition based method to easily convert MHA to its collaborative version without retraining.    

The paper is well-written and organized, the experiments are thorough. However, I have several concerns:
1)	In the Table 1, it will be more convincing, if the run time on CPU and GPU can be provided. 
2)	All the pre-trained models (BERT/DistilBERT/ALBERT) are evaluated on the GLUE dataset, the experiments on more challenging tasks like QA (e.g. SQuAD 1.1/2.0) should be added. 
3)	The proposed method seems to be not effective for pre-trained models, e.g. when the number of parameters is decreased from 108.5M to 96.6M, this reduction of model size is not that big, while the average score decreases from 83.2 to 77.6. 
4)	It would be interesting to add more analysis on the patterns of learned mixing matrix M for different tasks.
5)	In the Figure 3, the training time of the standard MHA with D_k = 256 is 0.0?
review_title: Interesting collaborative MHA mechanism

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper investigates the over-parameterization of attention heads in Transformer’s multi-head attention. The authors show that query-key projections are redundant because trained concatenated heads tend to compute their attention patterns on common features. They propose a reparameterization of multi-head attention allowing the parameters of queries and keys to be shared between heads: this is called “collaborative attention”. This attention can be applied either instead of the standard attention during training, or as a drop-in replacement for an already trained model. To use as a drop-in replacement, the method requires to use tensor decomposition and subsequent model fine-tuning.

------------------------------------------------------------------------------------------------------
Strengths

1) a nice analysis of PCA components showing that individual heads are not low-rank, but their concatenation is.
2) the paper is overall clear and the method is explained well.

------------------------------------------------------------------------------------------------------
Weaknesses

(main) While the main contribution is a more efficient attention layer without a significant drop in performance, this claim is not supported empirically. Since the method operated only within attention layers (reduces only dimensions of queries and keys), in terms of efficiency/quality trade-off it should be compared to other methods, e.g. simple head pruning. While the paper does not provide such comparison, it is clear from the results that the simple head pruning is likely to be superior (and is simpler implementation-wise). 

Namely, 
1) when used as a drop-in replacement, the method is more complicated than a simple head pruning, but not more effective. E.g., the proposed method reduces query/key dimension by a factor of 1.5/2/3 and keeps everything else intact. With simple head pruning, about 50% of the heads can be removed without sacrificing quality: in addition to queries and keys, this also halves values and the output projection matrix. 
2) when used in training (see the MT experiments), the results are also not better than post-hoc head pruning. The argument to support this new approach could be that the model is smaller in training, but since the parameter reduction is only in queries and keys, the decrease in the number of parameters is negligible for the whole model. For example, keeping the same quality it reduces the number of parameters from 6.1*10^6 to 5.4*10^6, which is not going to make a large difference.

(minor) As a side contribution, the authors claim to report the discrepancy between the theory and implementation of attention layers. Namely, while the original definition of attention layers does not have biases in linear projections for q/k/v in attention, the authors claim that implementations contain the bias terms, and spend some time showing how to model the biases in key and query layers properly.

However, the original Transformer implementation (tensor2tensor) does *not* have biases by default. This means that the authors are probably referring to some specific implementation, different from the original one provided by the Transformer’s authors. Therefore, I can not consider this as a contribution and think that this part is misleading for a reader.

P.S. Here is the tensor2tensor code I was referring to: https://github.com/tensorflow/tensor2tensor/blob/5f9dd2db6d7797162e53adf152310ed13e9fc711/tensor2tensor/layers/common_attention.py#L4416

------------------------------------------------------------------------------------------------------
Overall recommendation

Overall, I can not recommend accepting this paper. While there are some parts of the paper that I like, the main claim is not supported empirically: both in terms of baselines and the overall decrease in the number of parameters. Additionally, the part with the biases in attention implementation is misleading.

------------------------------------------------------------------------------------------------------
Update after author response
------------------------------------------------------------------------------------------------------

1) Context and content attention

Thank you for updating and saying that only some of the implementations include bias! I think now this part is not misleading and can be of interest. I still have some concerns that this part does not fit the whole story very well - but this is the matter of taste. In the current state, I think it is ok :)

2) On the comparison with head pruning and on the paper going beyond practical realm.

I agree with your comments, but I do think you should make it very clear in the paper. In the current state, the paper tried to make practical contributions and, since they mostly do not hold (e.g., head pruning is simpler in practice), it's hard to appreciate the paper's value. I think you need to modify the things you highlight, and with proper discussion it would be much better. For example, if you state explicitly that in practice pruning may be simpler, but your results say/illustrate something other than practical applications. You won't lose because of it; in fact, I think the opposite.

Overall, I think the paper has improved during the discussion period. In a hope that the authors address my later comments and discuss the pruning in the text, I'm raising the score.
review_title: A nice paper, but has problems with supporting the claims and some parts are misleading

Meta-review: This paper proposes an interesting collaborative multi-head attention (MHA) method to enable heads to share projections, which can reduce parameters and FLOPs of transformer-based models without hurting performance on En-De translation tasks. For pre-trained language models, a tensor decomposition method is used to easily covert the original MHA to its collaborative version without retraining. 

This paper receives 3 weak reject and 1 weak accept recommendations. On one hand, all the reviewers agree that the paper is well motivated and the proposed idea is interesting. On the other hand, all the reviewers also commented that the current empirical results and comparisons are weak, which are not enough to support the paper's main claim. From the current results, it is difficult to draw a conclusion that collaborative MHA is better. 

Specifically, (i) From Table 2, it can be seen that the proposed method is not effective for pre-trained models, i.e., even if the model size is not reduced much, the performance can be dropped significantly. (ii) More experiments, such as QA, more translation/generated tasks will make this paper more convincing. (iii) More rigorous experiments are needed to justify the practical value of the proposed method. If the authors try to emphasize that they go beyond practical realm, then probably a careful re-positioning of the paper is needed, which may not be a trivial task. 

The rebuttal unfortunately did not fully address the reviewers' main concerns. Therefore, the AC regrets that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject