Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This work discusses how to set the projection size for each head (head size) in multi-head attention module, especially Transformer. Theorem 1 is interesting, which points out a lower bound for the head size. The proposed method is to decouple the dependency between the head size and the embedding size. The experiments show that the proposed method is able to achieve comparable performance to BERT with fewer training cost.

The lower bound for the head size is a valuable result. However, the novelty is very limited. To decouple the dependency between the head size and the embedding size is not a novel point. In BERT/Transformer, it is set d_q=d_k=d_v=d, which is not a strict constraint. The only constraint in attention is to have d_q=d_k to allow dot product. Therefore, the proposed method is more like a tuning of hyper-parameters.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This work studies the head size <--> head number tradeoff in multihead attention. It argues and formally establishes that (1) the expressivity of an attention head is determined by its dimension and (b) fixing the head dimension, one gains additional expressive power by using more heads. In response to such observations, the paper proposes Fixed Multihead Attention, where the constraint that `head_size * number_of_heads = embedding_size` in standard multihead attention is lifted; and it allows for using more attention heads without making each head smaller. One can control the total amount of parameters by using smaller embedding sizes, making it comparable (in terms of #parameters) to standard multihead attention. Empirical results on language modeling and NLI tasks confirms the arguments. 

Pros:
- The arguments on head size and head number tradeoff could be inspiring to future works.
- A simple approach that proves strong in several NLP tasks.

Cons:
- The theoretical discussion imposes too strong assumptions that might make it less interesting in practice.
- No NMT experiments.
- The takeaway seems a bit trivial.

Details:
- Theorem 1 presents a rank-based view of each attention head's capacity, which is nice. Yet it is still unclear whether it is the case that the more expressive are the heads the better. For example, several recent works argues for specialized attention heads, i.e., each head has specific "job," which may not require it being very expressive [1, 2]. Further, other works shows that a low-rank P matrix could be beneficial [3, 4, 5], which contradicts the argument in this work. It would be nice if the authors and discuss this in the revision

(To be clear, I do believe this is still an open question, and do not think presenting a different view from previous works hurts the contribution of this work in any way.)

- Theorem 2. I didn't carefully check the proof. Why is it required that the V matrices for each head have the same product. For both Thm.1 and 2, it would be nice to see some discussion on how they translate into the models in practice.

- Can the authors compare the training/inference speed? It probably will be the same as standard transformers, but it would be nice to confirm.

- Figure 1: the caption says trying out embedding sizes from 256 to 512. But it seems that only 4 values are tried. Can the authors comment on this? Also, it is a bit awkward to plot a line chart out of 4 points. Same for Figure 2.

- It would be nice to see some NMT experiments. 

- The proposed method is so straightforward that I'm actually very surprised that this paper is the first trying this. The authors might need justify the technical contribution more.

(I'm on the fence for this one, but the system doesn't allow me to. I'm happy to revise the score if the authors can address my concerns.)


[1] Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. https://arxiv.org/abs/1905.09418

[2] Are Sixteen Heads Really Better than One? https://arxiv.org/abs/1905.10650

[3] Generating Long Sequences with Sparse Transformers. https://arxiv.org/abs/1904.10509

[4] Generating Long Sequences with Sparse Transformers. https://arxiv.org/pdf/1904.10509.pdf.

[5] Adaptively Sparse Transformers. https://arxiv.org/abs/1909.00015.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper proposes a “concise” version of the well-established Transformer model. The main proposal is to explicitly set the head size in the Transformer model instead of having to divide (share) representation ability amongst heads. 

This paper is poorly written and after an entire 2-page long-winded introduction, the reader is left wondering what is the main contribution of this work. The term “concise” is also not well-defined and left vague to readers. I re-read this paper multiple times and the only concluding finding I have is that this paper proposes an explicit way of setting the projection dimension regardless of the number of heads. 

After many mathematical formulations, theorems (seemingly ornamental, or handwavy actually), the final contribution seems to be to set the head size of BERT (size of each head) to 128. This is really trivial. The authors kept teasing a “different” way to do this, but this left the reader completely unsatisfied when the different way refers to explicitly setting each head to 128 and using a smaller model overall. 

The value 128 is derived from a theorem derived by the authors, which suggests that each head should at least be greater or equal than the sequence length (the sequence length here stated by the authors is 128). I’m not very convinced by the argument. While it is intuitive that each head has to be sufficiently large, being under-sized can be made up for with multiple heads. It is also not clear why every X and P must be expressed with transforms W_q and W_k. P here represents the affinity matrix between tokens in a sequence.  It does not make any sense to me to ensure that every variation of P can be expressed because P is literally the pairwise scores between every token in the fully-connected attention graph. 

While I did not have the luxury of time to parse the Appendix to validate the legitimacy of the proof, I think the overall shortcomings of the paper (highly non-readable, bad presentation and perhaps a fair attempt at masking the lack of contribution) warrants a clear reject from me. 

review_title: Official Blind Review #3

Meta-review: This paper studies tradeoffs in the design of attention-based architectures. It argues and formally establishes that the expressivity of an attention head is determined by its dimension and that fixing the head dimension, one gains additional expressive power by using more heads.

Reviewers were generally positive about the question under study here, but raised important concerns about the significance of the results and the take-home message in the current manuscript. The AC shares these concerns, and recommends rejection, while encouraging the authors to address the concerns raised during this discussion. 
Decision: Reject