Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The proposed mechanism disentangles the search and retrieval from the transformers architecture, where the retriever is no longer tied to a specific retriever that is in the same head - instead, there are multiple searches available, and multiple retrievers. Conditioned on the input, each search is "tied" to a (soft) single retriever. Then all results from all searches are concatenated.
main_review: Intuitively, the suggested method makes sense: it provides additional flexibility in the attention mechanism, since now the same retrievers can be re-used by different searchers, and since a search is not confined to a single retriever. Such added flexibility and re-usability could improve the generalization abilities of the attention mechanism.

The authors do a good job of evaluating the proposed mechanism, with a diverse selection of tasks: Reasoning over images (sort of clevr), logical reasoning, SCAN (length split) and language modeling and an additional specifically-designed task that requires contextual retrieval. For some of these datasets, models are evaluated on OOD samples. Since generalization seems to be the main motivation for the proposed architecture design, adding additional OOD tests where possible (e.g. as far as I understand CLEVR was only tested in i.i.d setting) could have made the results stronger. Nevertheless, experiments convincingly show an improvement in almost all cases.

My main concern regarding the experiments is that in all cases, parameters were shared between all transformer layers. Since in most common usages of the models parameters are not shared, it seems important to test how well does this method work with non-shared parameters - would the compositional attention still work well with different search/retrieve parameters?

summary_of_the_review: This paper proposes a modification to the transformer's attention mechanism, which shows improvement on multiple evaluation benchmarks of various types. I find this proposal to be convincingly useful, although evaluation on more o.o.d splits and experiments with non-shared parameters could have made results stronger.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a Compositional Attention to replace the standard multi-head attention. It argues that traditional multi-head attention has a rigid mapping between each query-key pair and the associated value. This leads to redundant parameters and less generalizability.

main_review: Strength: 

The author presents a comprehensive analysis of the limit of multi-head attention. They point out that the rigid mapping between query-key and value pairs can lead to the learning of redundant query-key matrices and value matrices. Instead, they propose to consider all pairs between query-key and value. They define query-key as a search and value as a retrieval and then introduce new parameters to compute the attention score to compute the weighted sum over the pairs of search to all retrievals.

It compares compositional attention with multi-head attention on a set of tasks including a proof-of-concept contextual retrieval task, Sort-of-CLEVER, logical reasoning tasks, language modeling, etc. Compositional attention has shown consistent improvement on these tasks.  

Weakness: 

They argue that the complexity overhead of compositional attention is lightweight. More importantly, the overhead can be mitigated by reducing the number of retrievals. The appendix shows the overhead analysis on the contextual retrieval task. It would be much more helpful and convincing to show such analysis on other real world tasks.

The paper lacks the comparison with other related works focusing on head pruning. As the author pointed out, in figure 11, the compositional attention may tend to learn to focus only on several pairs of search and retrieval pairing. This is very relevant to other head pruning works. It’s worth noting what’s the difference between compositional attention and other head pruning techniques. 

Another question remaining unclear is what kinds of tasks can benefit from compositional attention? Multi-head attention has been widely adopted in a lot of scenarios, such as pre-training, fine-tuning, various tasks with sufficient data or very little data. Though the author has tried to experiment on a set of tasks, it remains unclear what's the rule-of-thumb to replace compositional attention to multi-head attention.

summary_of_the_review: The paper focuses on improving parameter redundancy and generalizability in multi-head attention. The compositional attention is well motivated to mitigate these issues. The paper is well written and has shown improvement of compositional attention. That said, it's lacking analysis of the constraints on computation overhead of compositional attention on more real-world tasks under both sufficient data and low data scenario. It's a bit unclear of the rule-of-thumb to replace the multi-head attention with compositional attention.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the multi-head attention mechanism in Transformer. The paper first analysis the potential drawback of the rigid mapping between search and retrieval in standard attention heads, and then proposes compositional attention that disentangles search and retrieval and composes them in a dynamic and context-dependent manner. To evaluate the proposed method, the author conduct experiments on the standard Transformer model and Vision Transformer.

Overall, the paper is clearly written and the problem studied in this paper is interesting and well-motivated. The proposed compositional attention mechanism is rational. The experiment conducted in this paper is somewhat thorough.




main_review: Strengths:
-  The problem studied in this paper is interesting and well-motivated.
-  The paper points out the shortcomings of rigid search-and-retrieval coupling in standard multi-head attention.
-  The proposed compositional attention mechanism is rational.

Weaknesses:
-  The evaluation tasks selected in this paper are somewhat weird. The author should conduct experiments on some traditional or mainstream tasks  (or settings) of language or vision data.
- The paper should also compare with other variants of multi-head attention mechanisms.
- The paper should explore the effect of the proposed compositional attention mechanism on the latest pre-trained language models.






summary_of_the_review: The paper provides a somewhat novel insight into the shortcomings of standard multi-head attention, and also proposed a compositional attention mechanism. However, there are still some concerns about the evaluation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a mechanism for dynamically assigning query-key attention masks (searches) to one of several value matrices (retrievals), avoiding the rigid assignment of search to retrieval inherent in standard multi-head attention.
main_review: ### Strengths

- The paper identifies a current limitation of multi-head attention, and proposes an interesting method to overcome it.

- The paper shows results on a wide variety of experimental tasks.

- The paper is well-written and thorough.

- The illustrative experiment proposed in the paper is an intuitive, clear example of what types of problems Compositional Attention can help with


### Concerns

I would imagine that a Transformer without compositional search/retrieval could possibly compensate for that lack of single-operation compositionality through the stacked computation from the depth of the network.  However, most of the experiments use Universal Transformers as a baseline (or a single layer for the illustrative tasks).  Since the parameters are shared between layers, it seems like that might be a limitation compared to non-shared parameters when the focus is on compositional tasks.  Have you compared to Transformers without shared parameters between layers?  

The compositional attention adds overhead compared to a standard MHA.  Although the parameter efficiency argument is well made, there is little analysis of the computational tradeoffs.  How do the Compositional Attention models perform when matched against the baselines for runtime rather than parameter counts? (I understand that is a fraught task given different hardware optimizations/settings between models, but it is still an important consideration)  

On a related note, how might Compositional Attention scale to larger models with more searches and queries and more layers?  Does the runtime get much worse than the ~10% reported in the appendix when the models are closer to some of the bigger models reported in the literature?  I’m trying to evaluate how to think of the tradeoff between heavier attention operations run fewer times versus simpler operations with more depth.
summary_of_the_review: The paper presents an interesting idea, based on the promising property of compositionality, to increase the power of a single MHA computation and a resulting transformer architecture.  However, I am not quite convinced by the baselines and the analysis of the computational complexity.  I am happy to raise my score if my above concerns are addressed.


### Update after author comments

Thank you for the clear, detailed response and the additional experiments.  You have addressed my concerns, and I will raise my score.  Please consider including the results of the additional experiments and the related discussion of compositionality vs depth in the paper itself, even if just in an appendix.
review_title: null

Meta-review: This paper identifies a limitation with current attention in transformers where they scoring with query-key pairs is strongly tied to retrieving the value and proposes a more flexible configuration that subsumes the previous setup but provides more flexibility. The authors shows this leads to improvements in various settings.

Overall, all reviewers seem to agree there is interesting insight and results in this paper and it merits publication. Also the discussion helped stress important points regarding weight sharing and more. One concern is that the model was not evaluated on standard NLP/vision datasets (I assume alluding to GLUE/SuperGlue/SQuAD, etc.), and authors seem to hint that pre-training this is an issue for them computationally. This leaves open whether this indeed can and should replace the standard attention mechanism across the board, but is still very worthy of publication.
Decision: Accept (Spotlight)