Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors report the transformer model with global attention heads rather than the local ones as in other types of transformer architectures.
strengths_and_weaknesses: Strengths:

The paper is well-written and interesting to read, math is described sufficiently. Learning shared information over the transformer head can be a well scalable feature that can be used in other transformer-like architectures.

Weaknesses:

The authors performed several benchmarks, however, the improvement over the vanilla model is marginal. Even though the experiment was performed for 5 runs with various random seeds, it is hard to judge on this improvement as various parameters can affect performance.
questions: It would be interesting to see if the admixture of attention heads is applicable for the image domain as feature sharing is more natural there. Have the authors tried that?
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes to use a Transformer with a Finite Admixture of  Shared Heads.
The idea is to directly learn these attention matrices rather than the local ones as in other  transformers. At the core of the method they sample local attention matrices from an 73 admixture of a small number of global attention matrices. 
The paper shows numerous results on various benchmarks which support their claims. 

strengths_and_weaknesses: Strengths: 
I am not familiar with the literature, but the results are well established.
The experiments show advantages of their method, pointing to new directions of improving transformers.
The paper is fairly clearly written.

Weaknesses:
The authors' description of the experimental results are not accurate enough. 
I would be happy to see results on more benchmarks. 

questions: I see areas of similarity to the work: https://arxiv.org/abs/2110.08678 (Improving Transformers with Probabilistic Attention Keys). Please explain how your work differs.
limitations: Please see questions section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces FiSHformers, a Transformer model with a Finite Admixture of Shared Heads. FiSHformers calculate $M < H$ global heads and construct/sample the full attention heads from the $M$ heads, thus reducing a certain level of computations in constructing Transformer models.
strengths_and_weaknesses: Strengths:

Overall, the idea is interesting. The analysis from a probabilistic viewpoint and applying the admixture model to attention calculation is novel. The paper is well organized and presented. The experimental results presented in this paper show that Fishformers obtained comparable or better results than Transformer models with the calculation of global attention matrices. In addition, the authors show that the purposed structure can also work with some linear Transformers.

Weaknesses:

My main concern is about the significance of the work.

1. There are no theoretical guarantees that we can obtain the same or better performance with a strictly less number of global shared components.

2. Although the number of FLOPs is shown to be less (less global attention is calculated, depends on $H$) using FiSHFormer model and its variants, the computation complexity remains the same (depends on $N$).

3. The author also mentions the head redundancy issue, but $L_2$ distances among heads are more like a reflection but not determinating factors for the claim that FiSHFormer is more efficient than other Transformer variants (low-rank, parameter sharing heads, etc) in terms of the issue. 

questions: 1. In this paper, the authors assume $A_j$ comes from different distributions ( the learned attention heads are diverse ) but with the same M and $\theta$. I was wondering how this assumption is justified. 

2. Is it possible to measure and compare the sampled set of $A_j$s and original $A_j$s ? I mean, if there is a way to show that we can always construct original attention heads from sampled attention heads, or the former contains the latter, the claims on heads redundancy may be stronger. 

3. How to determine the number of $M$ for better or similar performance. 
limitations: No mentions of limitations, but do analyze the social impact.
review_title: null

Meta-review: metareview: This work proposes a version of transformers with an admixture of attention heads. The reviewers find the idea interesting. They find the paper to be well organized and presented, and with sufficient empirical support for the main conclusions. There was some initial concern over similarity to prior work, but the reviewer indicated this has been resolved in the discussion with the authors. I therefore recommend accepting the paper.
confidence: Less certain
Decision: Accept