Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes an approach to learn to share attention heads in multitask (multilingual, multi-domain) transformer models with multi-head attention layers. Given a multi-head attention layer with $H'$ heads being trained on a set of tasks, each task learns to use $H$ heads, and the choice of attention heads is dictated by learnt latent variables. 2 strategies are proposed for this attention head selection:
1. Subset selection, selects top-$H$ heads for each task.
2. Group selection, where $H'$ heads are divided into groups of $H'/H$ heads, and 1 head is selected per task from each group.

This approach is compared against baseline approaches with full parameter sharing (transformers for MT, speech-transformer for other tasks) and baselines enhanced with adapter modules [1] on multilingual Machine Translation, multilingual ASR, multi-domain ASR, multilingual speech translation and multi-domain speech translation. On all these tasks the proposed head-sharing approach shows modest improvements over the fully-shared baselines.

References:
[1] Simple scalable adaptation for neural machine translation, Bapna et al.
main_review: Pros:
1. The paper touches upon several recent research directions including routing strategies in sparsely gated networks [1, 2], parameter sharing in multilingual / multitask models [3, 4] and proposes an interesting extension of Mixture-of-Expert layers for multi-head attention. This should be of interest to practitioners working on large-scale pre-training and sparsely-gated models.
2. The set of tasks considered for evaluation is comprehensive, comprising of multilingual and multi-domain settings for speech-translation, ASR and text-translation.

Limitations:
1. A lack of comparison to fair baselines. For all evaluations the authors compare against a baseline transformer ( or speech transformer ) with $H$ heads and the baseline model enhanced with adapter layers. On the other hand, the proposed approach trains with $H'$ heads and selects $H$ heads per task ($H' > H$). As a result the proposed approach has a larger number of heads during training which might be contributing to the improved performance. It would have been interesting to have another baseline / roofline fully-shared transformer with $H'$ heads to compare against. For a fair comparison, another baseline could involve head-pruning [5] after the initial training of a transformer with $H'$ heads.
2. The value of 'learned' head selection is not clear. Would the performance of the model deteriorate if the $H$ out of the $H'$ attention heads were statically allocated to each task based on heuristic strategies [1], or based on language similarity, similar to the language-group based allocation in [6].

Other minor questions/comments:
1. Table 2: The WER on arabic seems extremely high. [7] suggests that this could be because of a mismatch between arabic-styles in the transcripts and the training data. Does it make sense to exclude arabic from your analysis?
2. Table 1: Would be great to include language-level performance (perhaps in the appendix) to heal understand which languages are benefitting from head-sharing.
3. Appendix: Some of the analysis regarding head-sharing depending more on the high-resourcedness of languages as against linguistic similarity are very interesting. It would be great to incorporate at least the key results into the main paper.

References: 

[1] Hash Layers For Large Sparse Models, Roller et al.

[2] Exploring Routing Strategies for Multilingual Mixture-of-Experts Models, Kudugunta et al.

[3] Latent multitask architecture learning, Ruder et al.

[4] SHARE OR NOT?
LEARNING TO SCHEDULE LANGUAGE-SPECIFIC CAPACITY FOR MULTILINGUAL TRANSLATION, Zhang et al.

[5] Are Sixteen Heads Really Better than One?, Michel et al.

[6] Beyond English-Centric Multilingual Machine Translation, Fan et al.

[7] The Multilingual TEDx Corpus for Speech Recognition and Translation, Salesky et al.

Edit: Following the discussion with the authors I would like to update my score to 7.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper address the problem of negative interference which happens when a single transformer model is shared among the different tasks in a multi-task learning setup specifically those of multilingual translation, speech recognition, and multi-domain speech translation.  The proposed approach is to adaptively make use of specific attention heads for a particular task among a pool of attention heads. In this way, the common attention heads between tasks can reinforce the positive transfer while the attention heads unique to each task will avoid the negative transfer. Extensive experiments on multiple tasks of multilingual translation and speech translation demonstrate improvements in performance over strong baselines.
main_review: 1. The proposed approach is an elegant way of avoiding the negative interference which is especially common when doing full parameter sharing in NLP models.

2. The paper contains extensive experiments in important tasks of language and speech translation, speech recognition and obtain good performance gains showing the usefulness of the proposed approach.

3. The paper is quite well-written and the related work section is comprehensive.

4. Presentation Improvements: The additional parameters introduced due to the incorporation of attention heads and in the two proposed approaches should be included in each of the results table.

**Question to the authors**:

1. Are the Transformer baselines in all the experiments trained with the same parameters as the ones contained in the adaptor models and head selection models? If not, do you think the baselines can yield stronger results when trained with more parameters.

limitations_and_societal_impact: The limitations are described in the conclusion section, but it is quite short. Limitations and Societal Impact should be their separate paragraphs.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a transformer architecture for multi-task learning, based on sharing some attention heads in multi-head attention between different tasks.
The model is supplied with more heads than necessary and a neural module using gumbel-softmax is used to select a subset of these heads for each task. 
A variant of this is described, where heads are distributed between groups and exactly one head is selected from each group.
main_review: The paper proposes a transformer architecture for multi-task learning, based on sharing some attention heads in multi-head attention between different tasks.
The model is supplied with more heads than necessary and a neural module using gumbel-softmax is used to select a subset of these heads for each task. 
A variant of this is described, where heads are distributed between groups and exactly one head is selected from each group.

The grouped model variant is sensible and gives good performance.
The paper is clear and well-described.
There is a good collection of experiments on different tasks and datasets.

The main weakness of this paper is in the weak baselines used for comparison:
1) The baseline models are constructed only using 4 attention heads, which seems unusually low given that transformers for text normally have more heads. BERT-base has 12 heads, BART is trained with 16.
Given that the proposed method introduces additional attention heads, it is not clear that the benefits are not simply due to the baseline having too few attention heads.

2) Also, only Bapna and Firat (2019) adapter architecture is used for comparison, while there are stronger architectures that have been published since then, both for adapters and multi-task learning in general. For example, comparison with the AdapterFusion architecture (Pfeiffer et al., 2020) would make the argument stronger.

3) Finally, reporting state-of-the-art results on these datasets for comparison would be informative, as it is currently unclear whether the reported results are in any way competitive with the general field or not.

When making claims about increased computation efficiency, then experiment timing information should also be reported.

While two experiments report the S2T transformer baseline in two different settings (separate and joint), the other experiments do not for some reason. It is also unclear which setting was used in the other experiments.

It would be interesting to add some analysis about how the selected heads differ between different tasks. In particular, demonstrating that the model is actively using the head selection mechanism would be good.

The proposed subset variant is not particularly well-motivated. If the position of different heads can be moved around in the final multi-head attention output, then the representation space will constantly change and not line up with the layers on top. This is addressed by the grouped variant, which indeed gives consistently better performance. 

The WMT shared task dataset should be referenced directly, not only through Liu et. al (2020) who used it for evaluation.

The paper contains spelling errors that should be corrected by a spell-checker or proof-reading, for example "hyperparamter" and "paramterizes".
limitations_and_societal_impact: The work is on core model optimization, so it does not have immediate potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method that learns a different set of attention head scores for multi-task data. The method is tested on several different sequence generation tasks that rely on the transformer models.

main_review: Originality: the method is somewhat related to the redundancy of attention heads in transformer models, but to my knowledge it’s the first work that tries to learn different attention weights under multilingual setting
Clarity: writing is generally clear although the method description over head selection is unclear
Significance: the paper addresses an important problem with multilingual parameter separation
Strength:
1. the method sounds intuitive and is generally clearly written
2. it seems to out perform the baselines on a variety of tasks
3. the method does not require much increase in extra parameters

Weakness:
The description over head selection strategy and its significance is kind of unclear
There is not much analysis on how the head attention distribution look like - it’s not clear whether the improvement is the effect of selecting a subset of attention head or the actual latent variable modeling

Questions:
1. It is not clear how the attention selection strategy works. My understanding is that there is already a q_\phi defined over the attention heads, so ideally it should be able to learn which of the attention heads should have probability of 0. Why is it so important to select a subset of heads before modeling the distribution over them?


limitations_and_societal_impact: yes
review_title: null

Meta-review: This paper focuses on modifying Transformer architecture to improve the performance of multilingual and multi-domain tasks. In particular, the paper modifies the multi-head attention and allows for different languages/domains to select different subsets of heads. This ensures that tasks corresponding to different domains/languages do not cause negative interference among them (which is a possibility when all parameters, including heads, are shared among all the tasks).

The paper proposes two strategies to select H out of H’ heads for each task and evaluates the obtained architecture on machine translation, automatic speech recognition, speech translations tasks. The paper compares the proposed architecture with natural baselines. Overall, the paper is well written and proposes novel and interesting contributions towards the underlying problem. There were some questions raised by the reviewers about additional baselines that the reviewers have addressed during the discussion phase. Including these results in the revised version will make the paper stronger. Also, consider improving the presentation in Section 3.2.
Decision: Accept (Poster)