Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this work, the authors present a conditional language-specific routing (CLSR) scheme for transformer-based multilingual NMT systems. They introduce a CLSR layer after every transformer encoder and decoder layer; each such layer is made up of hard gating functions conditioned on token representations that will either select a language-specific projection layer or a shared projection layer. Further, a budget is imposed on the language-specific capacity measured by aggregating the number of gates that allow for language-specific computations; this budget constraint forces the network to identify the sub-layers that will benefit most from being language-specific.

This is nice work. The proposed technique has been described clearly, the idea is intuitive and the experiments are pretty compelling. I have a couple of minor comments/suggestions for the authors.

* The authors show heat-maps of LSScore distribution in Figure 6 (Appendix B) which suggest that the LS capacity schedule might have little to do with linguistic characteristics. However, this might have to do with the multilingual model being trained on as many as 94 different languages. It seems plausible that linguistic similarities might govern LS capacity scheduling when there are fewer training languages to learn from. To check for this, it might be interesting to redo this experiment with the medium resource and low resource buckets containing 26-28 languages each.

* There are two (among many other) interesting things that stand out from the results in Tables 1 and 2. (1) From Table 1, the only setting where CLSR* (as well as "Top-Bottom" and "Dedicated") underperforms compared to the baseline is M2O for low-resource languages. It seems like the use of language-specific layers here has a strong adverse effect on performance (-4.56 with CLSR-L) which is largely offset by CLSR*. Some more insights based on the individual BLEU scores for each test language in the "Low" bin and whether there were certain languages that were largely responsible for the drop in performance would be interesting to the reader. (2) From M2O in Table 2, the win ratios of Top-Bottom are much lower when compared with Dedicated and CLSR* (61.54 vs. 84.62 vs. 84.62; 30.77 vs. 84.62 vs. 100).  Could the authors share their thoughts on why this drop might be appearing?
review_title: Nice work showing how to add language-specific modeling capacity to large multilingual NMT models in a principled manner

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The work proposes a hybrid architecture that has: (1) language-specific (LS) components; (2) as well as the components that are shared across all the languages -- a trade-off between specificity and generality.  A key conclusion of the work is that the best architectures typically are. the ones that have ~10-30% language-specific capacity. 

In terms of experimental work, the work uses WMT-14 and OPUS-100 datasets to show the proposed trade-off. 

In terms of exposition of the ideas, it's a well-written paper for the most part. 

One issue that the authors could improve on is clarifying how "the amount of LS computation" is measured. You have mentioned it several times in the abstract/intro and it's neither clear nor referenced (it could be the number of parameters, it could be the number of basic computations, etc). For a new reader, it takes quite a while to find that $p$ is defined in eq. 6 and defined as a budget contains. 

One other quibble is that all the trade-off figures are shown based BLEU/automatic metrics, which are known to be inaccurate. It would be nice to repeat one of the included evaluation with human judgments. 

Overall, I view this as a good contribution to pave the way towards stronger, but reasonably-sized multilingual models. This is partially assuming that the authors will stay true to their promise that "Source code and models will be released."


review_title: Cross-language parameter-sharing for multi-lingual translation 

final_score: rating: 7: Good paper, accept
review_content: Manual parameter sharing schemes are generally costly to come up with and when they are obtained for certain language pairs they do not necessarily generalize well to arbitrary language pairs in multilingual NMT. The idea of learning which parameters to share across languages in multilingual transformer models is original and potentially useful for designing and analyzing multilingual models in the context of NMT.  

**Strengths**:

The paper is well-written and easy to follow. The idea was (reasonably) well-positioned with respect to prior work and clearly presented. 

The technical merit is essentially in coming up with the budget constraint term in the loss function that forces the multilingual encoder-decoder "super-network" to use the desired percentage of language-specific computation using gating. 

A significant part of the contribution was in the analysis of the results, obtained by this learning-based parameter sharing approach, which was quite informative and revealed some interesting insights about where and when a language-specific computation is required. The takeaways should be of interest to researchers and practitioners interested in designing and analyzing multilingual NMT systems. 

**Weaknesses**: 

(1) Even though it is the first time such a method is applied in the context of NMT, the idea is not as much novel in the broader context of deep learning. Prior work has explored "learning-to-share"  strategies for parameter sharing in multi-task learning (see Ruder et al., AAAI 2018), and using gating/masking to control computational paths in a differentiable way (see Fan et al., ICLR 2019, Sukhbaatar et al., ACL 2019); it is clear that the focus is NMT but it should be worth mentioning/discussing such studies to better situate the work and to help the reader assess the actual contributions. 

(2) Another weakness is that the comparison with the vanilla and LS baselines does not seem to be properly controlled in terms of parameters. I appreciate that the authors do not read too much into it and focus more on the analysis of the results, but one thing that remains unanswered in this paper is how the proposed method fairs against multilingual baselines that utilize (roughly) the same number of parameters; currently, the best models outperform the LS baseline by ~28M and ~10M parameters on OPUS-100 and WMT-14 respectively. How important is this difference?

(3) In the experiment about linguistic similarity, it appears that the capacity schedule is the same across languages and the authors conclude from this that the schedule has little to do with linguistic characteristics. However, the main driving force in the choice of the language-specific computation is currently a single hyper-parameter p which is the same across languages; so, this will lead to choices that are good on average for all language pairs involved for a given *universal* budget. Do you think the conclusion would be still the same if a language-specific hyper-parameter p_l was used instead? 

review_title: Interesting contribution in the context of multilingual NMT

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: In this paper, the authors present a study of different aspects of language-specific model capacity for massively multilingual machine translation. To this end, language-specific behaviour is achieved via a combination of conditional computation to decide whether to use language-specific parameters or not and statically assigning experts for each languages. The language specific sub-layers are incorporated throughout the network. The training objective allow budgetary constraints on the amount of language-specific parameters. The paper does a systematic analysis on the role of language specific parameters using the proposed architecture. Based on the analysis, recommendations on design of multilingual NMT architectures are proposed and their efficacy validated experimentally. The study sheds light on the amount of language specific parameter sharing, their distribution in the network, impact of language, etc. 

I find the analysis presented in the paper very interesting and insightful - and distinguishes it from previous work in this area. The hypothesis are clearly stated and the experiments are well designed. The findings from the analysis are an important addition to the understanding of the role of language specific parameters in multilingual NMT. 

In terms of modelling, the work follows in the line of recent work on language-specific parameters for multilingual NMT. The deviation from existing work is mixing elements of conditional computation with language specific computation. I see this work more as an analysis on language-specific parameters for a particular LS-model rather than a novel architecture. It is not clear how this model would compare to other models using language specific parameters (sparsely gated mixture of experts (Lepikhin et al 2020), light-weight adapters (Bapna et al 2019)  ).


Questions: 

- Previous work has tried to combine both language-specific and shared parameters (Wang et al 2018), rather than making a binary choice between these. Did the authors compare with such an approach? 
- Since a major part of the model contains shared parameters, was there a need for new set of shared parameters along with the language-specific parameters. The gating decision could have been to bypass the language-specific sublayer or not.

References
Yining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu, and Chengqing Zong. Three strategies to improve one-to-many multilingual translation. EMNLP. 2018.
review_title: A systematic analysis of language specific parameters in multilingual translation

Meta-review: This paper proposes a conditional language-specific routing (CLSR)  mechanism for multilingual NMT, which also considers the trade-off between language specificity and generality.

All of the reviewers think this paper is interesting for both idea and empirical findings. Therefore, it is a clear acceptance.
Decision: Accept (Oral)