Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes experiments in training a non-autoregressive Transformer (glancing Transformer, or GLAT) in a multilingual MT setting. The use of a non-autoregressive model, plus token-level language tags, enables a code-switching strategy where a source sentence in a given language can generate a target sentence containing words from multiple specified languages. This is used to augment the multilingual training corpus with back-translated data consisting of code-switched synthetic source sentences paired with original monolingual target sentences. A multilingual GLAT model trained with this strategy is shown to outperform many baselines including monolingual GLAT and several autoregressive multilingual models. Ablation studies demonstrate that code-switched back-translation is essential for achieving good performance.

main_review: Strengths:
- Novel application of NAT to a multilingual setting, with an interesting new strategy for code-switched back-translation.
- Impressive gains over a fairly broad selection of baselines, including strong auto-regressive models.
- Potential to be a significant breakthrough in increasing both speed and accuracy of multilingual models.

Weaknesses:
- The capacity used is quite small for multilingual models. It certainly seems sub-optimal for the WMT-many setting, where both bilingual baselines are starting to pull away from the multilingual models. Since capacity is such a crucial parameter for multilingual models, experiments to show how this new technique compares as models grow would have made the paper more convincing.
- There are two new things in this paper: the application of NAT to a multilingual setting, and the use of code-switched back-translation. It would have been interesting to tease them apart using NAT architectures other than GLAT: how well do other recent NAT models work on multilingual data; how much do they benefit from code-switched back-translation (since that idea should apply to any NAT model)? The NAT baseline used in the paper is not competitive.
- There are many hyper-parameters involved in the setup. No information is supplied about how carefully they were tuned, or how they would generalize to new settings.
- Some problems with clarity, see details below.

Details:
- This is hard to understand: “	Such efficiency problem is more serious in multilingual setting because serving many translation directions in one model always leads to a higher throughput practically.”
- This statement should be qualified: “NAT generates translation outputs in parallel (Gu et al., 2018), which leads to significantly faster translation speed.” See: Kasai, Jungo, et al. "Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation." arXiv preprint arXiv:2006.10369 (2020).
- s3.2 isn’t very clear. The code-switched generation (middle panel in figure 2) seems to have abandoned the GLAT training idea, and just be the standard CMLM algorithm with language tags added to the masks to get code-switched output?
- s4.2 How exactly does switch-GLAT-w/o-glancing work?
- 4.4.2 "switch-GLAT is superior to all multilingual baselines on the average BLEU score.” - not really, it’s basically tied with Adapter and M-Transformer.

summary_of_the_review: The first application of non-autoregressive transformers to the important problem of multilingual MT, with a novel idea (code-switched back-translation) specific to that application. Results are generally impressive, but the experiments are missing crucial model-scaling data points, and they miss the opportunity to evaluate code-switched back-translation independent from the GLAT architecture the authors favour.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper extends non-autoregressive NMT models, particularly the GLAT model, from bilingual translation to multilingual translation. To enable such extension, the authors proposed token-level language tags for the decoder, code-switch decoder paired with back-translation, and also scheduled training between the traditional MLE loss and the BT loss. Experiments on several multilingual tasks based on WMT corpus show that the proposed model obtains better supervised translation quality than its auto-regressive counterparts and delivers better cross-lingual representations.
main_review: Strengths:

The authors examined the effectiveness of non-autoregressive NMT on multilingual translation, and obtained encouraging supervised translation performance on several data conditions.


Weaknesses:

* Some claims and proposals are not well supported by the current experiments;
* Experimental details are not always clear to me;
* Model comparison might be unfair and misleading, and more comparisons are required.


Comments:

1)	Firstly, at the core of the adaptation of the GLAT model to multilingual translation is the newly proposed code-switch decoder. This decoder uses token-level target language tag to produce code-switch translations. But the necessity of such design is not tested. Why use code-switch decoder not the vanilla GLAT decoder? A comparison to a vanilla multilingual GLAT should be given.
2)	Also, the authors propose code-switch back-translation to improve model’s performance, but don’t prove why such specific BT is favorable or necessary. What if using the vanilla BT to the NMT model? For example, this can be achieved via random online back-translation proposed in [1].
3)	The authors show that the proposed model improves cross-lingual representation compared to auto-regressive NMT models, but results in Table 3 show that BT delivers a large impact on such improvement without which switch-GLAT performs much worse. Then a question would be: what if applying BT to auto-regressive baselines as well (such as the above mentioned random online BT)?
4)	 Some experimental details are very unclear! How did you train your baseline models, CLSR and Adapter? What’s the hyperparameters? For CLSR, what’s your budget constraint? How did you set it? In addition, how did you train the MNAT baseline? Did you apply the recently introduced optimization techniques as in [2]?
5)	You mentioned in page 5 that knowledge distillation is used. What’s your teacher model, bilingual Transformer base or multilingual Transformer base? Did you also apply the KD to all baselines (autoregressive and non-autoregressive)? I would assume that the authors used bilingual models for KD and only applied it to the NAT models. In such a case, claiming multilingual NAT “outperforms” multilingual AT is non-convincing.
6)	As explained in the introduction, multilingual NMT benefits low-resource translation, which also enables zero-shot translation. Neither are explored in the current experiments. WMT also offers low-resource languages. Would switch-GLAT benefit these languages in a multilingual setup? What about switch-GLAT’s zero-shot ability?

[1] Zhang et al., 2020 Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
[2] Gu et al., 2020 Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade

summary_of_the_review: In short, this paper explores an interesting direction: applying non-autoregressive models to multilingual translation. But the experiments suffer from unclarity, unfair comparison and over claims. The paper could be largely improved with more analysis and comparisons.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed a novel method for the multilingual non-autoregressive machine translation (NAT) model. The key idea is to modify the decoder module of glancing Transformer (GLAT), making it generate the contextual code-switched translations, then use them to perform code-switch back-translation. To incorporate with the generation of contextual code-switch translations, the authors proposed to add token-level language tags to the first layer and the final output layer at each position. The empirical results show that: 1) the inference speed is faster than the autoregressive baseline model (M-Transformer). 2) the translation performance outperforms the NAT baselines in terms of WMT-EDF, WMT-EFZ benchmarks. Although the performance is slightly worse than the GLAT model on the WMT-Many benchmark, the author claim that the reason is due to the limitation of model capacity. 3) Two cross-lingual experiments and the visualization of representation prove that the proposed method has better cross-lingual capability compared to other multilingual models. 
main_review: **Strengths**: 
1. The paper presents a multilingual solution of NAT. The method works with GLAT and provides good insights on how to learn a good cross-lingual representation for NAT.


2. The proposed method speeds up the decoding process of the multilingual Transformer (M-Transformer). The generation quality of the proposed method is close to the bilingual autoregressive models (Transformer).


3. The author conducted comprehensive analytical studies on the improvement of the proposed method. It is clear to see how well the learned representations are aligned from the cross-lingual perspective. Besides, the word induction and parallel sentence retrieval task support the alignment ability in word-level and sentence-level, respectively.

**Weaknesses**: 
1. The proposed method builds on the top of the GLAT method, which limits its generality. According to the ablation study, the performance dropped a lot without using the glancing mechanism (w/o glancing in Table 1 and 2).


2. Although I can understand that how the proposed method works, the description text of Figure 2 seems to be problematic. In Section 3.1, 3.2, the authors said that the code-switch decoder and code-switch back-translation process are illustrated in the left and middle modules, respectively. It is quite confusing that why they do not match the upper annotations shown in Figure 2.


3. There may be an error of mathematical symbol in Section 3.3. What does $L_{CS}$ refer to? Should it be $L_{bt}$?


4.    Missed one result of representation visualization. I understand that the adapter cannot learn cross-lingual alignment well because it models language-specific capability. Since CLSR can learn the shared pathway and language-specific capability together, why the authors did not choose to compare it in Section 5.1?


5.    The explanation that why switch-GLAT performs slightly worse than GLAT does not support by the experiments. The authors claim that it “may be owing to the limitation of model capacity”, will a bigger model or a balanced training set be helpful to alleviate this problem? It should be verified by additional experiments.


6. In Section 3.2, the authors said that “In the ﬁrst iteration of $P_M$, the number of mixed languages is set to 1. Afterward, it will be increased to one-third of the total.” There are four languages in WMT-EDF and WMT-EFZ training sets, does it mean that the number of mixed languages is always set to 1 (4*1/3~=1) during the training process? In this case, how the cross-lingual alignment is learned?

summary_of_the_review: The paper is well-written and contributes a novel approach for multilingual translation in NAT. The design of the code-switch back-translation is reasonable for boosting the performance of multilingual translation and the empirical analysis is convincing. Experiments show that switch-GLAT is better than other NAT/multilingual baselines in limited benchmarks and has close performance compared to bilingual AT baseline.
review_title: null

Meta-review: This paper proposes several innovations for machine translation. The reviewers had several questions about the claims that were made and the authors addressed these and also acknowledged that some of their formulations (e.g. 'better') would need to be qualified. Overall, there are several interesting ideas that have been put together in a sensible way, but the story is not super consistent.

The detailed exchanges between the reviewers are authors are commendable!
Decision: Accept (Poster)