Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides an in-depth examination of language model based translation models and the impact of various parameters in the learning process. 

They pick the "classic" encoder-decoder based translation model as the baseline and propose several variations of language model based translation models to compare. This includes the two main categories of unidirectional and bidirectional language modelling of the source language. On top of that, they look into various setups such as which layers to use to pass the source encodings to the target generation phase of the translation, whether to share parameters between source and target, and changing the depth (number of transformer layers) and the width (feed forward dimension).

They evaluate on in-house and publicly available datasets and investigate how different setups compare in the supervised translation task, zero-shot learning, low-resource vs. high-resource language pairs, and bilingual to multilingual translation. 

main_review: This is a paper on investigating the gaps of information in a relatively new area of research in machine translation. The approach, experiment setup, and evaluation methods are reasonable and provide confident insight into the questions that the authors ask. 

Here are a number of questions for the authors:

- Your TopOnly setting is supposed to mimic how the encoder-decoder model uses the topmost-layer source encodings for translation. This setting indeed shows improvements and achieves lower perplexity on almost all language pairs. One can interpret that two fundamentally different architectures are still bound by similar features. Did you investigate why the topmost-layer is the most informative way (at least in the tried approaches) of transferring information from the source to the target generation? Following on that, looks like that the difference between the _Deep_ and _TopOnlyDeep_ variations is very small. What is your understanding on the difference of information captured by deeper _and_ multiple layers versus deeper and topmost layer? Did you compare _Deep_ and _TopOnly_ setups? This question is mostly about the PrefixLM model. 

-  You mention that the _TgtOnly_ setting is supposed to enrich the CasualLM model with additional information. However it
requires and occupies part of modeling capacity. Can you explain what you mean by occupying the modeling capacity?

- You conclude in one of the sections that "When there are fewer parameters, the model with inductive biases favoring translation will achieve better quality" and follow it with four observations two of which I want to discuss here. It is not entirely clear to me why "deeper LMs (_Deep_) rather than wider" and "training LMs without source-side language modeling loss (_TgtOnly_)" are both considered as high inductive biases. For the former, how is deeper models have more inductive biases than wider models? For the later, wouldn't the other way around be correct?

- You observe that the PrefixLM model performs surprisingly well on zero-shot directions. Do you have any intuitions or have you looked into finding _why_ that's the case?

- According to your experiments, it looks like CausalLM (unidirectional) model is not performing well, or at least as good as PrefixLM or EncoderDecoder models, in almost any settings. This is intuitive because the model has access to less information at each step. I'm wondering if you _can_ think of a scenario where such a model will be more effective.

- Since this is an examination work on multiple translation models and configurations, it would have been nice to provide a deeper analysis on the linguistic aspects of the experiments as well: Language pairs with different characteristics (morphologically rich languages for instance) work differently with different models. How different variations of the same model perform on different classes of languages is a significant question to investigate.




summary_of_the_review: I find this paper an interesting read. The comprehensive study into different variations sheds light on effectiveness of language model based translation models in different settings. The approach, experiment setup, and evaluation methods are reasonable and provide confident insight into the questions that the authors ask. 
This is valuable in the progress of research in developing new translation models. 
One downside of this work is that there is no novel approach proposed in this paper, as well as a more comprehensive study into existing models and combining various setups is more useful to the research community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper gives a quite thorough comparison between language-model-based (LM) and encoder-decoder-based (EncDec) architectures for neural machine translation (NMT). The investigated LMs are varied: different masking strategies, different origins of source features, and so on. The authors conducted experiments on representative bilingual and multilingual benchmarks, summarizing some findings based on the experimental results. 
main_review: This is an OK paper. The writing and presentation are clear, the experiments are relatively thorough, the related works are discussed comprehensively, the results are convincing, the claims are well supported by the results.

However, after a careful reading of the paper, I still could hardly find enough insights from the paper. ***How could the findings benefit the community?*** The design principle of EncDec-based NMT is the sub-optimal performance/efficiency of LM-based NMT, and this paper just says: "yes, the intuition is true", which is not exciting at all. One insight is the zero-shot performance of multilingual NMT, however, I do not think there are many differences between 4.80 BLEU scores and 7.95 BLEU scores, where are too low to make sense.

It would be nice if the authors could discuss the applicability of the findings in the author response and the future version. And better yet, directly utilizing the findings to enhance existing NMT systems. Furthermore, the paper topic is too narrow, it would be much better to extend to other language generations tasks, like dialogue and QA (not a weakness but a suggestion). 


summary_of_the_review: This is an OK paper but the current version is not exciting, which could hardly attract attention from the community. The authors should answer the question: how could the paper findings benefit the community?
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper empirically investigates the scaling and transfer of Transformer language model architectures (i.e., a unified LM architecture rather than encoder-decoder models) for machine translation. The paper examines choices of architectural designs wrt data scales and model sizes and evaluates by performance (BLEU) on bilingual, multilingual and zero-shot translation, as well as the power-law (Kaplan+20) of neural language models.  Experiments and analyses are extensive and findings are interesting. 


main_review: **Strengths**

1. The empirical evaluations of this paper are comprehensive and solid, the results and findings are interesting. 
2. The paper is well-organized and clearly written. 


**Weaknesses**

1. Despite good empirical efforts, the motivation of this paper seems somewhat unclear. Given that the encoder-decoder paradigm dominantly governs machine translation (also from the experiment part of this paper that LMs underperform EncDec most of the time.), for what reason should we need to consider a shift to a unified language model for such a seq2seq task? If the zero-shot transfer is the case, why not directly fix the off-target issue for EncDec and preserve the good of translation performance, instead of changing the paradigm? I feel like the authors didn't convey an incentive for this. 
2. The paper conducted extensive experiments to show how scaling affects LMs for MT, however, few suggestions based on the findings are given for future development of MT. 
3. The setting of CausalLM seems a bit weird that a unidirectional encoding behavior makes obviously no sense. 

**Questions**
1. Section 5 seems to mainly examine p and L_{\inf} In equation (7), whereas \alpha remains undetermined. How was the value of \alpha determined for diagrams in Figure 3? Get fitted from Figure 2 I guess? 

summary_of_the_review: This paper is a good empirical work investigating the unified LM for machine translation and shows some good findings. However, the motivation yet remains unclear, plus few insights are delivered. I feel like the paper is not bad in terms of empirical study, but feel a bit not interesting regarding the bar of ICLR. I tend to consider it as a borderline paper but are willing to change my mind if I did miss something upon the author response.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: There are more and more work using language model to conduct translation between languages. This paper examines scaling and transferring laws when implementing translation with LM architecture through extensive experiments. Some of these findings are straightforward and intuitive but not experimentally verified before, and some are not so obvious. The latter is especially important for the community. These findings include: LMs are generally inferior to the EncDec architecture in terms of MT performance; it does not have the advantage of speed either; while on the zero-shot transferring aspect, LMs often performs better than EncDec models; different LM architectures have different scaling properties, among which architecture differences usually have a significant impact on the performance of small-scale models, but the performance gap narrows as the number of parameters increases, and so on. We can find more in the paper.

main_review: Strengths: 
The authors delivered these findings via extensive experiments on different language pairs, data scales, model sizes, and translation task settings (bilingual, multilingual, zero-shot, etc.), so these findings have good reliability and generalization. Some of the findings are inspiring to the follow-up research on this topic.

Weaknesses: 
Some experiments require more careful design to make findings more reliable. Such as, in Figure 2(d), the Test Bleu of EndDec, CasualLM-Deep, and Prefix-TopOnly-Deep decrease as the parameters increase in the interval of #Params 0.19-0.43E8. This phenomenon is not in line with the general trend, also the intuition. I suspect that the experimental settings could be improved. Othervise, the authors need to give a reasonable explanation.

There is another question I would like to promote a discussion. That is how to measure the transferring capabilities. The conclusions drawn from Figure 6 is that LMs do not really facilitate the transfer to low-resource languages, because LM-based models do not outperform EncDec model in all #params scales. However, we cannot see the improvement each model gain through transferring over the baseline. These relative improvements over the baseline (not the absolute scores) may be more appropriate to measure the transferring capabilities.
summary_of_the_review: This paper has conducted extensive experiments to examine the scaling and transferring laws of LMs for machine translation and has concluded several interesting findings which could be inspiring to the future work.
review_title: null

Meta-review: This paper has conducted extensive experiments to examine the scaling and transferring laws of LMs for machine translation and has concluded several interesting findings which could be inspiring to the future work.

The main concerns from reviewers are that the novelty of this paper is not enough. In addition, the experiments are not well-designed and the clarity of this paper can be further improved. We hope the reviews can help authors improve their paper.
Decision: Reject