Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies scaling laws for NMT. It confirms and extends some of the existing work on scaling laws, and as such it is a valuable contribution to the field. In particular, new questions addressed include: does the encoder-decoder architecture of NMT share the same scaling law as decoder-only LMs? How does training data (backtranslated or not) affect scaling? 
main_review: Pros:
- I really enjoyed reading this paper. The research questions are clearly stated, and the results are fascinating to go through. Well-done!
- The number of experiments is impressive. This adds weight to the papers conclusions, which are certainly thought-provoking. For example, the conclusion that scaling law should be bivariate, treating the number of encoder and decoder layers separately, makes me think we should not train models of symmetric number of layers anymore. 

Cons: 
- The footnote on comparison to Gordon et. al. in EMNLP2021 is too terse and should be expanded. That paper studied scaling laws for NMT as well, but in a different parameter regime. Do your conclusions affirm, reject, or extend theirs? 
summary_of_the_review: This is a strong empirical paper that extends our understanding of scaling laws in deep learning. The research questions are clear and the results are convincing. I think it will be cited often in the future.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides interesting scaling properties for autoregressive neural machine translation models, which are validated on comprehensive experiments. The authors investigate the scaling properties from three perspectives:
1. Quantify the evolution of model quality as a function of model capacity for encoder-decoder NMT models.
2. How the composition of training/test data affects the scaling behavior of NMT models?
3. How generation quality evolves with the improvements in cross-entropy resulting from model scaling?
main_review: Pros:
1. The proposed function of model capacity for encoder-decoder NMT model performance is useful for parameter allocation of NMT models training.
2. This paper provides experiments to explore how the composition of training/test data affects the test/training loss for scaling NMT models. It's an important question to connect the other two questions.
3. It is important to investigate how generation quality evolves with the improvements in cross-entropy resulting from model scaling. The problem is of great value for both NMT practice and research.


Cons:
1. The authors have proposed three research questions, while the relationship among them is not clear.
2. The scaling property of the composition bias of the train/test sets has been less studied in previous work, and is most interesting to the reviewer. This part can be more convincing if the authors can provide more experiments and analyses.
summary_of_the_review: The research questions explored in the study are important and interesting, e.g., the scaling properties of composition bias and the influence of source/target naturalness on the scaling behaviors. However, the relationship among these research questions and findings is not clear.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This paper presents a study on the scaling power of NMT Transformer (encoder-decoder).

It shows: 
- how the scaling of encoder-decoder compares to language model
- that NMT behaves very differently on translationese texts 
- propose to model the scaling of NMT encoder-decoder

main_review: 
Strengths:
- the research questions answered in this paper are interesting and help to better understand the mechanics of NMT. As an expert in MT, I really found this paper interesting.
- findings are very insightful
- this paper will be impactful: it will guide NMT  (or any other kind of encoder-decoder model such a BART) practictionners to build more efficient and effective models 
- another proof that target-original test sets are not suitable for NMT evaluation (Figure 8 should go on Twitter when you publish your paper!)
- evaluation is not only performed with BLEU (BLEURT is also used)


Weaknesses:
- almost the entire evaluation is not reproducible: undisclosed datasets, use of a non-standard BLEU implementation, pre-processing of datasets not detailed, etc.
- the paper misses a related work section (more details than in the introduction). The reader may not know related work on language models for instance
summary_of_the_review: All the "strengths" pointed above justifies my recommendation for the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides scaling laws for machine translation models. They experiment on different language pairs and domains and focus on large-scale Transformer-based NMT models.

They find that the scaling behavior of cross-entropy loss can be described as a bivariate function of encoder and decoder size and make recommendations for optimal allocation of encoder/decoder capacity. They also find that “translationese” can affect the model scaling behavior. Finally, they study the correlations between cross-entropy loss and BLEU/BLEURT scores.
main_review: Strengths:
1. They conduct extensive experiments on four language pairs and five different test domains.
2. It is interesting that contrary to previous findings, this paper suggests that it is better to scale the decoder than the encoder, providing guidance for future research on deep NMT.
3. The analysis on the translationese provides insights on the correlations between cross-entropy loss and generation quality, and its effect on the scaling laws.

Weaknesses:
1. The scaling laws have several test set-specific parameters, which makes it questionable if the laws have practical values. For example, if we want to get the optimal allocation of encoder/decoder capacity using their method, we have to train models of different variants and fit their proposed formula, so why don’t we just do parameter search with the same training budget?
2. I appreciate that the authors make efforts to measure the outputs in both BLEU and BLEURT, but it would be better to include human evaluation results that can better reflect the generation quality of models.
3. The paper mainly focuses on the relationship between cross-entropy loss and the number of Transformer layers in large-scale settings, while there are a lot of other factors that can be considered. For example, it is unclear whether the scaling laws will hold if the architecture varies or if the hidden size changes.

summary_of_the_review: While the paper has some minor issues, it conducts extensive experiments in large-scale settings and the results can provide insights for future research, thus I’m leaning towards an acceptance of the paper
review_title: null

Meta-review: This is a strong empirical paper that studies scaling laws for NMT in terms of several new aspects, such as the model quality as a function of the encoder and decoder sizes, and how the composition of data affects scaling, etc. The extensive empricial results offer new insights to the questions and provide valuable guidance for future research on deep NMT. The datasets used in the study are non-public, which may make it hard to reproduce the evaluation.
Decision: Accept (Spotlight)