Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work provides a large-scale empirical study of the scaling properties of multitask/multilingual neural machine translation models. The work examined the dependence of the scaling law parameters on the task weights and demonstrated that the scaling exponent and the irreducible loss are independent of the task weightings. Based on the above observations and analysis, the paper provided a novel joint scaling law that captures the scaling behavior across different model sizes and task weightings and used it to define the notion of the effective fraction of parameters assigned to a task. Through experiments, this quantity captures task interactions robustly and is surprisingly invariant to task similarity. Finally, the paper draft a procedure that uses fi to estimate task performance trade-off bounds for all model scales.
strength_and_weaknesses: Strengths: 
1.	The purpose of this work is to finally gain a clear understanding of the behavior of these large-scale multi-task models by performing a large-scale study of the properties of models trained to solve multiple tasks. 
2.	This work provides a comprehensive analysis of the proposed methodology, allowing the reader to gain a detailed understanding of the work.

Weaknesses:
(1) The designed experiments are not comprehensive enough and need to cover more scenarios. 
(2) The purpose and details of the experimental design need to be described more clearly.

Questions:
1.	For the effective number of parameters, do different translation languages have overlapping parameters? Is it possible to provide a demonstration of the parametric distribution in experiments to judge the effectiveness of the effective number of parameters? 
2.	For translation quality experiments, using ChrF and BLUERT are conventional choices. I suggest that the effects of settings of different values be shown as a case study. Through different cases, MT literature may show different characteristics, which can inspire future research.
3.	For Equation 12, how are c1, c2, and c3 obtained?  What effect do these three parameters have on fi?
4.	For the experiment, I suggest that a set of experiments can be added. There was no correlation between the groups of languages in the translation task in this experiment. Its purpose is to test the robustness of the proposed method.

clarity,_quality,_novelty_and_reproducibility: Please see the previous section.
summary_of_the_review: Please see the previous section.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: From the assumption that more and more models are proposed and evaluated in a multi-task setting, this paper proposes to explore the multi-task scaling laws, with a focus on multilingual machine translation. This focus is motivated by the abundance of benchmarks for this task and the existence on previous work about the scaling laws of standard bilingual NMT systems.

They trained 200 NMT systems. 
Among the contributions:
- they propose a method to determine the number of parameters allocated to each translation direction in the model.
- they provide a method to predict the full task performance given the model size
- they found that the relationship between languages doesn't impact much the scaling behavior of the model



strength_and_weaknesses: Strengths:
- the paper is mostly clear and well-written. I had no issue to undetstand it.
- the method and experiments are reproducible. I am confident that I would be able to reproduce the shape of the curves but may be not with similar points since the evaluation framework used is not detailed enough.
- the paper delivers a comprehensive analysis of the scaling of multilingual NMT.

Weaknesses:
- I find very confusing that in this work multingual NMT is multi-task. NMT, multilingual or not, has only one task: translate. A "task" in this paper is a translation direction. In other word, the paper isn't investigating the multi-task, but multilingual, scaling laws. This is a critical issue I think since I have no idea how the conclusions/observations from the paper could be applied to a truly multi-task model.
- The paper isn't motivated enough. Why scaling laws for bilingual NMT are expected to be different from multilingual NMT? What are the challenges? What are the research questions to be addressed here?
- The contributions over previous work are unclear to me. The methodology from previous work on the scaling laws of bilingual NMT is applied to multilingual NMT. Experiments are performed and analyzed, but I'm unclear how this work changes the methodology (does it?). Maybe a a more extended section on the scaling laws of bilingual NMT, highlighting the limits of previous studies addressed in this paper, would be necessary to help the reader to spot the contributions.
- The limit of this work are not clearly discussed.
clarity,_quality,_novelty_and_reproducibility: All the clarity, quality, novelty, and reproducibility strengths and weakenesses are detailed in the section Strength And Weaknesses.
summary_of_the_review: The paper needs better motivation and should fully assume that it is about multilinguality rather than multi-tasking.  I strongly disagree that multilingual models are multi-task models.
The analysis and observations made from the experiments are very interesting and would be insightful for future work on multilingual NMT. On the other hand, it is unclear how different is the methology used by previous work on scaling laws and whether it needed special adaptations to the multilingual scenario.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper showed several interesting findings:
1. showed that the scaling laws fits well to the empirical test-cross entropy performance of a model at the end of the training.
2. showed that not all the fitted coefficients of the scaling laws vary that much when we change the sampling rate.  Specifically, alpha and L seem to be relatively constant regardless of the sampling rate. I found this very hard to understand why it happens but it is a very interesting finding.
3. based on the finding, the authors proposed a joint modeling scaling multitask scaling that has (much) less number of parameters (i.e. alpha and L are independent of the task weights) that still captures well the scaling behavior.
4. studied the effective number of parameters with the assumption that the fitted coefficients of the scaling laws are task-independent, showing that the fraction of parameters allocated to task only depends on beta and alpha (beta is sampling-rate specific while alpha is not). 
5. sketched a procedure that estimates the task performance trade-off frontier for all model scales
strength_and_weaknesses: - Strength

This paper is with interesting findings and solid experiment results. 

- Weakness

While the paper contributes interesting findings, I found found the explanation and the practical terms of these findings are not that well-explained. For instance, the authors show that there is a relatively little change of some fitted coefficients of the scaling laws, but why does it happen and what possibly it means in practice term are discussed in a too short way. Another example is Equation 10. I am not sure why Equation 10 means the fraction of parameters is independent of the model size. I hope the authors would elaborate more these things in the final version of the paper. I am happy to raise my scores to 8 if these weakness are addressed.
clarity,_quality,_novelty_and_reproducibility: - The paper is with clear writing. There should be more elaborations on the observations though.
- Extra question - I am interesting in this technical question if possible: with a batch size of 500K tokens, do you see any improvement by training the model with 500K gradient steps? (e.g. compared to training the model with 70K steps with the same batch size, for instance).
summary_of_the_review: A good paper, recommend to be accepted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies scaling laws of multitask neural models in the context of multilingual neural machine translation (MNMT). Scaling is studied in the context of model parameter scaling, in the large data and computational resources condition. The authors show empirically that the scaling law derived from prior work is quite accurate in their context, and that some parameters are invariant to task weighting, while the scaling factor (beta) is not. They introduce the notion of effective number of parameters, which turns out to be useful to compare various training setups and again show some invariance of the scaling behaviour w.r.t. domain or task/target language combination. This also provides a way to address the issue of how to balance the tasks/languages during learning in order to reach the best overall performance.
strength_and_weaknesses: This is an enjoyable paper addressing the important issue of how MNMT systems learning operates. The paper is clearly written, there obviously is a massive amount of experimental work behind these results, which show impressive agreement with the theoretical models adopted. I learned a significant amount by reading this paper.

The main obvious weaknesses are actually addressed in the "Future work". For example, given the claimed purpose of the paper, it would have been better to see more than two "tasks". There are also some unresolved questions, for example whether the distinct behaviour of XX->En vs. En->XX models (Sec. 3.3, last par.) is due to the choice of English (a particularly simple language) or reflects some intrinsic multitask learning feature. 
clarity,_quality,_novelty_and_reproducibility: The paper is overall clear and readable. Some important terms such as "capacity splitting behavior" are not really introduced of referenced, but the meaning is quite straightforward. The performance trade-off curves discussed in 3.4 (p.8) are never properly defined--Although the notion is fairly intuitive, I would recommend defining the concept properly instead of wasting space on Fig. 1, which is neither described nor referenced.

The paper is well organized, covers the claimed observations progressively and convincingly, and does a good job conveying its many empirical findings. Those offer novel insights into MNMT training, as far as I am aware.

Strict reproducibility of the paper's experiments seems difficult because most of the data is not available and the exact constitution of the model is not clear. However, the idea of the paper is clear and it should be straightforward for someone "skilled in the art" to replicate these experiments on their data. In fact, given the limitations acknowledged in the "Future work", it would be useful to have the authors of others see how this generalizes to different contexts.

Misc:
(p.2) f_i(w) is used in the last observation before Sec. 2, but never introduced before. (could be introduced in the second observation)
"Figure 4 curves" (14 lines from bottom of p.5) are likely Figure 2 curves.
(p.7) "English is the target language" -> English as the target language
summary_of_the_review: The paper studies a simple but effective model of how neural models learn multilingual machine translation, offering several non-trivial findings along the way. This will be useful to researchers looking for the best way to train MNMT models, and may have further implications for multitask learning in general.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper describes a large-scale empirical study of scaling in multilingual machine translation. In particular, the paper studies the relation to model size and language direction weights on model performance. The paper finds out that language similarity has little effect on performance, but the system is affected by what side of the translation English stands on.

Most of the findings of the paper are not terribly surprising or novel, but it is nice to have them confirmed in a systematic study. The paper may help researchers who are training multilingual NMT systems. 

The paper attempts to generalize from multilingual NMT to multitask training, but that's a reach. A more relevant generalization is a multilingual system for a particular task. The inclusion of "multitask" is confusing. The paper also fails to make any deeper hypothesis and conclusions -- for example is English special and the results of it being on different sides of the translation relevant only to English language pairs. Lastly, most of the observations of the paper are already established in the field. 
justification_for_why_not_higher_score: There is little in the way of novel and useful knowledge in the paper. The discussion of "multitask" methods is confusing and negatively impacts the validity of the results. Finally, the paper does not go into more depth into some of the unresolved mysteries of multilingual MT such as is there something special about English.
Decision: Reject