Title: Understanding Multi-Task Scaling in Machine Translation
Authors: Patrick Fernandes
Behrooz Ghorbani
Xavier Garcia
Markus Freitag
Orhan Firat
Venue: ICLR 2023
Keywords: scaling laws
machine translation
multilinguality
multi-task optimization

Abstract: In this work, we provide a large-scale empirical study of the scaling properties of multilingual (multitask) neural machine translation models. We examine how increases in the model size affect the model performance and investigate the role of the individual task weights on the scaling behavior. We find that these weights only affect the multiplicative factor of the scaling law and in particular, the scaling exponent is unaffected by them. Through a novel joint scaling law formulation, we compute the effective number of parameters allocated to each task and examine the role of language similarity in the scaling behavior of our models. We find minimal evidence that language similarity has any impact. In contrast, ``direction'' of the multilinguality plays a big role, with models translating from multiple languages into English having a larger number of effective parameters per task than their reversed counterparts. Finally, we leverage our observations to predict the performance of multilingual models trained with any language weighting at any scale, greatly reducing efforts required for task balancing in large multitask models. Our findings apply to both in-domain and out-of-domain test sets and to multiple evaluation metrics, such as ChrF and BLEURT.

TL;DR: We study the scaling behavior of multilingual, multi-task neural machine translation models. 

Review discussion:
This is an enjoyable paper addressing the important issue of how MNMT systems learning operates. The paper is clearly written, there obviously is a massive amount of experimental work behind these results, which show impressive agreement with the theoretical models adopted. I learned a significant amount by reading this paper.
The main obvious weaknesses are actually addressed in the "Future work".
For example, given the claimed purpose of the paper, it would have been better to see more than two "tasks". There are also some unresolved questions, for example whether the distinct behaviour of XX->En vs. En->XX models (Sec. 3.3, last par.) is due to the choice of English (a particularly simple language) or reflects some intrinsic multitask learning feature.
The paper is overall clear and readable. Some important terms such as "capacity splitting behavior" are not really introduced of referenced, but the meaning is quite straightforward. The performance trade-off curves discussed in 3.4 (p.8) are never properly defined--Although the notion is fairly intuitive, I would recommend defining the concept properly instead of wasting space on Fig. 1, which is neither described nor referenced.
The paper is well organized, covers the claimed observations progressively and convincingly, and does a good job conveying its many empirical findings. Those offer novel insights into MNMT training, as far as I am aware.
Strict reproducibility of the paper's experiments seems difficult because most of the data is not available and the exact constitution of the model is not clear.
However, the idea of the paper is clear and it should be straightforward for someone "skilled in the art" to replicate these experiments on their data.
In fact, given the limitations acknowledged in the "Future work", it would be useful to have the authors of others see how this generalizes to different contexts.
Misc:
(p.2) f_i(w) is used in the last observation before Sec. 2, but never introduced before. (could be introduced in the second observation)
"Figure 4 curves" (14 lines from bottom of p.5) are likely Figure 2 curves.
(p.7) "English is the target language" -> English as the target language
NO.
| “For example, given the claimed purpose of the paper, it would have been better to see more than two "tasks".
We agree that analyzing the setting where models are trained with more than two tasks is important. As an early effort to understand if our findings apply to more than two tasks, we trained various model sizes for to translate into *three* languages (EN->{DE,ZH,FR}), and compared the predictions using the scaling laws for models trained on two language-pairs (EN->{DE,ZH} and EN->{DE,FR}).
https://imgur.com/a/X5wOEwZ
Overall we find that (combination of) the joint scaling laws fitted on models trained on two language pairs predict well the performance of models trained for three language pairs, showing that the invariances found in previous sections generalize to settings with more than two tasks. These results also hint that computation of effective parameters counts for multi-task models with many tasks can be simplified and made more tractable by training models with much smaller subset of tasks.
We also show how the performance in german evolves as we change the probabilities in the other languages: https://imgur.com/a/rojVGhh
| “There are also some unresolved questions, for example whether the distinct behavior of XX->En vs. En->XX models (Sec. 3.3, last par.) is due to the choice of English (a particularly simple language) or reflects some intrinsic multitask learning feature.”
We agree that there are indeed some interesting questions regarding XX->En vs. En->XX. We believe that there is indeed a distinct behaviour caused by the differences between encoding vs decoding that are not only attributable to English as the target language, but leave a more extensive exploration of this question (by training multi-source models to languages other than English) to future work,
- Strength
This paper is with interesting findings and solid experiment results.
- Weakness
While the paper contributes interesting findings, I found found the explanation and the practical terms of these findings are not that well-explained.
For instance, the authors show that there is a relatively little change of some fitted coefficients of the scaling laws, but why does it happen and what possibly it means in practice term are discussed in a too short way.
Another example is Equation 10. I am not sure why Equation 10 means the fraction of parameters is independent of the model size. I hope the authors would elaborate more these things in the final version of the paper. I am happy to raise my scores to 8 if these weakness are addressed.
- The paper is with clear writing. There should be more elaborations on the observations though.
- Extra question - I am interesting in this technical question if possible: with a batch size of 500K tokens, do you see any improvement by training the model with 500K gradient steps? (e.g. compared to training the model with 70K steps with the same batch size, for instance).
NO.
|  “While the paper contributes interesting findings, I found found the explanation and the practical terms of these findings are not that well-explained.
For instance, the authors show that there is a relatively little change of some fitted coefficients of the scaling laws, but why does it happen and what possibly it means in practice term are discussed in a too short way”
We thank the reviewer for their comments. We fully agree that the empirical observations regarding constant scaling coefficients merit a more expansive discussion. We will further extend the discussion in Section 3.2 as follows:
“(...) As the figure suggests, the lines are all near parallel, suggesting that the scaling exponent is unchanged for all $p$.
This invariance in the estimates of $\\mathcal{L}_{\infty}$ and $\alpha$ has important implications: First, the fact that irreducible loss is independent of the sampling probability suggests that multi-task training does not significantly change the information content of the data for any of the tasks in the model. Given that we operate in the data-rich regime, this observation is not too surprising.
Nevertheless, this is a good sanity check to have.
Moreover, the invariance of the scaling exponents to the sampling probability suggests that, for high-resource tasks, simply scaling the model size is enough for delivering predictable gains across all the tasks in the model.
As such, attempting to achieve a perfect balance between the different tasks in the model might not be necessary at scale. As a concrete example, for our translation tasks, we observe the (reducible) test loss improves by a factor of $\approx 0.81$ with each doubling of the model size, regardless of the training data mixture.
Finally, given that even single-task models follow this very same trend, this also means that single-task scaling laws can be used to gauge the benefits of scaling multitask models."
|  “Another example is Equation 10. I am not sure why Equation 10 means the fraction of parameters is independent of the model size.
While the total number of effective parameters (Eq. 9) does indeed depend on the model size (since $N$ appears on the right-hand side), the **fraction** of effective parameters (Eq. 10) for a given task weighing $p$ is solely dependent on the scaling exponent $\alpha$ and scaling multipliers $\beta_p$ and $\beta_1$. Since these coefficients are constant for a given combination of tasks and independent of the model size, the fraction of effective parameters is also independent of model size and depends solely on the nature of the tasks and their interaction.
This means that if, for example, a mNMT model with 10M parameters has 60% effective parameters (6M) allocated to a particular language pair, then a model with 1B parameters trained for the same language-pairs/data will also have same 60% effective parameters (600M) associated with that same language pair.
We agree that this observation might not be clear from the current text. We will further elaborate on this point in an updated version of the paper.
|  “Extra question - I am interesting in this technical question if possible: with a batch size of 500K tokens, do you see any improvement by training the model with 500K gradient steps? (e.g. compared to training the model with 70K steps with the same batch size, for instance).”
We found that this is highly dependent on model size.
For example, for our smallest models (2-3 layers), there wouldn’t be much of a difference between training for 70k steps and 500k steps.
However bigger models require more steps for convergence, with 9L models requiring almost the 500k steps. This was the reason for increasing the number of training steps for larger models to 1M steps.
For reference, here is the cross-entropy evolution on both English->German and English->Chinese for a base (12L) model trained with 50% weight on each language-pair, where we can see that performance keeps improving way beyond the 70k steps.
https://imgur.com/a/uRhxPCf
Strengths:
- the paper is mostly clear and well-written. I had no issue to undetstand it.
- the method and experiments are reproducible. I am confident that I would be able to reproduce the shape of the curves but may be not with similar points since the evaluation framework used is not detailed enough.
- the paper delivers a comprehensive analysis of the scaling of multilingual NMT.
Weaknesses:
- I find very confusing that in this work multingual NMT is multi-task. NMT, multilingual or not, has only one task: translate. A "task" in this paper is a translation direction. In other word, the paper isn't investigating the multi-task, but multilingual, scaling laws. This is a critical issue I think since I have no idea how the conclusions/observations from the paper could be applied to a truly multi-task model.
- The paper isn't motivated enough. Why scaling laws for bilingual NMT are expected to be different from multilingual NMT? What are the challenges? What are the research questions to be addressed here?
- The contributions over previous work are unclear to me. The methodology from previous work on the scaling laws of bilingual NMT is applied to multilingual NMT. Experiments are performed and analyzed, but I'm unclear how this work changes the methodology (does it?). Maybe a a more extended section on the scaling laws of bilingual NMT, highlighting the limits of previous studies addressed in this paper, would be necessary to help the reader to spot the contributions.
- The limit of this work are not clearly discussed.
All the clarity, quality, novelty, and reproducibility strengths and weakenesses are detailed in the section Strength And Weaknesses.
NO.
We thank the reviewer for their comments.
|  “I find very confusing that in this work multingual NMT is multi-task. NMT, multilingual or not, has only one task: translate. A "task" in this paper is a translation direction. In other word, the paper isn't investigating the multi-task, but multilingual, scaling laws. This is a critical issue I think since I have no idea how the conclusions/observations from the paper could be applied to a truly multi-task model.”
Multilingual NMT has been framed as a multi-task optimization problem extensively in the past [1,2,3,4,5]. This stems from the fact that both learning to translate into/from two (or more) languages and learning two (or more) "real tasks" (e.g. translation and summarization) can both be seen as learning two (or more) different functions, and there is little difference in the mathematical formulation of the optimization problem.
However, we fully agree that there is a need to examine the scaling behavior of "truly" multi-task models. We set up this study as a first step in this path and we aim to explore more diverse setups in our future work.
[1] Luong et. al, 2015, Multi-task Sequence to Sequence Learning
[2] Firat et. al, 2016, Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism
[3] Arivazhagan et. al, 2019, Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
[4] Jean et. al, 2019, Adaptive Scheduling for Multi-Task Learning
[5] Wang et. al, 2020, Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
|  “The paper isn't motivated enough. Why scaling laws for bilingual NMT are expected to be different from multilingual NMT? What are the challenges? What are the research questions to be addressed here?”
Previous work around scaling laws for bilingual MT (or any other setting to our knowledge) studies how scale affects performance when we consider training/test data as single task, and if we considered the mNMT problem as single objective (say, by looking at the average cross-entropy on the different language pairs) we expect to see similar findings to previous work on scaling laws for bilingual MT. The key difference in our work comes from considering the training data not as a monolithic, single-task distribution with a single learning objective, but as the combination of several different tasks/language-pairs, and the learning problem as multi-objective optimization.
This comes with the added challenge of understanding how the weight of the different objectives (language pairs) affects the performance on the individual tasks/language-pairs, which we do by exploring different weightings in section 3. 2. This formulation also allows us to study how task similarity affects the scaling on the individual, something that wasn't possible with previous works.
|  “The contributions over previous work are unclear to me. The methodology from previous work on the scaling laws of bilingual NMT is applied to multilingual NMT. Experiments are performed and analyzed, but I'm unclear how this work changes the methodology (does it?). Maybe a a more extended section on the scaling laws of bilingual NMT, highlighting the limits of previous studies addressed in this paper, would be necessary to help the reader to spot the contributions.”
As mentioned, previous studies around scaling laws don’t consider the training objective as a multi-task optimization problem. This means that all the methodology surrounding the impact of the tasks’ weight on the loss is novel, including (1) the analysis of the impact of the task’s weight on scaling coefficients, (2) the proposed joint scaling law, (3) the analysis of the effective parameters for each tasks in a multi-task model and (4) the prediction of the full tasks’ performance trade-off frontier.
However, while we did mention that short-comings of previous work’s analysis in the introduction, we agree that the differences from previous work are not highlighted enough. We will include a more detailed description of these differences in the final version.
Strengths:
1.	The purpose of this work is to finally gain a clear understanding of the behavior of these large-scale multi-task models by performing a large-scale study of the properties of models trained to solve multiple tasks.
2.	This work provides a comprehensive analysis of the proposed methodology, allowing the reader to gain a detailed understanding of the work.
Weaknesses:
(1) The designed experiments are not comprehensive enough and need to cover more scenarios.
(2) The purpose and details of the experimental design need to be described more clearly.
Questions:
1.	For the effective number of parameters, do different translation languages have overlapping parameters? Is it possible to provide a demonstration of the parametric distribution in experiments to judge the effectiveness of the effective number of parameters?
2.	For translation quality experiments, using ChrF and BLUERT are conventional choices. I suggest that the effects of settings of different values be shown as a case study. Through different cases, MT literature may show different characteristics, which can inspire future research.
3.	For Equation 12, how are c1, c2, and c3 obtained?  What effect do these three parameters have on fi?
4.	For the experiment, I suggest that a set of experiments can be added. There was no correlation between the groups of languages in the translation task in this experiment. Its purpose is to test the robustness of the proposed method.
Please see the previous section.
NO.
We thank the reviewer for their comments.
| “The designed experiments are not comprehensive enough and need to cover more scenarios.”
What exact extra experiments would you like to see? It was unclear what these would cover and what questions they would help answer.
| “For the effective number of parameters, do different translation languages have overlapping parameters?”
The current formulation only allows calculating the effective number of parameters wrt a single-task model trained for a task/language-pair. In cases where both languages-pairs have high effective parameter ratio (say 75%/75% ratios for a model trained with 50%/50%) it’s natural to infer that this model must be sharing parameters between both tasks, but the scaling laws don’t allow predicting exactly what parameters are being shared.
| “Is it possible to provide a demonstration of the parametric distribution in experiments to judge the effectiveness of the effective number of parameters?”
Would it be possible to clarify what this distribution would be over? And how it would help measuring the “effectiveness” of the effective number of parameters?
| “For translation quality experiments, using ChrF and BLUERT are conventional choices. I suggest that the effects of settings of different values be shown as a case study. “
Is it possible to clarify what values we should vary and study for this case study?
| “For Equation 12, how are c1, c2, and c3 obtained? What effect do these three parameters have on fi?”
These parameters are fitted automatically along with the other parameters of the scaling law ($\alpha$, $\beta$, and $\mathcal{L}_\infty$). This functional form was inspired by the incomplete beta function, and the effects of these parameters are similar to the ones in the CDF of the beta distribution (see https://en.wikipedia.org/wiki/File:Beta_distribution_cdf.svg)
| “For the experiment, I suggest that a set of experiments can be added. There was no correlation between the groups of languages in the translation task in this experiment. Its purpose is to test the robustness of the proposed method.”
To clarify, would these extra experiments be with different language pairs? If so, we agree that they would certainly be useful, but given that we explored three different language-pair-combination setups, each with different properties (different languages, similar languages, and reverse direction), and the computation cost of each full setup, we leave the exploration of other setups to future work.

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: This paper describes a large-scale empirical study of scaling in multilingual machine translation. In particular, the paper studies the relation to model size and language direction weights on model performance. The paper finds out that language similarity has little effect on performance, but the system is affected by what side of the translation English stands on.

Most of the findings of the paper are not terribly surprising or novel, but it is nice to have them confirmed in a systematic study. The paper may help researchers who are training multilingual NMT systems. 

The paper attempts to generalize from multilingual NMT to multitask training, but that's a reach. A more relevant generalization is a multilingual system for a particular task. The inclusion of "multitask" is confusing. The paper also fails to make any deeper hypothesis and conclusions -- for example is English special and the results of it being on different sides of the translation relevant only to English language pairs. Lastly, most of the observations of the paper are already established in the field. 
justification_for_why_not_higher_score: There is little in the way of novel and useful knowledge in the paper. The discussion of "multitask" methods is confusing and negatively impacts the validity of the results. Finally, the paper does not go into more depth into some of the unresolved mysteries of multilingual MT such as is there something special about English.
justification_for_why_not_lower_score: N/A

Decision: Reject