Understanding Multi-Task Scaling in Machine Translation
Authors: Patrick Fernandes
Behrooz Ghorbani
Xavier Garcia
Markus Freitag
Orhan Firat
Venue: Submitted to ICLR 2023 (2023)
Keywords: scaling laws
machine translation
multilinguality
multi-task optimization

Abstract: In this work, we provide a large-scale empirical study of the scaling properties of multilingual (multitask) neural machine translation models. We examine how increases in the model size affect the model performance and investigate the role of the individual task weights on the scaling behavior. We find that these weights only affect the multiplicative factor of the scaling law and in particular, the scaling exponent is unaffected by them. Through a novel joint scaling law formulation, we compute the effective number of parameters allocated to each task and examine the role of language similarity in the scaling behavior of our models. We find minimal evidence that language similarity has any impact. In contrast, ``direction'' of the multilinguality plays a big role, with models translating from multiple languages into English having a larger number of effective parameters per task than their reversed counterparts. Finally, we leverage our observations to predict the performance of multilingual models trained with any language weighting at any scale, greatly reducing efforts required for task balancing in large multitask models. Our findings apply to both in-domain and out-of-domain test sets and to multiple evaluation metrics, such as ChrF and BLEURT.

Under review as a conference paper at ICLR 2023
UNDERSTANDING MULTI-TASK SCALING IN MACHINE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we provide a large-scale empirical study of the scaling properties of multilingual (multitask) neural machine translation models. We examine how increases in the model size affect the model performance and investigate the role of the individual task weights on the scaling behavior. We find that these weights only affect the multiplicative factor of the scaling law and in particular, the scaling exponent is unaffected by them. Through novel joint scaling law formulation, we compute the effective number of parameters allocated to each task and exam- ine the role of language similarity in the scaling behavior of our models. We find minimal evidence that language similarity has any impact. In contrast, "direction" of the multilinguality plays a significant role, with models translating from multi- ple languages into English having larger number of effective parameters per task than their reversed counterparts. Finally, we leverage our observations to predict the performance of multilingual models trained with any language weighting any scale, greatly reducing efforts required for task balancing in large multitask models. Our findings apply to both in-domain and out-of-domain test and multiple evaluation metrics, such as ChrF and BLEURT.
INTRODUCTION
Over the past few years, scaling has emerged as a popular and effective way to improve the perfor- mance of neural networks (Brown et al. 2020 Chowdhery et al. 2022 Lepikhin et al. 2020). Given the costs associated with training large ofate-of-thh-an neural models, much work has gone into un- derstanding their scaling properties and predicting the evolution of their performance with scale through scaling laws. Such scaling laws have been instrumental in guiding the model development efforts across variety of domains such as computer vision (Zhai et al. 2022), language modelling Kaplan et al. 2020 Hoffmann et al. 2022), and neural machine translation (Ghorbani et al. 2022).
Despite these impressive developments, of yet, most of the scaling laws studies available in the literature only focus on single-task models. On the contrary, current massive neural models are often trained to solve more than one task across
more modalities (Chowdhery et al.) 2022 Sanh et al.) 2022. Reed et al.] 2022). This disconnect from the current research frontier limits the applicability of the scaling laws guiding model development decisions. In particular, currently available scaling laws studies are unable to inform the decision process on how to balance the different tasks effectively at training time. Without such guidance, practitioners often have rely on cumbersome and costly approaches such as approximate grid search to inform their decision- making. Such approaches quickly become infeasible as the problem scale grows.
In this paper, we take the initial step towards developing quantitative understanding of the scaling behavior for multitask models. We choose multilingual neural machine translation (MNMT) as the setup for this initial study. This choice is motivated by several reasons: MNMT provides a popular setup with mature benchmarks and substantial literature on scaling (Lepikhin et al.) 2020 Costa-
jussà et al.) 2022 Bapna et al. 2022 Huang al. 2019). Moreover, recent results on scaling laws for
single-tat task MT models provide a natural starting point for our study (Ghorbani et al., 2022 Bansal
et al. 2022 Gordon et al. 202 Zhang et al. 2022). Finally, recent findings the optimization dynamics of MNMT models greatly simplify our study by removing the need to examine the role of the optimization algorithm in our results (Xin et al. 2022.

---

Under review as a conference paper at ICLR 2023
For our analysis, we train over 200 MNMT models (ranging from 20M to 1B non-embedding param- eters) and systematically examine their scaling behaviors. We focus our investigation on the data rich-compute rich regime where (i.e. language pairs and the model is trained to near convergence. Here, the main bottleneck in the model performance due to the lack of model capacity. We establish the following observations:
have access to vast amounts of training data for all the tasks
For each fixed task i and task weighting w, the evolution of the test cross-entropy loss (C) with model size (N) follows scaling law that resembles the scaling behavior single-task models:
LiNN; un) R / (.....)
T:20
Furthermore, we find that changes in the task weightings only affect the multiplicative factor B. The scaling exponent and the irreducible loss Lee are unaffected by these changes. In other words, scaling multi-task models will improve their performance in task at the same rate independently of its weight on the optimization objective.
We leverage these findings to propose scaling law that jointly predicts the performance for all tasks and weightings considered, and use it to examine how the model splits its capacity in between the tasks by computing the effective number of parameters allocated to each task (subsection 3.3)
We examine the popular belief that training multilingual models in similar languages is more effective than training models in unrelated languages. Surprisingly, for the high- resource language pairs considered, we don't t observe any significant differences in the scal- ing behavior of models trained translate from English into related languages (En (De, Fr}) with models trained in unrelated languages (En (De. Zh}). In contrast, we observe that models trained to translate from multiple languages into English (XX-=nn) benefit much more from multitasking compared to trained on translation out of English (En->XX).
In Section 3.4, we use simple approximations to fi (w)) to provide scaling law that predicts the full task performance trade-off frontier as a function of the model size (See Figure We describe how these predictions can be utilized for guiding task balancing in the development of massive models.
2 BACKGROUND
2.1 NEURAL SCALING LAWS
Recent research suggests that the performance of large neural models is well-predicted by smooth function of the fundamental problem parameters: the model size N the size of the training data D, and the amount of compute used for training C' (Hestness et al. 2017 Rosenfeld et al. 2019 Kaplan et al., 2020 Hernandez et al., [02T). The most relevant of these studies to ours is Ghorbani et al. (2022) where the authors study the effects of increasing the model size for single-task NMT models in the data-rich (D + x)), compute-rich (C - oo) regime. In this setting, the authors show that the following bivariate law describes the scaling behavior of encoder-decoder Transformers
C(N,, Na) = PN. N L.
Here, No and N₂ correspond the number of parameters in the encoder and decoder respectively and Loo corresponds to the irreducible loss associated with the task. {3, Pe, Pa, Lox ) are the param- eters of the scaling law that need to be empirically estimated from the data.
a et al. (2022) examine the question of optimally allocating parameters between In addition, Ghorbani
the encoder and the decoder. They show that in order to observe the optimal scaling behavior, one needs to proportionally scale the encoder and the decoder together. Under such scaling scheme,
Equation 2 simplifies to
L(N) = 3N + Loos
Ussin machine translation terminology, all language pairs are high-resource.
Following the literature conventions, we only consider the non-embedding layers when computing N.
2

---

Under review as a conference paper at ICLR 2023
which is similar to the scaling behavior observed in other domains such as computer vision Zhai et al. 2022) and autoregressive generative models (Henighan et al. 2020).
Based on these results, to achieve the optimal scaling behavior, adopt the proportional encoder- decoder scaling scheme for our experiments. A detailed overview of the size and architecture of our models is presented in Appendix
2.2 MULTITASK OPTIMIZATION
We focus our investigation on the supervised learning setup where the model parameters 0 E R2 are trained K different tasks simultaneously. multilingual MT, each task corresponds to translation for a different language pair. We denote the loss associated with task with (0).
Multitask models are often trained by minimizing a convex combination of the per-task losses:
K
((w) - arg min Σ wi w,Ci(0)
where w
> 0,
w w1=1
44)
1=1
==0
Here, w is a fixed vector of the task weights, determined apriori by the practitioner to emphasize her preferences on the balancing of the tasks. This so-called scalarization approach highly popular in the community due to its effectiveness and simplicity In fact, despite this simplicity, recent results on multitask optimization suggest that scalarization achieves performances on par or better than bespoke optimizers designed specifically for multitask models (Xin et al. 2022 Kurin et al. 2022).
In current large text models, such explicit scalarization is rare. Instead, scalarization is often im- plemented implicitly, by sampling observations from each task proportionally to that task's weight. Proportional sampling produces (in expectation) the same overall loss function explicit scalariza- tion but with much less engineering complexity.
3 EFFECTS OF SCALE IN MULTILINGUAL MT
EXPERIMENTAL SETUP
We use the (pre-LN) encoder-decodes Transformer architec- ture our models (Xiong et al.) 2020 Vaswani et al.) 2017). We train models of up to 8 sizes, approximately ranging from 20M to 1B (non-embeddingg parameters. When scaling encoder-decoder Transformers, to achieve the optimal scaling behavior, we scale the encoder and the decoder proportionally by increasing the model dimension and the number of layers in tandem. See Appendixx A for detailed overview.
Performance Trade-Off for Data-Rich Regime
1.0
0.9
0.8
2 Lose
0.7
0.5
For our experiments, we train two cohorts of models: En-+XX and XX-En. For En-XX cohort, we train multilingual model for translation from English (German (De), Chinese cohort,
0.4
0.3
0.4
0.6
0.8
1.0
Task
(Zh)) and {German (De), French (Fr)). For we present results for (De, Zh} Enn
Figure 1: Cartoon representation of the performance trade-off fron- tier for a hypothetical model.
We use the implicit scalarization approach train models; each observation the training batch chosen from first language pair with probability p and the second language pair with probability p. For our experiments, we choose from the set
p {0, 0.05, 0.1, 0.1, 0.5, 0.7, 0.95,
(5)
For En-+XX models, avoid confusing the model, we prepend language token the source sentence specifying the target language (e.g. <2de>). The models are trained using a per-token
*See (Boyd& & Vandenberghe 2004) for more detailed discussion of scalarization.
"To emphasize the fact that sampling-basen scalarization, we replace with in notation.
3

---

Under review as a conference paper at ICLR 2023
cross-entropy loss and the Adafactor optimizer (Shazeer & Stern 2018), using fixed batch size of 500K tokens. To mirror the compute-rich regime as closely as possible, we trained our models to near convergence. In practice, this translates to training our smaller models (< 500M parameters) for 500K gradient steps and our larger models for 1M steps.
To place our models in the data-rich regime, training our models. We filter this data using an online data selection procedure (Wang et al.) a (0188) and high-quality web-domain reference sets, extracting 600M sentences for each language pair. We tokenize this corpus by using a pretrained multilingual SentencePiece Kudo (2018) vocabulary, with a size of 128K sub-words.
massive in-house web-crawled dataset for
We measure the performance of models on both in-domain and out-of-domain test sets. For the in-domain test set, we extract 2000 sentences from the same in-house datasets used create the training (ensuring no overlap). For out-of-domain, we use newstest2019 (Barrault et al. 2019) consisting of 2000 sentence-pain extracted from aligned news documents.
3.2 RESULTS & ANALYSIS
Understanding Multitask Scaling We start our analysis by independently examining the model scaling behavior for each individual task weighting p in ( For each choice of p. we fit scaling law of the form
PN: N Spit + I.p..))
to the empirical (test) performance of models resulting from that task weighting.
Figure presents our findings for En-, Zh} models. Each point on the graph corresponds to the empirical test-cross entropy performance of a model at the end of the training We can see that our per-task-weighting laws are able to capture the scaling behavior of our multilingual models on both language pairs. As expected, when the weight for one of the languages is decreased, the performance of the models on that language decreases for scales. Our results suggest that the benefits of the increased model size for MNMT models well-described by power-law. See Appendix B for similar results for other language pair combinations.
English German 2R²= 99.98, JL 4/. 0.006)
English Chinese (R 99.97, |L -L]. 0.000)
DE only
pewerlaws
poweriaws empirical held out
1.4
empirical held-out
1.4
50/50
1.0
1.2
Test
1.0
0.8
0.4
1.0
0.0
0.2
0.4
only
+ Parameters (non-embeddings)
1e9
Parameters (non-embeddingg)
1e9
Figure 2: The evolution of the (in-domain) test cross-entropy loss with model size for (De Zh} models, as well the fitted scaling laws. These scaling laws are fitted separately for each task weighting. The color represents the weighting of the languages. The scaling laws are able to capture close 100% of the variation the data for both language pairs. Note that we don't show the zero-shot behavior.
Figure shows the fitted coefficients of the scaling laws for all p. The shaded area marks the one standard deviation uncertainty interval of our estimates Interestingly, we find that, across all values
For low probability tasks, apply convergence correction procedure make up for slow convergence. See the Apppendix for more details.
"We gauge the uncertainty in the cocfficients by examining the fluctuations in our estimates if our empirical Lid datapoints are perturbed by ./(0, o2). We choose conservative of 1% of the observed empirical loss for each data point.
4

---

Under review as a conference paper at ICLR 2023
English - German (R* 99.988, 8,, [L-4]- 0.006)
English - Chinese (R² 39.97, []-- 0.008)
DE only
10"
powerlaws
powerlaws empirical
empirical
10a
held-out
held-out
0.7
0.8
Entropy
0.6
0.6
50/50
0.4
2H
10a
10
10a (non- embedeling)
10*
only
Parameters mon-emmbdddings)
Figure 3: Log-log plot of the evolution the (in-domain) test cross-entropy loss as we scale. We subtract constant T jointly fitted for all the task weights (Fquation 7). All lines are nearly parallel, suggesting that the scaling exponent is unchanged for all p.
English - German
English Chinese
1.4
1.2
1.2
L.
1.0
Eitte Coatheient
0.8
0.6
0.6
0.4
0.2
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.3
1.0
Language Weight
Language Weight (p)
Figure 4: Coefficient values for German (left) and Chinese (right) as a function of the language weight, with the shaded region representing the standard deviation. The dashed lines represent the value of jointly fitted coefficients from Equation 7
of p. both the scaling exponent (a) and the irreducible loss (Coo) seem to be relatively unchanged. In particular, of our estimated a and Lx parameters are within two standard deviations of each other. In contrast, the multiplicative factor 3 seems to be highly sensitive to the choice of p.
Figure visually confirms the assertion that for our models ap and T.oo are effectively constant. Here, have subtracted fixed constant - from all the Figure 4 curves corresponding to the task We then plot results on log-log axes. As the figure suggests, the lines are all near parallel, suggesting that the scaling exponent is unchanged for all p. In practical terms this means that, for example, doubling the capacity of multitask model will reduce its loss by the same 2a factor, whether trained with 0.1 0.9 task weight. This also means that single-task scaling laws be used to gauge the benefits of scaling multitask models.
Jointly Modeling Multitask Scaling Based on the findings above, make the assumption that the scaling exponents and the irreducible losses are independent of the task weights, and propose joint scaling law of the form
L((ND) Ba L
Figure 5 shows the fit of this joint scaling law for En En-+(De. 7h) models evaluated on the in-domain test sets. Note that here, we fit total of 10 parameters for each task 8 for 8p, 's and two for as and ( contrast, in Figure 2 we used 24 overall parameters to capture the scaling behavior for each task. Despite this significant decrease in the number of total fitted parameters, observe that
5

---

Under review as a conference paper at ICLR 2023
joint laws able to almost completely capture the scaling behavior. We observe similar phe- nomenon for that the joint law accurately describes the scaling behavior of MNMT models.
test sets and other language pairs (see) Appendix C), further suggesting
English German (R*= 99.84, 0.026)
English Chinese (R*
0.030)
joint powerlaw empirical
only
empirical held-out
held-cut
1.6
Test Cress-Entrpp
1.2
14
1.0
0.8
only
Parameters (non-embeddings)
1c9
Parameters (non-embeedinng)
1c9
The joint scaling law of Equation 7 closely captures the scaling behavior of En (De, 7h} models. Test loss here is evaluated on in-domain test sets. See Appendix Cfor similar observations on [De, Fr} and (De, Zh} Zh models.
Figure 5:
3.3 EFFECTIVE NETWORK CAPACITY FOR MULTITASK MODELS
We leverage our joint scaling law to examine how MNMT models split their capacity in between the different tasks. We start by defining the notion of the effective number of parameters:
Definition. Consider a multitask model in which a task has been trained with weight We define the effective number of parameters allocated to i, N(G.)) to be equal the number of parameters necessary for single-task task model solety trained the multitask model.
reach the same (test loss) performance
(i.p)) Neff can be written as the solution of the equation
Mathematically,
C((N:)) CN(N (t.p), 1).
(8)
off
simple derivation yields that]
B1,t
N (i.p)
N.
(9)
off
Bp
Crucially, our calculations suggest that the fraction of parameters allocated to task i, which we denote by f((),, is independent of the model size:
f(()) =N((p)))) /N
31,5
at
(10)
3p.s
This observation yields a fundamental, scale-independent quantity that can be leveraged for under- standing the interactions between the different tasks in MNMT models.
Figure 6 shows the empirically estimated effective parameter ratios for models. Several obser- vations are order:
Consistency Across Domains: In Figure (left), we compare the capacity splitting behavior of the models on in-domain and out-of-domain (newstest19) test sets. Even though the scaling laws coefficients for in-domain and out-of-domain test sets differ, observe that the capacity splitting behavior mostly unchanged with different test sets. These findings hint at some measure of univer- sality across test domains how MNMT models divide their capacity and share their parameters.
See Appendix for details.

---

Under review as a conference paper at ICLR 2023
In-Domain vs Out-of-Domain
Different Target Language Combinatiors
XX - on -XX
100
0.8
0.8
0.8
0.6
Na.
N₂t
0.4
N6.4
0.4
Chinese Underaiint
Chinese jen (ee-
Chinese (eo (de.
German Immonieac11]
German (en (ele, R331
0.2
German (Col. 203 -an]
Chinese (nowsitec11)
French (en P31
Chinese (dl,, ab) -e0)
0.0
0.0
0.0
0.0
0.2
9.1
0.6
0.6
1.0
0.5
0.2
0.4
D.
0.6
1.0
0.0
0.2
0.4
0.5
1.5
Language Weight
Language Waight
Language Weight (6)
Figure 6: The effective fraction of parameters allocated to each task as estimated by our laws. Left: Comparison of the capacity splitting behavior of (De, Zh} models for in-domain and out-of-domain test sets. We observe minimal differences between the two setups. Center: Comparison of the capacity splitting behavior for EDe, Zh} and En En-(De, Fr} models. We don't observe any changes in the interaction between the tasks based language similarity. Right: Comparison of the capacity splitting behavior for translation to and from English. XX-FEn exhibit more synergy among the tasks.
our joint scaling
Consistency Across Languages Pairs: In Figure 6 (center), we compare the capacity splitting behavior of En- Zh} and En-> Fr] models. The conventional wisdom in the MT literature suggests that the tasks En-> (De, Fr] should exhibit a more positive interaction with each other compared En En-> Zh}. This is often justified by the intuition that representations aligned related languages and more aligned representations will encourage parameter sharing (Dabre al. 2017). Surprisingly, our results suggest that the interaction dynamics in En=+(De, Fr) and En-> (De Zhf models are not significantly different. In both settings, we observe a relatively neutral multitask behavior the performance MNMT of size N trained on task with (sampling) weight p is essentially similar to single-task model of size pN. In other words, there is minimal synergy among the tasks in both setups.
more
En-XX vs XX- En: In (right), we compare the interaction between the tasks when trans- lating out of English vs when translating to English. In stark contrast to the En-xXX setting, when translating into English, we observe significant positive synergy among the tasks. This observation aligns well with recent results the literature showing multilingual models achieving SOTA perfor- mance for translation to English (Chowdhery al., 2022 Lepikhin et al., 2020). It is unclear if this synergy arises as a specificity of having English is the target language or because multi-task encod- ing is intrinsically more amenable to parameter sharing than multi-task decoding. Understanding the exact dynamics giving rise to such positive interaction between the task is an exciting open question.
3.4 GUIDING TASK BALANCING
As discussed in the introduction, one of the areas where multitask (multilingual) scaling laws can be most impactful is in guiding task balancing/weiightng when training large multitask models, an open problem that has been studied extensively (Aharoni et al. 2019 Wang et al. (020). However, in its current form, our (joint) scaling law can only be use to decide between weightings that were for used for fitting it and cannot be used to predict performance on new, unseen weightings, as Sp. needs to be estimated empirically.
To extend to unseen task weightings, we instead focus estimating fi(+). Given access to Se(p), ac- curate prediction of (NV for any weighting can achieved by using the single-task scaling law:
C.(N:)) (f..())))))) +L.)
(11)
As observed in Section 3.3 f. f((p) has series of desirable properties that makes it easy to estimate: (i) it is invariant to test set and languages, (ii) it is smooth and generally well-behaved. As such, one can achieve accurate approximation of with just few data points.

---

Under review as a conference paper at ICLR 2023
Effective Number of Parameters
Predicted Performance Trade-off Frontiers
N=
1.0
German
joint powerlaw
Chinese
approx
1.8
0.8
y=x
Cross-Entroon (Chinese)
1.6
Nar
1.2
0.2
1.0
0.0
0.8
N= 200
0.8
0.6
1.4 Test Cross-Eniropy (German)
Language Weight (p)
Approximate joint scaling laws described by equations [I] and 12 almost perfectly capture the task interactions across all scales. Left: The fitted approximation f described in Equa- tion 12 Right: The predicted performance trade-off frontier (dashed lines) as well as the empirically observed trade-off values.
Figure 7:
We utilize this methodology to estimate the full task performance trade-off frontier for En-+ On-+(De (De. Zh} models. For estimating fi(-), we fit an approximate joint scaling law of the form Equation 11 where fi(-) is parameterized as
= - ppp
(12)
with C1 C22 C3 being fitted coefficients. Figure 7 demonstrates our results; our procedure able to almost perfectly capture the full task performance frontier across variety of model scales. With ac- cess to such accurate predictions of the performance frontier, practitioner can precisely determine how to weigh the individual tasks during the training based on her preference and target model size.
We should note that the choice of function class to fit fa(:) highly dependent on the practitioner's computational budget. In our case, we prioritized accuracy and used flexible function class of the form (2) for fitting. Such flexibility comes with cost of needing to compute more empirical values to reliably estimate f(-). In the scenarios with more limited computational budget, we have observed that even rudimentary linear approximations f are able to provide accurate representa- tions of the performance frontier. See Appendix E for examples.
Translation Quality Finally, we note that in the MT literature, quality is often measured via metrics such as BLEU (Papineni al., 2002), ChrlF (Popović 2015) and BLEURT (Sellam al., 2020) opposed to cross-entropy. since the latter doesn't i account for the problem of decoding translations from the models and sometimes found to not correlate with human preferences (Koehn & Knowles 2017). As such, MT practitioners might be concerned regarding the applicability of these results for practical applications. To ensure that our findings also apply to the quality of translations, we decode translations from our trained models using beam search (Graves) 2012) and evaluate how their quality changes as we scale the models, using ChrF and BLEURT.
Figure 8 (left) shows cross-entropy and ChrFF scores for the En->e language pair of our (D- Frj models, evaluated on the in-domain test set. find that this automatic metric has an almost- linear relationship with cross-entropy, hinting that our observations also generalize from cross- entropy to generation quality. Figure 8 (right) also shows the predicted ChrF performance trade- off frontier obtained by fitting joint scaling law (Equation 7 to the ChrF performance the in-domain test set (parametrizing the effective parameter fraction function as in in/Euatiio 12). Our procedure is able capture this trade-off frontier almost well the cross-entropy frontier. Sim- ilar findings for the BLEURT metric on out-of-distee -ii-tr--iitiittiion test sets can be found in Appendix F
8

---

Under review as a conference paper at ICLR 2023
ChrF Crass Entropy Correlation for German
Predicted Chrf Trade-Off Frantiers
-18
empircall held-ou
0.675
0.66
Ohi (Chinese)
0.450
0.425
0.62
0.400
0.350
0.58
0.325
0.8
1.0
2.4
only
0.54 0.56
0.58
0.50 0.62
0.64
0.66
0.68
No -20M
Test CrowseEntropy (German)
Ohff (German)
Figure 8: The generation quality behavior of our models as measured by ChrF. Left: We observe consistent positive correlations between ChrF and cross-entropy loss. Right: Our scaling laws can be used to generate accurate performance trade-off frontiers for ChrF.
4 CONCLUSIONS & FUTURE WORK
Current state-of-the-art large neural models are moving towards using as much data from as many domains and modalities as possible unlock exciting new capabilities. Unfortunately, as of yet, the research community does not have a clear understanding of the behavior of these multitask models scale. This in turn slows down the model development process since practitioners have to resort to trial and error for balancing their tasks in their models. In this paper, we attempted to take an initial step towards alleviating this problem by performing a large-scale study of the properties of models trained to solve multiple task.
In particular, we attempted to study this problem from the lens of multilingual machine translation. We showed that, for each task and each task weighting, power-law describes the evolution of the model test performance as a function of the model size. We examined the dependence of the scaling law parameters irreducible loss are independent of the task weightings. Using these observations, we provided a novel joint scaling law that succinetly captures the scaling behavior across different model sizes and task weightings and used it to define the notion of effective fraction of parameters assigned to a task (fa())). We showed that this quantity robustly captures the task interactions and is surprisingly invariant to the similarity of the tasks. In the end, we sketched a procedure to use fa to estimate the task performance trade-off frontier for all model scales.
the task weights and demonstrated that the scaling exponent and the
Future Work In this paper, we attempted to study the scaling behavior of multitask models. In order to keep our investigation tractable, we focused our study on MNMT models. Examining whether the conclusions of work apply to setups beyond translation is promising research direction. In the experiments presented in the paper, we focused only on the two-task scenario. We believe the presented results should be easily extendable to the multitask setup. We leave this to future work. Finally, to simplify the model scaling behavior, we focused our analysis to the data rich setup. However, in many applications, at least some of the tasks are mid- or low-resource. Extending these results to such scenarios is an interesting future direction.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Roce Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume / (Long and Short Papers), pp. 3874-3884 Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10.18653/v//119 1388. /RLLittppss/aclaannoooo..g
Yamini Bansal, B. Ghorbani, Ankush Garg, Biao Zhang, Maxim Krikun, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data scaling laws in nmt: The effect of noise and architecture. In ICMI., 2022.
Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Meng- meng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Axel- rod, Jason Riesa, Yuan Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apurva Shah, Yanping Huang, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. Building machine translation systems for the next thousand languages, 2022. URL https::/arxxis http:///axxiv..o///////////// owg//bgg//sooo.ooo
2205.0393
Loic Barrault, Ondfej Bojar, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philiip Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 con- ference on machine translation (WMT19). In Proceedings of the Fourth Conference on Ma- chine Translation (Volume 2: Shared Task Papers, Day 1), pp. 1-61, Florence, Italy, Au- gust 2019. Association for Computational Linguistics. doi: 10. 10.18653/11//11-53300 URL
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
doi: 10.1017/CB097705 | 1804441.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry. Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL /ttps::/proocedingg..r neurips...e..ppppp. http::/ww...enriiioocccaaa/////a wttpc//pppppee///E1... 1457c0d6bfcbb 9674 8bfb8ac1 42 42 42 .dd
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hycontuck Lim, Barret Zoph. Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022. URL wttphhtpp///m //axivv.//riii..//a owg. abs/ / 220022222222 02311
Marta R Costa-jussà, James Cross, Onur Çelebi. Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. 2022.
Raj Dabre, Fabien Cromieres, and Sadao Kurohashi. Enabling multi-source neural machine transla- tion by concatenating source sentences in multiple languages. In Proceedings of Machine Trans- lation Summit XVI: Research Track, pp. 96-107, Nagoya Japan, September 18- September 22
2017. httptt//p www.hhoolll......//... 2211 mt summi tommii-paapeee -aapers.e
10

---

Under review as a conference paper at ICLR 2023
B. Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier García, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. ArXiv, abs/2109.07740, 2022.
Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5915-5922, Online and Punta Cana, Dominican Republic, November 2021. Associatin for Computational Linguistics. doi: 10. 16633/v222222 wwtp:/02...mmmmmmm.mmmmmmmm URL https:://aclanttooloogg. org. wttp///e...mmmmmmmmmaa..
478
Alex Graves. Sequence transduction with recurrent neural networks, 2012.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. (tt::2000..401,, 2020.
Danny Hernandez, Jared Kaplan, T. J. Henighan, and Sam McCandlish. Scaling laws for transfer. ArXiv, abs/2102.01293, 2021.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv: 2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae. Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. URL wwt.hhttp//cccctm
2203 15556
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
Mareus Hutter. Learning curve theory. CoRR, abs/2102.04074, 2021. URL https://arxiv
org/ exq/abs/// awwaabsl0000 04074
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08366 2020. /ttp:/tt////
Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation. In Pro- ceedings of the First Workshop Neural Machine Translation, pp. 28-39, Vancouver, Au- gust 2017. Association for Computational Linguistics. doi:
URL
/ttpss////claathhllooo wwwp/rrg//11.. -
Taku Kudo. Subword regularization: Improving neural network translation models with multi- ple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 66-75, Melhourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. (1885533v/PPP 1007 URL https: //aclanthology.ozgg/////
1007
Vitaly Kurin, Alessandro De Palma, Ilya Kostrikov, Shimon Whiteson, and M Pawan Kumar. In de- fense of the unitary scalarization for deep multi-task learning, arXiv preprint arXiv:2201.044 0422,, 2022.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu. Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
11

---

Under review as a conference paper at ICLR 2023
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic
evaluation of machine translation. In Proceedings of the 40th Anmual Meeting of the Associa- tion for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/10730831107333... URL https:
Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the
Tenth Workshop on Statistical Machine Translation, pp. 392-395, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10.18653/11W115-30099 URL https:
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06/75. 2022.
Jonathan Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In International Conference Learning Representa- tions, 2019.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Repre- sentations, 2022. URL
net/ http://wwwwneetttuoneee//----/ Wow
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguisties, pp. 7881-7892, Online, July 2020. Association for Computational Linguis- tics. doi: 1..18863/1//222..cll 704. URL vel-main... 704
Noam M. Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. ArXiv, abs/1804.04235, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. Denoising neural machine translation training with trusted data and online data selection. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 133-143, Brussels, Belgium, October 2018 Association for Computational Linguistics. doi: 11653//1/W18-633 URL httpp:://aclanthoooooy... wtg.:/g.o...g...o. /W18 6314
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. Balancing training for multilingual neural ma- chine translation. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pp. 8526 8537, Online, July 2020. Association for Computational Linguis- URL htttps://aclantttete orgplooggyoog//0......
10200//11220..lmmmmm.. acl main. 754
tics. doi:
Derrick Xin, Behrooz Ghorbani, Ankush Garg, Orhan Firat, and Justin Gilmer. Do current multi- task optimization methods in deep learning even help? Advances in neural information processing systems, 2022.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp. 10533. PMLR, 2020.
12

---

Under review as a conference paper at ICLR 2023
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp-
12104-12115 12113, 2022.
Biao Zhang, Behrooz Ghorbani, Ankur Bapna, Yong Cheng, Xavier Garcia, Jonathan Shen, and Orhan Firat. Examining scaling and transfer of language model architectures for machine trans- lation, arxiv preprint arXiv:2202.00528, 2022.
13

---

Under review as a conference paper at ICLR 2023
A
MODEL SIZES AND HYPERPARAMETERS
Enc. Lavers Dec. Layers Embh Dim Heads
Head Dim
MLP dim
Vocab Size
#
Corrected Parameters
512
2048
128k
149,953.024
18,881,024
768
12
64
3072
128k
260,322.816
63,714,816
6
768
12
64
3072
128k
324,035, 328
127,427,
3072
128k
387.747,840
191.139.840
9
9
1024
16
64
4096
128k
601.931,776
339,787,776
12
115.193.144
453,049.344
12
12
1280
16
80
5120
128k
1,035,876,80 864
707, 707,869,,11 184
12
1536
16
96
6144
128k
LA12,528,,11
1,019,312.128
B INDIVIDUAL SCALING LAWS FITS
B. OUT-OF-DOMAIN
English - German (R² 99.95, |L [L- -= 0.017)
English Chinese (R² - -4).-(00000
0.011)
DE
1.8
poweriaws
poweriaws
only
2.0
Newsterii Cross Entropy
1.6
empirical
empirical
held-out
1.8
held-out
1.4
1.2
1.4
50/50
1.2
1.0
0.8
only
# Parameters (non-embeddings)
1c9
Parameters (non-embeddings)
1c9
The evolution with model size of the cross-entropy loss the newstest19 test set for (ne, Fr) models, as well as the fitted scaling laws. The color represents the weighting of the languages. Note that don't show the zero-shot behavior.
Figure 9:
English German (newstest19)
English Chinese (newstest19)
2,00
Fitedd Coefficiont: Values
1.4
1.50
1.25
1.0
1.00
0.8
0.75
0.4
0.50
0.2
0.6
0.8
0.2
0.6
0.8
Language Weight
Language Weight (p)
Figure 10: Coefficient values, for scaling laws fitted on newstest2019, for German (left) and French (right) as function of the language weight, with the shaded region representing the standard devia- tion. The dashed lines represent the value of jointly fitted coefficients from Equation
B.2 ENGL.IHH FRENCH
B.3 GERMAN,
14

---

Under review as a conference paper at ICLR 2023
English German (R² 99.88, [[-4]- 0.001)
English French (R² 99.82, -4 00.01)
1.5
only
power laws empincal
powerlews empirical
1.4
Test Cross Entropy
1.3
1.2
1.1
50/50
1.0
1.0
0.9
0.8
0.8
0.7
FR
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
only
(non-embeddings)
1e9
(nor-embbeddings)
1e9
Figure 11: The evolution the (in-domain) test cross-entropy loss with model size for En En-+ Fr} models, as well as the fitted scaling laws. The color represents the weighting of the languages. Note that we don't show the zero-shot behavior.
English - German
English - French
1.2
1.2
Fitted Confficient Values
0.8
0.8
0.6
0.6
0.4
0.2
0.8
1.0
0.4
0.6
Language Weight
Language Weight (p)
Figure 12:
Coefficient values for German (left) and French (right) as function of the language weight, with the shaded region representing the standard deviation. The dashed lines represent the value jointly fitted coefficients from Eqqation 7
German English =R²= 99.91, JL - -
= 0.007)
Chinese - Englishh (R² 99.888, [4-4]_=000 0.014)
DE only
1.15
powerlaws empirical
powerlaws empirical
1.10
Crozs Entropy
1.05
1.00
11
0.95
50/50
0.90
1.0
0.85
0.80
0.9
0.75
only
Parameters (non-ent embemdrredig))
1e9
Parameters (non embeediigg)))
1e9
Figure 13: The evolution of the (in-domain) test cross-entropy loss with model size for (De, Zh)-En models, well as the fitted scaling laws. The color represents the weighting of the languages. Note that we don't show the zero-shol behavior.
15

---

Under review as a conference paper at ICLR 2023
German + English
Chinese - Engllsh
2.0
2.0
Fitted Confiiiiott Values
1.5
1.5
10
1.0
0.5
0.5
0.0
0.0
-05
-0.5
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Language Weight
Language Weight
Figure 14: Coefficient values for German (left) and French (right) into English as a function of the language weight, with the shaded region representing the standard deviation. The dashed lines represent the value of jointly fitted coefficients from Equation
16

---

Under review as a conference paper at ICLR 2023
C JOINT SCALING LAW FITS
C.1
OUT-OF-DOMAIN
English German (R² 99.68, 1L-4. 0.042)
Engliih Chinese (R² 99.71. 4 0.054)
only
joint poweria av empinical held-out
joint powerlaw empirical held-out
2,0
Newsterr1 Crocs- Entroy
1.6
1.8
91
1.4
50/50
1.2
1.0
0.8
0.2
0.0
0.2
0.8
1.0
only
Parameters (nan-embedddngg ngs)
1c9
Parameters (nan-embeddlnngs)
109
Figure 15: The joint scaling law [Equation 7) fitted to models trained for En En- Zh} models. Test loss here is evaluated on the newstest2019 test set.
C.2 ENGLISH-+(GERMEN FRENCH]
English - German (R² 99.75. L]. 0.021)
English - French (R2 = 19.73.|1-44. 0.0199
DE only
joint powerlaw empirical
1.4
joint powerlaw empirical
1.4
1.3
Test Loss-kntropy Cros-Entropy
1.2
50/50
1.0
0.8
0.8
0.7
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
only
Parameters (nan-embeddinngs)
109
Parameters (non-ombeddings))
Figure 16:
The joint scaling law (Equation fitted models trained for En (De, Fr) models. Test loss here is evaluated on in-domain test sets.
C.3
(GERMAN, CHINESE) ENGLISH
17

---

Under review as a conference paper at ICLR 2023
German - English (R² 99.72, (4-41==000 0.011)
Chinese English (R² =9.662. (9) =.0.001999
joint poweriaw empirical
1.3
joint powerlaw empirical
Conyy
1.2
Test Crozs Entropy
1.00
0.95
0.90
0.85
0.80
0.9
0.75
only
Parameters (non-embeddings)
1e9
# Parameters (non-embeddings)
1e9
Figure 17: The joint scaling law Equation 7) fitted to models trained for (De, Zh} En models. Test loss here is evaluated in-domain test sets.
18

---

Under review as a conference paper at ICLR 2023
D DERIVATION OF THE EFFECTIVE NUMBER OF PARAMETERS
L (N::)) = 8p,iN
+
Ap.i
B1
N
+
L
$1,3
Pp.
BI,
N
+ (
B1,1
f1.
B1.
N
+ [
pp.s
Ntt di +
(i)
C.(Norr p)
E OTHER APPROXIMATIONS TO THE EFFECTIVE PARAMETER RATIO
We use linear approximation of the form
f((p) = cp - 1) + 1.
(13)
Effective Number of Parameters
Predicted Performance Trade-off off Frontiers
N # -18
Gemman
powerlaw
Chinese
1.8
joint poweriaw
approx
0.8
y-x
Test Cros-Entrpy (Chinese)
2.6
0.6
1.4
N 0.4
1.2
0.2
20M
0.0
0.4
0.6
0.8
1.6
Language Weight (p)
Test Cross-Entropy (German)
Figure 18: Approximate joint scaling laws described by equations II] and 13 is able to capture the task interactions across all scales well, even with single fitted coefficient for ratio function. Left: The fitted approximation f described in Equation 12 Right: The predicted performance trade-off frontier (dashed lines) as well the empirically observed trade-off values.
F TRANSLATION QUALITY
19

---

Under review as a conference paper at ICLR 2023
BLEURT-Cross Entropy Correlation for German
DE only
Predicted BLEURT Pareto Frontiers
0.80
empiniall
joint poweriaw
0.78
0.78
Nowstert BLEURT (Carmnn
Newstest19 (Chinese)
0.76
0.76
0.74
0.72
0.70
0.68
0.66
0.64
2H only
0.66
N8 -00
0.8
1.0 1.2
0.60
0.65
0.70
0.75
0.80
Newstest19 Cross Entropy (Germani
Newstest18 BLEURT (German)
Figure 19: (left) shows cross-entropy and BLEURT scores for the En-*De language pair of our En (ne, Fr) models, evaluated on the newstest19 test set. We find that this automatic metric has an cemnost-lineat relationship with cross-entropy. hinting that our observations also generalize from cross-entropy to generation quality. Figure 8 (right) also shows the predicted BLEURT performance trade-off frontier obtained by fitting our joint scaling law (Equation 7 to the BLEURT performance on the newstest 9 test set (parametrizing the effective parameter fraction function as in inuuuttton 12).
G CONVERGENCE CORRECTION
Due to implicil scalarization, models trained with very little task weight (< 0.1) will see less than a full epoch of that task's data, even when trained with 1M steps. our experiments saw that this was causing problems in the fit the scaling laws due to an undertraining of our largest models.
To mitigate this problem without training these models for prohibitively large number of steps, we apply recent findings in learning curve (Hutter 2021 to estimate the performance of largest models trained with p 0.05 task weight at convergence, by fitting power-law the performance evolution as training progresses, and predicting the performance of these models at 2.5M steps. This only affect two models per scenario considered.
20


Reviews:
[{"review_id": "t1fzaWhdvX", "reviewer": "Reviewer_3K3H", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper showed several interesting findings:\n1. showed that the scaling laws fits well to the empirical test-cross entropy performance of a model at the end of the training.\n2. showed that not all the fitted coefficients of the scaling laws vary that much when we change the sampling rate.  Specifically, alpha and L seem to be relatively constant regardless of the sampling rate. I found this very hard to understand why it happens but it is a very interesting finding.\n3. based on the finding, the authors proposed a joint modeling scaling multitask scaling that has (much) less number of parameters (i.e. alpha and L are independent of the task weights) that still captures well the scaling behavior.\n4. studied the effective number of parameters with the assumption that the fitted coefficients of the scaling laws are task-independent, showing that the fraction of parameters allocated to task only depends on beta and alpha (beta is sampling-rate specific while alpha is not). \n5. sketched a procedure that estimates the task performance trade-off frontier for all model scales", "summary_of_the_review": "A good paper, recommend to be accepted.", "strength_and_weaknesses": "- Strength\n\nThis paper is with interesting findings and solid experiment results. \n\n- Weakness\n\nWhile the paper contributes interesting findings, I found found the explanation and the practical terms of these findings are not that well-explained. For instance, the authors show that there is a relatively little change of some fitted coefficients of the scaling laws, but why does it happen and what possibly it means in practice term are discussed in a too short way. Another example is Equation 10. I am not sure why Equation 10 means the fraction of parameters is independent of the model size. I hope the authors would elaborate more these things in the final version of the paper. I am happy to raise my scores to 8 if these weakness are addressed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "khTfICCm3J", "reviewer": "Reviewer_JKLj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies scaling laws of multitask neural models in the context of multilingual neural machine translation (MNMT). Scaling is studied in the context of model parameter scaling, in the large data and computational resources condition. The authors show empirically that the scaling law derived from prior work is quite accurate in their context, and that some parameters are invariant to task weighting, while the scaling factor (beta) is not. They introduce the notion of effective number of parameters, which turns out to be useful to compare various training setups and again show some invariance of the scaling behaviour w.r.t. domain or task/target language combination. This also provides a way to address the issue of how to balance the tasks/languages during learning in order to reach the best overall performance.", "summary_of_the_review": "The paper studies a simple but effective model of how neural models learn multilingual machine translation, offering several non-trivial findings along the way. This will be useful to researchers looking for the best way to train MNMT models, and may have further implications for multitask learning in general.", "strength_and_weaknesses": "This is an enjoyable paper addressing the important issue of how MNMT systems learning operates. The paper is clearly written, there obviously is a massive amount of experimental work behind these results, which show impressive agreement with the theoretical models adopted. I learned a significant amount by reading this paper.\n\nThe main obvious weaknesses are actually addressed in the \"Future work\". For example, given the claimed purpose of the paper, it would have been better to see more than two \"tasks\". There are also some unresolved questions, for example whether the distinct behaviour of XX->En vs. En->XX models (Sec. 3.3, last par.) is due to the choice of English (a particularly simple language) or reflects some intrinsic multitask learning feature. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ikNny8xa-l", "reviewer": "Reviewer_TBUV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work provides a large-scale empirical study of the scaling properties of multitask/multilingual neural machine translation models. The work examined the dependence of the scaling law parameters on the task weights and demonstrated that the scaling exponent and the irreducible loss are independent of the task weightings. Based on the above observations and analysis, the paper provided a novel joint scaling law that captures the scaling behavior across different model sizes and task weightings and used it to define the notion of the effective fraction of parameters assigned to a task. Through experiments, this quantity captures task interactions robustly and is surprisingly invariant to task similarity. Finally, the paper draft a procedure that uses fi to estimate task performance trade-off bounds for all model scales.", "summary_of_the_review": "Please see the previous section.", "strength_and_weaknesses": "Strengths: \n1.\tThe purpose of this work is to finally gain a clear understanding of the behavior of these large-scale multi-task models by performing a large-scale study of the properties of models trained to solve multiple tasks. \n2.\tThis work provides a comprehensive analysis of the proposed methodology, allowing the reader to gain a detailed understanding of the work.\n\nWeaknesses:\n(1) The designed experiments are not comprehensive enough and need to cover more scenarios. \n(2) The purpose and details of the experimental design need to be described more clearly.\n\nQuestions:\n1.\tFor the effective number of parameters, do different translation languages have overlapping parameters? Is it possible to provide a demonstration of the parametric distribution in experiments to judge the effectiveness of the effective number of parameters? \n2.\tFor translation quality experiments, using ChrF and BLUERT are conventional choices. I suggest that the effects of settings of different values be shown as a case study. Through different cases, MT literature may show different characteristics, which can inspire future research.\n3.\tFor Equation 12, how are c1, c2, and c3 obtained?  What effect do these three parameters have on fi?\n4.\tFor the experiment, I suggest that a set of experiments can be added. There was no correlation between the groups of languages in the translation task in this experiment. Its purpose is to test the robustness of the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9P836jdKvcY", "reviewer": "Reviewer_49op", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "From the assumption that more and more models are proposed and evaluated in a multi-task setting, this paper proposes to explore the multi-task scaling laws, with a focus on multilingual machine translation. This focus is motivated by the abundance of benchmarks for this task and the existence on previous work about the scaling laws of standard bilingual NMT systems.\n\nThey trained 200 NMT systems. \nAmong the contributions:\n- they propose a method to determine the number of parameters allocated to each translation direction in the model.\n- they provide a method to predict the full task performance given the model size\n- they found that the relationship between languages doesn't impact much the scaling behavior of the model\n\n\n", "summary_of_the_review": "The paper needs better motivation and should fully assume that it is about multilinguality rather than multi-tasking.  I strongly disagree that multilingual models are multi-task models.\nThe analysis and observations made from the experiments are very interesting and would be insightful for future work on multilingual NMT. On the other hand, it is unclear how different is the methology used by previous work on scaling laws and whether it needed special adaptations to the multilingual scenario.\n", "strength_and_weaknesses": "Strengths:\n- the paper is mostly clear and well-written. I had no issue to undetstand it.\n- the method and experiments are reproducible. I am confident that I would be able to reproduce the shape of the curves but may be not with similar points since the evaluation framework used is not detailed enough.\n- the paper delivers a comprehensive analysis of the scaling of multilingual NMT.\n\nWeaknesses:\n- I find very confusing that in this work multingual NMT is multi-task. NMT, multilingual or not, has only one task: translate. A \"task\" in this paper is a translation direction. In other word, the paper isn't investigating the multi-task, but multilingual, scaling laws. This is a critical issue I think since I have no idea how the conclusions/observations from the paper could be applied to a truly multi-task model.\n- The paper isn't motivated enough. Why scaling laws for bilingual NMT are expected to be different from multilingual NMT? What are the challenges? What are the research questions to be addressed here?\n- The contributions over previous work are unclear to me. The methodology from previous work on the scaling laws of bilingual NMT is applied to multilingual NMT. Experiments are performed and analyzed, but I'm unclear how this work changes the methodology (does it?). Maybe a a more extended section on the scaling laws of bilingual NMT, highlighting the limits of previous studies addressed in this paper, would be necessary to help the reader to spot the contributions.\n- The limit of this work are not clearly discussed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject