On the Pareto Front of Multilingual Neural Machine Translation
Authors: Liang Chen
Shuming Ma
Dongdong Zhang
Furu Wei
Baobao Chang
Venue: NeurIPS 2023 poster (2023)
Keywords: Multilinugal Neural Machine Translation
Multitask Learning
Pareto Optimization
TL;DR: We propose the Double Power Law to predict the unique trade-off behavior in Multilingual NMT.

Abstract: In this work, we study how the performance of a given direction changes with its sampling ratio in Multilingual Neural Machine Translation (MNMT). By training over 200 multilingual models with various model sizes, data sizes, and language directions, we find it interesting that the performance of certain translation direction does not always improve with the increase of its weight in the multi-task optimization objective. Accordingly, scalarization method leads to a multitask trade-off front that deviates from the traditional Pareto front when there exists data imbalance in the training corpus, which poses a great challenge to improve the overall performance of all directions. Based on our observations, we propose the Double Power Law to predict the unique performance trade-off front in MNMT, which is robust across various languages, data adequacy, and the number of tasks. Finally, we formulate the sample ratio selection problem in MNMT as an optimization problem based on the Double Power Law. Extensive experiments show that it achieves better performance than temperature searching and gradient manipulation methods with only 1/5 to 1/2 of the total training budget. We release the code at https://github.com/pkunlp-icler/ParetoMNMT for reproduction.

On the Pareto Front of Multilingual Neural Machine Translation
Liang Chen
Shuming M Dongdong Zhang² Furu Wei? Baobao Chang"
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University
Microsoft Research
leo. wwang........m wwwn@outlook. com chbbēpku.edi edu. cn
{ (suumaa, dozhang, f
Abstract
In this work, study how the performance of given direction changes with its sampling ratio in Multilingual Neural Machine Translation (MNMT). By training over 200 multilingual models with various model sizes, data sizes, and language directions, we find it interesting that the performance certain translation direc- tion does not always improve with the increase of its weight in the multi-task optimization objective. Accordingly, scalarization method leads to a multitask trade-off front that deviates from the traditional Pareto front when there exists data imbalance in the training corpus, which poses a great challenge to improve the overall performance of all directions. Based on our observations, we pro- pose the Double Power Law to predict the unique performance trade-off front in MNMT, which is robust across various languages, data adequacy, and the number of tasks. Finally, we formulate the sample ratio selection problem in MNMT as optimization problem based the Double Power Law. In our experiments, it achieves better performance than temperature searching and gradient manipula- tion methods with only 1/5 1/2 of the total training budget. We release the code at
for reproduction.
Introduction
Multilingual Neural Machine Translation (MNMT) enables multiple translation directions in single model, significantly reducing deployment costs of translation systems and benefiting low-resource directions (Sennrich et all 20IS). MNMT can be framed we expect to improve the overall performance across translation directions. However, it is infeasible to find an optimal solution for all translation tasks because of the negative interference among trans- lation directions (L. cial 2019 all 2021; Wang et all, 1021). A common method to make trade-offs among different languages is scalarization, where combination of normalized weights is assigned to the loss of each direction during training. Xin et all (2(12)) empirically show that result in Pareto optimal solutions, where
multi-task learning problem, where
different weight combinations outperform another in all tasks. All such solutions could form a Pareto front as shown in Figure teranndss all (2123) show that there is a power law between the weight of one direction and its final generalization performance, indicating a positive relation between weight and performance.
solution could
In this work, empirically find that the positive relation together with the power law from Fermand at (2023) does not hold true in more realistic MNMT scenario where the data for different directions is highly imbalanced. The performance of direction may decrease its weight increases.
"Equal Contribution. Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

It raises our interests that 1) What causes the inconsistency phenomena? 2) Can we model the trade- off behavior among different directions? 3) How should we set the weights for each direction?
To answer these questions, we first identify the problem in MNMT that the scalarization method could not generate Pareto optimal solutions when there exist low-resource directions, which is named the collapse of Pareto front. Secondly, to quantify how the performance of a given direction changes with its sampling ratio, we propose the Double Power Law. The law takes the number of training data into consideration and explicitly models the relationship between capacity occupation and the potential of over-fitting given direction, which has the form:
((,,D)) (k (k.p) + (D7 + b) (q D)
M
Capacity Occupation
Intrinsic Over-fitting
Bins Term
where F; is the predicted performance of task i measured by cross-entropy loss, is the sampling
ratio, Dj is the number of training examples and M is constant bias term. The rest are fixed
parameters to estimate. Experiments justify the robustness of the law under various settings.
Based on the Double Power Law (DPL., we propose approach to compute the optimal sampling ratios. Our experiments demonstrate that DPL outperforms directly search for the optimal temper- ature and gradient manipulation methods for MNMT, achieving better accuracy (+0.3 BLEU) with only 1/5 to 1/2 training cost in our experiments.
To summarize, the contribution of this work is threefold:
1. We identify and analyze the collapse of Pareto Front phenomenon in MNMT.
2. We propose the Double Power Law to model the trade-off among different directions in MNMT, which is robust on languages, data adequacy, and number of directions.
3. Based on the Double Power Law, we propose method to automatically compute the op- timal sampling ratio combination, eliminating the need for manual tuning of the sampling temperature and achieving better performance in BLEU, COMET and BertScore metrics.
2 Collapse of Pareto Front in Multilingual Neural Machine Translation
2.1 Pareto Front in MNMT
Pareto Front In multi-task learning, where K is the number of objectives and (.) is the loss of task i under solution parameterized by 0 E R", we say solution 02 Pareto dominates solution 0, if 1 viiis K, (.2) < L. (0i) and 3 < < K, L,02) < Li (01). The Pareto front is set of solutions where no solution Pareto dominates another. As shown in Figure II, we give two toy examples of the possible shape of Pareto front when K and K 2.
K=3
K=2
Tast
9,
Toy examples of Pareto front. When there are three tasks (K=3), we color the Pareto front according to task 3's loss. We also plot the contours, which also Pareto fronts in 2-dimension.
Figure 1:
Scalarization (Boyd and Vandenberghe, 2004) is a traditional method to optimize the multi-task learning model where set of predefined normalized weights is applied to each task. If there is litle interaction among different tasks, that one task's loss would drop when increasing its weight, equation @ with different could guarantee solutions that can form Pareto front (Xin 2022).
((w) arg min L(O; w) s.t. L(0:w) = Σ w: C.), w > 0,
Σ w
(2)

---

Generulization Performance Trade-offf En (DeZb]
Generalization Performance
(Fr,2h)
Fraee
Parcto Froud
A0%
High High
High- High
High Low
High-Low
26
samyring
74.BLEU
24
samyling ratlo
ZH-LEE
24
22
22
89%
70%
40%
Jas,
65% 18% x
20 NM
100%
20
26 FR-GLEU
28
30
32
DE-BLEU
Figure 2: Generalization performance trade-off curves under different data distributions. The blue line stands for English (Germaa--......6... Chinese-10M) (left) and English-((French 1OM. Chinese-10M)rrighlee The orange line stands for the result when replacing the high-resource The Pareto front collapses (sampling ratio in-
10M to creases while performance drops) when changing the high-resource direction to a resouree one.
In -n (Fr-lligh De-lligh, Zh-Highh
En -e (Frsttigh, De-lligh, Zb-Low)
In - (Fr-ttigh, De-Low, 2h-l.ww)
Crasa
211 Dras
2H Dras-Drisp)
Figure 3: Generalization Performance trade-off curves under different data distributions under the 3-direction setting. The left figure shows Pareto front where one direction's 's performance will always decrease when another direction's 's performance increases. However, the performance of low- resource direction would increase in step with the high-resource direction's 's at some points.
Multi-task Learning for MNMT MNMT can be framed as a multi-task learning problem where one model can do the translation among multiple directions (Dong et all, 2015: Aharoni all 2019). To prevent the high-resource directions from dominating the training and to improve the low-resource directions' performance, scalarization is implemented through proportional sampling, where the sampling ratio of examples for task in a training batch is wy Currently, there is no consensus on how to directly specific w; for each task, temperature-based sampling more commonly used where w; Σ P)
p!/T 1r Pt denotes the ratio of the training examples from task i.
2.2 The Collapse of Pareto Front Widely Exists
We use datasets provided in WMT10 (Wang et all 20205) and WMT19 (Barrault et al, 2019) to conduct the MNMT experiment. The description of dataset listed in Appendix [ In the 2-task setting, we experiment with the sampling ratio of one task ranging from 10% to 90%. In the 3- task setting, we experiment with each task ranging from 10% to 80% resulting in 32 different ratio combinations. More detail about the training is in Appendix B. The experiment results and major findings in this section are illustrated in Figure A and
The collapse exists in different language pairs In Figure ZI we can see that the generalization per- formance tra situation is different between the High-High and High-Low data settings. Under the High-High setting, the solutions from scalarization could form Pareto front where one task's 's performance increases with the decrease the other task. This expected and also in line with the recent results from Xin et all (2022); Fernandes et all (2023). However, the Pareto front disappears when we replace the English Taglish-~Chinese Chinese- 10M with Anglish-CChiesse. restrict the directions to high- high-resouree ones they also get Pareto optimal results, Xxm ct
Fernandes et all (2023)
3

---

also find that there exists global optimal point in imbalanced multilingual translation however do not give it an explanation. Understanding the reason can help build better MNMT system since the multilingual training data is highly imbalanced in reality (Haddooaw et all, 2021: team et all, 2022).
In Figure D. when increasing the sampling ratio for one High- Righ-Resource direction, its performance continually goes up. While increasing the sampling ratio for one Low- Resource direction from 10% to 90%, its performance first goes up and then gradually goes down. This is the
Trade-off curves in models with different sizes
Data
High Low
Model Size
-- Base
Medium
direct cause of the collapse of Pareto front. The collapse also exists when there are more than two languages. As shown in Figure E we draw the 3-task trade-off front us- ing linear interpolation. The front in the left where all three directions are high-resource, is most similar to the example 3D Pareto front in Figure When there exists low-resource directions shown in the middle and the right Figure E, the Pareto front would disappear.
Longo
22
20
32.5
35.0
FA-BLEU
Figure 4: Performance (BLEU) trade- off curves for (Freinhh>(Frenchh Chi- nese) under different model sizes and data distributions.
It indicates that the relation between High and Low re- sources directions is not zero-sum game and there exists a sampling ratio range where different tasks can improve or deteriorate simultaneously. For example, in Figure D this range could be 40% to 90% sampling ratio for the low-resource Eoggish-to-Chineee direction.
The collapse exists in models with different sizes Figure @ shows the trade-off curves on models with different sizes. The Pareto front would collapse when there exist low-resource directions.
The collapse of Pareto front complicates the optimization of MNMT because the sampling ratio must be carefully set or the performance might fall in the collapsed area where the model's capacity is not fully exploited and there exist other solutions that can Pareto dominates the current performance.
2.3 Explore the Reason for the Collapse
The collapse is closely related to inadequate data. If only consider the performance of low- resource direction in Figure 2 the performance of the low-resource direction briefly improves with an increase in the sampling ratio from 10% 20%, but then gradually deteriorates the sampling ratio continues to increase from 20% to 90%. This is in contrast to the performance of high- directions, which continue improve with an increase in sampling ratio from 10% 90%.
The effect of increasing the sampling ratio for a task is twofold: on the one hand, it gives it more weight and the model would allocate more capacity to optimize its perfor- mance, on the other hand, it also increases its risk of over- fitting. For directions with abundant training examples, the over-fitting term is less significant its performance would keep improving with more sampling ratio. How- ever, for low-resource directions, the risk of over-fitting would be dominant for a high sampling ratio because of the lack of training examples. A small sampling ratio serves as a form of regularization for these low-resource directions. This difference accounts for the contrasting performance trends observed between low- resource and high-resource directions.
Ratio(De:Ili)
Encoder Last
Decuder Last
1:9
21.80/30.0 96
32,96/42. 98
11.87/20.12
25.88/33.59
9:1
10.43/14.48
23.88/27.88
Table 1: The sharpness for En- [De]
En En-(HH) tasks. The result indicates that with more high-resource data, the model tends to come to flatter local minima after training.
It has been shown that the model would be less likely over-fit and generalize better when it reaches a flatter local minimum after the gradient descent optimization (Keskar et all 2016 Klemberg et all 2018; Forei et all, 2I2I). To justify our assumption, following Keskar el all (2)))) we compute the partial local curvature i.c. sharpness of the optimized model through the trace of the Hessian matrix as shown in Equation E. where X stands for held-out evaluation data and Y stands for the label. We compute the sharpness of the FFN layer at different transformer layers.
Sharpness (fo, X, X,Y) Y) = fo
(3)

---

As shown in Table II we compute the sharpness of our MNMT model on English-+ Germann, Hindi] ) task with different sampling ratio combinations. The result shows that with more sampling ratio for high-resource direction, the model could reach a flatter local minimum for both tasks.
The results of the generalization performance and sharpness experiments indicate that adding weight for a certain translation direction is a double-edged sword. It forces the model to optimize more towards minimizing the training loss of this task, the same time increasing its over-fitting risk, especially for the low-resource directions.
3 The Double Power Law
In the previous section, we discussed the collapse of Pareto front phenomena and its impact on MNMT. The collapse presents challenge for determining the sampling ratio combination, a poor combination can adversely affect the final performance by deviating from potential Pareto solutions. Additionally, we identified over-fitting as sigmificant problem in MNMT training especially for low-resource directions. While straightforward solution would be to grid search the sampling ratio combination, becomes impractical with increasing directions due to the exponentially growing search space. In this section, we are going to answer the following questions:
Is there pattern behind the collapse phenomena? Can we predict the performance trade-off curve among different directions?
Fernandes et al (2023) adopt single power law inspired by the scaling literature in NMT (Ghorbani stall 202TE: Gordon etall (02ID to predict how the sampling ratio affect the performance MNMT. However, it assumes that all directions have unlimited data, which does not reflect the diverse data adequacy in reality for supervised MNMT training. Under Eernandes et al (2023))s 's power law, the trade-off curve of any two directions can form Pareto front, which could not capture the collapse.
We find out that: the generalization performance F of the i-th direction in MNMT follows a double power law, which takes the sampling ratio p and the number of training examples D; as input:
FV D.) (k 1 + (D().) (q p)* + +
(4)
Capacity Oscupation
Over-fitting
where F; is measured through evaluation cross-entropy loss. k at q, B are fixed parameters, (D( is monotonically decreasing function of the number of training examples D:. _ and 0 are bias terms that are relevant to the direction and the model.
3.1 Capacity Occupation and Intrinsic Over-fitting Terms
Based on our previous discussion, the generalization performance for certain direction not only depends on its sampling ratio but also on its number of training examples. So we set two terms to jointly model the generalization performance, namely Capacity Occupation term and Intrinsic Over- fitting term. As shown Equation A similar to the power law in Fernandes all (2023), Capacity Occupation term reflect that with larger sampling ratio, the model would assign more capacity for the task to minimize its training loss, which could possibly improve the generalization perfor- mance. Intrinsic Over-fitting term reflects the fact that with a larger sampling ratio, the model is more likely to over-fit on this task, which would harm its generalization performance. Meanwhile, the scale of Intrinsic Over-fitting term negatively related to the number of training examples. With fewer training examples, the influence of Intrinsic Over-fitting would be more significant.
3.2 Parameter Estimation
We estimate the parameters of the double power law in three steps. We conduct a series of English~ (french-10M. German-260Rg experiments with English-Freenc--10n sampling ra- tio ranging from 10% to 90%. 1) We estimate the parameters of the Capacity Occupation term results by ignoring the Intrinsic Over-fitiing term because it has minor ef- with English-PFrench fect when D. large for high-resource direction. 2) We fix the parameters of the Capacity Oc- cupation term and estimate the parameters of the Intrinsic Over-fitting term with the results of
5

---

EN-2/FR-IONM ZH-260K)
EN--(ER-IOM, 2HO-)
(N- ZH-IMM
EN- (OM, ZH- 2H-
Generalization Cross Einrroyy
EN-o-FR-IOM. HI- 260K)
EN- 1RM- [LL)660K)
ENo (FR-LOM, DE-166M)
ENo LOM, DE
pnsart Orea Fringy
mncuttt Caso Fareys
Curve
Observation
Figure 5: Evaluation of Equation under different languages and data-adequacy settings. The curve predicts how one direction's 's generalization performance changes with its sampling ratio. The Double Power Law is robust under different scenarios.
English- -----GG-mmmm000 3) We fix all parameters in Equation except for the value of (D). We suppose that (D) also subjects to the form of power law and conduct the same series of ex- periments on English { (French- I0M, German- IM) and English~ (French- 10M, German-4.6MM get values of G (D) given different D; and estimate the parameters in (D). Thus, the double power law could be summarized below, where _ stands for the sum of bias terms
(1) and O!
Fi(p D.)
(k- (k.p) Capacity Occupation
+ (D] +b (q p)
(5)
Intrinsic Over-fitting
Bias Term
Following the three steps, estimate" the parameters of the double power law for base-size model. For convenience, we present the estimated double power law below:
F. D) (0.07 -0.20 + (D-0.33 0.33 0.50) (1.18 p) MLO
(6)
The law indicates an interesting result: the generalization performance trade-off among directions in MNMT is mostly relative to the training data size in each direction. The difference in languages only influences the bias term.
3.3 Evaluation and Discussion
We evaluate the estimated double power law in different task and data size combinations as shown in Figure 5. For different task settings, we only tune the bias term ( in Equation R As shown in Figure 5L the double power law can generally well reflect how the performance one task changes with its sampling ratio for both of the high-resource and low-resource directions.
Generalization to more directions We also evaluate the double power law when there are more than 2 languages. We conduct a three-direction MNMT experiment on English-> [French-10M, German-4.6M, Chinese-260K), where English-> Chinese is the low-resource direction. We fix the sampling ratio of English~> German to 0.1 and change the sampling ratio of English French from 0.1 to 0.8 and tune the bias term (1) in Equation & to fit the performance curve of English-> French and English~> Chinese. As shown in Figure the double power law can also well reflect the performance trade-off with more directions.
we scipy.optimize.cure fit function from the scipy library. The unit of training examples is million.

---

EN-x(FR-IOM, DE-4,6M, ZH-260K]
EN-> (N--(FR- DE-4,6M, ZH-200K)
Entropy
EN- Cross LRDaR
P(DE--011
piDE
30
43"
Generaizaaionn
23-
4.0
30-
Curve
Oberxation
Figure We examine Equation Bunder the 3-direction setting. Double Power Law can well describe the performance trend of both the high-resource (FR-10M) and low-resource directions (ZH-260K).
The trade-off curve is language-agnostic As shown in Figure Fand & by adjusting the bias term in Equation the predicted curve can well describe the trade-off with different languages setting, both the similar(FR and DE) and the distant (FR and ZH) ones. In other words, the shape of the trade-off curve more related to the number of training examples in each direction rather than the language of the direction, which mostly influences the bias term of the curve. This result is in line with Xin et all (2022); Shaham et all (2022) that the training data size plays a more important role than language similarity the trade-off of different directions' performance.
Critical point for low-resource directions When look at the predicted performance curve and the observations for low (ow-resource directions, it all shows a "U" shape. It means there exists a critical point when increasing the sampling ratio for low-resource direction. On the right side of the critical point, the Capacity Occupation term has more influence on the generalization performance. The Intrinsic Over-fitting term becomes dominant after the sampling ratio goes beyond the critical point. While for high-resource directions, the curve is monotonic.
3.4 Application of the Double Power Law
We adopt the Double Power Law to compute the optimal sampling ratio combination for MNMT. For task with n directions, is evaluated by the weighted sum of each direction's 's performance. r; the weight of the i-th task and Σ 1.
Given the evaluation metrics' weights r and the number of training examples in each task d, the goal is to compute the optimal sampling ratio combination p, where we assume that all directions follow the double power law in Equation !! Thus, the optimization objective becomes:
arg min d( s.t. C(p:r; (nF(ddd)) p>0, Σp:=1 =1 (7)
4 Experiments
4.1
Settings
We conduct experiments on a two-direction English + |German-4666 Hindi-260K) task and four-direction English + [French-1OM, German-4.66 Chinese-260K, Hindi-260K) task using the same training configuration as in Section 2.2 in our base size model.
We two different evaluation metrics: 1) Arithmetic mean (r; 1). 2) Customized metric where we are interested in improving the performance of low-resource direction i (r: = 1, Tiyy = 0).
We solve Equation @ with the estimated double power law as in Equation G. Even though we do not know the bias term Mxe in Equation E. does influence the computation the optimal sampling ratio combination P. With the computed p. train the MNMT model according to Appendix (H

---

(a) 4-direction (Arithmetic mean)
Methods
Cross-Entrops 1
BLEU
TrainingTime
FR
DE
ZH
AVG
FR
AVG
Temperature baseed Sampling
T=1
1.58
1.55
4.73
3.21
2.77
28.3
28.2
15.7
19.7
1.0x
T=2
1.59
1.61
3.93
2.59
2,43
27.7
28.1
20.5
10.9
21.8
1,0x
T=5
1.65
1.63
3.83
2.56
2.42
27.2
28.0
21.1
10.8
21.8
1.0x
T=10
1.70
1.76
3.77
2.48
2.43
25.8
27.2
21.8
11.4
21.6
1.0x
T=100
1.75
1.83
3.79
2.54
2.48
25.4
26.7
22.1
11.0
21.3
1.0x
Gradient Monipulation
PCGrad
1.57
1.59
4.78
3,28
2.81
28.1
28.5
15.3
19.6
3.2x
Gradient Vaccine
1.58
1.62
4.94
3.41
2.89
28.0
28.5
15.0
19.4
3.2x
DPL. (Arithmetic mean)
1.64
1.63
3.83
2,53
2.41
27.3
28.0
20.9
11.3
21.9
1.0k
(b) 2-direction (Arithmetic mean & customized metric)
Cross-eatrop
BLEU
Training Time
DE
III
AVG
DE
III
AVG
Temperature based Sampling
Tel
1.41
3.20
2.31
36.6
22.5
1.0k
T=2
1.44
2.56
2.00
35.8
11.1
23.4
1.0x
Tw5
1.55
2.51
2.03
34.1
12.2
23.1
1.0x
T=10
1.6
2.59
2.10
33.0
11.7
22.3
1.0k
T=100
Gradient Monipulation
PCGrad
1.42
3.18
2.30
36.6
8.3
22.5
2.1x
Gradient Vaccine
1.43
2.95
2.19
36.4
23.1
DPL. (Arithmetic mean)
1.46
2.52
1.99
35.4
12.0
23.7
1.0x
DPL (FH)
1.57
Table Results of the main experiments for the application of Double Power Law. We report the average results of three random seeds for each method.
4.2 Baselines
Temperature-basee Sampling We search E 21, 2, 10, 100), which are most frequently used
in NMT literature. After choosing T, the sampling ratio r/ =1,/"/ / EE / where ps denotes the ratio of the training examples from task i the whole training set.
1/2
Gradient Manipulation We also compare our method to gradient manipulation methods that are reported to improve MNMT performance: 1) PCGrad(Yu all (020) and 2) Gradient Vac- cine(Wang et all, 2021). We employ the default configuration of the methods described in their corresponding papers. Gradient manipulation methods require computing the cosine similarity of gradients at each optimization step and adjusting the gradient direction to resolve conflicts, resulting an increased computational budget along with the increase of directions.
4.3 Results
As shown in Table D. we report both the generalization cross-entropy and detokenized BLEU score using sacrebleu for different sampling methods.
Under the arithmetic mean metric, DPL reaches similar generalization cross-entropy compared to the best result when tuning temperature (T=2) and gains +0.3 average BLEU scores than T=2 the 2-direction setting. In the 4-direction setting, DPL also gains better results(+O.1 BLEU) compared to the best result when tuning temperature (T=5). The best sampling temperature varies for differ- ent MNMT tasks while DPL could surpass the corresponding best results by just computing once, reducing the searching time to 1/5 origin.
Comparing to the Gradient Manipulation methods, DPL leads to much better average performance than the best method in both experiments (+0.6 BLEU in 2-direction. +2.3 BLEU 4-direction),
2 aref's Chreff:IIcase:miie www.ase:mmieeelefffooooooooo htt.://hww......tooooooooeesssoo

---

Metric
Method
FR
DE
ZH
HI
AVG
T=5 (best T)
71.2
74.4
70.5
65.5
70.4
BERT-Score
DPL(Ours)
71
74.1
71.6
66.3
70.8
T=5 (best T)
54.4
52,4
46.8
45.2
49.7
COMET
DPL(Ours) 53.9
52.
48.2
46.5
50.2
Table 3: Model-Basee evaluation results of the 4-direction multilingual NMT experiments. DPL. method consistently outperforms the best temperature-based method in terms of average results.
with only 1/2 and 1/3 training time. The result also in line with Xin et (2022) that simple scalarization could beat the complex optimization method if set the weights properly.
We also evaluate the customized metric in the 2-direction experiment, where we hope to best enhance the performance of the low-resource direction. It turns out that DPL more effective (+0.4 BLEU) in improving the low-resource direction than tuning the temperature (p<0.05).
In addition to using the BLEU score for evaluation, we also utilize model-based evaluation tech- niques, specifically Bert-Score (Zhang et all 202() and COMET (Rer steeetal all, 2020). Research in- dicates that these metrics often align more closely with human evaluations than their model-free counterparts. As detailed in Table & DPI. outperforms the best temperature-based method, showing an average improvement of 0.4 in BERT-Score (p<0.05) and 0.5 in COMET score (p<0.05).
The results justify that through the Double Power Law, we can compute the sampling ratio combi- nation that best satisfies our expectation without manually searching the sampling ratios.
4.4 Discussion
The Double Power Law of model with different sizes As shown in Table El we evaluate the how the power terms of Double Power Law change in models with different sizes.
Among the parameters in the Double Power Law, we are more inter- ested power terms in the Capacity Occupation(a) and Intrinsic Over-fitting(3)) term since they better reflect the sensitivity of per- formance changes against sampling ratio. We keep the other param- eters the same as in Equation K and estimate the power terms in the Medium and Large size models.
Model Size
Ct
/
Base(64M)
0.20
1.21
Medium{ (02M)
0.25
2.41
Large(317M)
0.27
3.43
Table 4: The estimated power terms of for different models.
It shows that the power term(33 in the Intrinsic Over-fitting term increases rapidly with the growth in model size. While the power term(a in the Capacity Occupation term changes much slower. For larger model, over-fitting phe- nomena of low-resource direction will be more severe and the final performance would be more likely to fall into the collapsed area when increasing the sampling ratio for low-resource directions.
Generalization of DPL to more diverse Directions To better test the generality of computed as Equation B. we ran experiments on 10 languages in X-EN from WMT10 (Calnson Buich 2010). Dataset details are in Appendix Q. Figure 0 indicates Pareto Front Collapse in low-resource directions, with the double power law fitting well across data/language conditions (₁ values of 0.86 and 0.92 for low and high resources). As for the difference, we observed a more pronounced pareto front collapse in X-EN than in EN-X. By only adjusting the 7 term in DPL to -0.25, the fit improves for X-EN directions 2r 0.86 to 0.91 for low-resources; 0.92 to 0.97 for high-resources), highlighting the significant effect of the Intrinsic Over-fitting term in X- X-EN directions.
The results combined show that we can rely on the instantiated DPL when the model and the training papaeeparammtters (e.g., batch size, total update steps are not changed. The DPL. robust when introducing more languages changing the translation directions.

---

(~*2-0.86)
X-+EN High Resource Diractions (r*2=0.92)
dreen
3350
Iren
325
en
3.00
ro-en
Voard Froo Faimpy
Iren
eten
Valid Croos Estrrey
2.75
2.50
2.25
2.00
2-
2.50
0.2
Proportson
Proportion
(a) As the proportion increases, the validation loss goes down then goes up for low-resource lan- guages.
(b) The validation loss goes down when the propor- tion of high-resource direction increases.
Figure 7: Evaluation of the Double Power Law under X->EN translation setting, with more diverse languages. The points are the actual value and the solid lines are the predicted values for one direction at certain sampling ratio.
Related Work
Many previous work studied the trade-off among different directions and how to achieve a better overall performance in MNMT. Abaroni al (2019); Shaham at (MI22) empirically study how different settings influence the positive and negative interference among different directions. Xin al (2022)firstly incorporate Pareto optimity to the analysis of MNMT system. |ermandes et (2025) propose to use single power law to model how one direction's 's performance changes with its sampling ratio under the assumption that all directions have unlimited data.
Another related line works study multi-task optimization for MNMT. Yu al (2020); Wang all (202I) propose to dynamically altering the conflict gradients eliminate the negative interfer- ence among tasks. |Wang et all (20207); Zhou al (2021) explore how to dynamically adjust the optimization objective along with the training process better schedule the training for different directions. Li and Gong (202I) focus on the imbalance nature of MNMT task and propose an op- timization method based the local curvature of the loss landscape. However, Xin et all (2022) recently show that simple scalariaztion method could beat many multitask optimization methods for MNMT if the sampling ratio for different tasks are correctly set.
While we focus on how to set the optimal weights, modeling how the size of neural network influ- ences the performance also attracts much attention recently, i.e. the neural scaling law (Kaplan et all 2020: Hoffmann et all, 2022: OpenAl, 2023).. In the field of NMT, Ghorbani et all (2021a); a) Cordon etal (202T) show that the scaling behavior of cross-entropy loss follows a power law with the model size as input while Fernandes et al (2024) extends the conclusions to multilingual setting.
Conclusion
In this work, we start from our observations that the trade-off front is no longer Pareto front in Mul- tilingual Neural Machine Translation when there exists low-resource directions, i.e. the collapse of Pareto front. We analyse the reason causing the phenomena and find that there exists a balance between model capacity occupation and the intrinsic risk to over-fit for certain direction. Based on the findings, we propose the Double Power Law to model how the performance of given direc- tion changes with its sampling ratio, which is robust across different task settings and is capable of capturing the collapse phenomena. Finally, we frame the sampling ratio selection problem for Mul- tilingual Neural Machine Translation as an optimization problem, which greatly reduces the cost for manually tuning the sampling temperature and reaches better results.
10

---

7 Acknowledgements
We thank all reviewers for their helpful advice. This paper supported by the National Key Research and Development Program of China under Grant No.2020AAA0106700,, the National Science Foun- dation of China under Grant No.61936012.
References
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 3874-3884, Minneapolis, Minnesota. Association for Computational Linguisties.
Loic Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Gra- ham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMTT9), In Proceedings of the Fourth Conference Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy, Association for Computa- tional Linguistics.
Stephen P. Boyd and Lieven Vandenberghe. 2004. Convex optimization. IEEE Transactions on Automatic Control, 51:1859-1859.
Chris Burcch, Philipp Koehn, Christof Monz, Kay Peterson, and Omar Zaidan, editors. 2010. Proceedings of the Joint Fifth Workshop Statisfical Machine Translation and MetricsMA7R. Association for Computational Linguistics, Uppsala, Sweden.
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation, In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 1723-1732 1732, Beijing, China. Association for Compu- tational Linguisties.
Patrick Fernandes, B. Ghorbani, Xavier García, Markus Freitag, and Orhan Firat. 2023. Scaling laws for multilingual neural machine translation. ArXiv, abs/2302.09650.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 2020, Sharpness-aware mini- mization for efficiently improving generalization. ArXiv, abs/2010.01412.
B. Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Gareía, Ciprian Chelba, and Colin Cherry. 2021a. Scaling laws for neural machine translation. ArXiv, abs/2109.07740.
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. 2021b. Scaling laws for neural machine translation, CoRR, abs/2109.07740..
Mitchell Gordon, Kevin Duh, and Jared Kaplan. 2021. Data and parameter scaling laws for neuura machine
machine translation. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 5915-5922, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Barry Haddow, Rachel Bawden, Antonio Valerio Miceli Barone, Jindvrich Helcl, and Alexandra Birch. 2021. Survey of low-resource machine translation. Computational Linguistics, 48:673- 732.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack Rae, Oriol Vinyals, and L.. Sifre. 2022. Training compute-optimal large language models. ArXiv, abs/2203.15556.
11

---

Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. 2020. Scaling laws for neural language models. ArXiv, 020/2001083611
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. 2016. On large-batch training for deep learning: Generalization gap and sharp minima. ArXiv, abs/1 (tbs//609.000 609.04836.
Bobby Kleinberg, Yuanzhi Li, and Yang Yuan. 2018. An alternative view: When does SGD es- cape local minima? In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2698- 2698-2707. PMLR.
Xian Li and Hongyu Gong. 2021. Robust optimization for multilingual translation with imbalanced data. In Neural Information Processing Systems.
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and Sam Tak Wu Kwong. 2019. Pareto multi- task learning. In Neural Information Processing Systems.
OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08874.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of
NAACL-HLT 2019: Demonstrations.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. ArXiv, abs/1511.06609,
Uri Shaham, Maha Elbayad, Vedanuj Goswami, Omer Levy, and Shruti Bhosale. 2022. Causes and cures for interference in multilingual translation. ArXiv, abs/2212.07530.
NIlb team, Marta Ruiz Costa-jusså, James Cross, Onur eCelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Alison Youngblood, Bapi Akula, Lote Barrault, Gabriel Mejia Gonza- lez. Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon L. Spruit, C. Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami. Francisco Guzm' an, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. ArXiv, abs/2207.0461
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz. Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. 2020a. Balancing training for multilingual neural machine translation, In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pages 8526-8537, Online. Association for Computational Linguistics.
Yiren Wang, ChengXiang Zhai, and Hany Hassan. 2020b. Multi-askk learning for muluulngul neu- Tal machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1022-1034, Online. Association for Computational Lin- guistics.
Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. 2021. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. ICLR.
Derrick Xin, B. Ghorhani, Ankush Garg, Orhan Firat, and Justin Gilmer. 2022. Do current multi-task optimization methods in deep learning even help? ArXiv, abs/2209.11379,
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. ArXiv, abs/2001.06782.
12

---

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi 2020. Beriscoret Evaluating text generation with bert, In International Conference on Learning Representations.
Chunting Zhou, Daniel Lévy, Xian Li, Marjan Ghazvininejad, and Graham Neubig. 2021. Distribu- tionally robust multilingual machine translation. ArXiv, 10./2100.0400..
13

---

A Datasets for Main Experiments
Direction
Dataset
Num. of Training Examples
Evaluation Dataset
Eeglish-to-Frenchh
WMT10
{10M} (10M, 1M, 260K)
newtest15
English-to- Chincee WMT19
newtest19
English-to-Germen WMT10
(4.6M. IM, 260K)
newtest18
English-to-Hinds
WMT10
(260k)
newtest14
Table 5: The datasets description for the main experiments. We randomly choose subset of the full training set of a direction to form a smaller one.
B Training Details of Multilingual NMT experiments
In this section, we would provide concrete details of our experimental setup for reproduction.
Similar to Xin all (2022)'s setting, conduct our experiments three Transformer (Vaswani atall 2017) models with different sizes as shown in Table
Size
Enc. Layer
Dec. Layer
Hid. Dim
FFN Dim
Max Steps
Max Tokens per Batch
Base (64M)
3
512
2048
100k
20k
Medium (102M)
512
2048
100k
20k
Lurge (317M)
1024
4096
400k
Table 6: Overview of model sizes and optimization hyper- paremeearameters.. The Large-size model is trained with a larger batch-size and training steps or it would fail converge. For 4-task experiment, we double the max training steps due to the increase of training data.
For all of our model, use 64k sentence piece vocabulary. All models trained with 4k warm- up steps with the learning rate linearly increasing from to 3e -4 then decreasing with inverse_sqri learning rate scheduler. The label smoothing term is set 0.1 following the NMT literature con- vention. Evaluation is done every 5k steps, and we choose the best checkpoint with lowest average validation loss before examining the final generalization performance.
C Datasets for Generality Testing
Language Pair Num. of Training Examples
Evaluation Dataset
fr-en cs-en de-en zh-en hi-en
10M
newtest
5M
newtest16
4.6M
newtest16
260K
newtest19
260K
newtest14
gu-en
80k
newtest19
tr-en
180k
newtest16
ro-en Iv-en
500K
newtest16
300K
newtest17
et-en
300k
newtest18
Table Dataset deseription for multilingual x-en translation experiments. The dataset taken from WMT10. The dataset includes languages from diverse language family and data-adequacy.
We use Cairseq(Oit MUT the training framework
14


Reviews:
[{"review_id": "IWMUiwodfG", "reviewer": "Reviewer_4R4K", "summary": "This work proposes a power law for multilingual machine translation, a magic formula and constants to predict the performance of multilingual machine translation, that trades the model capacity and generalization based on the number of parameters and the training data sizes. The law is derived from the observation that the Pareto front, the performance curve by treating each language direction as an objective, collapses when the training data is highly imbalanced. They hypothesize that the root cause is the trade of the risk of overfitting when upsampling low resource, and thus, derived the power low. Experimental results show that the simple law achieved better results when compared with other competitive methods, e.g., simply tweaking temperature parameter to affect sampling ration and/or gradient manipulation-based methods.", "questions": "* Does the double power law hold for X-to-en directions? How about X-to-X directions in general?", "limitations": "This wok needs discussion on the applicability to a model with more diverse languages.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The analysis of Pareto front is very interesting by treating each language direction as an objective, and it clearly shows the collapse when training data size is largely skewed. \n\n* The derived power law is very simple, yet backed by the observation of the Pareto front analysis.\n\n* Extensive analysis on English-to-{French,German,Chinese,Hindi} directions and the empirical results show the effectiveness of the proposed power law.", "weaknesses": "* It is experimented only for en-to-X directions, and not X-to-en directions. I'd like to know if the law is also applicable to those directions, in addition to X-to-X directions.\n\n* Similarly, it would be nice to investigate zero-shot directions by focusing on language families to see if transferability will also follow similar trends.\n\n* Translation qualities are measured only by BLEU. It would be good to add different metrics, e.g., COMET or BERTScore, to see if similar trends are happening to those metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we really appreciate your effort in reviewing our paper, and thank you for your helpful comments. We are glad that you recognize our analysis of Pareto front very interesting, the simplicity of the derived power law and the effectiveness of its application. We address your questions below:\n\n> W1/Q1/L1: It is experimented only for en-to-X directions, and not X-to-en directions. I'd like to know if the law is also applicable to those directions, in addition to X-to-X directions.             Does the double power law hold for X-to-en directions? How about X-to-X directions in general?            This wok needs discussion on the applicability to a model with more diverse languages.\n> \n\nWe appreciate your interest in the robustness of our findings and would like to address your concerns. We added additional experiments covering 10 diverse languages:  French (FR), German (DE), Estonian (ET), Finnish (FI), Romanian (RO), Turkish (TR), Gujarati (GU), Hindi (HI), Chinese (ZH), and Latvian (LV). The dataset description is listed in Table 1 from the **general response PDF.** \n\n For X-EN directions, as shown in Figure 1 and 2 in the **general response PDF**, there is clear Pareto Front Collapse phenomenon for all low-resource directions. And the double power law can fit well (i.e., r^2=0.86 for low-resources and r^2=0.92 for high-resources) under different data-adequacy/language conditions. The results confirm that Pareto Front Collapse phenomenon and the double power law hold for both X-to-en and en-to-X directions across these diverse languages, further strengthening our claim of the power law's effectiveness in multilingual machine translation with more diverse languages and directions.\n\n> W2: Similarly, it would be nice to investigate zero-shot directions by focusing on language families to see if transferability will also follow similar trends.\n> \n\nWe conducted zero-shot experiments on the MNMT model trained on 10 from-EN and 10 to-EN directions. We discovered that in the zero-shot experiment, there are significant off-target issues, such as sentences being translated into incorrect languages or not being translated at all. These issues make it challenging to obtain a clear comparison of translation quality under such conditions. Even zero-shot direction among high-resource languages would suffer (e.g., fr→de 36% off-target ratio). We would discuss about it in the revised manuscript.\n\n> W3: Translation qualities are measured only by BLEU. It would be good to add different metrics, e.g., COMET or BERTScore, to see if similar trends are happening to those metrics.\n> \n\nWe recognize that relying exclusively on BLEU scores may not offer a comprehensive understanding of translation quality. In response to your suggestion, we have incorporated model-based metric, BERTScore(https://github.com/Tiiiger/bert_score), into our evaluation. The results demonstrate that the Pareto Front Collapse phenomenon and the trends in BERTScore changes are consistent with those observed for BLEU scores and cross-entropy loss. Please refer to the general response for the detailed BERTScore results."}, {"review_id": "SfLwiEPkP7", "reviewer": "Reviewer_mnLR", "summary": "This paper proposes to model the scaling law of multilingual machine translation using the double power law, which considers both the sampling ratio of a language and the size of the corpus. The motivation is that simply increasing the sampling ratio of low-resource language might lead to decreased performance for both high and low-resource languages due to overfitting. The paper then uses the estimated formula to improve multilingual NMT that jointly trains on two or three languages.", "questions": "Have you considered using more true low-resource languages, such as some languages listed here: https://www.statmt.org/wmt21/large-scale-multilingual-translation-task.html", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and it includes a good motivation and analysis section on why we should consider dataset size in the scaling law estimation\n2. The resulting method is compared to strong baselines on multilingual NMT system for 2 to 4 languages. The method does seem to bring some improvement over the baseline.", "weaknesses": "1. The experiment is restricted to maximum of 4 languages, which is rather limited. It would be more helpful and convincing to the readers if there are multilingual NMT experiment for 10+ languages\n2. The paper focuses on addressing the overfitting issue with low-resource languages, but the experiment only includes Hindi and Chinese as the “low-resource” languages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we really appreciate your effort in reviewing our paper, and thank you for your helpful comments. We are glad that you recognize our analysis of Pareto front well motivated and our method effective. We address your questions below:\n\n> W1/W2/Q1: The experiment is restricted to maximum of 4 languages, which is rather limited. It would be more helpful and convincing to the readers if there are multilingual NMT experiment for 10+ languages.  / The paper focuses on addressing the overfitting issue with low-resource languages, but the experiment only includes Hindi and Chinese as the “low-resource” languages.  / Have you considered using more true low-resource languages …\n> \n\nWe have expanded our analysis by conducting additional experiments that encompass 10 language languages such as French (FR), German (DE), Estonian (ET), Finnish (FI), Romanian (RO), Turkish (TR), Gujarati (GU), Hindi (HI), Chinese (ZH), and Latvian (LV). The dataset description is listed in Table 1 from  the **general response PDF.** \n\nAs illustrated in Figures 1 and 2 in the **general response PDF**, a clear Pareto Front Collapse phenomenon is evident for all low-resource language directions in the X-EN direction. Moreover, the double power law demonstrates a strong fit (r^2=0.86 for low-resource languages and r^2=0.92 for high-resource languages, with r^2 values closer to 1 indicating a better fit) across various data-adequacy and language conditions. The direction that has the least data(gu-en,80k) shows the strongest collapse phenomenon.\n\nThese findings substantiate the presence of the Pareto Front Collapse phenomenon and the effectiveness of double power law in both X-to-EN and EN-to-X language directions across a diverse range of languages. .\n"}, {"review_id": "oPZpjTcKN7", "reviewer": "Reviewer_wMmE", "summary": "This paper investigates the multi-task scaling laws for multilingual machine translation (mNMT) in the presence of *limited data* for at least one of the tasks/language pairs.\n\nExtending Fernandes et. al. (2022) work on the data-rich scenario, the authors found that their proposed multi-task scaling law doesn't hold when data for different languages pairs is imbalanced: in this unbalanced scenario, there is a collapse of the *Pareto frontier*: increasing the weight of one LP doesn't not necessarily increase the performance of that model in that LP.\n\nTo address this, they introduced a novel *Double Power Law* (DPL) that introduces an extra term to Fernandes et. al. (2022)'s law that takes into account *overfitting*. Based on this DPL, they also porpose a method for finding the optimal weighting of the LPs, showing that it outperforms other multi-task optimizers according to BLEU.", "questions": "- In the DPL equation (5), the “capacity occupatio ratio” seems to the same as the “effective parameter ratio” in Fernandes et al. (2023), but its functional form is different (for example, the main on in the paper had 3 fittable parameters). Why did you change it?", "limitations": "See above", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "of your tasks is low-resource. While this paper focus on mNMT, this is aksi the case (implicitly) for most LLMs trained today. \n\nPrior work by Fernandes et. al. (2023) characterized the trade-off in the case where all tasks have pratically unlimited data, and introduced a multi-task scaling law for balancing tasks/LPs in this case. However, as the authors point out, in real life most tasks (language pairs in MT) have very limited data, and so the introduced DPL scaling law has the potential to have a lot of practical impact in balancing languages/tasks during the training of models. Their analysis relating sharpness and overfitting to the collapse of the pareto frontier is also quite interesting.", "weaknesses": "Despite the potential impact of the this work, I think it is quite limited by its small analysis of impact of *data* term: they introduce powerlaw term for data and add an overfitting term. However the proposed Double Power Law is given very little theoretical motivation, and little empirical validation (with non of the standard fitting metrics like r2 reported). This is exharcebated by the fact that the data terms are fitted with two or three different possible data sizes, and its hard to trust the scaling law without some validations. \n\nNevertheless the current findings are already quite prevalent, but the analysis is further limited by the fact they only explored one “direction” of multiliguality: and language pairs are *from* English into XXX. Fernandes et. al. (2023) also explored the opposite XX→EN direction, and found different behavior in the Pareto curves, so understanding what happens for LR in that direction is quite pertinent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we really appreciate your effort in reviewing our paper, and thank you for your helpful comments. We are glad that you recognize our work 1) having the potential to have a lot of practical impact in balancing languages/tasks during the training of models 2) analysis relating sharpness and overfitting quite interesting. We address your questions below:\n\n> W1: Despite the potential impact of this work, I think it is quite limited by its small analysis of the impact of the *data* term: they introduce a powerlaw term for data and add an overfitting term. However, the proposed Double Power Law is given very little theoretical motivation, and little empirical validation (with non of the standard fitting metrics like r2 reported).\n> \n\nWe have added the result of the r^2 evaluation to our fitting results. As shown in Figure1/2  in the **general response PDF,** the double power law can fit well (r^2=0.86 for low-resources, r^2=0.92 for high-resources, r^2 closer to 1 suggests a better fit) under different data-adequacy/language conditions. \n\n> W2: Nevertheless the current findings are already quite prevalent, but the analysis is further limited by the fact they only explored one “direction” of multiliguality: and language pairs are *from* English into XXX. Fernandes et. al. (2023) also explored the opposite XX→EN direction, and found different behavior in the Pareto curves, so understanding what happens for LR in that direction is quite pertinent.\n> \n\n\nWe conducted additional experiments on X-EN directions with more diverse languages. The conclusion is the same as EN-X directions that there also exists Pareto Front Collapse phenomenon in low-resource directions and the double power law can well fit the results. Please refer to the general response for more details.\n\nAs for the difference, we find that the pareto front collapse phenomenon is more severe for X-EN directions than EN-X directions. Fixing all the parameters of DPL except for the $\\gamma$ term, increasing it to -0.25 would result in the best fit for X-EN directions (r^2: 0.86 → 0.91 for low-resources,  r^2=0.92 → 0.97 for high-resources). It means the Intrinsic Over-fitting term has more impact on the final performance in X-EN directions. We will add the discussion to the revised manuscript.\n\n> Q1: In the DPL equation (5), the “capacity occupation ratio” seems to the same as the “effective parameter ratio” in Fernandes et al. (2023), but its functional form is different (for example, the main on in the paper had 3 fittable parameters). Why did you change it?\n> \n\nWe think there might be some misunderstanding about DPL equation (5), the “capacity occupation ratio” **has the same functional form** as the “effective parameter ratio” in Fernandes et al. (2023), which has only **two** fittable parameters $k$ and $\\alpha$ while $p$  is the input variable."}, {"review_id": "wIA7ZWnVzZ", "reviewer": "Reviewer_hKkr", "summary": "This paper explores the influence of language pair sampling weights on the ultimate performance of Multilingual Neural Machine Translation (MNMT), specifically through the lens of the Pareto front concept. Existing literature suggests that under certain conditions, diverse sampling weights can yield solutions that constitute a Pareto front. Here, the performance of a specific task is positively related to its assigned weight. \n\nHowever, this work introduces the concept of Pareto front collapse, where the performance of a low-resource direction might deteriorate as its weight escalates, especially under data-imbalance scenarios. To explain the reason for the collapse, the authors argue that increasing the sampling weight for a low-resource task not only heightens model capacity for it but also amplifies the risk of overfitting.\n\nTo predict the performance trade-off curve among different directions, the authors propose a Double Power Law (DPL) that considers capacity occupation and intrinsic over-fitting. Experimental results show that DPL captures the performnce curve successfully under various setting (including Pareto front collapse) and can used to determined the optimal sampling ratio for MNMT.", "questions": "Main questions are listed in *Weaknesses*. Another question here is why Figure 2 and Figure 3 describe Pareto front collapse using different indicators, BLEU, and cross-entropy.", "limitations": "Main limitations are listed *Weaknesses*.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This paper delves into a more realistic scenario of MNMT compared to previous works, wherein the data sizes of different language pairs are extremely imbalanced , and highlights the issue of the Pareto front collapse in this situation.\n* The paper provides an insightful explanation for the collapse: low-resource tasks bring about the risk of overfitting as their weights are increased. This interpretation aligns intuitively with empirical results.\n* The proposed DPL can predict the task performance given the data size and sampling weight which can be further used to select the optimal samlping ratios.", "weaknesses": "* The paper's assessment of translation performance primarily leans on BLEU, which has shown to be less reliable in terms of aligning with human judgment compared to metrics like COMET and BLEURT. This reliance on BLEU could be justified if experiments were conducted on truly low-resource languages unsupported by these advanced metrics. However, the languages featured in the study (English, German, French) are all high-resource, making this justification less feasible.\n* Despite considering data imbalance scenarios, the study mostly focuses on translations from English to other languages, thus straying from a truly realistic scenario. The paper lacks supporting evidence that its findings can be generalized to more diverse cases, such as English to other languages, or non-English-centric MNMT.\n* The authors have not included two crucial summaries about the DPL:\n  * When can one rely on an already instantiated DPL or, conversely, when do the parameters of the DPL need to be re-estimated: e.g., when the model size changes, more language pairs are introduced, or when transitioning from English-centric to non-English-centric?\n  * How many times the model needs to be retrained to re-estimate the parameters of DPL in different cases? What if I only need to estimate the sample ratios (no bias terms)?\n* The training time recorded in Table 2 does not factor in time spent on hyperparameter searching, such as temperature-based sampling or DPL parameter estimation. Therefore, clarification on the above two points is crucial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we really appreciate your effort in reviewing our paper, and thank you for your helpful comments. We are glad that you recognize our work 1) delves into a more realistic scenario of MNMT 2) provides an insightful explanation for the collapse 3) The DPL is effective in predicting the task performance. We address your questions below:\n\n> W1: The paper's assessment of translation performance primarily leans on BLEU, which has shown to be less reliable in terms of aligning with human judgment compared to metrics like COMET and BLEURT. This reliance on BLEU could be justified if experiments were conducted on truly low-resource languages unsupported by these advanced metrics. However, the languages featured in the study (English, German, French) are all high-resource, making this justification less feasible.\n> \n\nWe acknowledge that BLEU scores may not provide a holistic insight into translation quality. In response to your recommendation, we have integrated the model-based metric, BERTScore (**https://github.com/Tiiiger/bert_score**), into our evaluation process. The outcomes reveal that the Pareto Front Collapse phenomenon and the BERTScore trends align with the patterns observed for BLEU scores and cross-entropy loss.\n\nIn the experimental section (Table 2 of the original paper), the DPL method results in a BERTScore improvement of 0.4 points (4-direction average of 70.4 → 70.8, p<0.05) and 1.0 points (2-direction average of 75.4 → 76.4, p<0.05) when compared to the best Temperature-based Sampling method. We will incorporate these findings into the revised manuscript.\n\n Please refer to the general response for the detailed BERTScore results.\n\n> W2: Despite considering data imbalance scenarios, the study mostly focuses on translations from English to other languages, thus straying from a truly realistic scenario. The paper lacks supporting evidence that its findings can be generalized to more diverse cases, such as English to other languages (We suppose other to English here?), or non-English-centric MNMT.\n> \n\nWe have included X-EN directions and more languages such as French (FR), German (DE), Estonian (ET), Finnish (FI), Romanian (RO), Turkish (TR), Gujarati (GU), Hindi (HI), Chinese (ZH), and Latvian (LV) to the experiments. \n\nFor X-EN directions, as shown in Figure-1/2 in the **general response PDF**, there is a clear Pareto Front Collapse phenomenon for all low-resource directions. And the double power law can fit well (r^2=0.86 for low-resources, r^2=0.92 for high-resources, r^2 closer to 1 suggests a better fit) under different data-adequacy/language conditions. The results confirm that the Pareto Front Collapse phenomenon and the double power law hold for both X-to-en and en-to-X directions across these diverse languages, further strengthening our claim of the power law's effectiveness in multilingual machine translation with more diverse languages and directions. \n\n> W3: When can one rely on an already instantiated DPL or, conversely, when do the parameters of the DPL need to be re-estimated?  How many times the model needs to be retrained to re-estimate the parameters of DPL in different cases?\n> \n\nWe can rely on the instantiated DPL when the model and the training hyper-parameters (e.g., batch size, total update steps …) are not changed. The DPL is robust when introducing more languages or changing the translation directions, which is shown in the previous answer. When changing from EN-X to X-EN directions, we would get a better fitting result if we simply re-estimating (by retraining the model for 4 times by changing one direction’s sampling ratio from 0.1 to 0.9 equally) the $\\gamma$ term of DPL (r^2: 0.86 → 0.91 for low-resources,  r^2=0.92 → 0.97 for high-resources), leaving other terms fixed.  When changing the model size, all parameters need to be re-estimated with the method listed in Section 3.2 of the original paper. It costs about 15 times to retrain the model with different ration combinations. In fact, when the estimation of DPL is done, we don’t need to re-estimate the parameters of DPL when adding new languages to the model as long as the model size is not changed.\n\n> W4: The training time recorded in Table 2 does not factor in time spent on hyperparameter searching, such as temperature-based sampling or DPL parameter estimation. Therefore, clarification on the above two points is crucial.\n> \n\n**Once** the estimation of DPL is done, we did not re-estimate it again in the experiment section. The same DPL is transferred to the 2-direction and the 4-direction setting. While for temperature-based sampling, we need to conduct the search whenever the number of directions or languages changes. We would clarify it in the revised manuscript.\n\n> Q1: Why Figure 2 and Figure 3 describe Pareto front collapse using different indicators, BLEU, and cross-entropy.\n> \n\nWe use different indicators to show that the Pareto front collapse phenomenon is consistent under different evaluation metrics. We would clarify it in the revised manuscript."}, {"review_id": "g7PM4lbf9s", "reviewer": "Reviewer_CMx8", "summary": "This paper frames the sampling ratio selection problem of MNMT as an optimization problem, and introduces a double power law to control the sampling ratio for different languages. The author argue that traditional method assumes unlimited training data which does not hold true for low resource settings, thus both model capacity occupation and intrinsic risk of overfitting should be taken into consideration during training data sampling.  \nExperiments on multiple language direction shows better accuracy (+0.3 BLEU) with only half training cost.\n", "questions": "1. So if we change the corpus size of certain language direction, should the parameters of the double power be re-estimated, and what is the time cost for this?", "limitations": "I think the two major limitation of this paper are the ignorance of data quality issue in MNMT training and the less convincing performance comparing with temperature based methods.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is an upgrade on the Fernandes et.al 2023 paper, from single power law to double power law, \ntaking both model capacity occupation and intrinsic risk of overfitting into consideration. \nI think over the years the sampling strategy of MNMT has always been on data balancing, while the overfitting risk for low resource language has largely been ignored. And this paper addresses an important problem in this direction.\nI like the organization of this paper, starting from an empirical phenomena, find the problem and solve it with a well-grounded method.\n", "weaknesses": "However, I believe there is still room for improvement in this paper:\n1.  I think the double power law is still not enough to resolve the sampling problem. One important aspect is the data quality: so in the low resource setting, the quality of training data is often worse comparing with the rich resource setting, and the noisy data would also cause the performance drop (or local optimum ) during training. So if we are talking about a more realistic MNMT scenario, the data quality issue should definitely be considered.  I would like to see more analysis in this direction.\n\n2. There are some confusing claims in the paper :  A. in the intro: \"achieving better accuracy (+0.3 BLEU) with only half training cost\". \nBut in the table 2, the statics shows something different : for example in (a),  comparing with temperature based method,  BLEU +0.1, training time both 1.0x,  with Gradient based method, BLEU +2.4, training time 1.0x vs 3.2x.   So I am not sure which one is the key comparison to look at.  Besides, when comparing the training cost, the time to estimate the parameters of the double power law should also be included.  B. in abstract, the author claims \"by training over 200 multilingual models\", but there is no evidence in the experiments, only a two-direction model and a four-direction model.   So I think the author should be more careful in making these claims. \n\n3. The improvement over traditional method is marginal. Comparing with Temperature based method, the proposed method doesn't show enough improvement in both BLEU score and training time.\n\n\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we really appreciate your effort in reviewing our paper, and thank you for your helpful comments. We are glad that you recognize our work 1) addresses an important problem in this direction 2) with good organization of the paper 3) solves the problem with a well-grounded method. We address your questions below:\n\n> W1:  I think the double power law is still not enough to resolve the sampling problem. One important aspect is the data quality: so, in the low resource setting, the quality of training data is often worse compared with the rich resource setting, and the noisy data would also cause the performance drop (or local optimum) during training. So, if we are talking about a more realistic MNMT scenario, the data quality issue should definitely be considered. I would like to see more analysis in this direction.\n> \n\n**We found that the DPL is robust across directions with different qualities.** In the original paper, to ensure the robustness of our findings, we used two kinds of low-resources data. The one like en-hi is the true low-resource direction, and the one en-zh260k is sampled from the larger en-zh10M, which is actually a high-resources direction. The data creation process was listed in Appendix A, we would make it more clear in the paper. As shown in Figure-5 from the paper, DPL can well fit the en-hi, en-zh260k, and en-zh10M data settings.  It shows that the DPL is robust under different data-quality settings and data size plays a more important role when changing the sampling ratios. \n\nTo further test DPL’s robustness under different data-quality settings, we have conducted additional experiments covering 10 diverse language languages: French (FR), German (DE), Estonian (ET), Finnish (FI), Romanian (RO), Turkish (TR), Gujarati (GU), Hindi (HI), Chinese (ZH), and Latvian (LV). For X-EN directions, as shown in Figures 1 and 2 in the **general response PDF**, there is a clear Pareto Front Collapse phenomenon for all low-resource directions. And the double power law can fit well (r^2=0.86 for low-resources, r^2=0.92 for high-resources, r^2 closer to 1 suggests a better fit) under different data-adequacy/language conditions. The results confirm that the Pareto Front Collapse phenomenon and the double power law hold for both X-to-en and en-to-X directions across these diverse languages, further strengthening our claim of the power law's effectiveness in multilingual machine translation with more diverse languages, directions, and data-quality settings.\n\n> W2: There are some confusing claims in the paper.\n> \n\nThanks for pointing out. A) When using the Temperature-based Sampling method, we need to **search** the best sampling ratio every time we have a new MNMT setting, in our experiment we search 5 different temperatures so the total training time for the Temperature-based Sampling method is 5x. While for the DPL method, the parameters of DPL **do not need to be re-estimated** in our experiment, the same DPL parameter is transferred among different experiment settings and we only need to train the model once. We would make the comparison more clear in the revised paper. B) by saying “200 multilingual models” in the abstract, we mean the total number of models trained in this paper to get all the conclusions including the Pareto Front Collapse phenomenon and the DPL. We would make it more clear in the revised paper.\n\n> W3: The improvement over the traditional method is marginal. Compared with Temperature based method, the proposed method doesn't show enough improvement in both the BLEU score and training time.\n> \n\nCompared with Temperature based method, which needs searching the best Temperature by training the models for 5 times in different MNMT experiment settings, DPL does not need re-estimate the parameters and only needs to train the model once when involving new directions or changing corpus size. In our experiments, the same DPL parameters are transferred among the 2-direction and 4-direction experiments, which is the same as Equation-6. We added an additional model-based metric BERTScore to evaluate the performance, and DPL shows consistent improvement over 0.4 points (4-direction avg 70.4 → 70.8, p<0.05) and 1.0 points (2-direction avg 75.4 → 76.4, p<0.05).  Please refer to the general response for the detailed BERTScore results.\n\n> Q1: So if we change the corpus size of a certain language direction, should the parameters of the double power be re-estimated, and what is the time cost for this?\n> \n\nWhen changing the corpus size of a certain direction, the parameters of DPL **do not need to be re-estimated.**  DPL already takes the corpus size as its one input variable when predicting the performance as shown in Equation 5 from the paper. The $D_i$ term is the corpus size of a certain direction."}]

Author rebuttal:
# General Response for all Reviewers

Dear Reviewers and ACs,

We sincerely appreciate the reviewers' effort in reviewing our paper and the AC for reviewing and organizing discussions! We are encouraged to hear that our analysis of Pareto front in MNMT very interesting (R1,R3), well motivated(R2), insightful(R4) and addressing an important problem(R5)

We have submitted specific responses that address each reviewer's concerns and questions. We also added a **general response PDF** which includes the results for the added experiments. In addition, we would like to highlight some shared points that arise in the reviews:

1. Whether the Pareto Front Collapse phenomenon exsits in the X-EN directions and more diverse languages.

The answer is yes! We conducted additional experiments covering 10 diverse languages in X-EN directions: French (FR), German (DE), Estonian (ET), Finnish (FI), Romanian (RO), Turkish (TR), Gujarati (GU), Hindi (HI), Chinese (ZH), and Latvian (LV). The dataset description is listed in Table 1 from the **general response PDF.**  For X-EN directions, as shown in Figure 1 and 2 in the **general response PDF**, there is clear Pareto Front Collapse phenomenon for all low-resource directions. And the double power law can fit well (i.e., r^2=0.86 for low-resources and r^2=0.92 for high-resources) under different data-adequacy/language conditions.

2. Model-based evaluation metric.

We realize that relying solely on BLEU scores may not fully capture translation quality. Thus, we've included BERTScore in our evaluation. The results show consistent trends with BLEU scores and cross-entropy loss regarding the Pareto Front Collapse phenomenon. In our experiments (Table 2 in the original paper), the DPL method leads to a 0.4-point (4-direction avg: 70.4 → 70.8, p<0.05) and 1.0-point (2-direction avg: 75.4 → 76.4, p<0.05) BERTScore improvement compared to the best Temperature-based Sampling method.

The detailed result for BERTScore is listed below:

| 2-d (bert-score) | DE | HI | AVG |
| --- | --- | --- | --- |
| T=2 (best T) | 78.5 | 72.3 | 75.4 |
| DPL(Ours) | 78.2 | 74.6 | 76.4 |

| 4-d (bert-score) | FR | DE | ZH | HI | AVG |
| --- | --- | --- | --- | --- | --- |
| T=5 (best T) | 71.2 | 74.4 | 70.5 | 65.5 | 70.4 |
| DPL(Ours) | 71 | 74.1 | 71.6 | 66.3 | 70.8 |

We hope to get valuable comments from the reviewers, and we are ready and glad to engage in active discussions.

**Please download the PDF file in this response to see the figures of the X-EN experiments' result.**

Meta-review: Paper investigates multi-task scaling laws for multilingual neural machine translation by extending Fernandes et al. 2022 and proposes a new parametrization of the scaling laws, called Double Power Law, that is able to predict out of sample points in the multilingual translation experiments. By using the proposed scaling law, authors report improved quality on multilingual 2-English and out-of-English translation scenarios covering at most 10 languages, measure both automatic (BLEU) and model based metrics (COMET, BERTscore).

Reviewers raised issues in regards to experimental design (number of languages, directionality of translation), evaluation protocols (number of languages, lack of model based metrics) and formulation/parametrization of the scaling law. Authors addressed all the issues raised by reviewers hence reviewers updated their scores.

With the current form, paper is adding practical value to the multilingual translation, and llm training in multilingual setup with its updated scaling law (DPL). Accept as a poster.
Decision: Accept (poster)