Title: On the Pareto Front of Multilingual Neural Machine Translation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multilinugal Neural Machine Translation

Multitask Learning

Pareto Optimization
Abstract: In this work, we study how the performance of a given direction changes with its sampling ratio in Multilingual Neural Machine Translation (MNMT). By training over 200 multilingual models with various model sizes, data sizes, and language directions, we find it interesting that the performance of certain translation direction does not always improve with the increase of its weight in the multi-task optimization objective. Accordingly, scalarization method leads to a multitask trade-off front that deviates from the traditional Pareto front when there exists data imbalance in the training corpus, which poses a great challenge to improve the overall performance of all directions. Based on our observations, we propose the Double Power Law to predict the unique performance trade-off front in MNMT, which is robust across various languages, data adequacy, and the number of tasks. Finally, we formulate the sample ratio selection problem in MNMT as an optimization problem based on the Double Power Law. Extensive experiments show that it achieves better performance than temperature searching and gradient manipulation methods with only 1/5 to 1/2 of the total training budget. We release the code at https://github.com/pkunlp-icler/ParetoMNMT for reproduction.

On the Pareto Front of Multilingual Neural Machine Translation

Abstract

In this work, we study how the performance of a given direction changes with its sampling ratio in Multilingual Neural Machine Translation (MNMT). By training over 200 multilingual models with various model sizes, data sizes, and language directions, we find it interesting that the performance of certain translation direction does not always improve with the increase of its weight in the multi-task optimization objective. Accordingly, scalarization method leads to a multitask trade-off front that deviates from the traditional Pareto front when there exists data imbalance in the training corpus, which poses a great challenge to improve the overall performance of all directions. Based on our observations, we propose the Double Power Law to predict the unique performance trade-off front in MNMT, which is robust across various languages, data adequacy, and the number of tasks. Finally, we formulate the sample ratio selection problem in MNMT as an optimization problem based on the Double Power Law. In our experiments, it achieves better performance than temperature searching and gradient manipulation methods with only 1/5 to 1/2 of the total training budget. We release the code at https://github.com/pkunlp-icler/ParetoMNMT for reproduction.

1 Introduction

Multilingual Neural Machine Translation (MNMT) enables multiple translation directions in a single model, significantly reducing deployment costs of translation systems and benefiting low-resource directions (Sennrich et al., 2015). MNMT can be framed as a multi-task learning problem, where we expect to improve the overall performance across translation directions. However, it is infeasible to find an optimal solution for all translation tasks because of the negative interference among translation directions (Lin et al., 2019; Yu et al., 2020; Wang et al., 2021). A common method to make trade-offs among different languages is scalarization, where a combination of normalized weights is assigned to the loss of each direction during training. Xin et al. (2022) empirically show that different weight combinations can result in Pareto optimal solutions, where no one solution could outperform another in all tasks. All such solutions could form a Pareto front as shown in Figure 1. Fernandes et al. (2023) show that there is a power law between the weight of one direction and its final generalization performance, indicating a positive relation between weight and performance.

In this work, we empirically find that the positive relation together with the power law from Fernandes et al. (2023) does not hold true in a more realistic MNMT scenario where the data for different directions is highly imbalanced. The performance of a direction may decrease as its weight increases.

It raises our interests that 1) What causes the inconsistency phenomena? 2) Can we model the tradeoff behavior among different directions? 3) How should we set the weights for each direction?

To answer these questions, we first identify the problem in MNMT that the scalarization method could not generate Pareto optimal solutions when there exist low-resource directions, which is named the collapse of Pareto front . Secondly, to quantify how the performance of a given direction changes with its sampling ratio, we propose the Double Power Law. The law takes the number of training data into consideration and explicitly models the relationship between capacity occupation and the potential of over-fitting in a given direction, which has the form:

where F i is the predicted performance of task i measured by cross-entropy loss, p is the sampling ratio, D i is the number of training examples and M ( i ) ∞ is a constant bias term. The rest are fixed parameters to estimate. Experiments justify the robustness of the law under various settings.

Based on the Double Power Law (DPL), we propose an approach to compute the optimal sampling ratios. Our experiments demonstrate that DPL outperforms directly search for the optimal temperature and gradient manipulation methods for MNMT, achieving better accuracy (+0.3 BLEU) with only 1/5 to 1/2 training cost in our experiments.

To summarize, the contribution of this work is threefold:

We identify and analyze the collapse of Pareto Front phenomenon in MNMT.

We propose the Double Power Law to model the trade-off among different directions in MNMT, which is robust on languages, data adequacy, and number of directions.

Based on the Double Power Law, we propose a method to automatically compute the optimal sampling ratio combination, eliminating the need for manual tuning of the sampling temperature and achieving better performance in BLEU, COMET and BertScore metrics.

2 Collapse of Pareto Front in Multilingual Neural Machine Translation

2.1 Pareto Front in MNMT

Pareto Front In multi-task learning, where K is the number of objectives and L i ( θ ) is the loss of task i under solution parameterized by θ ∈ R n , we say solution θ 2 Pareto dominates solution θ 1 if ∀ 1 ≤ i ≤ K, L i ( θ 2 ) ≤ L i ( θ 1 ) and ∃ 1 ≤ i ≤ K, L i ( θ 2 ) < L i ( θ 1 ) . The Pareto front is a set of solutions where no solution Pareto dominates another. As shown in Figure 1, we give two toy examples of the possible shape of Pareto front when K = 3 and K = 2 .

Task 3 Loss

K=2

+

2.0

1.6

1.2

0.8

0.4

0.5

0.4

0.1

0.3

0.2

Task1 Loss

0.3

0.4

0.1

0.2

Task 2 Los5s

1.0

0.8

ask 2 Loss

0.6

0.4

0.2

0.0

0.0

0.2

0.4

Task 1 LosS

0.6

0.8

1.0

0.5

Scalarization (Boyd and Vandenberghe, 2004) is a traditional method to optimize the multi-task learning model where a set of predefined normalized weights w is applied to each task. If there is little interaction among different tasks , that one task's loss would drop when increasing its weight, equation 2 with different w could guarantee solutions that can form a Pareto front (Xin et al., 2022).

Generalization Performance Trade-off in En → {De,Zh}

90%

80%

70%

60% 50%

40%

30%

20%

10%

90%

80%

70%

60%

50%

40%

30%

20%

10%

x % Zh sampling ratio

90%

80%

70%

60%

50%

40%

30% 20%

10%

x % Zh sampling ratio

90%

80%

70%

60%

50%

40%

30%

20%

10%

Generalization Performance Trade-off in En → {Fr,Zh}

28

Data

High-High

26

Data

High-High

High-Low

High-Low

ZH-BLEU

26

ZH-BLEU

24

24

22

22

20

X

20

25

30

35

40

20

22

24

26

28

30

32

DE-BLEU

FR-BLEU

En → {Fr-High, De-High, Zh-High}

En → {Fr-High, De-Low, Zh-Low}

ZH Cross-Entropy

En → {Fr-High, De-High, Zh-Low}

ZH Cross-Entropy

ZH Cross-Entropy

5.0

5.0

5.5

3.0

2.5

3.5

4.5

4.0

2.5

3.0

3.5

4.5

4.0

2.5

3.0

3.5

4.5

4.0

5.0

1.5

2.0

1.5

2.0

2.0

1.5

1.9

2.0

2.2

2.4

2.2

2.4

1.31.41.51.61.71.8

DE Cross-Entropy

1.9

2.0

1.3

1.4

1.5

1.6

Cross-Entropy

1.7

1.8

1.4

DE Cross-Entropy

1.6

1.8

2.0

2.2

2.4

1.4

1.6

Cross-Entropy

1.8

2.0

1.4

DE Cross-Entropy

1.61.8

2.0

2.2

2.4

1.4

1.6

Cross-Entropy

1.8

2.0

Multi-task Learning for MNMT MNMT can be framed as a multi-task learning problem where one model can do the translation among multiple directions (Dong et al., 2015; Aharoni et al., 2019). To prevent the high-resource directions from dominating the training and to improve the low-resource directions' performance, scalarization is implemented through proportional sampling, where the sampling ratio of examples for task i in a training batch is w i . Currently, there is no consensus on how to directly set specific w i for each task, temperature-based sampling is more commonly used where w i = p 1 /T i ∑ K j p 1 /T j , p i denotes the ratio of the training examples from task i .

2.2 The Collapse of Pareto Front Widely Exists

We use datasets provided in WMT10 (Wang et al., 2020b) and WMT19 (Barrault et al., 2019) to conduct the MNMT experiment. The description of dataset is listed in Appendix A. In the 2-task setting, we experiment with the sampling ratio of one task ranging from 10% to 90%. In the 3task setting, we experiment with each task ranging from 10% to 80% resulting in 32 different ratio combinations. More detail about the training is in Appendix B. The experiment results and major findings in this section are illustrated in Figure 2,3 and 4.

The collapse exists in different language pairs In Figure 2, we can see that the generalization performance trade-off situation is different between the High-High and High-Low data settings. Under the High-High setting, the solutions from scalarization could form a Pareto front where one task's performance increases with the decrease in the other task. This is expected and also in line with the recent results from Xin et al. (2022); Fernandes et al. (2023). However, the Pareto front disappears when we replace the English → Chinese-10M with English → Chinese-260K. Fernandes et al. (2023) restrict the directions to high-resource ones so they also get Pareto optimal results, Xin et al. (2022)

also find that there exists a global optimal point in imbalanced multilingual translation however do not give it an explanation. Understanding the reason can help build a better MNMT system since the multilingual training data is highly imbalanced in reality(Haddow et al., 2021; team et al., 2022).

In Figure 2, when increasing the sampling ratio for one High-Resource direction, its performance continually goes up. While increasing the sampling ratio for one LowResource direction from 10% to 90%, its performance first goes up and then gradually goes down. This is the direct cause of the collapse of Pareto front. The collapse also exists when there are more than two languages. As shown in Figure 3, we draw the 3-task trade-off front using linear interpolation. The front in the left where all three directions are high-resource, is most similar to the example 3D Pareto front in Figure 1. When there exists low-resource directions as shown in the middle and the right in Figure 3, the Pareto front would disappear.

It indicates that the relation between High and Low resources directions is not a zero-sum game and there exists a sampling ratio range where different tasks can improve or deteriorate simultaneously. For example, in Figure 2,

Base

Medium

Large

High-High

High-Low

Trade-off curves in models with different sizes

32

Data

30

X---X---X-X

28

ZH-BLEU

26

24

22

20

20.0

22.5

25.0

27.5

30.0

32.5

35.0

FR-BLEU

this range could be 40% to 90% sampling ratio for the low-resource English-to-Chinese direction.

The collapse exists in models with different sizes Figure 4 shows the trade-off curves on models with different sizes. The Pareto front would collapse when there exist low-resource directions.

The collapse of Pareto front complicates the optimization of MNMT because the sampling ratio must be carefully set or the performance might fall in the collapsed area where the model's capacity is not fully exploited and there exist other solutions that can Pareto dominates the current performance.

2.3 Explore the Reason for the Collapse

The collapse is closely related to inadequate data. If we only consider the performance of lowresource direction in Figure 2, the performance of the low-resource direction briefly improves with an increase in the sampling ratio from 10% to 20%, but then gradually deteriorates as the sampling ratio continues to increase from 20% to 90%. This is in contrast to the performance of high-resource directions, which continue to improve with an increase in sampling ratio from 10% to 90%.

The effect of increasing the sampling ratio for a task is twofold: on the one hand, it gives it more weight and the model would allocate more capacity to optimize its performance, on the other hand, it also increases its risk of overfitting. For directions with abundant training examples, the over-fitting term is less significant so its performance would keep improving with more sampling ratio. However, for low-resource directions, the risk of over-fitting would be dominant for a high sampling ratio because of the lack of training examples. A small sampling ratio serves as a form of regularization for these low-resource

directions. This difference accounts for the contrasting performance trends observed between lowresource and high-resource directions.

It has been shown that the model would be less likely to over-fit and generalize better when it reaches a flatter local minimum after the gradient descent optimization (Keskar et al., 2016; Kleinberg et al., 2018; Foret et al., 2020). To justify our assumption, following Keskar et al. (2016), we compute the partial local curvature i.e. sharpness of the optimized model through the trace of the Hessian matrix as shown in Equation 3, where X stands for held-out evaluation data and Y stands for the label. We compute the sharpness of the FFN layer at different transformer layers.

As shown in Table 1, we compute the sharpness of our MNMT model on English → {German, Hindi} task with different sampling ratio combinations. The result shows that with more sampling ratio for high-resource direction, the model could reach a flatter local minimum for both tasks.

The results of the generalization performance and sharpness experiments indicate that adding weight for a certain translation direction is a double-edged sword. It forces the model to optimize more towards minimizing the training loss of this task, at the same time increasing its over-fitting risk, especially for the low-resource directions.

3 The Double Power Law

In the previous section, we discussed the collapse of Pareto front phenomena and its impact on MNMT.The collapse presents a challenge for determining the sampling ratio combination, as a poor combination can adversely affect the final performance by deviating from potential Pareto solutions. Additionally, we identified over-fitting as a significant problem in MNMT training especially for low-resource directions. While a straightforward solution would be to grid search the sampling ratio combination, it becomes impractical with increasing directions due to the exponentially growing search space. In this section, we are going to answer the following questions:

Is there a pattern behind the collapse phenomena? Can we predict the performance trade-off curve among different directions?

Fernandes et al. (2023) adopt single power law inspired by the scaling literature in NMT (Ghorbani et al., 2021b; Gordon et al., 2021) to predict how the sampling ratio affect the performance of MNMT. However, it assumes that all directions have unlimited data, which does not reflect the diverse data adequacy in reality for supervised MNMT training. Under Fernandes et al. (2023)'s power law, the trade-off curve of any two directions can form a Pareto front, which could not capture the collapse.

We find out that: the generalization performance F of the i -th direction in MNMT follows a double power law, which takes the sampling ratio p and the number of training examples D i as input:

where F i is measured through evaluation cross-entropy loss. k, α, q, β are fixed parameters, G ↓ ( D ) is a monotonically decreasing function of the number of training examples D i . L ( i ) ∞ and O ( i ) ∞ are bias terms that are relevant to the direction and the model.

3.1 Capacity Occupation and Intrinsic Over-fitting Terms

Based on our previous discussion, the generalization performance for a certain direction not only depends on its sampling ratio but also on its number of training examples. So we set two terms to jointly model the generalization performance, namely Capacity Occupation term and Intrinsic Overfitting term. As shown in Equation 4, similar to the power law in Fernandes et al. (2023), Capacity Occupation term can reflect that with a larger sampling ratio, the model would assign more capacity for the task to minimize its training loss, which could possibly improve the generalization performance. Intrinsic Over-fitting term reflects the fact that with a larger sampling ratio, the model is more likely to over-fit on this task, which would harm its generalization performance. Meanwhile, the scale of Intrinsic Over-fitting term is negatively related to the number of training examples. With fewer training examples, the influence of Intrinsic Over-fitting would be more significant.

3.2 Parameter Estimation

We estimate the parameters of the double power law in three steps. We conduct a series of English → { French-10M, German-260K } experiments with English → French-10M sampling ratio ranging from 10% to 90%. 1) We estimate the parameters of the Capacity Occupation term with English → French's results by ignoring the Intrinsic Over-fitting term because it has minor effect when D i is large for high-resource direction. 2) We fix the parameters of the Capacity Occupation term and estimate the parameters of the Intrinsic Over-fitting term with the results of

EN->{FR-10M, ZH-260K}

EN->{FR-10M, ZH-260K}

EN->{FR-10M, ZH-1M}

EN->{FR-10M, ZH-10M}

EN->{FR-10M, HI-260K}

EN->{FR-10M, HI-260K}

EN->{FR-10M, DE-4.6M}

EN->{FR-10M, DE-4.6M}

Generalization Cross Entropy

p

p

p

p

p

p

p

p

EN->FR Cross Entropy

EN->HI Cross Entropy

EN->FR Cross Entropy

EN->DE Cross Entropy

EN->FR Cross Entropy

EN->ZH Cross Entropy

EN->ZH Cross Entropy

EN->ZH Cross Entropy

Predicted Curve

Observation

4.0

3.4

3.5

3.2

5.0

5.0

3.0

2.8

3.0

4.5

4.5

2.5

2.6

4.0

4.0

2.4

2.0

2.2

3.5

3.5

2.0

3.0

1.5

0.0

0.2

0.4

0.6

0.8

1.0

1.8

0.0

0.2

0.4

0.6

0.8

1.0

3.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

4.0

3.50

3.0

3.5

3.5

3.25

3.00

2.8

2.6

3.0

3.0

2.75

2.4

2.2

2.5

2.5

2.50

2.25

2.0

2.0

2.0

1.8

2.00

1.5

0.0

0.2

0.4

0.6

0.8

1.0

1.75

0.0

0.2

0.4

0.6

0.8

1.0

1.6 *

0.0

0.2

0.4

0.6

0.8

1.0

1.5

0.0

0.2

0.4

0.6

0.8

1.D

English → German-260K. 3) We fix all parameters in Equation 4 except for the value of G ↓ ( D ) . We suppose that G ↓ ( D ) also subjects to the form of power law and conduct the same series of experiments on English →{ French-10M, German-1M } and English →{ French-10M, German-4.6M } to get values of G ↓ ( D ) given different D i and estimate the parameters in G ↓ ( D ) . Thus, the double power law could be summarized below, where M ( i ) ∞ stands for the sum of bias terms L ( i ) ∞ and O ( i ) ∞ :

Following the three steps, we estimate 1 the parameters of the double power law for our base-size model. For convenience, we present the estimated double power law below:

The law indicates an interesting result: the generalization performance trade-off among directions in MNMT is mostly relative to the training data size in each direction. The difference in languages only influences the bias term.

3.3 Evaluation and Discussion

We evaluate the estimated double power law in different task and data size combinations as shown in Figure 5. For different task settings, we only tune the bias term M ( i ) ∞ in Equation 6. As shown in Figure 5, the double power law can generally well reflect how the performance of one task changes with its sampling ratio for both of the high-resource and low-resource directions.

Generalization to more directions We also evaluate the double power law when there are more than 2 languages. We conduct a three-direction MNMT experiment on English → { French-10M, German-4.6M, Chinese-260K}, where English → Chinese is the low-resource direction. We fix the sampling ratio of English → German to 0.1 and change the sampling ratio of English → French from 0.1 to 0.8 and tune the bias term M ( i ) ∞ in Equation 6 to fit the performance curve of English → French and English → Chinese. As shown in Figure 6, the double power law can also well reflect the performance trade-off with more directions.

Generalization Cross Entropy

EN->{FR-10M, DE-4.6M, ZH-260K}

EN->{FR-10M, DE-4.6M, ZH-260K}

Predicted Curve

Observation

p

p

p(DE)=0.1

p(DE)=0.1

EN->FR Cross Entropy

EN->ZH Cross Entropy

★

5.4

★

5.0

5.2

5.0

4.5

4.8

4.0

4.6

4.4

3.5

4.2

4.0

3.0

★

3.8

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

The trade-off curve is language-agnostic As shown in Figure 5and 6, by adjusting the bias term in Equation 6, the predicted curve can well describe the trade-off with different languages setting, both the similar(FR and DE) and the distant (FR and ZH) ones. In other words, the shape of the trade-off curve is more related to the number of training examples in each direction rather than the language of the direction, which mostly influences the bias term of the curve. This result is in line with Xin et al. (2022); Shaham et al. (2022) that the training data size plays a more important role than language similarity in the trade-off of different directions' performance.

Critical point for low-resource directions When we look at the predicted performance curve and the observations for low-resource directions, it all shows a "U" shape. It means there exists a critical point when increasing the sampling ratio for low-resource direction. On the right side of the critical point, the Capacity Occupation term has more influence on the generalization performance. The Intrinsic Over-fitting term becomes dominant after the sampling ratio goes beyond the critical point. While for high-resource directions, the curve is monotonic.

3.4 Application of the Double Power Law

We adopt the Double Power Law to compute the optimal sampling ratio combination for MNMT. For a task with n directions, it is evaluated by the weighted sum of each direction's performance. r i is the weight of the i -th task and ∑ n i =1 r i = 1 .

Given the evaluation metrics' weights r and the number of training examples in each task d , the goal is to compute the optimal sampling ratio combination ˆ p , where we assume that all directions follow the double power law F in Equation 4. Thus, the optimization objective becomes:

4 Experiments

4.1 Settings

We conduct experiments on a two-direction English → { German-4.6M, Hindi-260K } task and a four-direction English →{ French-10M, German-4.6M, Chinese-260K, Hindi-260K } task using the same training configuration as in Section 2.2 in our base size model.

We use two different evaluation metrics: 1) Arithmetic mean ( r i = 1 n ). 2) Customized metric where we are interested in improving the performance of low-resource direction i ( r i = 1 , r j = i = 0 ).

̸

We solve Equation 7 with the estimated double power law as in Equation 6. Even though we do not know the bias term M ∞ in Equation 6, it does not influence the computation of the optimal sampling ratio combination ˆ p . With the computed ˆ p , we train the MNMT model according to Appendix B.

4.2 Baselines

Temperature-based Sampling We search T ∈ { 1 , 2 , 5 , 10 , 100 } , which are most frequently used in NMT literature. After choosing T, we set the sampling ratio p ′ i = p 1 /T i / ∑ K j p 1 /T j where p i denotes the ratio of the training examples from task i to the whole training set.

Gradient Manipulation We also compare our method to gradient manipulation methods that are reported to improve MNMT performance: 1) PCGrad(Yu et al., 2020) and 2) Gradient Vaccine(Wang et al., 2021). We employ the default configuration of the methods described in their corresponding papers. Gradient manipulation methods require computing the cosine similarity of gradients at each optimization step and adjusting the gradient direction to resolve conflicts, resulting in an increased computational budget along with the increase of directions.

4.3 Results

As shown in Table 2, we report both the generalization cross-entropy and detokenized BLEU score using sacrebleu 2 for different sampling methods.

Under the arithmetic mean metric, DPL reaches similar generalization cross-entropy compared to the best result when tuning temperature (T=2) and gains +0.3 average BLEU scores than T=2 in the 2-direction setting. In the 4-direction setting, DPL also gains better results(+0.1 BLEU) compared to the best result when tuning temperature (T=5). The best sampling temperature varies for different MNMT tasks while DPL could surpass the corresponding best results by just computing once, reducing the searching time to 1/5 of origin.

Comparing to the Gradient Manipulation methods, DPL leads to a much better average performance than the best method in both experiments (+0.6 BLEU in 2-direction , +2.3 BLEU in 4-direction),

with only 1/2 and 1/3 training time. The result is also in line with Xin et al. (2022) that simple scalarization could beat the complex optimization method if we set the weights properly.

Wealso evaluate the customized metric in the 2-direction experiment, where we hope to best enhance the performance of the low-resource direction. It turns out that DPL is more effective (+0.4 BLEU) in improving the low-resource direction than tuning the temperature (p<0.05).

In addition to using the BLEU score for evaluation, we also utilize model-based evaluation techniques, specifically Bert-Score (Zhang et al., 2020) and COMET (Rei et al., 2020). Research indicates that these metrics often align more closely with human evaluations than their model-free counterparts. As detailed in Table 3, DPL outperforms the best temperature-based method, showing an average improvement of 0.4 in BERT-Score (p<0.05) and 0.5 in COMET score (p<0.05).

The results justify that through the Double Power Law, we can compute the sampling ratio combination that best satisfies our expectation without manually searching the sampling ratios.

4.4 Discussion

The Double Power Law of model with different sizes As shown in Table 4, we evaluate the how the power terms of Double Power Law change in models with different sizes.

Among the parameters in the Double Power Law, we are more interested in power terms in the Capacity Occupation( α ) and Intrinsic Over-fitting( β ) term since they better reflect the sensitivity of performance changes against sampling ratio. We keep the other parameters the same as in Equation 6 and estimate the power terms in the Medium and Large size models.

It shows that the power term( β ) in the Intrinsic Over-fitting term increases rapidly with the growth in model size. While the power

term( α ) in the Capacity Occupation term changes much slower. For larger model, over-fitting phenomena of low-resource direction will be more severe and the final performance would be more likely to fall into the collapsed area when increasing the sampling ratio for low-resource directions.

Generalization of DPL to more diverse Directions To better test the generality of DPL computed as Equation 6, we ran experiments on 10 languages in X-EN from WMT10 (Callison-Burch et al., 2010). Dataset details are in Appendix C. Figure 7 indicates a Pareto Front Collapse in low-resource directions, with the double power law fitting well across data/language conditions ( r 2 values of 0.86 and 0.92 for low and high resources). As for the difference, we observed a more pronounced pareto front collapse in X-EN than in EN-X. By only adjusting the γ term in DPL to -0.25, the fit improves for X-EN directions ( r 2 : 0.86 to 0.91 for low-resources; 0.92 to 0.97 for high-resources), highlighting the significant effect of the Intrinsic Over-fitting term in X-EN directions.

The results combined show that we can rely on the instantiated DPL when the model and the training hyper-parameters (e.g., batch size, total update steps ) are not changed. The DPL is robust when introducing more languages or changing the translation directions.

6

★

hi-en

gu-en

5

★

★

tr-en

ro-en

ValidCrossEntropy

Iv-en

et-en

4

m

0.0

0.2

0.4

0.6

0.8

1.0

Proportion

(a) As the proportion increases, the validation loss goes down then goes up for low-resource languages.

3.50

★

fr-en

3.25

★

★

de-en

cs-en

3.00

CrossEntropy

2.75

2.50

Valid

2.25

2.00

1.75

1.50

0.0

0.2

0.4

0.6

0.8

1.0

Proportion

(b) The validation loss goes down when the proportion of high-resource direction increases.

5 Related Work

Many previous work studied the trade-off among different directions and how to achieve a better overall performance in MNMT. Aharoni et al. (2019); Shaham et al. (2022) empirically study how different settings influence the positive and negative interference among different directions. Xin et al. (2022)firstly incorporate Pareto optimity to the analysis of MNMT system. Fernandes et al. (2023) propose to use a single power law to model how one direction's performance changes with its sampling ratio under the assumption that all directions have unlimited data.

Another related line of works study multi-task optimization for MNMT. Yu et al. (2020); Wang et al. (2021) propose to dynamically altering the conflict gradients eliminate the negative interference among tasks. Wang et al. (2020a); Zhou et al. (2021) explore how to dynamically adjust the optimization objective along with the training process to better schedule the training for different directions. Li and Gong (2021) focus on the imbalance nature of MNMT task and propose an optimization method based on the local curvature of the loss landscape. However, Xin et al. (2022) recently show that simple scalariaztion method could beat many multitask optimization methods for MNMTif the sampling ratio for different tasks are correctly set.

While we focus on how to set the optimal weights, modeling how the size of neural network influences the performance also attracts much attention recently, i.e. the neural scaling law (Kaplan et al., 2020; Hoffmann et al., 2022; OpenAI, 2023). In the field of NMT, Ghorbani et al. (2021a); Gordon et al. (2021) show that the scaling behavior of cross-entropy loss follows a power law with the model size as input while Fernandes et al. (2023) extends the conclusions to multilingual setting.

6 Conclusion

In this work, we start from our observations that the trade-off front is no longer Pareto front in Multilingual Neural Machine Translation when there exists low-resource directions, i.e. the collapse of Pareto front. We analyse the reason causing the phenomena and find that there exists a balance between model capacity occupation and the intrinsic risk to over-fit for certain direction. Based on the findings, we propose the Double Power Law to model how the performance of a given direction changes with its sampling ratio, which is robust across different task settings and is capable of capturing the collapse phenomena. Finally, we frame the sampling ratio selection problem for Multilingual Neural Machine Translation as an optimization problem, which greatly reduces the cost for manually tuning the sampling temperature and reaches better results.

References

Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 3874-3884, Minneapolis, Minnesota. Association for Computational Linguistics.

Loïc Barrault, Ondˇ rej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) , pages 1-61, Florence, Italy. Association for Computational Linguistics.

Stephen P. Boyd and Lieven Vandenberghe. 2004. Convex optimization. IEEE Transactions on Automatic Control , 51:1859-1859.

Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, and Omar Zaidan, editors. 2010. Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR . Association for Computational Linguistics, Uppsala, Sweden.

Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 1723-1732, Beijing, China. Association for Computational Linguistics.

Patrick Fernandes, B. Ghorbani, Xavier García, Markus Freitag, and Orhan Firat. 2023. Scaling laws for multilingual neural machine translation. ArXiv , abs/2302.09650.

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 2020. Sharpness-aware minimization for efficiently improving generalization. ArXiv , abs/2010.01412.

Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier García, Ciprian Chelba, and Colin Cherry. 2021a. Scaling laws for neural machine translation. ArXiv , abs/2109.07740.

Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. 2021b. Scaling laws for neural machine translation. CoRR , abs/2109.07740.

Mitchell A Gordon, Kevin Duh, and Jared Kaplan. 2021. Data and parameter scaling laws for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 5915-5922, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Barry Haddow, Rachel Bawden, Antonio Valerio Miceli Barone, Jindvrich Helcl, and Alexandra Birch. 2021. Survey of low-resource machine translation. Computational Linguistics , 48:673732.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and L. Sifre. 2022. Training compute-optimal large language models. ArXiv , abs/2203.15556.

Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. 2020. Scaling laws for neural language models. ArXiv , abs/2001.08361.

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. 2016. On large-batch training for deep learning: Generalization gap and sharp minima. ArXiv , abs/1609.04836.

Bobby Kleinberg, Yuanzhi Li, and Yang Yuan. 2018. An alternative view: When does SGD escape local minima? In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 2698-2707. PMLR.

Xian Li and Hongyu Gong. 2021. Robust optimization for multilingual translation with imbalanced data. In Neural Information Processing Systems .

Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and Sam Tak Wu Kwong. 2019. Pareto multitask learning. In Neural Information Processing Systems .

OpenAI. 2023. Gpt-4 technical report. ArXiv , abs/2303.08774.

Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations .

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation.

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. ArXiv , abs/1511.06709.

Uri Shaham, Maha Elbayad, Vedanuj Goswami, Omer Levy, and Shruti Bhosale. 2022. Causes and cures for interference in multilingual translation. ArXiv , abs/2212.07530.

Nllb team, Marta Ruiz Costa-jussà, James Cross, Onur cCelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Alison Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon L. Spruit, C. Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm'an, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. ArXiv , abs/2207.04672.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc.

Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. 2020a. Balancing training for multilingual neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 8526-8537, Online. Association for Computational Linguistics.

Yiren Wang, ChengXiang Zhai, and Hany Hassan. 2020b. Multi-task learning for multilingual neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1022-1034, Online. Association for Computational Linguistics.

Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. 2021. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. ICLR .

Derrick Xin, B. Ghorbani, Ankush Garg, Orhan Firat, and Justin Gilmer. 2022. Do current multi-task optimization methods in deep learning even help? ArXiv , abs/2209.11379.

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. ArXiv , abs/2001.06782.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore:

Evaluating text generation with bert. In International Conference on Learning Representations .

Chunting Zhou, Daniel Lévy, Xian Li, Marjan Ghazvininejad, and Graham Neubig. 2021. Distributionally robust multilingual machine translation. ArXiv , abs/2109.04020.

A Datasets for Main Experiments

B Training Details of Multilingual NMT experiments

In this section, we would provide concrete details of our experimental setup for reproduction.

Similar to Xin et al. (2022)'s setting, we conduct our experiments on three Transformer (Vaswani et al., 2017) models with different sizes as shown in Table 6

For all of our model, we use a 64k sentence piece vocabulary. All models are trained 3 with 4k warmup steps with the learning rate linearly increasing from 0 to 3 e -4 then decreasing with inverse_sqrt learning rate scheduler. The label smoothing term is set to 0.1 following the NMT literature convention. Evaluation is done every 5k steps, and we choose the best checkpoint with lowest average validation loss before examining the final generalization performance.

C Datasets for Generality Testing

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper frames the sampling ratio selection problem of MNMT as an optimization problem, and introduces a double power law to control the sampling ratio for different languages. The author argue that traditional method assumes unlimited training data which does not hold true for low resource settings, thus both model capacity occupation and intrinsic risk of overfitting should be taken into consideration during training data sampling.  \nExperiments on multiple language direction shows better accuracy (+0.3 BLEU) with only half training cost.\n\nSTRENGTHS:\nThis paper is an upgrade on the Fernandes et.al 2023 paper, from single power law to double power law, \ntaking both model capacity occupation and intrinsic risk of overfitting into consideration. \nI think over the years the sampling strategy of MNMT has always been on data balancing, while the overfitting risk for low resource language has largely been ignored. And this paper addresses an important problem in this direction.\nI like the organization of this paper, starting from an empirical phenomena, find the problem and solve it with a well-grounded method.\n\nWEAKNESSES:\nHowever, I believe there is still room for improvement in this paper:\n1.  I think the double power law is still not enough to resolve the sampling problem. One important aspect is the data quality: so in the low resource setting, the quality of training data is often worse comparing with the rich resource setting, and the noisy data would also cause the performance drop (or local optimum ) during training. So if we are talking about a more realistic MNMT scenario, the data quality issue should definitely be considered.  I would like to see more analysis in this direction.\n\n2. There are some confusing claims in the paper :  A. in the intro: \"achieving better accuracy (+0.3 BLEU) with only half training cost\". \nBut in the table 2, the statics shows something different : for example in (a),  comparing with temperature based method,  BLEU +0.1, training time both 1.0x,  with Gradient based method, BLEU +2.4, training time 1.0x vs 3.2x.   So I am not sure which one is the key comparison to look at.  Besides, when comparing the training cost, the time to estimate the parameters of the double power law should also be included.  B. in abstract, the author claims \"by training over 200 multilingual models\", but there is no evidence in the experiments, only a two-direction model and a four-direction model.   So I think the author should be more careful in making these claims. \n\n3. The improvement over traditional method is marginal. Comparing with Temperature based method, the proposed method doesn't show enough improvement in both BLEU score and training time.\n\nQUESTIONS:\n1. So if we change the corpus size of certain language direction, should the parameters of the double power be re-estimated, and what is the time cost for this?\n\nLIMITATIONS:\nI think the two major limitation of this paper are the ignorance of data quality issue in MNMT training and the less convincing performance comparing with temperature based methods.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the influence of language pair sampling weights on the ultimate performance of Multilingual Neural Machine Translation (MNMT), specifically through the lens of the Pareto front concept. Existing literature suggests that under certain conditions, diverse sampling weights can yield solutions that constitute a Pareto front. Here, the performance of a specific task is positively related to its assigned weight. \n\nHowever, this work introduces the concept of Pareto front collapse, where the performance of a low-resource direction might deteriorate as its weight escalates, especially under data-imbalance scenarios. To explain the reason for the collapse, the authors argue that increasing the sampling weight for a low-resource task not only heightens model capacity for it but also amplifies the risk of overfitting.\n\nTo predict the performance trade-off curve among different directions, the authors propose a Double Power Law (DPL) that considers capacity occupation and intrinsic over-fitting. Experimental results show that DPL captures the performnce curve successfully under various setting (including Pareto front collapse) and can used to determined the optimal sampling ratio for MNMT.\n\nSTRENGTHS:\n* This paper delves into a more realistic scenario of MNMT compared to previous works, wherein the data sizes of different language pairs are extremely imbalanced , and highlights the issue of the Pareto front collapse in this situation.\n* The paper provides an insightful explanation for the collapse: low-resource tasks bring about the risk of overfitting as their weights are increased. This interpretation aligns intuitively with empirical results.\n* The proposed DPL can predict the task performance given the data size and sampling weight which can be further used to select the optimal samlping ratios.\n\nWEAKNESSES:\n* The paper's assessment of translation performance primarily leans on BLEU, which has shown to be less reliable in terms of aligning with human judgment compared to metrics like COMET and BLEURT. This reliance on BLEU could be justified if experiments were conducted on truly low-resource languages unsupported by these advanced metrics. However, the languages featured in the study (English, German, French) are all high-resource, making this justification less feasible.\n* Despite considering data imbalance scenarios, the study mostly focuses on translations from English to other languages, thus straying from a truly realistic scenario. The paper lacks supporting evidence that its findings can be generalized to more diverse cases, such as English to other languages, or non-English-centric MNMT.\n* The authors have not included two crucial summaries about the DPL:\n  * When can one rely on an already instantiated DPL or, conversely, when do the parameters of the DPL need to be re-estimated: e.g., when the model size changes, more language pairs are introduced, or when transitioning from English-centric to non-English-centric?\n  * How many times the model needs to be retrained to re-estimate the parameters of DPL in different cases? What if I only need to estimate the sample ratios (no bias terms)?\n* The training time recorded in Table 2 does not factor in time spent on hyperparameter searching, such as temperature-based sampling or DPL parameter estimation. Therefore, clarification on the above two points is crucial.\n\nQUESTIONS:\nMain questions are listed in *Weaknesses*. Another question here is why Figure 2 and Figure 3 describe Pareto front collapse using different indicators, BLEU, and cross-entropy.\n\nLIMITATIONS:\nMain limitations are listed *Weaknesses*.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the multi-task scaling laws for multilingual machine translation (mNMT) in the presence of *limited data* for at least one of the tasks/language pairs.\n\nExtending Fernandes et. al. (2022) work on the data-rich scenario, the authors found that their proposed multi-task scaling law doesn't hold when data for different languages pairs is imbalanced: in this unbalanced scenario, there is a collapse of the *Pareto frontier*: increasing the weight of one LP doesn't not necessarily increase the performance of that model in that LP.\n\nTo address this, they introduced a novel *Double Power Law* (DPL) that introduces an extra term to Fernandes et. al. (2022)'s law that takes into account *overfitting*. Based on this DPL, they also porpose a method for finding the optimal weighting of the LPs, showing that it outperforms other multi-task optimizers according to BLEU.\n\nSTRENGTHS:\nof your tasks is low-resource. While this paper focus on mNMT, this is aksi the case (implicitly) for most LLMs trained today. \n\nPrior work by Fernandes et. al. (2023) characterized the trade-off in the case where all tasks have pratically unlimited data, and introduced a multi-task scaling law for balancing tasks/LPs in this case. However, as the authors point out, in real life most tasks (language pairs in MT) have very limited data, and so the introduced DPL scaling law has the potential to have a lot of practical impact in balancing languages/tasks during the training of models. Their analysis relating sharpness and overfitting to the collapse of the pareto frontier is also quite interesting.\n\nWEAKNESSES:\nDespite the potential impact of the this work, I think it is quite limited by its small analysis of impact of *data* term: they introduce powerlaw term for data and add an overfitting term. However the proposed Double Power Law is given very little theoretical motivation, and little empirical validation (with non of the standard fitting metrics like r2 reported). This is exharcebated by the fact that the data terms are fitted with two or three different possible data sizes, and its hard to trust the scaling law without some validations. \n\nNevertheless the current findings are already quite prevalent, but the analysis is further limited by the fact they only explored one “direction” of multiliguality: and language pairs are *from* English into XXX. Fernandes et. al. (2023) also explored the opposite XX→EN direction, and found different behavior in the Pareto curves, so understanding what happens for LR in that direction is quite pertinent.\n\nQUESTIONS:\n- In the DPL equation (5), the “capacity occupatio ratio” seems to the same as the “effective parameter ratio” in Fernandes et al. (2023), but its functional form is different (for example, the main on in the paper had 3 fittable parameters). Why did you change it?\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes to model the scaling law of multilingual machine translation using the double power law, which considers both the sampling ratio of a language and the size of the corpus. The motivation is that simply increasing the sampling ratio of low-resource language might lead to decreased performance for both high and low-resource languages due to overfitting. The paper then uses the estimated formula to improve multilingual NMT that jointly trains on two or three languages.

STRENGTHS:
1. The paper is well written and it includes a good motivation and analysis section on why we should consider dataset size in the scaling law estimation
2. The resulting method is compared to strong baselines on multilingual NMT system for 2 to 4 languages. The method does seem to bring some improvement over the baseline.

WEAKNESSES:
1. The experiment is restricted to maximum of 4 languages, which is rather limited. It would be more helpful and convincing to the readers if there are multilingual NMT experiment for 10+ languages
2. The paper focuses on addressing the overfitting issue with low-resource languages, but the experiment only includes Hindi and Chinese as the “low-resource” languages.

QUESTIONS:
Have you considered using more true low-resource languages, such as some languages listed here: https://www.statmt.org/wmt21/large-scale-multilingual-translation-task.html

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a power law for multilingual machine translation, a magic formula and constants to predict the performance of multilingual machine translation, that trades the model capacity and generalization based on the number of parameters and the training data sizes. The law is derived from the observation that the Pareto front, the performance curve by treating each language direction as an objective, collapses when the training data is highly imbalanced. They hypothesize that the root cause is the trade of the risk of overfitting when upsampling low resource, and thus, derived the power low. Experimental results show that the simple law achieved better results when compared with other competitive methods, e.g., simply tweaking temperature parameter to affect sampling ration and/or gradient manipulation-based methods.\n\nSTRENGTHS:\n* The analysis of Pareto front is very interesting by treating each language direction as an objective, and it clearly shows the collapse when training data size is largely skewed. \n\n* The derived power law is very simple, yet backed by the observation of the Pareto front analysis.\n\n* Extensive analysis on English-to-{French,German,Chinese,Hindi} directions and the empirical results show the effectiveness of the proposed power law.\n\nWEAKNESSES:\n* It is experimented only for en-to-X directions, and not X-to-en directions. I'd like to know if the law is also applicable to those directions, in addition to X-to-X directions.\n\n* Similarly, it would be nice to investigate zero-shot directions by focusing on language families to see if transferability will also follow similar trends.\n\n* Translation qualities are measured only by BLEU. It would be good to add different metrics, e.g., COMET or BERTScore, to see if similar trends are happening to those metrics.\n\nQUESTIONS:\n* Does the double power law hold for X-to-en directions? How about X-to-X directions in general?\n\nLIMITATIONS:\nThis wok needs discussion on the applicability to a model with more diverse languages.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}