SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Liang Chen $^{1*}$  Shuming Ma $^{2*}$  Dongdong Zhang $^{2}$  Furu Wei $^{2}$  Baobao Chang $^{1\dagger}$
$^{1}$ National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University
leo.liang.chen@outlook.com chbb@pku.edu.cn
{shumma,dozhang,fuwei}@microsoft.com
In this work, we study how the performance of a given direction changes with its sampling ratio in Multilingual Neural Machine Translation (MNMT). By training over 200 multilingual models with various model sizes, data sizes, and language directions, we find it interesting that the performance of certain translation direction does not always improve with the increase of its weight in the multi-task optimization objective. Accordingly, scalarization method leads to a multitask trade-off front that deviates from the traditional Pareto front when there exists data imbalance in the training corpus, which poses a great challenge to improve the overall performance of all directions. Based on our observations, we propose the Double Power Law to predict the unique performance trade-off front in MNMT, which is robust across various languages, data adequacy, and the number of tasks. Finally, we formulate the sample ratio selection problem in MNMT as an optimization problem based on the Double Power Law. In our experiments, it achieves better performance than temperature searching and gradient manipulation methods with only 1/5 to 1/2 of the total training budget. We release the code at https://github.com/pkunlp-icler/ParetoMNMT for reproduction.
Multilingual Neural Machine Translation (MNMT) enables multiple translation directions in a single model, significantly reducing deployment costs of translation systems and benefiting low-resource directions (Sennrich et al., 2015). MNMT can be framed as a multi-task learning problem, where we expect to improve the overall performance across translation directions. However, it is infeasible to find an optimal solution for all translation tasks because of the negative interference among translation directions (Lin et al., 2019; Yu et al., 2020; Wang et al., 2021). A common method to make trade-offs among different languages is scalarization, where a combination of normalized weights is assigned to the loss of each direction during training. Xin et al. (2022) empirically show that different weight combinations can result in Pareto optimal solutions, where no one solution could outperform another in all tasks. All such solutions could form a Pareto front as shown in Figure 1. Fernandes et al. (2023) show that there is a power law between the weight of one direction and its final generalization performance, indicating a positive relation between weight and performance.
In this work, we empirically find that the positive relation together with the power law from Fernandes et al. (2023) does not hold true in a more realistic MNMT scenario where the data for different directions is highly imbalanced. The performance of a direction may decrease as its weight increases.
It raises our interests that 1) What causes the inconsistency phenomena? 2) Can we model the tradeoff behavior among different directions? 3) How should we set the weights for each direction?
To answer these questions, we first identify the problem in MNMT that the scalarization method could not generate Pareto optimal solutions when there exist low-resource directions, which is named the collapse of Pareto front. Secondly, to quantify how the performance of a given direction changes with its sampling ratio, we propose the Double Power Law. The law takes the number of training data into consideration and explicitly models the relationship between capacity occupation and the potential of over-fitting in a given direction, which has the form:
$$
\mathcal {F} _ {i} (p, D _ {i}) = \underbrace {(k \cdot p) ^ {- \alpha}} _ {\text {C a p a c i t y O c c u p a t i o n}} + \underbrace {\left(D _ {i} ^ {\gamma} + b\right) \cdot (q \cdot p) ^ {\beta}} _ {\text {I n t r i n s i c O v e r - f i t t i n g}} + \underbrace {M _ {\infty} ^ {(i)}} _ {\text {B i a s T e r m}} \tag {1}
$$
where  $\mathcal{F}_i$  is the predicted performance of task  $i$  measured by cross-entropy loss,  $p$  is the sampling ratio,  $D_{i}$  is the number of training examples and  $M_{\infty}^{(i)}$  is a constant bias term. The rest are fixed parameters to estimate. Experiments justify the robustness of the law under various settings.
Based on the Double Power Law (DPL), we propose an approach to compute the optimal sampling ratios. Our experiments demonstrate that DPL outperforms directly search for the optimal temperature and gradient manipulation methods for MNMT, achieving better accuracy  $(+0.3$  BLEU) with only 1/5 to 1/2 training cost in our experiments.
To summarize, the contribution of this work is threefold:
1. We identify and analyze the collapse of Pareto Front phenomenon in MNMT.
2. We propose the Double Power Law to model the trade-off among different directions in MNMT, which is robust on languages, data adequacy, and number of directions.
3. Based on the Double Power Law, we propose a method to automatically compute the optimal sampling ratio combination, eliminating the need for manual tuning of the sampling temperature and achieving better performance in BLEU, COMET and BertScore metrics.
2 Collapse of Pareto Front in Multilingual Neural Machine Translation
2.1 Pareto Front in MNMT
Pareto Front In multi-task learning, where  $K$  is the number of objectives and  $\mathcal{L}_i(\theta)$  is the loss of task  $i$  under solution parameterized by  $\pmb{\theta} \in \mathbb{R}^n$ , we say solution  $\theta_2$  Pareto dominates solution  $\theta_1$  if  $\forall 1 \leq i \leq K, \mathcal{L}_i(\theta_2) \leq \mathcal{L}_i(\theta_1)$  and  $\exists 1 \leq i \leq K, \mathcal{L}_i(\theta_2) < \mathcal{L}_i(\theta_1)$ . The Pareto front is a set of solutions where no solution Pareto dominates another. As shown in Figure 1, we give two toy examples of the possible shape of Pareto front when  $K = 3$  and  $K = 2$ .
Figure 1: Toy examples of Pareto front. When there are three tasks  $(\mathrm{K} = 3)$ , we color the Pareto front according to task 3's loss. We also plot the contours, which are also Pareto fronts in 2-dimension.
Scalarization (Boyd and Vandenberghe, 2004) is a traditional method to optimize the multi-task learning model where a set of predefined normalized weights  $\boldsymbol{w}$  is applied to each task. If there is little interaction among different tasks, that one task's loss would drop when increasing its weight, equation 2 with different  $\boldsymbol{w}$  could guarantee solutions that can form a Pareto front (Xin et al., 2022).
$$
\hat {\boldsymbol {\theta}} (\boldsymbol {w}) = \arg \min  _ {\boldsymbol {\theta}} \mathcal {L} (\boldsymbol {\theta}; \boldsymbol {w}) \quad \text {s . t .} \quad \mathcal {L} (\boldsymbol {\theta}; \boldsymbol {w}) = \sum_ {i = 1} ^ {K} w _ {i} \mathcal {L} _ {i} (\boldsymbol {\theta}), \quad \boldsymbol {w} > 0, \quad \sum_ {i} w _ {i} = 1 \tag {2}
$$
Figure 2: Generalization performance trade-off curves under different data distributions. The blue line stands for English  $\rightarrow$  {German-4.6M, Chinese-10M} (left) and English  $\rightarrow$  {French-10M, Chinese-10M} (right). The orange line stands for the result when replacing the high-resource English  $\rightarrow$  Chinese-10M to English  $\rightarrow$  Chinese-260K. The Pareto front collapses (sampling ratio increases while performance drops) when changing the high-resource direction to a low-resource one.
Figure 3: Generalization Performance trade-off curves under different data distributions under the 3-direction setting. The left figure shows a Pareto front where one direction's performance will always decrease when another direction's performance increases. However, the performance of low-resource direction would increase in step with the high-resource direction's at some points.
Multi-task Learning for MNMT MNMT can be framed as a multi-task learning problem where one model can do the translation among multiple directions (Dong et al., 2015; Aharoni et al., 2019). To prevent the high-resource directions from dominating the training and to improve the low-resource directions' performance, scalarization is implemented through proportional sampling, where the sampling ratio of examples for task  $i$  in a training batch is  $w_{i}$ . Currently, there is no consensus on how to directly set specific  $w_{i}$  for each task, temperature-based sampling is more commonly used where  $w_{i} = \frac{p_{i}^{1 / T}}{\sum_{j}^{K} p_{j}^{1 / T}}$ ,  $p_{i}$  denotes the ratio of the training examples from task  $i$ .
2.2 The Collapse of Pareto Front WidelyExists
We use datasets provided in WMT10 (Wang et al., 2020b) and WMT19 (Barrault et al., 2019) to conduct the MNMT experiment. The description of dataset is listed in Appendix A. In the 2-task setting, we experiment with the sampling ratio of one task ranging from  $10\%$  to  $90\%$ . In the 3-task setting, we experiment with each task ranging from  $10\%$  to  $80\%$  resulting in 32 different ratio combinations. More detail about the training is in Appendix B. The experiment results and major findings in this section are illustrated in Figure 2,3 and 4.
The collapse exists in different language pairs In Figure 2, we can see that the generalization performance trade-off situation is different between the High-High and High-Low data settings. Under the High-High setting, the solutions from scalarization could form a Pareto front where one task's performance increases with the decrease in the other task. This is expected and also in line with the recent results from Xin et al. (2022); Fernandes et al. (2023). However, the Pareto front disappears when we replace the English  $\rightarrow$  Chinese-10M with English  $\rightarrow$  Chinese-260K. Fernandes et al. (2023) restrict the directions to high-resource ones so they also get Pareto optimal results, Xin et al. (2022)
also find that there exists a global optimal point in imbalanced multilingual translation however do not give it an explanation. Understanding the reason can help build a better MNMT system since the multilingual training data is highly imbalanced in reality(Haddow et al., 2021; team et al., 2022).
In Figure 2, when increasing the sampling ratio for one High-Resource direction, its performance continually goes up. While increasing the sampling ratio for one Low-Resource direction from  $10\%$  to  $90\%$ , its performance first goes up and then gradually goes down. This is the direct cause of the collapse of Pareto front. The collapse also exists when there are more than two languages. As shown in Figure 3, we draw the 3-task trade-off front using linear interpolation. The front in the left where all three directions are high-resource, is most similar to the example 3D Pareto front in Figure 1. When there exists low-resource directions as shown in the middle and the right in Figure 3, the Pareto front would disappear.
It indicates that the relation between High and Low resources directions is not a zero-sum game and there exists a sampling ratio range where different tasks can improve or deteriorate simultaneously. For example, in Figure 2,
this range could be  $40\%$  to  $90\%$  sampling ratio for the low-resource English-to-Chinese direction.
Figure 4: Performance (BLEU) tradeoff curves for English  $\rightarrow$  {French, Chinese} under different model sizes and data distributions.
The collapse exists in models with different sizes Figure 4 shows the trade-off curves on models with different sizes. The Pareto front would collapse when there exist low-resource directions.
The collapse of Pareto front complicates the optimization of MNMT because the sampling ratio must be carefully set or the performance might fall in the collapsed area where the model's capacity is not fully exploited and there exist other solutions that can Pareto dominate the current performance.
2.3 Explore the Reason for the Collapse
The collapse is closely related to inadequate data. If we only consider the performance of low-resource direction in Figure 2, the performance of the low-resource direction briefly improves with an increase in the sampling ratio from  $10\%$  to  $20\%$ , but then gradually deteriorates as the sampling ratio continues to increase from  $20\%$  to  $90\%$ . This is in contrast to the performance of high-resource directions, which continue to improve with an increase in sampling ratio from  $10\%$  to  $90\%$ .
The effect of increasing the sampling ratio for a task is twofold: on the one hand, it gives it more weight and the model would allocate more capacity to optimize its performance, on the other hand, it also increases its risk of overfitting. For directions with abundant training examples, the over-fitting term is less significant so its performance would keep improving with more sampling ratio. However, for low-resource directions, the risk of over-fitting would be dominant for a high sampling ratio because of the lack of training examples. A small sampling ratio serves as a form of regularization for these low-resource
directions. This difference accounts for the contrasting performance trends observed between low-resource and high-resource directions.
It has been shown that the model would be less likely to over-fit and generalize better when it reaches a flatter local minimum after the gradient descent optimization (Keskar et al., 2016; Kleinberg et al., 2018; Foret et al., 2020). To justify our assumption, following Keskar et al. (2016), we compute the partial local curvature i.e. sharpness of the optimized model through the trace of the Hessian matrix as shown in Equation 3, where  $X$  stands for held-out evaluation data and  $Y$  stands for the label. We compute the sharpness of the FFN layer at different transformer layers.
Ratio(De:Hi)	Encoder Last	Decoder Last
1:9	21.80/30.96	32.96/42.98
5:5	11.87/20.12	25.88/33.59
9:1	10.43/14.48	23.88/27.86
Table 1: The sharpness for  $\mathrm{En}\rightarrow \{\mathrm{De}\}$  /  $\mathrm{En}\rightarrow \{\mathrm{Hi}\}$  tasks. The result indicates that with more high-resource data, the model tends to come to a flatter local minima after training.
$$
\operatorname {S h a r p n e s s} \left(f _ {\theta}, X, Y\right) = t r \left(\nabla^ {2} f _ {\theta} (X, Y)\right) \tag {3}
$$
As shown in Table 1, we compute the sharpness of our MNMT model on English  $\rightarrow$  {German, Hindi} task with different sampling ratio combinations. The result shows that with more sampling ratio for high-resource direction, the model could reach a flatter local minimum for both tasks.
The results of the generalization performance and sharpness experiments indicate that adding weight for a certain translation direction is a double-edged sword. It forces the model to optimize more towards minimizing the training loss of this task, at the same time increasing its over-fitting risk, especially for the low-resource directions.
3 The Double Power Law
In the previous section, we discussed the collapse of Pareto front phenomena and its impact on MNMT. The collapse presents a challenge for determining the sampling ratio combination, as a poor combination can adversely affect the final performance by deviating from potential Pareto solutions. Additionally, we identified over-fitting as a significant problem in MNMT training especially for low-resource directions. While a straightforward solution would be to grid search the sampling ratio combination, it becomes impractical with increasing directions due to the exponentially growing search space. In this section, we are going to answer the following questions:
Is there a pattern behind the collapse phenomena? Can we predict the performance trade-off curve among different directions?
Fernandes et al. (2023) adopt single power law inspired by the scaling literature in NMT (Ghorbani et al., 2021b; Gordon et al., 2021) to predict how the sampling ratio affects the performance of MNMT. However, it assumes that all directions have unlimited data, which does not reflect the diverse data adequacy in reality for supervised MNMT training. Under Fernandes et al. (2023)'s power law, the trade-off curve of any two directions can form a Pareto front, which could not capture the collapse.
We find out that: the generalization performance  $\mathcal{F}$  of the  $i$ -th direction in MNMT follows a double power law, which takes the sampling ratio  $p$  and the number of training examples  $D_{i}$  as input:
$$
\mathcal {F} _ {i} (p, D _ {i}) = \underbrace {(k \cdot p) ^ {- \alpha} + L _ {\infty} ^ {(i)}} _ {\text {C a p a c i t y O c c u p a t i o n}} + \underbrace {\mathcal {Q} ^ {\downarrow} (D _ {i}) \cdot (q \cdot p) ^ {\beta} + O _ {\infty} ^ {(i)}} _ {\text {I n t r i n s i c O v e r - f i t t i n g}} \tag {4}
$$
where  $\mathcal{F}_i$  is measured through evaluation cross-entropy loss.  $k, \alpha, q, \beta$  are fixed parameters,  $\mathcal{G}^{\downarrow}(D)$  is a monotonically decreasing function of the number of training examples  $D_i$ .  $L_{\infty}^{(i)}$  and  $O_{\infty}^{(i)}$  are bias terms that are relevant to the direction and the model.
3.1 Capacity Occupation and Intrinsic Over-fitting Terms
Based on our previous discussion, the generalization performance for a certain direction not only depends on its sampling ratio but also on its number of training examples. So we set two terms to jointly model the generalization performance, namely Capacity Occupation term and Intrinsic Overfitting term. As shown in Equation 4, similar to the power law in Fernandes et al. (2023), Capacity Occupation term can reflect that with a larger sampling ratio, the model would assign more capacity for the task to minimize its training loss, which could possibly improve the generalization performance. Intrinsic Over-fitting term reflects the fact that with a larger sampling ratio, the model is more likely to over-fit on this task, which would harm its generalization performance. Meanwhile, the scale of Intrinsic Over-fitting term is negatively related to the number of training examples. With fewer training examples, the influence of Intrinsic Over-fitting would be more significant.
3.2 Parameter Estimation
We estimate the parameters of the double power law in three steps. We conduct a series of English  $\rightarrow$  {French-10M, German-260K} experiments with English  $\rightarrow$  French-10M sampling ratio ranging from  $10\%$  to  $90\%$ . 1) We estimate the parameters of the Capacity Occupation term with English  $\rightarrow$  French's results by ignoring the Intrinsic Over-fitting term because it has minor effect when  $D_{i}$  is large for high-resource direction. 2) We fix the parameters of the Capacity Occupation term and estimate the parameters of the Intrinsic Over-fitting term with the results of
Figure 5: Evaluation of Equation 6 under different languages and data-adequacy settings. The curve predicts how one direction's generalization performance changes with its sampling ratio. The Double Power Law is robust under different scenarios.
English  $\rightarrow$  German-260K. 3) We fix all parameters in Equation 4 except for the value of  $\mathcal{G}^{\downarrow}(D)$ . We suppose that  $\mathcal{G}^{\downarrow}(D)$  also subjects to the form of power law and conduct the same series of experiments on English  $\rightarrow$  {French-10M, German-1M} and English  $\rightarrow$  {French-10M, German-4.6M} to get values of  $\mathcal{G}^{\downarrow}(D)$  given different  $D_{i}$  and estimate the parameters in  $\mathcal{G}^{\downarrow}(D)$ . Thus, the double power law could be summarized below, where  $M_{\infty}^{(i)}$  stands for the sum of bias terms  $L_{\infty}^{(i)}$  and  $O_{\infty}^{(i)}$ :
$$
\mathcal {F} _ {i} (p, D _ {i}) = \underbrace {(k \cdot p) ^ {- \alpha}} _ {\text {C a p a c i t y O c c u p a t i o n}} + \underbrace {\left(D _ {i} ^ {\gamma} + b\right) \cdot (q \cdot p) ^ {\beta}} _ {\text {I n t r i n s i c O v e r - f i t t i n g}} + \underbrace {M _ {\infty} ^ {(i)}} _ {\text {B i a s T e r m}} \tag {5}
$$
Following the three steps, we estimate<sup>1</sup> the parameters of the double power law for our base-size model. For convenience, we present the estimated double power law below:
$$
\mathcal {F} _ {i} (p, D) = \left(0. 0 7 \cdot p\right) ^ {- 0. 2 0} + \left(D ^ {- 0. 3 3} - 0. 5 0\right) \cdot \left(1. 1 8 \cdot p\right) ^ {1. 2 1} + M _ {\infty} ^ {(i)} \tag {6}
$$
The law indicates an interesting result: the generalization performance trade-off among directions in MNMT is mostly relative to the training data size in each direction. The difference in languages only influences the bias term.
3.3 Evaluation and Discussion
We evaluate the estimated double power law in different task and data size combinations as shown in Figure 5. For different task settings, we only tune the bias term  $M_{\infty}^{(i)}$  in Equation 6. As shown in Figure 5, the double power law can generally well reflect how the performance of one task changes with its sampling ratio for both of the high-resource and low-resource directions.
Generalization to more directions We also evaluate the double power law when there are more than 2 languages. We conduct a three-direction MNMT experiment on English  $\rightarrow$  {French-10M, German-4.6M, Chinese-260K}, where English  $\rightarrow$  Chinese is the low-resource direction. We fix the sampling ratio of English  $\rightarrow$  German to 0.1 and change the sampling ratio of English  $\rightarrow$  French from 0.1 to 0.8 and tune the bias term  $M_{\infty}^{(i)}$  in Equation 6 to fit the performance curve of English  $\rightarrow$  French and English  $\rightarrow$  Chinese. As shown in Figure 6, the double power law can also well reflect the performance trade-off with more directions.
we use scipy.optimize.curve_fit function from the scipy library. The unit of training examples is million.
Figure 6: We examine Equation 6 under the 3-direction setting. Double Power Law can well describe the performance trend of both the high-resource (FR-10M) and low-resource directions (ZH-260K).
The trade-off curve is language-agnostic As shown in Figure 5 and 6, by adjusting the bias term in Equation 6, the predicted curve can well describe the trade-off with different languages setting, both the similar(FR and DE) and the distant (FR and ZH) ones. In other words, the shape of the trade-off curve is more related to the number of training examples in each direction rather than the language of the direction, which mostly influences the bias term of the curve. This result is in line with Xin et al. (2022); Shaham et al. (2022) that the training data size plays a more important role than language similarity in the trade-off of different directions' performance.
Critical point for low-resource directions When we look at the predicted performance curve and the observations for low-resource directions, it all shows a "U" shape. It means there exists a critical point when increasing the sampling ratio for low-resource direction. On the right side of the critical point, the Capacity Occupation term has more influence on the generalization performance. The Intrinsic Over-fitting term becomes dominant after the sampling ratio goes beyond the critical point. While for high-resource directions, the curve is monotonic.
3.4 Application of the Double Power Law
We adopt the Double Power Law to compute the optimal sampling ratio combination for MNMT. For a task with  $n$  directions, it is evaluated by the weighted sum of each direction's performance.  $r_i$  is the weight of the  $i$ -th task and  $\sum_{i=1}^{n} r_i = 1$ .
Given the evaluation metrics' weights  $\boldsymbol{r}$  and the number of training examples in each task  $\boldsymbol{d}$ , the goal is to compute the optimal sampling ratio combination  $\hat{\boldsymbol{p}}$ , where we assume that all directions follow the double power law  $\mathcal{F}$  in Equation 4. Thus, the optimization objective becomes: