SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\hat {\boldsymbol {p}} = \arg \min  _ {\boldsymbol {p}} \mathcal {L} (\boldsymbol {p}; \boldsymbol {r}; \boldsymbol {d}) \quad \text {s . t .} \quad \mathcal {L} (\boldsymbol {p}; \boldsymbol {r}; \boldsymbol {d}) = \sum_ {i = 1} ^ {n} r _ {i} \mathcal {F} _ {i} \left(p _ {i}, d _ {i}\right), \quad \boldsymbol {p} > 0, \quad \sum_ {i} p _ {i} = 1 \tag {7}
$$
We conduct experiments on a two-direction English  $\rightarrow$  {German-4.6M, Hindi-260K} task and a four-direction English  $\rightarrow$  {French-10M, German-4.6M, Chinese-260K, Hindi-260K} task using the same training configuration as in Section 2.2 in our base size model.
We use two different evaluation metrics: 1) Arithmetic mean  $(r_i = \frac{1}{n})$ . 2) Customized metric where we are interested in improving the performance of low-resource direction  $i$  ( $r_i = 1, r_{j \neq i} = 0$ ).
We solve Equation 7 with the estimated double power law as in Equation 6. Even though we do not know the bias term  $M_{\infty}$  in Equation 6, it does not influence the computation of the optimal sampling ratio combination  $\hat{p}$ . With the computed  $\hat{p}$ , we train the MNMT model according to Appendix B.
(a) 4-direction (Arithmetic mean)
Methods	Cross-Entropy ↓	BLEU ↑	TrainingTime
FR	DE	ZH	HI	AVG	FR	DE	ZH	HI	AVG
Temperature-based Sampling
T=1	1.58	1.55	4.73	3.21	2.77	28.3	28.2	15.7	6.4	19.7	1.0x
T=2	1.59	1.61	3.93	2.59	2.43	27.7	28.1	20.5	10.9	21.8	1.0x
T=5	1.65	1.63	3.83	2.56	2.42	27.2	28.0	21.1	10.8	21.8	1.0x
T=10	1.70	1.76	3.77	2.48	2.43	25.8	27.2	21.8	11.4	21.6	1.0x
T=100	1.75	1.83	3.79	2.54	2.48	25.4	26.7	22.1	11.0	21.3	1.0x
Gradient Manipulation
PCGrad	1.57	1.59	4.78	3.28	2.81	28.1	28.5	15.3	6.3	19.6	3.2x
Gradient Vaccine	1.58	1.62	4.94	3.41	2.89	28.0	28.5	15.0	5.9	19.4	3.2x
DPL (Arithmetic mean)	1.64	1.63	3.83	2.53	2.41	27.3	28.0	20.9	11.3	21.9	1.0x
(b) 2-direction (Arithmetic mean & customized metric)
Methods	Cross-Entropy ↓	BLEU ↑	Training Time
DE	HI	AVG	DE	HI	AVG
Temperature-based Sampling
T=1	1.41	3.20	2.31	36.6	8.4	22.5	1.0x
T=2	1.44	2.56	2.00	35.8	11.1	23.4	1.0x
T=5	1.55	2.51	2.03	34.1	12.2	23.1	1.0x
T=10	1.60	2.59	2.10	33.0	11.7	22.3	1.0x
T=100	1.68	2.69	2.19	27.1	10.0	18.5	1.0x
Gradient Manipulation
PCGrad	1.42	3.18	2.30	36.6	8.3	22.5	2.1x
Gradient Vaccine	1.43	2.95	2.19	36.4	9.8	23.1	2.1x
DPL (Arithmetic mean)	1.46	2.52	1.99	35.4	12.0	23.7	1.0x
DPL (rHI = 1)	1.57	2.45	2.01	34.4	12.6	23.5	1.0x
Table 2: Results of the main experiments for the application of Double Power Law. We report the average results of three random seeds for each method.
Temperature-based Sampling We search  $T \in \{1, 2, 5, 10, 100\}$ , which are most frequently used in NMT literature. After choosing  $T$ , we set the sampling ratio  $p_i' = p_i^{1/T} / \sum_{j}^{K} p_j^{1/T}$  where  $p_i$  denotes the ratio of the training examples from task  $i$  to the whole training set.
Gradient Manipulation We also compare our method to gradient manipulation methods that are reported to improve MNMT performance: 1) PCGrad(Yu et al., 2020) and 2) Gradient Vaccine(Wang et al., 2021). We employ the default configuration of the methods described in their corresponding papers. Gradient manipulation methods require computing the cosine similarity of gradients at each optimization step and adjusting the gradient direction to resolve conflicts, resulting in an increased computational budget along with the increase of directions.
As shown in Table 2, we report both the generalization cross-entropy and detokenized BLEU score using sacrebleu $^2$  for different sampling methods.
Under the arithmetic mean metric, DPL reaches similar generalization cross-entropy compared to the best result when tuning temperature  $(\mathrm{T} = 2)$  and gains  $+0.3$  average BLEU scores than  $\mathrm{T} = 2$  in the 2-direction setting. In the 4-direction setting, DPL also gains better results  $(+0.1$  BLEU) compared to the best result when tuning temperature  $(\mathrm{T} = 5)$ . The best sampling temperature varies for different MNMT tasks while DPL could surpass the corresponding best results by just computing once, reducing the searching time to 1/5 of origin.
Comparing to the Gradient Manipulation methods, DPL leads to a much better average performance than the best method in both experiments (+0.6 BLEU in 2-direction, +2.3 BLEU in 4-direction),
2nrefs:1|case:mixedleff:noltok:13alsmooth:explversion:2.1.0
Metric	Method	FR	DE	ZH	HI	AVG
BERT-Score	T=5 (best T)	71.2	74.4	70.5	65.5	70.4
DPL(Ours)	71	74.1	71.6	66.3	70.8
COMET	T=5 (best T)	54.4	52.4	46.8	45.2	49.7
DPL(Ours)	53.9	52.1	48.2	46.5	50.2
with only 1/2 and 1/3 training time. The result is also in line with Xin et al. (2022) that simple scalarization could beat the complex optimization method if we set the weights properly.
We also evaluate the customized metric in the 2-direction experiment, where we hope to best enhance the performance of the low-resource direction. It turns out that DPL is more effective (+0.4 BLEU) in improving the low-resource direction than tuning the temperature (p<0.05).
In addition to using the BLEU score for evaluation, we also utilize model-based evaluation techniques, specifically Bert-Score (Zhang et al., 2020) and COMET (Rei et al., 2020). Research indicates that these metrics often align more closely with human evaluations than their model-free counterparts. As detailed in Table 3, DPL outperforms the best temperature-based method, showing an average improvement of 0.4 in BERT-Score  $(p < 0.05)$  and 0.5 in COMET score  $(p < 0.05)$ .
The results justify that through the Double Power Law, we can compute the sampling ratio combination that best satisfies our expectation without manually searching the sampling ratios.
The Double Power Law of model with different sizes As shown in Table 4, we evaluate the how the power terms of Double Power Law change in models with different sizes.
Among the parameters in the Double Power Law, we are more interested in power terms in the Capacity Occupation  $(\alpha)$  and Intrinsic Over-fitting  $(\beta)$  term since they better reflect the sensitivity of performance changes against sampling ratio. We keep the other parameters the same as in Equation 6 and estimate the power terms in the Medium and Large size models.
It shows that the power term  $(\beta)$  in the Intrinsic Over-fitting term increases rapidly with the growth in model size. While the power
term  $(\alpha)$  in the Capacity Occupation term changes much slower. For larger model, over-fitting phenomena of low-resource direction will be more severe and the final performance would be more likely to fall into the collapsed area when increasing the sampling ratio for low-resource directions.
Table 3: Model-Based evaluation results of the 4-direction multilingual NMT experiments. DPL method consistently outperforms the best temperature-based method in terms of average results.
Model Size	α	β
Base(64M)	0.20	1.21
Medium(102M)	0.25	2.41
Large(317M)	0.27	3.43
Table 4: The estimated power terms of for different models.
Generalization of DPL to more diverse Directions To better test the generality of DPL computed as Equation 6, we ran experiments on 10 languages in X-EN from WMT10 (Callison-Burch et al., 2010). Dataset details are in Appendix C. Figure 7 indicates a Pareto Front Collapse in low-resource directions, with the double power law fitting well across data/language conditions ( $r^2$  values of 0.86 and 0.92 for low and high resources). As for the difference, we observed a more pronounced pareto front collapse in X-EN than in EN-X. By only adjusting the  $\gamma$  term in DPL to -0.25, the fit improves for X-EN directions ( $r^2$ : 0.86 to 0.91 for low-resources; 0.92 to 0.97 for high-resources), highlighting the significant effect of the Intrinsic Over-fitting term in X-EN directions.
The results combined show that we can rely on the instantiated DPL when the model and the training hyper-parameters (e.g., batch size, total update steps) are not changed. The DPL is robust when introducing more languages or changing the translation directions.
(a) As the proportion increases, the validation loss goes down then goes up for low-resource languages.
(b) The validation loss goes down when the proportion of high-resource direction increases.
Figure 7: Evaluation of the Double Power Law under X->EN translation setting, with more diverse languages. The points are the actual value and the solid lines are the predicted values for one direction at certain sampling ratio.
Many previous work studied the trade-off among different directions and how to achieve a better overall performance in MNMT. Aharoni et al. (2019); Shaham et al. (2022) empirically study how different settings influence the positive and negative interference among different directions. Xin et al. (2022) firstly incorporate Pareto optimality to the analysis of MNMT system. Fernandes et al. (2023) propose to use a single power law to model how one direction's performance changes with its sampling ratio under the assumption that all directions have unlimited data.
Another related line of works study multi-task optimization for MNMT. Yu et al. (2020); Wang et al. (2021) propose to dynamically altering the conflict gradients eliminate the negative interference among tasks. Wang et al. (2020a); Zhou et al. (2021) explore how to dynamically adjust the optimization objective along with the training process to better schedule the training for different directions. Li and Gong (2021) focus on the imbalance nature of MNMT task and propose an optimization method based on the local curvature of the loss landscape. However, Xin et al. (2022) recently show that simple scalarization method could beat many multitask optimization methods for MNMT if the sampling ratio for different tasks are correctly set.
While we focus on how to set the optimal weights, modeling how the size of neural network influences the performance also attracts much attention recently, i.e. the neural scaling law (Kaplan et al., 2020; Hoffmann et al., 2022; OpenAI, 2023). In the field of NMT, Ghorbani et al. (2021a); Gordon et al. (2021) show that the scaling behavior of cross-entropy loss follows a power law with the model size as input while Fernandes et al. (2023) extends the conclusions to multilingual setting.
In this work, we start from our observations that the trade-off front is no longer Pareto front in Multilingual Neural Machine Translation when there exists low-resource directions, i.e. the collapse of Pareto front. We analyse the reason causing the phenomena and find that there exists a balance between model capacity occupation and the intrinsic risk to over-fit for certain direction. Based on the findings, we propose the Double Power Law to model how the performance of a given direction changes with its sampling ratio, which is robust across different task settings and is capable of capturing the collapse phenomena. Finally, we frame the sampling ratio selection problem for Multilingual Neural Machine Translation as an optimization problem, which greatly reduces the cost for manually tuning the sampling temperature and reaches better results.
We thank all reviewers for their helpful advice. This paper is supported by the National Key Research and Development Program of China under Grant No.2020AAA0106700, the National Science Foundation of China under Grant No.61936012.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with bert.* In *International Conference on Learning Representations*.
Chunting Zhou, Daniel Lévy, Xian Li, Marjan Ghazvininejad, and Graham Neubig. 2021. Distribu-tionally robust multilingual machine translation. ArXiv, abs/2109.04020.
A Datasets for Main Experiments
Direction	Dataset	Num. of Training Examples	Evaluation Dataset
English-to-French	WMT10	{10M}	newtest15
English-to-Chinese	WMT19	{10M, 1M, 260K}	newtest19
English-to-German	WMT10	{4.6M, 1M, 260K}	newtest18
English-to-Hindi	WMT10	{260k}	newtest14
B Training Details of Multilingual NMT experiments
In this section, we would provide concrete details of our experimental setup for reproduction.
Similar to Xin et al. (2022)'s setting, we conduct our experiments on three Transformer (Vaswani et al., 2017) models with different sizes as shown in Table 6
Table 5: The datasets description for the main experiments. We randomly choose a subset of the full training set of a direction to form a smaller one.
Size	Enc. Layer	Dec. Layer	Hid. Dim	FFN Dim	Max Steps	Max Tokens per Batch
Base (64M)	3	3	512	2048	100k	20k
Medium (102M)	6	6	512	2048	100k	20k
Large (317M)	6	6	1024	4096	400k	80k
For all of our model, we use a 64k sentence piece vocabulary. All models are trained<sup>3</sup> with 4k warm-up steps with the learning rate linearly increasing from 0 to  $3e^{-4}$  then decreasing with inverse_sqrt learning rate scheduler. The label smoothing term is set to 0.1 following the NMT literature convention. Evaluation is done every 5k steps, and we choose the best checkpoint with lowest average validation loss before examining the final generalization performance.
C Datasets for Generality Testing
Table 6: Overview of model sizes and optimization hyper-parameters. The Large-size model is trained with a larger batch-size and training steps or it would fail to converge. For 4-task experiment, we double the max training steps due to the increase of training data.
Language Pair	Num. of Training Examples	Evaluation Dataset
fr-en	10M	newtest13
cs-en	5M	newtest16
de-en	4.6M	newtest16
zh-en	260K	newtest19
hi-en	260K	newtest14
gu-en	80k	newtest19
tr-en	180k	newtest16
ro-en	500K	newtest16
lv-en	300K	newtest17
et-en	300k	newtest18
Table 7: Dataset description for multilingual x-en translation experiments. The dataset is taken from WMT10. The dataset includes languages from diverse language family and data-adequacy.
<sup>3</sup>We use fairseq(Ott et al., 2019) as the training framework