SlowGuess/ACL_2020_OCR huggingface dataset:
Prior works on multilingual models generally focus on improving the average performance of the model on all supported languages (Arivazhagan et al., 2019; Conneau et al., 2019). The formulation of MultiDDS reflects this objective by defining the aggregation of  $n$  dev sets using Eq. 3, which is simply the average of dev risks. However, average performance might not be the most desirable objective under all practical usage settings. For example, it may be desirable to create a more egalitarian system that performs well on all languages, or a more specialized system that does particularly well on a subset of languages.
In this section, we examine the possibility of using MultiDDS to control the priorities of the multilingual model by defining different dev set aggregation methods that reflect these priorities. To do so, we first train the model for 10 epochs using regular MultiDDS, then switch to a different dev set aggregation method. Specifically, we compare MultiDDS with three different priorities:
Regular: this is the standard MultiDDS that optimizes all languages throughout training using the average dev risk aggregation in Eq. 3
Low: a more egalitarian system that optimizes the average of the four languages with the worst dev perplexity, so that MultiDDS can focus on optimizing the low-performing languages
High: a more specialized system that optimizes the four languages with the best dev perplexity, for MultiDDS to focus on optimizing the high-performing languages
We performed experiments with these aggregation methods on the Diverse group, mainly because there is more performance trade-off among these languages. First, in Tab. 3 we show the average BLEU over all languages, and find that MultiDDS with different optimization priorities still maintains competitive average performance compared to the baseline. More interestingly, in Fig. 2, we plot the BLEU score difference of High and Low compared to Regular for all 8 languages. The languages are ordered on the  $x$ -axis from left to right in decreasing perplexity. Low generally performs better on the low-performing languages on the left, while High generally achieves the best performance on the high-performing languages on
8531
Method	Avg.	aze	bel	glg	slk	tur	rus	por	ces
M2O	Prop. MultiDDS-S	24.88	11.20	17.17	27.51	28.85	23.09*	22.89	41.60	26.80
25.52	12.20*	19.11*	29.37*	29.35*	22.81	22.78	41.55	27.03
O2M	Temp. MultiDDS-S	16.61	6.66	11.29	21.81	18.60	11.27	14.92	32.10	16.26
17.32	6.59	12.39*	21.65	20.61*	11.58	15.26*	33.52*	16.98*
bos	mar	hin	mkd	ell	bul	fra	kor
M2O	Prop. MultiDDS-S	26.68	23.43	10.10	22.01	31.06	35.62*	36.41*	37.91*	16.91
27.00	25.34*	10.57	22.93*	32.05*	35.27	35.77	37.30	16.81
O2M	Temp. MultiDDS-S	17.94	14.73*	4.93	15.49	20.59	24.82	26.60	29.74*	6.62
18.24	14.02	4.76	15.68*	21.44	25.69*	27.78*	29.60	7.01*
Table 1: BLEU scores of the best baseline and MultiDDS-S for all translation settings. MultiDDS-S performs better on more languages. For each setting, bold indicates the highest value, and * means the gains are statistically significant with  $p < {0.05}$  .
Method	M2O	O2M
Related	Diverse	Related	Diverse
Baseline	Uni. (τ=∞)	22.63	24.81	15.54	16.86
Temp. (τ=5)	24.00	26.01	16.61	17.94
Prop. (τ=1)	24.88	26.68	15.49	16.79
Ours	MultiDDS	25.26	26.65	17.17	18.40
MultiDDS-S	25.52	27.00	17.32	18.24
Table 2: Average BLEU for the baselines and our methods. Bold indicates the highest value.
Setting	Baseline	MultiDDS-S
Regular	Low	High
M2O	26.68	27.00	26.97	27.08
O2M	17.94	18.24	17.95	18.55
Table 3: Average BLEU of the best baseline and three MultiDDS-S settings for the Diverse group. MultiDDS-S always outperform the baseline.
the right, with results most consistent in the O2M setting. This indicates that MultiDDS is able to prioritize different predefined objectives.
It is also worth noting that low-performing languages are not always low-resource languages. For example, Korean (kor) has the largest amount of training data, but its BLEU score is among the lowest. This is because it is typologically very different from English and the other training languages. Fig. 2 shows that Low is still able to focus on improving kor, which aligns with the predefined objective. This fact is not considered in baseline methods that only consider data size when sampling from the training datasets.
6.5 Learned Language Distributions
In Fig. 3, we visualize the language distribution learned by MultiDDS throughout the training process. Under all settings, MultiDDS gradually increases the usage of LRLs. Although initialized
Figure 2: The difference between Low and High optimization objectives compared to Regular for the Diverse language group. MultiDDS successfully optimize for different priorities. left: M2O; right: O2M.
with the same distribution for both one-to-many and many-to-one settings, MultiDDS learns to up-sample the LRLs more in the one-to-many setting, likely due to the increased importance of learning language-specific decoders in this setting. For the Diverse group, MultiDDS learns to decrease the usage of Korean (kor) the most, probably because it is very different from other languages in the group.
6.6 Effect of Stabilized Rewards
Next, we study the effect of the stabilized reward proposed in § 2. In Fig. 4, we plot the regular reward (used by MultiDDS) and the stable reward (used by MultiDDS-S) throughout training. For all settings, the reward in MultiDDS and MultiDDS-S follows the similar trend, while the stable reward used in MultiDDS-S has consistently less variance.
MultiDDS-S also results in smaller variance in the final model performance. We run MultiDDS and MultiDDS-S with 4 different random seeds, and record the mean and variance of the average BLEU score. Tab. 4 shows results for the Diverse group, which indicate that the model performance achieved using MultiDDS-S has lower
8532
Figure 3: Language usage by training step. Left: many-to-one; Right: one-to-many; Top: related language group; Bottom: diverse language group.
Method	M2O	O2M
Mean	Var.	Mean	Var.
MultiDDS	26.85	0.04	18.20	0.05
MultiDDS-S	26.94	0.02	18.24	0.02
Table 4: Mean and variance of the average BLEU score for the Diverse group. The models trained with MultiDDS-S perform better and have less variance.
variance and a higher mean than MultiDDS.
Additionally, we compare the learned language distribution of MultiDDS-S and MultiDDS in Fig. 5. The learned language distribution in both plots fluctuates similarly, but MultiDDS has more drastic changes than MultiDDS-S. This is also likely due to the reward of MultiDDS-S having less variance than that of MultiDDS.
Our work is related to the multilingual training methods in general. Multilingual training has a rich history (Schultz and Waibel, 1998; Mimno et al., 2009; Shi et al., 2010; Tackstrom et al., 2013), but has become particularly prominent in recent years due the ability of neural networks to easily perform multi-task learning (Dong et al., 2015; Plank et al., 2016; Johnson et al., 2016). As stated previously, recent results have demonstrated the importance of balancing HRLs and LRLs during multilingual training (Arivazhagan et al., 2019; Conneau et al.,
Figure 4: Variance of reward. Left: M2O; Right: O2M; Top: Related language group; Bottom: Diverse language group.
Figure 5: Language usage for the M2O-Diverse setting. Left: MultiDDS-S; Right: MultiDDS. The two figures follow similar trends while MultiDDS changes more drastically.
2019), which is largely done with heuristic sampling using a temperature term; MultiDDS provides a more effective and less heuristic method. Wang and Neubig (2019); Lin et al. (2019) choose languages from multilingual data to improve the performance on a particular language, while our work instead aims to train a single model that handles translation between many languages. (Zaremoodi et al., 2018; Wang et al., 2018, 2019a) propose improvements to the model architecture to improve multilingual performance, while MultiDDS is a model-agnostic and optimizes multilingual data usage.
Our work is also related to machine learning methods that balance multitask learning (Chen et al., 2018b; Kendall et al., 2018). For example, Kendall et al. (2018) proposes to weigh the training loss from a multitask model based on the uncertainty of each task. Our method focuses on optimizing the multilingual data usage, and is both somewhat orthogonal to and less heuristic than such loss weighting methods. Finally, our work is related to meta-learning, which is used in hyperpa
8533
rameter optimization (Baydin et al., 2018), model initialization for fast adaptation (Finn et al., 2017), and data weighting (Ren et al., 2018). Notably, Gu et al. (2018) apply meta-learning to learn an NMT model initialization for a set of languages, so that it can be quickly fine-tuned for any language. This is different in motivation from our method because it requires an adapted model for each of the language, while our method aims to optimize a single model to support all languages. To our knowledge, our work is the first to apply meta-learning to optimize data usage for multilingual objectives.
In this paper, we propose MultiDDS, an algorithm that learns a language scorer to optimize multilingual data usage to achieve good performance on many different languages. We extend and improve over previous work on DDS (Wang et al., 2019b), with a more efficient algorithmic instantiation tailored for the multilingual training problem and a stable reward to optimize multiple objectives. MultiDDS not only outperforms prior methods in terms of overall performance on all languages, but also provides a flexible framework to prioritize different multilingual objectives.
Notably, MultiDDS is not limited to NMT, and future work may consider applications to other multilingual tasks. In addition, there are other conceivable multilingual optimization objectives than those we explored in § 6.4.
The first author is supported by a research grant from the Tang Family Foundation. This work was supported in part by NSF grant IIS-1812327. The authors would like to thank Amazon for providing GPU credits.
Roeah Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In NAACL.
Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A Smith. 2016. Many languages, one parser. TACL, 4:431-444.
Naveen Arivazhagan, Ankur Bapna, Orhan First, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural
machine translation in the wild: Findings and challenges. In arxiv.
Atilim Gunes Baydin, Robert Cornish, David Martínez-Rubio, Mark Schmidt, and Frank Wood. 2018. Online learning rate adaptation with hypergradient descent. In ICLR.
Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2018a. Adversarial deep averaging networks for cross-lingual sentiment classification. In ACL.
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018b. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In ICML.
Benoit Colson, Patrice Marcotte, and Gilles Savard. 2007. An overview of bilevel optimization. Annals OR, 153(1).
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. In EMNLP.
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In ACL.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML.
Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. 2018. Meta-learning for low-resource neural machine translation. In EMNLP.
Melvin Johnson et al. 2016. Google's multilingual neural machine translation system: Enabling zero-shot translation. In TACL.
Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR.
Alexandre Klementiev, Ivan Titov, and Binod Bhattacharai. 2012. Inducing crosslingual distributed representations of words. In Proceedings of COLING 2012, pages 1459-1474.
Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In EMNLP.
Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learning. In ACL.
8534
Chaitanya Malaviya, Graham Neubig, and Patrick Littell. 2017. Learning language representations for typology prediction. In EMNLP.
David M. Mimno, Hanna M. Wallach, Jason Naradowsky, David A. Smith, and Andrew McCallum. 2009. Polylingual topic models. In EMNLP.
Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. EMNLP.
Toan Q. Nguyen and David Chiang. 2018. Transfer learning across low-resource, related languages for neural machine translation. In *NAACL*.
Toan Q. Nguyen and Julián Salazar. 2019. Transformers without tears: Improving the normalization of self-attention. In IWSLT.
Robert Östling and Jörg Tiedemann. 2017. Continuous multilinguality with language vectors. In EACL.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In *NAACL: Demonstrations*.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL.
Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In ACL.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In WMT.
Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In NAACL.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. 2018. Learning to reweight examples for robust deep learning. In ICML.
Tanja Schultz and Alex Waibel. 1998. Multilingual and crosslingual speech recognition. In Proc. DARPA Workshop on Broadcast News Transcription and Understanding. CiteSeer.
Lei Shi, Rada Mihalcea, and Mingjun Tian. 2010. Cross language text classification by model translation and semi-supervised learning. In EMNLP.
Oscar Tackström, Dipanjan Das, Slav Petrov, Ryan McDonald, and Joakim Nivre. 2013. Token and type constraints for cross-lingual part-of-speech tagging. In TACL.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Xinyi Wang and Graham Neubig. 2019. Target conditioned sampling: Optimizing data selection for multilingual neural machine translation. In ACL.
Xinyi Wang, Hieu Pham, Philip Arthur, and Graham Neubig. 2019a. Multilingual neural machine translation with soft decoupled encoding. In ICLR.
Xinyi Wang, Hieu Pham, Paul Mitchel, Antonis Anastasopoulos, Jaime Carbonell, and Graham Neubig. 2019b. Optimizing data usage via differentiable rewards. In *arxiv*.
Yining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu, and Chengqing Zong. 2018. Three strategies to improve one-to-many multilingual translation. In EMNLP.
Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In EMNLP.
Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A. Smith, and Jaime Carbonell. 2018. Neural crosslingual named entity recognition with minimal resources. In EMNLP.
Poorya Zaremoodi, Wray L. Buntine, and Gholamreza Haffari. 2018. Adaptive knowledge sharing in multitask learning: Improving low-resource neural machine translation. In ACL.
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low resource neural machine translation. In EMNLP.
8535
A.1 Effect of Step-ahead Reward
Setting	Baseline	MultiDDS
Moving Ave.	Step-ahead
M2O	24.88	25.19	25.26
O2M	16.61	17.17	17.17
In this section, we list the details of preprocessing and hyperparameters we use for the experiments.
- We use 6 encoder and decoder layers, with 4 attention heads
- The embedding size is set to 512, and the feedforward layer has a dimension of 1024
- We use the dropout rate of 0.3
- The batch size is set to 9600 tokens
- We use label smoothing with rate of 0.1
- We use the scaled  $l_{2}$  normalization before residual connection, which is shown to be helpful for small data (Nguyen and Salazar, 2019)
A.3 Dataset statistics
Table 5: Ave. BLEU for the Related language group. The step-ahead reward proposed in the paper is better or comparable with the moving average, and both are better than the baseline.
Language	Train	Dev	Test
aze	5.94k	671	903
bel	4.51k	248	664
glg	10.0k	682	1007
slk	61.5k	2271	2445
tur	182k	4045	5029
rus	208k	4814	5483
por	185k	4035	4855
ces	103k	3462	3831
Table 6: Statistics of the related language group.
Language	Train	Dev	Test
bos	5.64k	474	463
mar	9.84k	767	1090
hin	18.79k	854	1243
mkd	25.33k	640	438
ell	134k	3344	4433
bul	174k	4082	5060
fra	192k	4320	4866
kor	205k	4441	5637
Table 7: Statistics of the diverse language group.
A.4 Detailed Results for All Settings
8536
Method	Avg.	aze	bel	glg	slk	tur	rus	por	ces
Uni. (τ =∞)	22.63	8.81	14.80	25.22	27.32	20.16	20.95	38.69	25.11
Temp. (τ = 5)	24.00	10.42	15.85	27.63	28.38	21.53	21.82	40.18	26.26
Prop. (τ = 1)	24.88	11.20	17.17	27.51	28.85	23.09	22.89	41.60	26.80
MultiDDS	25.26	12.20	18.60	28.83	29.21	22.24	22.50	41.40	27.22
MultiDDS-S	25.52	12.20	19.11	29.37	29.35	22.81	22.78	41.55	27.03
Table 8: BLEU score of the baselines and our method on the Related language group for many-to-one translation
Method	Avg.	bos	mar	hin	mkd	ell	bul	fra	kor
Uni. (τ =∞)	24.81	21.52	9.48	19.99	30.46	33.22	33.70	35.15	15.03
Temp. (τ = 5)	26.01	23.47	10.19	21.26	31.13	34.69	34.94	36.44	16.00
Prop. (τ = 1)	26.68	23.43	10.10	22.01	31.06	35.62	36.41	37.91	16.91
MultiDDS	26.65	25.00	10.79	22.40	31.62	34.80	35.22	37.02	16.36
MultiDDS-S	27.00	25.34	10.57	22.93	32.05	35.27	35.77	37.30	16.81
Table 9: BLEU score of the baselines and our method on the Diverse language group for many-to-one translation
Method	Avg.	aze	bel	glg	slk	tur	rus	por	ces
Uni. (τ =∞)	15.54	5.76	10.51	21.08	17.83	9.94	13.59	30.33	15.35
Temp. (τ = 5)	16.61	6.66	11.29	21.81	18.60	11.27	14.92	32.10	16.26
Prop. (τ = 1)	15.49	4.42	5.99	14.92	17.37	12.86	16.98	34.90	16.53
MultiDDS	17.17	6.24	11.75	21.46	20.67	11.51	15.42	33.41	16.94
MultiDDS-S	17.32	6.59	12.39	21.65	20.61	11.58	15.26	33.52	16.98
Table 10: BLEU score of the baselines and our method on the Related language group for one-to-many translation
Method	Avg.	bos	mar	hin	mkd	ell	bul	fra	kor
Uni. (τ =∞)	16.86	14.12	4.69	14.52	20.10	22.87	25.02	27.64	5.95
Temp. (τ = 5)	17.94	14.73	4.93	15.49	20.59	24.82	26.60	29.74	6.62
Prop. (τ = 1)	16.79	6.93	3.69	10.70	15.77	26.69	29.59	33.51	7.49
MultiDDS	18.40	14.91	4.83	14.96	22.25	24.80	27.99	30.77	6.75
MultiDDS-S	18.24	14.02	4.76	15.68	21.44	25.69	27.78	29.60	7.01
Table 11: BLEU score of the baselines and our method on the Diverse language group for one-to-many translation
8537