SlowGuess/ACL_2020_OCR huggingface dataset:
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019a. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810-1822, Florence, Italy. Association for Computational Linguistics.
Xinyi Wang, Hieu Pham, Philip Arthur, and Graham Neubig. 2019b. Multilingual neural machine translation with soft decoupled encoding. In International Conference on Learning Representations.
Yining Wang, Long Zhou, Jiajun Zhang, Feifei Zhai, Jingfang Xu, and Chengqing Zong. 2019c. A compact and language-sensitive multilingual translation method. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1213-1223, Florence, Italy. Association for Computational Linguistics.
Biao Zhang, Ivan Titov, and Rico Sennrich. 2019. Improving deep transformer with depth-scaled initialization and merged attention. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 898-909, Hong Kong, China. Association for Computational Linguistics.
Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. 2016. Variational neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 521-530, Austin, Texas. Association for Computational Linguistics.
Zaixiang Zheng, Hao Zhou, Shujian Huang, Lei Li, Xin-Yu Dai, and Jiajun Chen. 2020. Mirrorgenerative neural machine translation. In International Conference on Learning Representations.
A OPUS-100: The OPUS Multilingual Dataset
Table 8 lists the languages (other than English) and numbers of sentence pairs in the English-centric multilingual dataset.
B Model Settings
We optimize model parameters using Adam  $(\beta_{1} = 0.9, \beta_{2} = 0.98)$  (Kingma and Ba, 2015) with label smoothing of 0.1 and scheduled learning rate (warmup step  $4\mathrm{k}$ ). We set the initial learning rate to 1.0 for bilingual models, but use 0.5 for multilingual models in order to stabilize training. We apply dropout to residual layers and attention weights, with a rate of  $0.1 / 0.1$  for 6-layer Transformer models and  $0.3 / 0.2$  for deeper ones. We group sentence
Table 8: Numbers of training, validation, and test sentence pairs in the English-centric multilingual dataset.
Language	Train	Valid	Test
af	Afrikaans	275512	2000	2000
am	Amharic	89027	2000	2000
an	Aragonese	6961	0	0
ar	Arabic	1000000	2000	2000
as	Assamese	138479	2000	2000
az	Azerbaijani	262089	2000	2000
be	Belarusian	67312	2000	2000
bg	Bulgarian	1000000	2000	2000
bn	Bengali	1000000	2000	2000
br	Breton	153447	2000	2000
bs	Bosnian	1000000	2000	2000
ca	Catalan	1000000	2000	2000
cs	Czech	1000000	2000	2000
cy	Welsh	289521	2000	2000
da	Danish	1000000	2000	2000
de	German	1000000	2000	2000
dz	Dzongkha	624	0	0
el	Greek	1000000	2000	2000
eo	Esperanto	337106	2000	2000
es	Spanish	1000000	2000	2000
et	Estonian	1000000	2000	2000
eu	Basque	1000000	2000	2000
fa	Persian	1000000	2000	2000
fi	Finnish	1000000	2000	2000
fr	French	1000000	2000	2000
fy	Western Frisian	54342	2000	2000
ga	Irish	289524	2000	2000
gd	Gaelic	16316	1605	1606
gl	Galician	515344	2000	2000
gu	Gujarati	318306	2000	2000
ha	Hausa	97983	2000	2000
he	Hebrew	1000000	2000	2000
hi	Hindi	534319	2000	2000
hr	Croatian	1000000	2000	2000
hu	Hungarian	1000000	2000	2000
hy	Armenian	7059	0	0
id	Indonesian	1000000	2000	2000
ig	Igbo	18415	1843	1843
is	Icelandic	1000000	2000	2000
it	Italian	1000000	2000	2000
ja	Japanese	1000000	2000	2000
ka	Georgian	377306	2000	2000
kk	Kazakh	79927	2000	2000
km	Central Khmer	111483	2000	2000
kn	Kannada	14537	917	918
ko	Korean	1000000	2000	2000
ku	Kurdish	144844	2000	2000
ky	Kyrgyz	27215	2000	2000
li	Limburgan	25535	2000	2000
lt	Lithuanian	1000000	2000	2000
Language	Train	Valid	Test
lv	Latvian	1000000	2000	2000
mg	Malagasy	590771	2000	2000
mk	Macedonian	1000000	2000	2000
ml	Malayalam	822746	2000	2000
mn	Mongolian	4294	0	0
mr	Marathi	27007	2000	2000
ms	Malay	1000000	2000	2000
mt	Maltese	1000000	2000	2000
my	Burmese	24594	2000	2000
nb	Norwegian Bokmål	142906	2000	2000
ne	Nepali	406381	2000	2000
nl	Dutch	1000000	2000	2000
nn	Norwegian Nynorsk	486055	2000	2000
no	Norwegian	1000000	2000	2000
oc	Occitan	35791	2000	2000
or	Oriya	14273	1317	1318
pa	Panjabi	107296	2000	2000
pl	Polish	1000000	2000	2000
ps	Pashto	79127	2000	2000
pt	Portuguese	1000000	2000	2000
ro	Romanian	1000000	2000	2000
ru	Russian	1000000	2000	2000
rw	Kinyarwanda	173823	2000	2000
se	Northern Sami	35907	2000	2000
sh	Serbo-Croatian	267211	2000	2000
si	Sinhala	979109	2000	2000
sk	Slovak	1000000	2000	2000
sl	Slovenian	1000000	2000	2000
sq	Albanian	1000000	2000	2000
sr	Serbian	1000000	2000	2000
sv	Swedish	1000000	2000	2000
ta	Tamil	227014	2000	2000
te	Telugu	64352	2000	2000
tg	Tajik	193882	2000	2000
th	Thai	1000000	2000	2000
tk	Turkmen	13110	1852	1852
tr	Turkish	1000000	2000	2000
tt	Tatar	100843	2000	2000
ug	Uighur	72170	2000	2000
uk	Ukrainian	1000000	2000	2000
ur	Urdu	753913	2000	2000
uz	Uzbek	173157	2000	2000
vi	Vietnamese	1000000	2000	2000
wa	Walloon	104496	2000	2000
xh	Xhosa	439671	2000	2000
yi	Yiddish	15010	2000	2000
yo	Yoruba	10375	0	0
zh	Chinese	1000000	2000	2000
zu	Zulu	38616	2000	2000