SlowGuess/NAACL_2021_OCR huggingface dataset:
Surafel M Lakew, Alija Erofeeva, Matteo Negri, Marcello Federico, and Marco Turchi. 2018. Transfer learning in multilingual neural machine translation with dynamic vocabulary. arXiv preprint arXiv:1811.01137.
Surafel M Lakew, Alina Karakanta, Marcello Federico, Matteo Negri, and Marco Turchi. 2019. Adapting multilingual neural machine translation to unseen languages. arXiv preprint arXiv:1910.13998.
Bei Li, Yinqiao Li, Chen Xu, Ye Lin, Jiqiang Liu, Hui Liu, Ziyang Wang, Yuhao Zhang, Nuo Xu, Zeyang Wang, et al. 2019. The niutrans machine translation systems for wmt19. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 257-266.
Patrick Littell, Chi-kiu Lo, Samuel Larkin, and Darlene Stewart. 2019. Multi-source transformer for kazakh-russian-english neural machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 267-274.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.08210.
David Lopez-Paz and Marc'Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In Advances in neural information processing systems, pages 6467-6476.
Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In *Psychology of learning and motivation*, volume 24, pages 109-165. Elsevier.
Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. arXiv preprint arXiv:1808.04189.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Janani Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? arXiv preprint arXiv:1804.06323.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010.
Mark Bishop Ring. 1994. *Continual learning in reinforcement environments*. Ph.D. thesis, University of Texas at Austin, Austin, Texas 78712.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan First, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan, and Yonghui Wu. 2020. Leveraging monolingual data with self-supervision for multilingual neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2827-2835, Online. Association for Computational Linguistics.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pretraining for language generation. In International Conference on Machine Learning, pages 5926-5936.
A.1 Data statistics and details
We outline the counts, domains, test set, and BLEU scores of our original translation model on the 24 languages in Table 6. We do the same for the unseen languages in Table 7. All the Paracrawl data is from v6.0.
A.2 Adaption schemes
We now explain in detail our configurations:
Monolingual data for a single language In this case, we compute the probabilities following the temperature-based sampling scheme that we would have obtained had we computed with this data in the first place. Then we proceed to set the sampling probability of the new monolingual to  $30\%$  and rescale the remaining probabilities so that they add up to 1.
Monolingual data for a single language coupled with back-translation In order to properly utilize back-translation, we first train the model for 10k step in the same fashion as the previous paragraph. Then, we use offline backtranslation on the new monolingual data to generate pseudo-parallel data. We then treat this data as authentic and include it in the model. We set the sampling probability of the pseudo-parallel data to be  $10\%$ , we reset the sampling probability of the monolingual data to  $10\%$ , and rescale the rest so that they sum up to 1. We then continue training for an additional 20k steps, amounting to a total of 30k steps.
Monolingual & parallel data for a single language We multiply the probabilities of the new parallel data by a factor of 10, set the sampling probability of the monolingual data to  $10\%$  then rescale the reamining probabilities so that they are normalized. We then train for 15k steps.
Monolingual & parallel data for all four languages We do not use the same scaling as before, since this would aggressively undersample the original language pairs. Instead, we first average the total probabilities for the new parallel data, multiply it by 5 and then assign this probability to each of the parallel datasets. We then fix the probability of sampling the new monolingual datasets to be  $5\%$  each. We then train for 30k steps
A.3 Token overlap analysis
We first verify that the results in Table 1 apply for different vocabulary sizes. We compute analogous tables for vocabulary size of  $32k$  and  $128k$  tokens in Table 4 and 5 respectively.
Next, we examine which tokens are lost during the vocabulary substitution. Since the Sentencepiece library does not provide an easy way to acquire frequency scores for BPE models after training, we instead use the order of the tokens as a proxy for the relative ranking obtained by sorting the tokens by frequency. For each language, we produce violin plots for the indices in the original
# languages in base model	bn	pl	kk	ps	bn+pl+kk+ps
1	53.3%	49.3%	48.4%	47.7%	22.8%
5	83.0%	81.1%	81.6%	78.0%	51.9%
10	89.7%	87.4%	88.9%	85.6%	65.1%
15	92.8%	92.1%	90.2%	88.9%	72.1%
20	94.7%	90.3%	92.9%	92.2%	79.0%
24	95.6%	95.3%	95.5%	93.2%	83.8%
Table 4: Token overlap between vocabularies (consisting of  $32\mathrm{k}$  tokens) before & after the inclusion of a new language.
#languages in base model	bn	pl	kk	ps	bn+pl+kk+ps
1	53.5%	45.1%	43.5%	47.2%	21.1%
5	85.1%	80.7%	81.6%	81.7%	54.0%
10	91.0%	87.5%	89.3%	88.4%	67.4%
15	93.6%	91.8%	91.4%	91.5%	74.6%
20	95.1%	90.3%	93.3%	93.5%	79.4%
24	95.5%	94.2%	95.4%	93.9%	82.8%
Table 5: Token overlap between vocabularies (consisting of 128k tokens) before & after the inclusion of a new language.
vocabulary which are not in the adapted vocabulary for that language in Figure 3.
Critically, we observe that most of the tokens lost are towards the end of spectrum, suggesting that the model is mostly discarding infrequent tokens. Notably, it cannot discard the tail due to our requirement of full character coverage, which introduces a variety of rare Unicode characters as tokens that reside in the tail.
Violin plots for indices not in adapted vocabularies.
Indices in original vocabularies.
Figure 3: Violin plots for the indices in the original vocabulary that do not appear in the adapted vocabulary. Note that all language configurations, most of the indices that do not appear in the overlap are towards the infrequent side of the spectrum.
Language	Monolingual data (# of lines)	Parallel data (# of examples)	Domain (Monolingual data)	Domain (Parallel data)	Test set	Language family	BLEU en-xx	BLEU xx-en
Bg	39610418	4111172	NewsCrawl	Paracrawl	TED	Slavic	32.43	35.77
Cs	81708712	64336053	NewsCrawl	WMT	WMT'18	Slavic	18.42	28.60
Da	4139992	6370432	Wiki	Paracrawl	TED	Germanic	38.81	42.87
De	333313278	4508785	NewsCrawl	WMT	WMT'14	Germanic	23.63	30.38
El	8332782	5298946	NewsCrawl	Paracrawl	TED	Hellenic	29.03	34.40
Es	53874815	15182374	NewsCrawl	WMT	WMT'13	Romance	31.74	33.23
Et	5367030	2175873	NewsCrawl	WMT	WMT'18	Uralic	16.99	27.53
Fi	21520558	6587448	NewsCrawl	WMT	WMT'19	Uralic	16.95	27.08
Fr	87063385	40853298	NewsCrawl	WMT	WMT'14	Romance	35.04	36.13
Gu	816575	155798	NewsCrawl	WMT	WMT'19	Indo-Aryan	10.92	20.91
Hi	23611899	313748	NewsCrawl	WMT	WMT'14	Indo-Aryan	13.36	18.98
Hr	6814690	6814690	NewsCrawl	Paracrawl	TED	Slavic	25.31	34.81
Hu	40879784	4963481	NewsCrawl	Paracrawl	TED	Uralic	15.90	24.25
It	2836989	2747344	NewsCrawl	Paracrawl	TED	Romance	31.87	36.59
Lt	2836989	635146	NewsCrawl	WMT	WMT'19	Baltic	11.56	30.82
Lv	11338472	637599	NewsCrawl	WMT	WMT'17	Baltic	17.16	22.69
Pt	9392574	20677300	NewsCrawl	Paracrawl	TED	Romance	33.25	41.79
Ro	21033306	610320	NewsCrawl	WMT	WMT'16	Romance	27.18	36.92
Ru	93827187	38492126	NewsCrawl	WMT	WMT'19	Slavic	22.20	34.70
Sk	3040748	3303841	Wiki	Paracrawl	TED	Slavic	22.59	29.52
Sl	2669157	1923589	Wiki	Paracrawl	TED	Slavic	21.06	25.73
Ta	708500	736479	NewsCrawl	WMT	WMT'20	Dravidian	6.29	12.06
Table 6: Details on the original 24 languages considered. For Tamil, we did not have access to the test set, so we used newsdev2019 instead. The BLEU scores are from the our original translation model.
Language	Monolingual data (# of lines)	Parallel data (# of examples)	Domain (Monolingual data)	Domain (Parallel data)	Test set	Language family
bn	3918906	27584	Newscrawl	PMIndia	PMIndia	Indo-Aryan
kk	4032908	222424	Newscrawl + Wiki Dumps	WMT	WMT	Turkic
pl	3788276	5001447	Newscrawl	WMT	WMT	Slavic
ps	6969911	1134604	Newscrawl + CommonCrawl	WMT	WMT	Indo-Iranian
Table 7: Details on the additional 4 languages considered for adaptation. For Polish, we did not have access to the test set so we used the dev set instead.