SlowGuess/NAACL_2021_OCR huggingface dataset:
Xavier Garcia and Aditya Siddhant and Orhan First and Ankur P. Parikh
{xgarcia,adisid,orhanf,aparikh}@google.com
Unsupervised translation has reached impressive performance on resource-rich language pairs such as English-French and English-German. However, early studies have shown that in more realistic settings involving low-resource, rare languages, unsupervised translation performs poorly, achieving less than 3.0 BLEU. In this work, we show that multilinguality is critical to making unsupervised systems practical for low-resource settings. In particular, we present a single model for 5 low-resource languages (Gujarati, Kazakh, Nepali, Sinhala, and Turkish) to and from English directions, which leverages monolingual and auxiliary parallel data from other high-resource language pairs via a three-stage training scheme. We outperform all current state-of-the-art unsupervised baselines for these languages, achieving gains of up to 14.4 BLEU. Additionally, we outperform strong supervised baselines for various language pairs as well as match the performance of the current state-of-the-art supervised model for  $\mathsf{Ne}\rightarrow \mathsf{En}$ . We conduct a series of ablation studies to establish the robustness of our model under different degrees of data quality, as well as to analyze the factors which led to the superior performance of the proposed approach over traditional unsupervised models.
Neural machine translation systems (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015; Wu et al., 2016) have demonstrated state-of-the-art results for a diverse set of language pairs when given large amounts of relevant parallel data. However, given the prohibitive nature of such a requirement for low-resource language pairs, there has been a growing interest in unsupervised machine translation (Ravi and Knight, 2011) and its neural counterpart, unsupervised neural machine translation (UNMT) (Lample et al., 2018a; Artetxe et al., 2018), which leverage only
monolingual source and target corpora for learning. Bilingual unsupervised systems (Lample and Conneau, 2019; Artetxe et al., 2019; Ren et al., 2019; Li et al., 2020a) have achieved surprisingly strong results on high-resource language pairs such as English-French and English-German.
However, these works only evaluate on high-resource language pairs with high-quality data, which are not realistic scenarios where UNMT would be utilized. Rather, the practical potential of UNMT is in low-resource, rare languages that may not only lack parallel data but also have a shortage of high-quality monolingual data. For instance, Romanian (a typical evaluation language for unsupervised methods) has 21 million lines of high-quality in-domain monolingual data provided by WMT. In contrast, for an actual low-resource language, Gujarati, WMT only provides 500 thousand lines of monolingual data (in news domain) and an additional 3.7 million lines of monolingual data from Common Crawl (noisy, general-domain).
Given the comparably sterile setups UNMT has been studied in, recent works have questioned the usefulness of UNMT when applied to more realistic low-resource settings. Kim et al. (2020) report BLEU scores of less than 3.0 on low-resource pairs and Marchisio et al. (2020) also report dramatic degradation under domain shift.
However, the negative results shown by the work above only study bilingual unsupervised systems and do not consider multilinguality, which has been well explored in supervised, zero-resource and zero-shot settings (Johnson et al., 2017; Firat et al., 2016a,b; Chen et al., 2017; Neubig and Hu, 2018; Gu et al., 2018; Liu et al., 2020; Ren et al., 2018; Zoph et al., 2016) to improve performance for low-resource languages. The goal of this work is to study if multilinguality can help UNMT be more robust in the low-resource, rare language setting.
In our setup (Figure 1), we have a single model
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1126-1137 June 6-11, 2021. ©2021 Association for Computational Linguistics
for 5 target low-resource unsupervised directions (that are not associated with any parallel data): Gujarati, Kazakh, Nepali, Sinhala, and Turkish. These languages are chosen to be studied for a variety of reasons (discussed in §3) and have been of particular challenge to unsupervised systems. In our approach, as shown in Figure 1, we also leverage auxiliary data from a set of higher resource languages: Russian, Chinese, Hindi, Arabic, Tamil, and Telugu. These higher resource languages not only possess significant amounts of monolingual data but also auxiliary parallel data with English that we leverage to improve the performance of the target unsupervised directions<sup>1</sup>.
Existing work on multilingual unsupervised translation (Liu et al., 2020; Garcia et al., 2020; Li et al., 2020b; Bai et al., 2020), which also uses auxiliary parallel data, employs a two-stage training scheme consisting of pre-training with noisy reconstruction objectives and fine-tuning with on-the-fly (iterative) back-translation and cross-translation terms (§4). We show this leads to sub-optimal performance for low-resource pairs and propose an additional intermediate training stage in our approach. Our key insight is that pre-training typically results in high  $\mathrm{X}\rightarrow \mathrm{En}$  (to English) performance but poor  $\mathrm{En}\rightarrow \mathrm{X}$  (from English) results, which makes finetuning unstable. Thus, after pre-training, we propose an intermediate training stage that leverages offline back-translation (Sennrich et al., 2016) to generate synthetic data from the  $\mathrm{X}\rightarrow \mathrm{En}$  direction to boost  $\mathrm{En}\rightarrow \mathrm{X}$  accuracy.
Our final results show that our approach outperforms a variety of supervised and unsupervised baselines, including the current state-of-the-art supervised model for the  $\mathrm{Ne} \rightarrow \mathrm{En}$  language pair. Additionally, we perform a series of experimental studies to analyze the factors that affect the performance of the proposed approach, as well as the performance in data-starved settings and settings where we only have access to noisy, multi-domain monolingual data.
Multilinguality has been extensively studied in the supervised literature and has been applied to the related problem of zero-shot translation (Johnson et al., 2017; First et al., 2016a; Arivazhagan et al.,
Figure 1: A pictorial depiction of our setup. The dashed edge indicates the target unsupervised language pairs that lack parallel training data. Full edges indicate the existence of parallel training data.
2019a; Al-Shedivat and Parikh, 2019). Zero-shot translation concerns the case where direct (source, target) parallel data is lacking but there is parallel data via a common pivot language to both the source and the target. For example, in Figure 1,  $\mathrm{Ru} \leftrightarrow \mathrm{Zh}$  and  $\mathrm{Hi} \leftrightarrow \mathrm{Te}$  would be zero-shot directions.
In contrast, a defining characteristic of the multilingual UNMT setup is that the source and target are disconnected in the graph and one of the languages is not associated with any parallel data with English or otherwise.  $\mathrm{En}\leftrightarrow \mathrm{Gu}$  or  $\mathrm{En}\leftrightarrow \mathrm{Kk}$  are such example pairs as shown in Figure 1.
Recently Guzmán et al. (2019); Liu et al. (2020) showed some initial results on multilingual unsupervised translation in the low-resource setting. They tune language-specific models and employ a standard two-stage training scheme (Lample and Conneau, 2019), or in the case of Liu et al. (2020) directly fine-tuning on a related language pair (e.g. Hi→En) and then test on the target X→En pair (e.g. Gu→En). In contrast our approach trains one model for all the language pairs targeted and employs a three stage training scheme that leverages synthetic parallel data via offline back-translation.
Offline backtranslation (Sennrich et al., 2016) was originally used for unsupervised translation (Lample et al., 2018b; Artetxe et al., 2019), especially with phrase-based systems.
2.1 Terminology
There is some disagreement on the definition of multilingual unsupervised machine translation, which we believe arises from extrapolating unsu
This makes our setting considerably more challenging than the zero-shot/zero resource setting. See §2 and §2.1 for a discussion.
Domain	En	Tr News	Kk News	Gu News	Ne Wiki	Si Wiki	Te Wiki	Ta Wiki	Hi IITB	Ru UN	Ar UN	Zh UN
Monolingual	News	233M	17M	1.8M	530K	-	-	2.5M	2.3M	32.6M	93.8M	9.2M	4.7M
Wikipedia	-	-	-	384K	92k	155k	-	-	-	-	-	22.7M
Crawled	-	-	7.1M	3.7M	3.5M	5.1M	-	-	-	-	-	-
Auxiliary parallel (w/ English)	Mixed	-	205k	225k	10k	564k	647k	290K	350K	1.5M	23.2M	9.2M	15.8M
In-domain (%)	-	-	100%	20.2%	11.4%	2.0%	2.9%	-	-	-	-	-	-
Table 1: The amount and domain of the data used in these experiments. For the unsupervised language pairs, we additionally included the domain of the development and test sets. For Arabic, we took the 18.4M samples from the UN Corpus and divided it in two, treating one half of it as unpaired monolingual data. We include the amount of parallel data for the unsupervised language pairs, which is only utilized for our in-house supervised baseline.
pervised translation to multiple languages. In the case of only two languages, the definition is clear: unsupervised machine translation consists of the case where there is no parallel data between the source and target languages. However, in a setting with multiple languages, there are multiple scenarios which satisfy this condition. More explicitly, suppose that we want to translate between languages  $\mathcal{X}$  and  $\mathcal{Y}$  and we have access to data from another language  $\mathcal{Z}$ . Then, we have three possible scenarios:
- We possess parallel data for  $(\mathcal{X},\mathcal{Z})$  and  $(\mathcal{Z},\mathcal{Y})$  which would permit a 2-step supervised baseline via the pivot. Existing literature (Johnson et al., 2017; First et al., 2016b) has used the term "zero-shot" and "zero-resource" to refer specifically to this setup.
- We have parallel data for  $(\mathcal{X},\mathcal{Z})$  but only monolingual data in  $\mathcal{V}$ , as considered in (Li et al., 2020b; Liu et al., 2020; Garcia et al., 2020; Bai et al., 2020; Guzmán et al., 2019; Artetxe et al., 2020). Note that the pivot-based baseline above is not possible in this setup.
- We do not have any parallel data among any of the language pairs, as considered in (Liu et al., 2020; Sun et al., 2020).
We believe the first setting is not particularly suited for the case where either  $\mathcal{X}$  or  $\mathcal{Y}$  are true low-resource languages (or extremely low-resource languages), since it is unlikely that these languages possess any parallel data with any other language. On the other hand, we usually assume that one of these languages is English and we can commonly find large amounts of parallel data for English with other high-resource auxiliary languages. For these reasons, we focus on the second setting for the rest of this work.
Arguably, the existence of the auxiliary parallel data provides some notion of indirect supervision
that is not present when only utilizing monolingual data. However, this signal is weaker than the one encountered in the zero-shot setting, since it precludes the 2-step supervised baseline. As a result, recent work (Artetxe et al., 2020; Guzmán et al., 2019; Garcia et al., 2020; Liu et al., 2020) has also opted to use the term "unsupervised". We too follow this convention and use this terminology, but we emphasize that independent of notation, our goal is to study the setting where only the (extremely) low-resource languages of interest possess no parallel data, whether with English or otherwise.
3 Choice of languages
The vast majority of works in UNMT (multilingual or otherwise) have focused on traditionally high-resource languages, such as French and German. While certain works simulate this setting by using only a smaller subset of the available monolingual data, such settings neglect common properties of true low-resource, rare languages: little-to-no lexical overlap with English and noisy data sources coming from multiple domains. Given the multifaceted nature of what it means to be a low-resource language, we have chosen a set of languages with many of these characteristics. We give a detailed account of the available data in Table 1.
Target unsupervised directions: We select Turkish (Tr), Gujarati (Gu), and Kazakh (Kk) from WMT. The latter two possess much smaller amounts of data than most language pairs considered for UNMT e.g. French or German. In order to vary the domain of our test sets, we additionally include Nepali (Ne) and Sinhala (Si) from the recently-introduced FLoRes dataset (Guzmán et al., 2019), as the test sets for these languages are drawn from Wikipedia instead of news. Not only do these languages possess monolingual data in amounts comparable to the low-resource languages from WMT, the subset of in-domain monolingual data for both languages make up less than  $5\%$  of
the available monolingual data of each language.
Auxiliary languages: To choose our auxiliary languages that contain both monolingual data and parallel data with English, we took into account linguistic diversity, size, and relatedness to the target directions. Russian shares the same alphabet with Kazakh, and Hindi, Telugu, and Tamil are related to Gujarai, Nepali and Sinhala. Chinese, while not specifically related to any of the target language, is high resource and considerably different in structure from the other languages.
For a given language pair  $(\mathrm{X},\mathrm{Y})$  of languages  $\mathrm{X}$  and  $\mathrm{Y}$ , we possess monolingual datasets  $\mathcal{D}_{\mathrm{X}}$  and  $\mathcal{D}_{\mathrm{Y}}$ , consisting of unpaired sentences of each language.
Neural machine translation In supervised neural machine translation, we have access to a parallel dataset  $\mathcal{D}_{\mathrm{X\times Z}}$  consisting of translation pairs  $(x,z)$ . We then train a model by utilizing the cross-entropy objective:
$$
\mathcal {L} _ {\mathrm {c r o s s - e n t r o p y}} (x, y) = - \log p _ {\theta} (y | x)
$$
where  $p_{\theta}$  is our translation model. We further assume  $p_{\theta}$  follows the encoder-decoder paradigm, where there exists an encoder  $\mathsf{Enc}_{\theta}$  which converts  $x$  into a variable-length representation which is passed to a decoder  $p_{\theta}(y|x) \coloneqq p_{\theta}(y|\mathsf{Enc}_{\theta}(x))$ .
Unsupervised machine translation In this setup, we no longer possess  $\mathcal{D}_{\mathrm{X\times Y}}$ . Nevertheless, we may possess auxiliary parallel datasets such as  $\mathcal{D}_{\mathrm{X\times Z}}$  for some language  $\mathbb{Z}$ , but we enforce the constraint that we do not have access to analogous dataset  $\mathcal{D}_{\mathrm{Y\times Z}}$ . Current state-of-the-art UNMT models divide their training procedure into two phases: i) the pre-training phase, in which an initial translation model is learned through a combination of language modeling or noisy reconstruction objectives (Song et al., 2019; Lewis et al., 2019; Lample and Conneau, 2019) applied to the monolingual data; ii) the fine-tuning phase, which resumes training the translation model built from the pre-training phase with a new set of objectives, typically centered around iterative back-translation i.e. penalizing a model's error in round-trip translations. We outline the objectives below:
Pre-training objectives We use the MASS objective (Song et al., 2019), which consists of masking<sup>2</sup>
2We choose a starting index of less than half the length  $l$  of the input and replace the next  $l / 2$  tokens with a [MASK]
a contiguous segment of the input and penalizing errors in the reconstruction of the masked segment. If we denote the masking operation by MASK, then we write the objective as follows:
$$
\mathcal {L} _ {\mathrm {M A S S}} (x) = - \log p _ {\theta} (x | \mathrm {M A S K} (x), l _ {x})
$$
where  $l_{x}$  denotes the language indicator of example  $x$ . We also use cross-entropy on the available auxiliary parallel data.
Fine-tuning objectives We use on-the-fly back-translation, which we write explicitly as:
$$
\mathcal {L} _ {\mathrm {b a c k - t r a n s l a t i o n}} (x, l _ {y}) = - \log p _ {\theta} (x | \tilde {y} (x), l _ {x})
$$
where  $\tilde{y}(x) = \operatorname{argmax}_y p_\theta(y|x, l_y)$  and we apply a stop-gradient to  $\tilde{y}(x)$ . Computing the mode  $\tilde{y}(x)$  of  $p_\theta(\cdot|x, l_y)$  is intractable, so we approximate this quantity with a greedy decoding procedure. We also utilize cross-entropy, coupled with cross-translation (Garcia et al., 2020; Li et al., 2020b; Xu et al., 2019; Bai et al., 2020), which ensures cross-lingual consistency:
$$
\mathcal {L} _ {\text {c r o s s - t r a n s l a t i o n}} (x, y, l _ {z}) = - \log p _ {\theta} (y | \tilde {z} (x), l _ {y})
$$
where  $\tilde{z}(x) = \operatorname{argmax}_{z} p_{\theta}(z|x, l_z)$ .
For the rest of this work, we assume that we want to translate between English (En) and some low-resource languages which we denote by X. In our early experiments, we found that proceeding to the fine-tuning stage immediately after pre-training with MASS provided sub-optimal results (see §7.2), so we introduced an intermediate stage which leverages synthetic data to improve performance. This yields a total of three stages, which we describe below.
5.1 First stage of training
In the first stage, we leverage monolingual and auxiliary parallel data, using the MASS and cross-entropy objectives on each type of dataset respectively. We describe the full procedure in Algorithm 1.
token. The starting index is randomly chosen to be 0 or  $l / 2$  with  $20\%$  chance for either scenario otherwise it is sampled uniformly at random.
Algorithm 1 STAGE 1 & 2
Input: Datasets  $\mathfrak{D}$ , number of steps  $N$ , parameterized family of translation models  $p_{\theta}$
1: Initialize  $\theta \gets \theta_0$
2: for step in 1, 2, 3, ...,  $N$  do
3: Choose dataset  $D$  at random from  $\mathfrak{D}$ .
4: if  $D$  consists of monolingual data then
5: Sample batch  $x$  from  $D$ .
6: MASS Loss:  $\mathrm{ml}\gets \mathcal{L}_{\mathrm{MASS}}(x)$
7: Update:  $\theta \gets$  optimizer_update(ml,  $\theta$ ).
8: else if  $D$  consists of auxiliary parallel data then
9: Sample batch  $(x,z)$  from  $D$
10:  $\mathrm{tl}\gets \mathcal{L}_{\mathrm{cross - entropy}}(x,z) + \mathcal{L}_{\mathrm{cross - entropy}}(z,x).$
11: Update:  $\theta \gets$  optimizer_update  $(\mathrm{tl},\theta)$
12: end if
13: end for
5.2 Second stage of training
Once we have completed the first stage, we will have produced an initial model capable of generating high-quality  $\mathrm{X} \rightarrow \mathrm{En}$  (to English) translations for all of the low-resource pairs we consider, also known as many-to-one setup in multilingual NMT (Johnson et al., 2017). Unfortunately, the model does not reach that level of performance for the  $\mathrm{En} \rightarrow \mathrm{X}$  translation directions, generating very low-quality translations into these low-resource languages. Note that, this phenomenon is ubiquitously observed in multilingual models (Firat et al., 2016a; Johnson et al., 2017; Aharoni et al., 2019). This abysmal performance could have dire consequences in the fine-tuning stage, since both on-the-fly back-translation and cross-translation rely heavily on intermediate translations. We verify that this is in fact the case in §7.2.
Instead, we exploit the strong  $\mathrm{X}\rightarrow \mathrm{En}$  performance by translating subsets of the monolingual data of the low-resource languages using our initial model and treat the result as pseudo-parallel datasets for the language pairs  $\mathrm{En}\rightarrow \mathrm{X}$ . More explicitly, given a sentence  $x$  from a low-resource language, we generate an English translation  $\tilde{y}_{\mathrm{En}}$  with our initial model and create a synthetic translation-pair  $(\tilde{y}_{\mathrm{En}}, x)$ . We refer to this procedure as offline back-translation (Sennrich et al., 2015). We add these datasets to our collection of auxiliary parallel corpora and repeat the training procedure from the first stage (Algorithm 1), starting from the last checkpoint. Note that, while offline back-translated (synthetic) data is commonly used for zero-resource translation (Firat et al., 2016b; Chen et al., 2017), it is
worth emphasizing the difference here again, that in the configuration studied in this paper, we do not assume the existence of any parallel data between  $\mathrm{En} \leftrightarrow \mathrm{X}$ , which is exploited by such methods.
Upon completion, we run the procedure a second time, with a new subset of synthetic data of twice the size for the  $\mathrm{En}\rightarrow \mathrm{X}$  pairs. Furthermore, since the translations from English have improved, we take disjoint subsets $^4$  of the English monolingual data and generate corpora of synthetic  $\mathrm{X}\rightarrow \mathrm{En}$  translation pairs that we also include in the second run of our procedure.
5.3 Third stage of training
For the third and final stage of training, we use back-translation of the monolingual data and cross-translation on the auxiliary parallel data. We also leverage the synthetic data through the cross-entropy objective. We present the procedure in detail under Algorithm 2.
6 Main experiment
In this section, we describe the details of our main experiment. As indicated in Figure 1, we consider five languages (Nepali, Sinhala, Gujarati, Kazakh, Turkish) as the target unsupervised language pairs with English. We leverage auxiliary parallel data from six higher-resource languages (Chinese, Russian, Arabic, Hindi, Telugu, Tamil) with English. The domains and counts for the datasets considered can be found in Table 1 and a more detailed discussion on the source of the data and the preprocessing steps can be found in the Appendix. In the following subsections, we provide detailed descriptions of the model configurations, training parameters, evaluation and discuss results of our main experiment.
6.1 Datasets and preprocessing
We draw most of our data from WMT. The monolingual data comes from News Crawl when available. For all the unsupervised pairs except Turkish, we supplement the News Crawl datasets with monolingual data from Common Crawl and Wikipedia<sup>7</sup>.
41 million lines of English per low-resource language.
<sup>5</sup>For Nepali, Sinhala and Gujarati, we use Hindi as the pivot language. For Turkish, we use Arabic and for Kazakh, we use Russian.
<sup>6</sup>http://data.statmt.org/news-crawl/
<sup>7</sup>We used the monolingual data available from https://github.com/facebookresearch/fores for Nepali and Sinhala in order to avoid any data leakage from the test sets.
3We utilize  $10\%$  of the monolingual data for each lowresource language.
Model	newstest2019 Gu ↔ En	newstest2019 Kk ↔ En	newstest2017 Tr ↔ En
No parallel data	Kim et al. (2020)	0.6	0.6	0.8	2.0	-	-
No parallel data for{Gu,Kk,Tr}	Stage 1 (Ours)	4.4	19.3	3.9	14.8	8.4	15.9
Stage 2 (Ours)	16.4	20.4	9.9	15.6	20.0	20.5
Stage 3 (Ours)	16.4	22.2	10.4	16.4	19.8	19.9
With parallel data for{Gu,Kk,Tr}	Mult. MT Baseline (Ours)	15.5	19.3	9.5	15.1	18.1	22.0
mBART (Liu et al., 2020)	0.1	0.3	2.5	7.4	17.8	22.5
Table 2: BLEU scores of various supervised and unsupervised models on the WMT newtest sets. The bolded numbers are the best unsupervised scores and the underlined numbers represent the best supervised scores. For any  $\mathrm{X} \leftrightarrow \mathrm{Y}$  language pair, the  $\mathrm{X} \rightarrow \mathrm{Y}$  translation results are listed under each  $\mathrm{Y}$  column, and vice-versa.
Model	FLoRes devtest	FLoRes devtest
Ne ↔ En	Si ↔ En
No parallel data	Guzmán et al. (2019)	0.1	0.5	0.1	0.1
No parallel data with{Ne,Si}	Liu et al. (2020)	-	17.9	-	9.0
Guzmán et al. (2019)	8.3	18.3	0.1	0.1
Stage 1 (Ours)	3.3	18.3	1.4	11.5
Stage 2 (Ours)	8.6	20.8	7.7	15.7
Stage 3 (Ours)	8.9	21.7	7.9	16.2
With parallel data for{Ne,Si}	Mult. MT Baseline (Ours)	8.6	20.1	7.6	15.3
Liu et al. (2020)	9.6	21.3	9.3	20.2
Guzmán et al. (2019)	8.8	21.5	6.5	15.1