SlowGuess/ACL_2020_OCR huggingface dataset:
Biao Zhang $^{1}$  Philip Williams $^{1}$  Ivan Titov $^{1,2}$  Rico Senrich $^{3,1}$
$^{1}$ School of Informatics, University of Edinburgh
$^{2}$ ILLC, University of Amsterdam
$^{3}$ Department of Computational Linguistics, University of Zurich
B.Zhang@ed.ac.uk, {pwillia4,ititov}@inf.ed.ac.uk, sennrich@cl.uzh.ch
Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. In this paper, we explore ways to improve them. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics, and overcome this bottleneck via language-specific components and deepening NMT architectures. We identify the off-target translation issue (i.e. translating into a wrong target language) as the major source of the inferior zero-shot performance, and propose random online backtranslation to enforce the translation of unseen training language pairs. Experiments on OPUS-100 (a novel multilingual dataset with 100 languages) show that our approach substantially narrows the performance gap with bilingual models in both one-to-many and many-to-many settings, and improves zero-shot performance by  $\sim 10$  BLEU, approaching conventional pivot-based methods.
With the great success of neural machine translation (NMT) on bilingual datasets (Bahdanau et al., 2015; Vaswani et al., 2017; Barrault et al., 2019), there is renewed interest in multilingual translation where a single NMT model is optimized for the translation of multiple language pairs (Firat et al., 2016a; Johnson et al., 2017; Lu et al., 2018; Aharoni et al., 2019). Multilingual NMT eases model deployment and can encourage knowledge transfer among related language pairs (Lakew et al., 2018; Tan et al., 2019), improve low-resource translation (Ha et al., 2016; Arivazhagan et al., 2019b),
Source	Jusqu'à ce qu'on trouve le moment clé, celui où tu as su que tu l'aimais.
Reference	Bis wir den unverkennbaren Moment gefun-den haben, den Moment, wo du wusstest, du liebst ihn.
Zero-Shot	Jusqu'à ce qu'on trouve le moment clé, celui où tu as su que tu l'aimais.
Source	Les États membres ont été consultés et ontapprovéd cette proposition.
Reference	Die Mitgliedstaaten wurden konsultiert undspruchen sich für diesen Vorschlag aus.
Zero-Shot	Les Member States have been consultedésand have approved this proposal.
Table 1: Illustration of the off-target translation issue with French  $\rightarrow$  German zero-shot translations with a multilingual NMT model. Our baseline multilingual NMT model often translates into the wrong language for zero-shot language pairs, such as copying the source sentence or translating into English rather than German.
and enable zero-shot translation (i.e. direct translation between a language pair never seen in training) (Firat et al., 2016b; Johnson et al., 2017; Al-Shedivat and Parikh, 2019; Gu et al., 2019).
Despite these potential benefits, multilingual NMT tends to underperform its bilingual counterparts (Johnson et al., 2017; Arivazhagan et al., 2019b) and results in considerably worse translation performance when many languages are accommodated (Aharoni et al., 2019). Since multilingual NMT must distribute its modeling capacity between different translation directions, we ascribe this deteriorated performance to the deficient capacity of single NMT models and seek solutions that are capable of overcoming this capacity bottleneck. We propose language-aware layer normalization and linear transformation to relax the representation constraint in multilingual NMT models. The linear transformation is inserted in-between the encoder and the decoder so as to facilitate the induction of language-specific translation correspond
<sup>1</sup>We release our code at https://github.com/bzhangGo/zero. We release the OPUS-100 dataset at https://github.com/EdinburghNLP/opus-100-corpus.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1628-1639 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
dences. We also investigate deep NMT architectures (Wang et al., 2019a; Zhang et al., 2019) aiming at further reducing the performance gap with bilingual methods.
Another pitfall of massively multilingual NMT is its poor zero-shot performance, particularly compared to pivot-based models. Without access to parallel training data for zero-shot language pairs, multilingual models easily fall into the trap of off-target translation where a model ignores the given target information and translates into a wrong language as shown in Table 1. To avoid such a trap, we propose the random online backtranslation (ROBT) algorithm. ROBT finetunes a pretrained multilingual NMT model for unseen training language pairs with pseudo parallel batches generated by back-translating the target-side training data. We perform backtranslation (Sennrich et al., 2016a) into randomly picked intermediate languages to ensure good coverage of  $\sim 10,000$  zero-shot directions. Although backtranslation has been successfully applied to zero-shot translation (Firat et al., 2016b; Gu et al., 2019; Lakew et al., 2019), whether it works in the massively multilingual set-up remained an open question and we investigate it in our work.
For experiments, we collect OPUS-100, a massively multilingual dataset sampled from OPUS (Tiedemann, 2012). OPUS-100 consists of 55M English-centric sentence pairs covering 100 languages. As far as we know, no similar dataset is publicly available. We have released OPUS-100 to facilitate future research. We adopt the Transformer model (Vaswani et al., 2017) and evaluate our approach under one-to-many and many-to-many translation settings. Our main findings are summarized as follows:
- Increasing the capacity of multilingual NMT yields large improvements and narrows the performance gap with bilingual models. Low-resource translation benefits more from the increased capacity.
- Language-specific modeling and deep NMT architectures can slightly improve zero-shot
translation, but fail to alleviate the off-target translation issue.
- Finetuning multilingual NMT with ROBT substantially reduces the proportion of off-target translations (by  $\sim 50\%$ ) and delivers an improvement of  $\sim 10$  BLEU in zero-shot settings, approaching the conventional pivot-based method. We show that finetuning with ROBT converges within a few thousand steps.
Pioneering work on multilingual NMT began with multitask learning, which shared the encoder for one-to-many translation (Dong et al., 2015) or the attention mechanism for many-to-many translation (Firat et al., 2016a). These methods required a dedicated encoder or decoder for each language, limiting their scalability. By contrast, Lee et al. (2017) exploited character-level inputs and adopted a shared encoder for many-to-one translation. Ha et al. (2016) and Johnson et al. (2017) further successfully trained a single NMT model for multilingual translation with a target language symbol guiding the translation direction. This approach serves as our baseline. Still, this paradigm forces different languages into one joint representation space, neglecting their linguistic diversity. Several subsequent studies have explored different strategies to mitigate this representation bottleneck, ranging from reorganizing parameter sharing (Blackwood et al., 2018; Sachan and Neubig, 2018; Lu et al., 2018; Wang et al., 2019c; Vázquez et al., 2019), designing language-specific parameter generators (Platanios et al., 2018), decoupling multilingual word encodings (Wang et al., 2019b) to language clustering (Tan et al., 2019). Our language-specific modeling continues in this direction, but with a special focus on broadening normalization layers and encoder outputs.
Multilingual NMT allows us to perform zero-shot translation, although the quality is not guaranteed (Firat et al., 2016b; Johnson et al., 2017). We observe that multilingual NMT often translates into the wrong target language on zero-shot directions (Table 1), resonating with the 'missing ingredient problem' (Arivazhagan et al., 2019a) and the spurious correlation issue (Gu et al., 2019). Approaches to improve zero-shot performance fall into two categories: 1) developing novel cross-lingual regularizers, such as the alignment regularizer (Arivazhagan et al., 2019a) and the consistency regularizer (Al
Note that backtranslation actually converts the zero-shot problem into a zero-resource problem. We follow previous work and continue referring to zero-shot translation, even when using synthetic training data.
3Previous studies (Aharoni et al., 2019; Arivazhagan et al., 2019b) adopt in-house data which was not released.
4https://github.com/EdinburghNLP/ opus-100-corpus
Shedivat and Parikh, 2019); and 2) generating artificial parallel data with backtranslation (Firat et al., 2016b; Gu et al., 2019; Lakew et al., 2019) or pivot-based translation (Currey and Heafield, 2019). The proposed ROBT algorithm belongs to the second category. In contrast to Gu et al. (2019) and Lakew et al. (2019), however, we perform online back-translation for each training step with randomly selected intermediate languages. ROBT avoids decoding the whole training set for each zero-shot language pair and can therefore scale to massively multilingual settings.
Our work belongs to a line of research on massively multilingual translation (Aharoni et al., 2019; Arivazhagan et al., 2019b). Aharoni et al. (2019) demonstrated the feasibility of massively multilingual NMT and reported encouraging results. We continue in this direction by developing approaches that improve both multilingual and zero-shot performance. Independently from our work, Arivazhagan et al. (2019b) also find that increasing model capacity with deep architectures (Wang et al., 2019a; Zhang et al., 2019) substantially improves multilingual performance. A concurrent related work is (Bapna and First, 2019), which introduces task-specific and lightweight adaptors for fast and scalable model adaptation. Compared to these adaptors, our language-aware layers are jointly trained with the whole NMT model from scratch without relying on any pretraining.
3 Multilingual NMT
We briefly review the multilingual approach (Ha et al., 2016; Johnson et al., 2017) and the Transformer model (Vaswani et al., 2017), which are used as our baseline. Johnson et al. (2017) rely on preponding tokens specifying the target language to each source sentence. In that way a single NMT model can be trained on the modified multilingual dataset and used to perform multilingual translation. Given a source sentence  $\mathbf{x} = (x_{1}, x_{2}, \ldots, x_{|\mathbf{x}|})$ , its target reference  $\mathbf{y} = (y_{1}, y_{2}, \ldots, y_{|\mathbf{y}|})$  and the target language token  $t^5$ , multilingual NMT translates under the encoder-decoder framework (Bahdanau et al., 2015):
$$
\mathbf {H} = \operatorname {E n c o d e r} ([ t, \mathbf {x} ]), \tag {1}
$$
$$
\mathbf {S} = \operatorname {D e c o d e r} (\mathbf {y}, \mathbf {H}), \tag {2}
$$
where  $\mathbf{H} \in \mathbb{R}^{|\mathbf{x}| \times d} / \mathbf{S} \in \mathbb{R}^{|\mathbf{y}| \times d}$  denote the encoder/decoder output.  $d$  is the model dimension.
We employ the Transformer (Vaswani et al., 2017) as the backbone NMT model due to its superior multilingual performance (Lakew et al., 2018). The encoder is a stack of  $L = 6$  identical layers, each containing a self-attention sublayer and a point-wise feedforward sublayer. The decoder follows a similar structure, except for an extra cross-attention sublayer used to condition the decoder on the source sentence. Each sublayer is equipped with a residual connection (He et al., 2015), followed by layer normalization (Ba et al., 2016, LN( $\cdot$ )):
$$
\bar {\mathbf {a}} = \mathrm {L N} (\mathbf {a} \mid \mathbf {g}, \mathbf {b}) = \frac {\mathbf {a} - \mu}{\sigma} \odot \mathbf {g} + \mathbf {b}, \tag {3}
$$
where  $\odot$  denotes element-wise multiplication,  $\mu$  and  $\sigma$  are the mean and standard deviation of the input vector  $\mathbf{a} \in \mathbb{R}^d$ , respectively.  $\mathbf{g} \in \mathbb{R}^d$  and  $\mathbf{b} \in \mathbb{R}^d$  are model parameters. They control the sharpness and location of the regularized layer output  $\bar{\mathbf{a}}$ . Layer normalization has proven effective in accelerating model convergence (Ba et al., 2016).
Despite its success, multilingual NMT still suffers from 1) insufficient modeling capacity, where including more languages results in reduction in translation quality (Aharoni et al., 2019); and 2) off-target translation, where models translate into a wrong target language on zero-shot directions (Arivazhagan et al., 2019a). These drawbacks become severe in massively multilingual settings and we explore approaches to alleviate them. We hypothesize that the vanilla Transformer has insufficient capacity and search for model-level strategies such as deepening Transformer and devising language-specific components. By contrast, we regard the lack of parallel data as the reason behind the off-target issue. We resort to data-level strategy by creating, in online fashion, artificial parallel training data for each zero-shot language pair in order to encourage its translation.
Deep Transformer One natural way to improve the capacity is to increase model depth. Deeper neural models are often capable of inducing more generalizable ('abstract') representations and capturing more complex dependencies and have shown encouraging performance on bilingual translation (Bapna et al., 2018; Zhang et al., 2019; Wang
${}^{5}t$  is in the form of “  $< 2\mathrm{X} >$  ” where  $\mathrm{X}$  is a language name, such as  $< 2\mathrm{{EN}} >$  meaning translating into English.
et al., 2019a). We adopt the depth-scaled initialization method (Zhang et al., 2019) to train a deep Transformer for multilingual translation.
Language-aware Layer Normalization Regardless of linguistic differences, layer normalization in multilingual NMT simply constrains all languages into one joint Gaussian space, which makes learning more difficult. We propose to relax this restriction by conditioning the normalization on the given target language token  $t$  (LALN for short) as follows:
$$
\bar {\mathbf {a}} = \operatorname {L N} \left(\mathbf {a} \mid \mathbf {g} _ {t}, \mathbf {b} _ {t}\right). \tag {4}
$$
We apply this formula to all normalization layers, and leave the study of conditioning on source language information for the future.
Language-aware Linear Transformation Different language pairs have different translation correspondences or word alignments (Koehn, 2010). In addition to LALN, we introduce a target-language-aware linear transformation (LALT for short) between the encoder and the decoder to enhance the freedom of multilingual NMT in expressing flexible translation relationships. We adapt Eq. (2) as follows:
$$
\mathbf {S} = \operatorname {D e c o d e r} (\mathbf {y}, \mathbf {H} \mathbf {W} _ {t}), \tag {5}
$$
where  $\mathbf{W}_t \in \mathbb{R}^{d \times d}$  denotes model parameters. Note that adding one more target language in LALT brings in only one weight matrix. Compared to existing work (First et al., 2016b; Sachan and Neubig, 2018), LALT reaches a better trade-off between expressivity and scalability.
Random Online Backtranslation Prior studies on backtranslation for zero-shot translation decode the whole training set for each zero-shot language pair (Gu et al., 2019; Lakew et al., 2019), and scalability to massively multilingual translation is questionable – in our setting, the number of zero-shot translation directions is 9702.
We address scalability by performing online backtranslation paired with randomly sampled intermediate languages. Algorithm 1 shows the detail of ROBT, where for each training instance  $(\mathbf{x}_k,\mathbf{y}_k,t_k)$ , we uniformly sample an intermediate language  $t_k^\prime$  ( $t_k\neq t_k^\prime$ ), back-translate  $\mathbf{y}_k$  into
$t_k^\prime$  to obtain  $\mathbf{x}_k^\prime$ , and train on the new instance  $(\mathbf{x}_k^\prime, \mathbf{y}_k, t_k)$ . Although  $\mathbf{x}_k^\prime$  may be poor initially (translations are produced on-line by the model being trained), ROBT still benefits from the translation signal of  $t_k^\prime \rightarrow t_k$ . To reduce the computational cost, we implement batch-based greedy decoding for line 7.
5 OPUS-100
Recent work has scaled up multilingual NMT from a handful of languages to tens or hundreds, with many-to-many systems being capable of translation in thousands of directions. Following Aharoni et al. (2019), we created an English-centric dataset, meaning that all training pairs include English on either the source or target side. Translation for any language pair that does not include English is zero-shot or must be pivoted through English.
We created OPUS-100 by sampling data from the OPUS collection (Tiedemann, 2012). OPUS-100 is at a similar scale to Aharoni et al. (2019)'s, with 100 languages (including English) on both sides and up to 1M training pairs for each language pair. We selected the languages based on the volume of parallel data available in OPUS.
The OPUS collection is comprised of multiple corpora, ranging from movie subtitles to GNOME
We also attempted to factorize  $\mathbf{W}_t$  into smaller matrices/vectors to reduce the number of parameters. Unfortunately, the final performance was rather disappointing.
ID	Model Architecture	L	#Param	BLEU94	WR	BLEU4
1	Transformer, Bilingual	6	106M	-	-	20.90
2	Transformer, Bilingual	12	150M	-	-	22.75
3	Transformer	6	106M	24.64	ref	18.95
4	3 + MATT	6	99M	23.81	20.2	17.95
5	4 + LALN	6	102M	24.22	28.7	18.50
6	4 + LALT	6	126M	27.11	72.3	20.28
7	4 + LALN + LALT	6	129M	27.18	75.5	20.08
8	4	12	137M	25.69	81.9	19.13
9	7	12	169M	28.04	91.5	19.93
10	7	24	249M	29.60	92.6	21.23
Table 2: Test BLEU for one-to-many translation on OPUS-100 (100 languages). "Bilingual": bilingual NMT, "L": model depth (for both encoder and decoder), "#Param": parameter number, "WR": win ratio (\%) compared to ref (\%), MATT: the merged attention (Zhang et al., 2019). LALN and LALT denote the proposed language-aware layer normalization and linear transformation, respectively. "BLEU $_{94}$ /BLEU $_{4}$ " : average BLEU over all 94 translation directions in test set and  $\mathrm{En} \rightarrow \mathrm{De} / \mathrm{Zh} / \mathrm{Br} / \mathrm{Te}$ , respectively. Higher BLEU and WR indicate better result. Best scores are highlighted in bold.
documentation to the Bible. We did not curate the data or attempt to balance the representation of different domains, instead opting for the simplest approach of downloading all corpora for each language pair and concatenating them. We randomly sampled up to 1M sentence pairs per language pair for training, as well as 2000 for validation and 2000 for testing. To ensure that there was no overlap (at the monolingual sentence level) between the training and validation/test data, we applied a filter during sampling to exclude sentences that had already been sampled. Note that this was done cross-lingually, so an English sentence in the Portuguese-English portion of the training data could not occur in the Hindi-English test set, for instance.
OPUS-100 contains approximately 55M sentence pairs. Of the 99 language pairs, 44 have 1M sentence pairs of training data, 73 have at least 100k, and 95 have at least 10k.
To evaluate zero-shot translation, we also sampled 2000 sentence pairs of test data for each of the 15 pairings of Arabic, Chinese, Dutch, French, German, and Russian. Filtering was used to exclude sentences already in OPUS-100.
We perform one-to-many (English-X) and many-to-many (English-X  $\cup$  X-English) translation on OPUS-100 (  $|\mathcal{T}|$  is 100). We apply byte pair encoding (BPE) (Sennrich et al., 2016b; Kudo and Richardson, 2018) to handle multilingual words with a joint vocabulary size of 64k. We randomly
shuffle the training set to mix instances of different language pairs. We adopt BLEU (Papineni et al., 2002) for translation evaluation with the toolkit SacreBLEU (Post, 2018). We employ the langdetect library to detect the language of translations, and measure the translation-language accuracy for zero-shot cases. Rather than providing numbers for each language pair, we report average BLEU over all 94 language pairs with test sets  $\mathrm{(BLEU_{94})}$ . We also show the win ratio (WR), counting the proportion where our approach outperforms its baseline.
Apart from multilingual NMT, our baselines also involve bilingual NMT and pivot-based translation (only for zero-shot comparison). We select four typologically different target languages (German/De, Chinese/Zh, Breton/Br, Telugu/Te) with varied training data size for comparison to bilingual models as applying bilingual NMT to each language pair is resource-consuming. We report average BLEU over these four languages as  $\mathrm{BLEU}_4$ . We reuse the multilingual BPE vocabulary for bilingual NMT.
We train all NMT models with the Transformer base settings (512/2048, 8 heads) (Vaswani et al., 2017). We pair our approaches with the merged attention (MATT) (Zhang et al., 2019) to reduce training time. Other details about model settings are in the Appendix.
6.2 Results on One-to-Many Translation
Table 2 summarizes the results. The inferior performance of multilingual NMT (3) against its
<sup>7</sup>For efficiency, we only use 200 sentences per language pair for validation in our multilingual experiments.
$^{8}$ Signature: BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+version.1.4.1
https://github.com/Mimino666/ langdetect
ID	Model Architecture	L	#Param	w/o ROBT	w/ ROBT
BLEU94	WR	BLEU4	BLEU94	WR	BLEU4
1	Transformer, Bilingual	6	110M	-	-	20.28	-	-	-
2	Transformer	6	110M	19.50	ref	15.35	18.75	4.3	14.73
3	2 + MATT	6	103M	18.49	5.3	14.90	17.85	6.4	14.38
4	3 + LALN + LALT	6	133M	21.39	78.7	18.13	20.81	69.1	17.45
5	3	12	141M	20.77	94.7	16.08	20.24	84.0	15.80
6	4	12	173M	22.86	97.9	19.25	22.39	97.9	18.23
7	4	24	254M	23.96	100.0	19.83	23.36	97.9	19.45
Table 3: English→X test BLEU for many-to-many translation on OPUS-100 (100 languages). "WR": win ratio (%) compared to ref (② w/o ROBT). ROBT denotes the proposed random online backtranslation method.
ID	Model Architecture	L	#Param	w/o ROBT	w/ ROBT
BLEU94	WR	BLEU4	BLEU94	WR	BLEU4
1	Transformer, Bilingual	6	110M	-	-	21.23	-	-	-
2	Transformer	6	110M	27.60	ref	23.35	27.02	14.9	22.50
3	2 + MATT	6	103M	26.90	2.1	22.78	26.28	4.3	21.53
4	3 + LALN + LALT	6	133M	27.50	37.2	23.05	27.22	23.4	23.30
5	3	12	141M	29.15	98.9	24.15	28.80	91.5	24.03
6	4	12	173M	29.49	97.9	24.53	29.54	96.8	25.43
7	4	24	254M	31.36	98.9	26.03	30.98	95.7	26.78
Table 4: X→English test BLEU for many-to-many translation on OPUS-100 (100 languages). "WR": win ratio (%) compared to ref (② w/o ROBT).
bilingual counterpart (①) reflects the capacity issue  $(-1.95\mathrm{BLEU}_4)$ . Replacing the self-attention with MATT slightly deteriorates performance  $(-0.83\mathrm{BLEU}_{94}$  ③→④); we still use MATT for more efficiently training deep models.
Our ablation study (4-7) shows that enriching the language awareness in multilingual NMT substantially alleviates this capacity problem. Relaxing the normalization constraints with LALN gains 0.41  $\mathrm{BLEU}_{94}$  with  $8.5\%$  WR  $(4\rightarrow 5)$ . Decoupling different translation relationships with LALT delivers an improvement of 3.30  $\mathrm{BLEU}_{94}$  and  $52.1\%$  WR  $(4\rightarrow 6)$ . Combining LALT and LALN demonstrates their complementarity (+3.37  $\mathrm{BLEU}_{94}$  and  $+55.3\%$  WR,  $(4\rightarrow 7)$ , significantly outperforming the multilingual baseline (+2.54  $\mathrm{BLEU}_{94}$ ,  $(3\rightarrow 7)$ ), albeit still behind the bilingual models (-0.82  $\mathrm{BLEU}_4$ ,  $(1\rightarrow 7)$ ).