SlowGuess/NAACL_2021_OCR huggingface dataset:
Qi Liu†, Matt J. Kusner†*, Phil Blunsom‡,
$^{\ddagger}$ University of Oxford  $^{\diamond}$ DeepMind
†University College London *The Alan Turing Institute
$\ddagger$  {firstname.lastname}@cs.ox.ac.uk
†m.kusner@ucl.ac.uk
We propose a data augmentation method for neural machine translation. It works by interpreting language models and phrasal alignment causally. Specifically, it creates augmented parallel translation corpora by generating (path-specific) counterfactual aligned phrases. We generate these by sampling new source phrases from a masked language model, then sampling an aligned counterfactual target phrase by noting that a translation language model can be interpreted as a Gumbel-Max Structural Causal Model (Oberst and Sontag, 2019). Compared to previous work, our method takes both context and alignment into account to maintain the symmetry between source and target sequences. Experiments on IWSLT'15 English  $\rightarrow$  Vietnamese, WMT'17 English  $\rightarrow$  German, WMT'18 English  $\rightarrow$  Turkish, and WMT'19 robust English  $\rightarrow$  French show that the method can improve the performance of translation, backtranslation and translation robustness.
Neural machine translation (NMT) models (Kalchbrenner and Blunsom, 2013; Bahdanau et al., 2014; Vaswani et al., 2017) have reached state-of-the-art performance on various benchmarks. However, these models frequently rely on large-scale parallel corpora for training, exhibiting degraded performance on low-resource languages (Zoph et al., 2016). Further, modern NMT systems are often brittle, as noises (e.g. grammatical errors) can cause significant mistranslations (Sakaguchi et al., 2017; Michel and Neubig, 2018).
Data augmentation is a promising direction to overcome these issues. It works by enlarging the number of data points for training without manually collecting new data. It is widely used to improve diversity and robustness and to avoid overfitting on small datasets. Even though data augmentation (e.g. image flipping, cropping and blurring) has
Figure 1: We interpret a translation language model  $p(Y_{j}|\mathcal{X},\mathcal{Y}_{-j})$ $(\mathcal{V}_{-j}$  means that phrase  $Y_{j}$  has been removed from sequence  $\mathcal{V}$  ) as a causal model. The randomness of the causal model comes from unobserved variables  $\mathbf{G}$ . For data augmentation, we sample counterfactual parallel sequences based on the causal effects singled-out by an unsupervised alignment model (i.e., the black arrows from  $\mathcal{X}$  to  $\mathcal{V}$  above).
become a standard technique in computer vision (Krizhevsky et al., 2012; Huang et al., 2017; Chen et al., 2020), it is non-trivial to apply in machine translation since even a slight modification to a sequence can result in drastic changes in its syntax and semantics. Indeed there is relatively little work in this direction due to these difficulties (Sennrich et al., 2016; Fadaee et al., 2017; Wang et al., 2018; Gao et al., 2019; Xia et al., 2019; Kobayashi, 2018). Further, work based on word replacement either ignores the contexts of replaced words or breaks the alignment between source and target sequences, both detrimental for generating high-quality data.
In this paper we observe that a translation language model can be interpreted as a causal model, as described in Figure 1. Doing so allows us to ask counterfactual questions of the form: Given source and target sequences, if a phrase in the source sequence is changed, how would the target sequence change? We propose a data augmentation method for machine translation that generates counterfactual parallel translation data. To ensure these counterfactuals are close to the original data we sample a new source phrase from a masked language
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 187-197 June 6-11, 2021. ©2021 Association for Computational Linguistics
model. We then consider the (path-specific) counterfactual target phrase that is aligned to that source phrase (given by an unsupervised phrasal alignment method). The idea is that this augmentation procedure exposes inductive biases in existing language models that enables new translation models to learn more efficiently and exhibit more robust generalisation. Specifically, our augmentation procedure performs the following three steps:
1. We utilize unsupervised phrasal alignment (e.g. Neubig et al. (2011) and Dyer et al. (2013)) to obtain correspondences between source and target phrases.
2. A source phrase is removed and then resampled according to a trained masked language model (Devlin et al., 2018; Raffel et al., 2019).
3. We perform (path-specific) counterfactual inference on the causal model given by a trained translation language model (Lample and Conneau, 2019) to resample only the aligned target phrase, given the changed source phrase.
Different from prior work, our approach takes advantage of both source/target context and alignment for data augmentation. Experiments on IWSLT'15 English  $\rightarrow$  Vietnamese, WMT'17 English  $\rightarrow$  German, and WMT'18 English  $\rightarrow$  Turkish show that our method improves the translation performance on both high-resource and low-resource datasets. We additionally demonstrate that our method complements existing approaches such as backtranslation (Sennrich et al., 2015a). Finally, we demonstrate that our method improves translation robustness (we evaluate this on the WMT'19 English  $\rightarrow$  French robustness dataset).
In this section we describe background on neural machine translation (NMT), phrasal alignment, and causal modelling.
Neural machine translation. Given a set of parallel sequences,  $\mathcal{S} = \{(\mathcal{X}^i,\mathcal{Y}^i)\}_{i = 1}^N$ , NMT maximizes the log-likelihood of  $\mathcal{V}$  given  $\mathcal{X}$ , assuming each  $(\mathcal{X}^i,\mathcal{Y}^i)$  pair is independently and identically distributed:
$$
\max  _ {\theta} \sum_ {(\mathcal {X} ^ {i}, \mathcal {Y} ^ {i}) \in \mathcal {S}} \log p _ {\theta} (\mathcal {Y} ^ {i} | \mathcal {X} ^ {i}).
$$
However, paired sequences are usually expensive to collect, as it requires an expert to translate sequences  $\mathcal{X}^i$  into another language  $\mathcal{Y}^i$ . Data augmentation aims to generate new parallel sequences  $(\hat{\mathcal{X}}^i,\hat{\mathcal{Y}}^i)$  without manually collecting new data.
Phrasal alignment. Phrasal alignment identifies the translation relationships among phrases in parallel sequences. Given a parallel sequence  $(\mathcal{X},\mathcal{Y})$  where  $\mathcal{X} = (X_1 = x_1,X_2 = x_2,\dots,X_{|\mathcal{X}|} = x_{|\mathcal{X}|})$  and  $\mathcal{Y} = (Y_1 = y_1,Y_2 = y_2,\dots,Y_{|\mathcal{Y}|} = y_{|\mathcal{Y}|})$ $(X / Y$  and  $x / y$  denote a phrase and its value, respectively), phrasal alignment  $h$  learns a mapping that projects each position  $i$  of  $\mathcal{X}$  to a position  $j$  of  $\mathcal{V}$ , i.e.  $j = h(i)$ . In this paper, we use pialign (Neubig et al., 2011) to obtain alignments.
Causal modelling. We formulate causality using the structural causal model (SCM) framework of Pearl (2003). Each SCM is a set of structural equations represented by a graph. The edges of this graph specify the inputs and outputs of the structural equations. Specifically, a variable  $V_{i}$  is caused by a set of observable parent variables  $pa(V_{i})$  and unobserved variables  $\mathbf{U}_i$  if there exists a deterministic or stochastic) structural equation  $f_{i}$ :
$$
V _ {i} = f _ {i} \left(p a \left(V _ {i}\right), \mathbf {U} _ {i}\right).
$$
If the structural equations  $f$  are identified, it is possible to compute a causal quantity called counterfactuals. Counterfactuals are questions that, given the current state of the world, ask what would have changed if some variable  $V$  had been different. For example, "Would a person have been able to obtain a visa if they had been born in a different country?" Formally we denote the counterfactual value of a variable  $V_{i}$ , had another variable  $W \in pa(V_{i})$  been  $\hat{w}$  (i.e., compared to its observed value  $w$ ) as  $V_{i,W \leftarrow \hat{w}}$ . To compute counterfactuals we can follow a three-step procedure (for more details see Chapter 4 of Pearl et al. (2016)): 1. **Abduction:** Given a prior distribution on unobserved variables  $p(\mathbf{U}_i)$ , compute the posterior given all observed variables  $\mathbf{V} = \mathbf{v}$ :  $p(\mathbf{U}_i|\mathbf{V} = \mathbf{v})$ ; 2. **Action:** Modify the structural equation for  $V_{i}$ , so that  $W$  is fixed to the counterfactual value  $\hat{w}$  (the modified equation is denoted as  $f_{i,\hat{w}}$ ); 3. **Prediction:** Compute the distribution  $p(V_{i,W \leftarrow \hat{w}}|\mathbf{V} = \mathbf{v})$  using  $p(\mathbf{U}_i|\mathbf{V} = \mathbf{v})$ , the observed variables  $\mathbf{v}$ , and the modified structural equation  $f_{i,\hat{w}}$ .
Figure 2: The three steps of Translation-Counterfactual Word Replacement. See text for details.
Our goal is to take an input sequence pair  $(\mathcal{X},\mathcal{Y})$  and create augmented data from it. We aim to do so by removing phrases, resampling them in the source sequence, and computing the counterfactual effect of doing so in the target sequence. We argue that for any such augmentation method for NMT, it is crucial to leverage both contextual and alignment information, for the following reasons. (1) Context: As contextual information is widely used to disambiguate words (Peters et al., 2018) and generate realistic-looking sequences (Zellers et al., 2019), it is critical to utilize contextual information to obtain grammatically-correct and semantically-sound sequences. (2) Alignment: Phrasal alignment plays a critical role in statistical machine translation (Brown et al., 1993; Vogel et al., 1996). As phrasal alignment provides information about which phrase in the source sequence produces a phrase in the target sequence, a data augmentation algorithm which disregards alignment risks breaking the symmetry between source and target sequences. To this end, in Section 3.1, we introduce a technique called Translation-Counterfactual Word Replacement (TCWR) for leveraging both context and alignment to replace phrases in source and target sequences. In Section 3.2, we propose a new data augmentation algorithm based on this replacement technique. In Section 3.3, we describe the architectures used to parameterize the models.
3.1 Translation-Counterfactual Word Replacement
Consider the sequence pair  $(\mathcal{X},\mathcal{Y})$  in Figure 2. A translation language model that learns  $p(Y_{j}|\mathcal{X},\mathcal{Y}_{-j})$  (where  $\mathcal{V}_{-j}$  indicates the sequence  $\mathcal{V}$  with  $Y_{j}$  removed) for all  $j\in \{1,\dots,|\mathcal{V}|\}$  induces a causal graph on this pair. Specifically it is fully connected, in the following way: (a) all phrases in  $\mathcal{X}$  cause all phrases in  $\mathcal{V}$ , (b) all phrases in  $\mathcal{V}$
cause all other phrases in  $\mathcal{V}$  (these connections are signified by the wide gray arrow in Figure 2). Additionally, there are unobserved variables  $\mathbf{G}_i^{\mathcal{X}},\mathbf{G}_i^{\mathcal{Y}}$  that cause each individual phrase (more on this below). We choose this fully connected structure to take contexts of each phrase into account. Note that this graph is cyclic, yet the counterfactual distribution we care about is identifiable given the posterior of the unobserved variables (which we describe below) and the known equations of the causal model (i.e., the translation language model).
Consider that we have an alignment between  $\mathcal{X}$  and  $\mathcal{Y}$ , which singles-out the causal effects shown with black arrows in Figure 2. Our idea is to derive a new sequence pair  $(\hat{\mathcal{X}}, \hat{\mathcal{Y}})$  by computing a counterfactual. We propose to calculate the counterfactual corresponding to a single alignment, i.e. a path-specific counterfactual: "What would  $Y_{j}$  have looked like, had  $X_{i} = \hat{x}_{i}$  instead of  $x_{i}$ , given that  $Y_{j}$  is aligned to  $X_{i}$ , and all other phrases  $\mathcal{X}_{-i}, \mathcal{Y}_{-j}$  had been held constant?". This allows us to consider 1. Context: By holding all other phrases constant we control for the specific context around the changed phrases  $\hat{x}_{i}, \hat{y}_{j}^{1}$ ; 2. Alignment: The derived counterfactual is based on the direct effect of  $X_{i}$  on  $Y_{j}$ , where this singled-out link is identified from an alignment.
We now outline the three steps to calculate the counterfactual. The example in Figure 2 is used for illustration. The goal is to sample from the following counterfactual distribution:  $p(Y_{2,\{X_3 \leftarrow \hat{x}_3, \mathcal{X}_{-3} \leftarrow \mathbf{x}_{-3}, \mathcal{Y}_{-2} \leftarrow \mathbf{y}_{-2}\}} | \mathcal{X}, \mathcal{Y})$  with the translation language model, which describes "What would  $Y_2$  have looked like, had  $X_3 = \hat{x}_3$  instead of  $x_3$ , given that  $Y_2$  is aligned to  $X_3$ , and all other phrases  $\mathcal{X}_{-3}, \mathcal{Y}_{-2}$  had been held constant?" For ease of illustration, we assume both  $X_3$  and  $Y_2$
Further, the posterior of unobserved random variables  $\mathbf{G}_j^{\mathcal{Y}}$  given  $\mathcal{X},\mathcal{Y}$  will encode additional context w.r.t. the original sequence pair  $\mathcal{X},\mathcal{Y}$ .
contain one token after Byte Pair Encoding (BPE) segmentation (Sennrich et al., 2015b). In Section 3.3, we explain how we use a sequence-to-sequence model to generate phrases containing multiple tokens after segmentation.
1. Abduction. The goal of the abduction step is to estimate any unobserved variables that impact the counterfactual. As our translation language model specifies a categorical distribution  $p(Y_{2}|\mathcal{X},\mathcal{Y}_{-2})$ , this unobserved randomness, i.e. the prior of  $\mathbf{G}_2^{\mathcal{Y}}$ , takes the form of a Gumbel random vector. This is due to the fact that random sampling from a categorical distribution can be done via a procedure called the Gumbel-Max Trick (Maddison et al., 2014b).
Definition 3.1 (Gumbel-Max Trick). Two steps are required to sample from a categorical distribution  $p(Y)$  with  $K$  categories: 1. Sample  $g_1, \ldots, g_K \sim Gumbel(0,1)$ . Each  $g_k$  can be computed as  $g_k = -\log(-\log u_k)$  where  $u_k \sim \text{Uniform}(0,1)$ ; 2. Compute  $y = \arg \max_{k=1,\ldots,K} \log p(Y=k)+g_k$ .
As such, sampling from the translation language model  $p(Y_{2}|\mathcal{X},\mathcal{Y}_{-2})$  with vocabulary size  $|V|$  can be written as follows,
$$
\begin{array}{l} y_{2} = \operatorname *{arg  max}_{k = 1,\ldots ,|V|}\log p(Y_{2} = k|\mathcal{X},\mathcal{Y}_{-2}) + g_{k}, \\ s. t. g _ {k} \sim \operatorname {G u m b e l} (0, 1). \\ \end{array}
$$
The abduction step samples from the posterior distribution over these Gumbel random variables, given the observed pair  $(\mathcal{X},\mathcal{Y})$ , i.e.,  $p(\mathbf{G}_2^{\mathcal{Y}}|\mathcal{X},\mathcal{Y})$ . Fortunately, sampling from the posterior is straightforward to do in two steps (Maddison et al., 2014a; Maddison and Tarlow, 2017): 1. Let  $y_{2} = k^{*}$ . Sample  $\hat{g}_{k^{*}}\sim \mathrm{Gumbel}(0,1)$ ; 2. For the remaining  $k$ , compute the probabilities from the model  $p(Y_{2} = k|\mathcal{X},\mathcal{Y}_{-2})$ , and sample from the distribution  $\hat{g}_k\sim \mathrm{Gumbel}(\log p(Y_2 = k|\mathcal{X},\mathcal{Y}_{-2}),1)$  truncated within the range  $(-\infty ,\hat{g}_{k^{*}})$ . The resulting samples  $[\hat{g}_1,\dots ,\hat{g}_{|V|}]$  are from the posterior  $p(\mathbf{G}_2^{\mathcal{Y}}|\mathcal{X},\mathcal{Y})$ . We describe these steps in more detail in Algorithm 1.
2. Action. In this step, we replace a phrase  $x_{3}$  in the source sequence with a substitute phrase  $\hat{x}_{3}$ . While any replacement leads to a valid counterfactual, we propose to sample  $\hat{x}_{3}$  as
$$
\hat {x} _ {3} \sim p (X _ {3} | \mathcal {X} _ {- 3}),
$$
where  $p(X_{3}|\mathcal{X}_{-3})$  is given by a trained masked language model. By sampling from a distribution
conditioned on the remaining phrases in  $\mathcal{X}$ , we sample a realistic replacement word for  $X_{3}$ . In Figure 2, we sample  $\hat{x}_3 = \text{'apple'}$  in place of  $x_{3} = \text{'house'}$ .
3. Prediction. Given the posterior samples  $[\hat{g}_1,\dots ,\hat{g}_{|V|}]\sim p(\mathbf{G}_2^{\mathcal{Y}}|\mathcal{X},\mathcal{Y})$  and the substitute phrase  $\hat{x}_3$ , we can compute the counterfactual distribution of interest  $p(Y_{2,\{X_{3}\leftarrow \hat{x}_{3},\mathcal{X}_{-3}\leftarrow \mathbf{x}_{-3},\mathcal{Y}_{-2}\leftarrow \mathbf{y}_{-2}\}}|\mathcal{X},\mathcal{Y})$ , via the trained translation language model. We do so by computing:
$$
\hat {y} _ {2} = \underset {k = 1, \dots , | V |} {\arg \max } \log p (Y _ {2} = k | \hat {x} _ {3}, \mathcal {X} _ {- 3}, \mathcal {Y} _ {- 2}) + \hat {g} _ {k}. \tag {1}
$$
The sample  $\hat{y}_2$  from the counterfactual distribution is based on the direct effect of  $X_3$  on  $Y_2$ . We remark that the causal model we consider was first introduced by Oberst and Sontag (2019) and called the Gumbel-Max Structural Causal Model. Our insight here is that counterfactuals from this model can be used as an effective data augmentation method for machine translation.
Given the above procedure to replace phrases, we propose a new data augmentation method, shown in Algorithm 2. The algorithm takes an input pair of sequences  $(\mathcal{X},\mathcal{Y})$  and loops through every phrase  $X_{i}\in \mathcal{X}$ . At each iteration with probability  $c$  it replaces the phrase pair  $(x_{i},y_{j})$  with  $(\hat{x}_i,\hat{y}_j)$ .
3.3 Training Language Models
We introduce a special [MASK] token (Devlin et al., 2018) to represent a removed phrase for parameterizing both  $p(X_{i}|\mathcal{X}_{-i})$  and  $p(Y_{j}|\mathcal{X},\mathcal{Y}_{-j})$  as:
Output:A new pair  $(\mathcal{X},\mathcal{Y})$
$$
\hat {\mathcal {X}}, \hat {\mathcal {Y}} = \mathcal {X}, \mathcal {Y}
$$
$$
p _ {\theta_ {1}} \left(X _ {i} \mid X _ {1}, \dots , X _ {i} = [ \mathrm {M A S K} ], \dots , X _ {| X |}\right) \tag {2}
$$
$$
p _ {\theta_ {2}} \left(Y _ {j} \mid \mathcal {X}, Y _ {1}, \dots , Y _ {j} = [ \mathrm {M A S K} ], \dots , Y _ {| Y |}\right). \tag {3}
$$
Eq. 2 only requires monolingual datasets, which are abundant. On the other hand, Eq. 3 requires parallel corpora to train. We parameterize Eq. 3 using a variant of the translation language model (Lample and Conneau, 2019). The main difference is that only phrases in target sequences are masked, whereas Lample and Conneau (2019) mask both source and target tokens, with the goal of learning bilingual relations. Another difference is that a phrase with consecutive tokens is masked, while masked tokens in Lample and Conneau (2019) are not necessarily consecutive.
To better tackle unknown and rare tokens, we adopt BPE to segment phrases into tokens. As the number of tokens is undetermined during generation, we use a sequence-to-sequence Transformer model (Vaswani et al., 2017) to encode inputs and decode tokens one by one until a special end-of-sequence symbol is encountered.
More specifically, given a sequence of  $N$  tokens  $(t_1,\dots,t_N)$ , the sequence contains a special [MASK] token signifying a masked phrase.
Each token  $t_i$  is first projected into its embedding  $\mathbf{e}_{t_i}$ , which is a sum of its token embedding, position embedding, and language embedding, inspired by XLM (Lample and Conneau, 2019). Then, a Transformer encoder is applied to encode the tokens into their hidden representations  $\mathbf{H} \in \mathbb{R}^{N \times o}$  (where  $o$  denotes the hidden size), i.e.  $\mathbf{H} = \mathrm{Encoder}(\mathbf{e}_{t_1}, \dots, \mathbf{e}_{t_N})$ . The hidden representation of [MASK],  $\mathbf{h}_{[\mathrm{MASK}]} \in \mathbb{R}^o$ , is fed into a Transformer decoder to predict the tokens of the masked phrase.
We learn our models  $p_{\theta_1}(X_i|\mathcal{X}_{-i})$ $p_{\theta_2}(Y_j|\mathcal{X},\mathcal{Y}_{-j})$  by maximizing the following objectives:
$$
\mathbb {E} _ {\mathcal {X} \sim \mathcal {D}} \left[ \mathbb {E} _ {i \sim \operatorname {U n i f o r m} (1, \dots , | \mathcal {X} |)} \left[ \log p _ {\theta_ {1}} \left(X _ {i} \mid \mathcal {X} _ {- i}\right) \right] \right]
$$
$$
\mathbb {E} _ {(\mathcal {X}, \mathcal {Y}) \sim \mathcal {S}} \left[ \mathbb {E} _ {j \sim \operatorname {U n i f o r m} (1, \dots , | \mathcal {Y} |)} [ \log p _ {\theta_ {2}} (Y _ {j} | \mathcal {X}, \mathcal {Y} _ {- j}) ] \right].
$$
Here  $\mathcal{D}$  is a monolingual dataset and  $S$  is a parallel corpus.
4.1 Data Augmentation for NMT
We categorize previous work on data augmentation for NMT into two classes, word replacement and backtranslation.
Word replacement. WordDropout (Sennrich et al., 2016) randomly zeros out word embeddings in order to introduce noises. BPEDropout (Provilkov et al., 2020) stochastically corrupts the segmentation procedure of BPE, leading to different subword segmentations with the same BPE vocabulary. RAML (Norouzi et al., 2016) applies a reward-augmented maximum likelihood objective, which essentially augments target sequences with sequences sampled based on metrics, such as edit distance and BLEU score (Wang et al., 2018). SwitchOut (Wang et al., 2018) extends RAML, augmenting both source and target sequences by randomly replacing words with noisy words sampled from a uniform distribution. These works do not take context and alignment into account. TDA (Fadaee et al., 2017) first uses two uni-directional language models to replace a word in the source sequence, before replacing the corresponding word based on a bilingual lexicon. TDA does not consider contexts in target sequences and relies on a high-quality bilingual lexicon. SCDA (Gao et al., 2019) uses a soft augmentation approach, where
Dataset	# Sequences	# Words	# Chars
News Commentary	0.46M	10.05M	63.96M
News crawl 2010	6.8M	0.14B	0.83B
the one-hot representation of a word is replaced by a soft distribution of words given by a language model. DADA (Cheng et al., 2019) uses gradient information to generate adversarial sequences for more robust NMT. AdvAug (Cheng et al., 2020) extends DADA, where embeddings of virtual sequences are sampled from an adversarial distribution for augmentation. SCDA and AdvAug ignore the alignment information, thereby breaking the symmetry of source and target sequences. While DADA takes both context and alignment into account, it replaces multiple words in source and target sequences simultaneously, which risks generating unnatural sequences. In this paper, we utilize both alignment and contextual information to sequentially replace aligned phrases for better performance.
Backtranslation. The idea of backtranslation dates back to statistical machine translation (Goutte et al., 2009; Bojar and Tamchyna, 2011). Senrich et al. (2016) use backtranslation, where monolingual sequences in the target language are translated into the source language, and obtain substantial improvements on the WMT and IWSLT tasks. Currey et al. (2017) apply backtranslation to low-resource languages, finding that even low-quality translations due to limited parallel corpora are beneficial. He et al. (2016) propose a dual learning framework, where the primal task (source-to-target translation) and the dual task (target-to-source translation) teach each other through a reinforcement learning process until convergence. Edunov et al. (2018) scale backtranslation to millions of monolingual data and obtain state-of-the-art performance on WMT'14 English  $\rightarrow$  German. Xia et al. (2019) use a two-step pivoting method for improving back-translation on low-resource languages. We show that TCWR can be used together with backtranslation and obtain further improvements.
We now describe the improvements with the data augmentation based on TCWR.
Table 1: The statistics of the monolingual datasets.
Dataset	# Train	# Dev	# Test
WMT'18 En-Tr	206K	3007	3000
WMT'17 En-De	5.85M	2,999	3,004
IWSLT'15 En-Vi	133K	1,553	1,268
WMT'19 Robust En-Fr	36,058	852	1,401
Europarl-v7 En-Fr	2M	-	-
Table 2: The statistics of the parallel corpora.
5.1 Language Model Details