SlowGuess/NAACL_2022_OCR huggingface dataset:
On Synthetic Data for Back Translation*
Jiahao Xu $^{1}$ , Yubin Ruan $^{2}$ , Wei Bi $^{2}$ , Guoping Huang $^{2}$ , Shuming Shi $^{2}$ , Lihui Chen $^{1}$ , Lemao Liu $^{2}$
$^{1}$ Nanyang Technological University,  $^{2}$ Tencent AI Lab
1{jiahao004@e.,elhchen@}ntu.edu.sg
$^{2}$  {brantruan, victoriabi, donkeyhuang, shumingshi, redmondliu} @tencent.com
Back translation (BT) is one of the most significant technologies in NMT research fields. Existing attempts on BT share a common characteristic: they employ either beam search or random sampling to generate synthetic data with a backward model but seldom work studies the role of synthetic data in the performance of BT. This motivates us to ask a fundamental question: what kind of synthetic data contributes to BT performance? Through both theoretical and empirical studies, we identify two key factors on synthetic data controlling the back-translation NMT performance, which are quality and importance. Furthermore, based on our findings, we propose a simple yet effective method to generate synthetic data to better trade off both factors so as to yield a better performance for BT. We run extensive experiments on WMT14 DE-EN, EN-DE, and RU-EN benchmark tasks. By employing our proposed method to generate synthetic data, our BT model significantly outperforms the standard BT baselines (i.e., beam and sampling based methods for data generation), which proves the effectiveness of our proposed methods.
Since the birth of neural machine translation (NMT) (Bahdanau et al., 2014; Sutskever et al., 2014) back translation (BT) (Sennrich et al., 2016a) has quickly become one of the most significant technologies in natural language processing (NLP) research field. This is because 1) it provides a simple yet effective approach to advance the supervised NMT by leveraging monolingual data (Edunov et al., 2018) and it also serves as a key learning objective in unsupervised NMT (Artetxe et al., 2018; Lample et al., 2018); 2) back translation even plays a significant role in other NLP re
search fields beyond translation such as paraphrasing (Mallinson et al., 2017) and style transfer (Prabhumoye et al., 2018; Zhang et al., 2018).
Back translation consists of two steps, namely synthetic corpus generation with a backward model and parameter optimization for the forward model. Various contributions have been made on improving back translation, for instance, iterative back translation (Hoang et al., 2018), tagged back translation (Caswell et al., 2019), confidence weighting (Wang et al., 2019), data diversification (Nguyen et al., 2020). Although these efforts differ in some aspects, all of them share a common characteristic: they employ a default way to generate synthetic data in the first step of BT which is either beam search or random sampling with a backward model. Seldom work studies the consequences of synthetic corpus to back translation and hence it is unclear how synthetic data influences the final performance of BT.
The early study empirically suggests the quality of the synthetic corpus is vital for BT performance (Sennrich et al., 2016a). However, recent studies illustrate better test performance can be achieved by low quality synthetic corpus (Edunov et al., 2018). This contradictory observation indicates the quality of synthetic data is not the only element that affects the BT performance. Hence, this fact naturally raises a fundamental question: what kind of synthetic data contributes to back translation performance?
In this paper, we attempt to take a step forward toward the above fundamental question. To this end, we start from a critical objective in semi-supervised learning, which is defined by the marginal distribution of a target language. Then we derive an approximate lower bound of the objective function, which is closely related to the objective of back translation. Corresponding to this lower bound, we theoretically find two related elements for maximizing such a lower bound: quality of syn
*This work was done during the internship of the first author at Tencent AI Lab. The code is available at https://github.com/Jiahao004/Data-for-BT
Human Language Technologies, pages 419 - 430
thetic bilingual data and importance weight of its source. Since both elements are mutually exclusive to some extent, it may induce contradictory observation if one judges the BT performance according to a single element. In addition, such a theoretical explanation is supported by our empirical experiments. Furthermore, based on our findings, we propose a new heuristic approach to generate synthetic data whose both elements are better balanced so as to yield improvements over both sampling and beam search based methods. Extensive experiments on three WMT14 tasks show that our BT consistently outperforms the standard sampling and beam search based baselines by a significant margin.
Our contributions are three folds:
1. We point out that importance weight and quality of synthetic candidates are two key factors that affect the NMT performance.
2. We propose a simple yet effective method for synthetic corpus generation, which could better balance the quality and importance of synthetic data.
3. Our experiments prove the effectiveness of the aforementioned strategy, it outperforms beam or sampling decoding methods on three benchmark tasks.
2 Revisiting Back Translation
NMT builds a probabilistic model  $p(y|x;\theta)$  with neural networks parameterized by  $\theta$ , which is used to translate a sentence  $x$  in source language  $\mathcal{X}$  to a sentence  $y$  in target language  $\mathcal{Y}$ . The standard wisdom to train the model is to minimize the following objective function over a given bilingual corpus  $\mathcal{B} = \{(x_i, y_i)\}$ :
$$
\ell (\mathcal {B}; \theta) = \sum_ {\left(x _ {i}, y _ {i}\right) \in \mathcal {B}} \log p \left(y _ {i} \mid x _ {i}; \theta\right) \tag {1}
$$
Recently Senrich et al. (2016a) propose a remarkable method called Back Translation (BT) to improve NMT by using a monolingual corpus  $\mathcal{M}$  in target language  $\mathcal{Y}$  besides  $\mathcal{B}$  and back translation becomes one of the most successful techniques in NMT (Fadaee and Monz, 2018; Edunov et al., 2018). At a high level, back translation can be considered as a semi-supervised method because it leverages both labeled and unlabeled data. Suppose  $p(x|y;\pi)$  is the backward translation model
whose parameter  $\pi$  is optimized over  $\mathcal{B}$ , the key idea of back translation can be summarized as the following two steps:
- Synthetic Corpus Generation: It firstly back-translates each target sentence  $y \in \mathcal{M}$  to  $\hat{x}$  obtain a synthetic bilingual corpus  $\{(\hat{x},y) \mid y \in \mathcal{M}\}$  by  $p(x|y;\pi)$ .
- Parameter Optimization: It combines both authentic corpus  $\mathcal{B}$  and the synthetic corpus and then optimizes the parameter  $\theta$  by minimizing the loss
$$
\ell (\mathcal {B}; \theta) + \sum_ {y \in \mathcal {M}} \log p (y | \hat {x}; \theta) \tag {2}
$$
To make BT more efficient, the standard configuration is widely adopted: each sentence  $y$  is required to generate a single source  $\hat{x}$  and both two steps are performed for a single pass. We follow this standard in this paper for generality but our idea in this paper is straightforward to apply to other configurations such as (Graça et al., 2019; Hoang et al., 2018; Nguyen et al., 2020).
In the first step, there are two main strategies to generate the synthetic corpus, i.e., deterministically decoding and randomly sampling with  $p(x|y; \pi)$ . The first strategy aims to search the best candidate as follows,
$$
\hat {x} ^ {b} = \arg \max  p (\hat {x} | y; \pi) \tag {3}
$$
The above optimization is achieved by the beam search decoding, which can be regarded as a degenerated shortest path problem with respect to the  $\log p(\hat{x} |y;\pi)$  with limited routing attempts. The alternative strategy is random sampling: it randomly samples a token with respect to the distribution estimated by a back-translation model at each decoding step. Such a process can be modelled by,
$$
\hat {x} ^ {s} = \operatorname {r a n d} \left\{p (\hat {x} | y; \pi) \right\} \tag {4}
$$
Research Question Prior work points out (Sennrich et al., 2016a) that the synthetic corpus with high quality is beneficial to the final performance of back translation. However, the recent studies (Edunov et al., 2018) find that NMT models with unsatisfactory BLEU score corpus, for instance, the corpus generated by sampling based strategy, also establish the state-of-the-art (SOTA) achievement among back-translation NMT models.
This contradictory fact indicates that the quality of synthetic corpus is not the sole element for back translation. This motivates us to study a fundamental question for back translation: what kind of synthetic corpus is beneficial to back translation?
3 Understanding Synthetic Data by Two Factors
To answer the fundamental question presented in the previous section, we first start from the marginal likelihood objective defined on the target language  $\mathcal{V}$ , and then we theoretically explain two factors (i.e., quality and importance) that are highly related to the training objective of back translation. Finally, we empirically explain why synthetic corpus with low quality may lead to better performance than synthetic corpus with high quality by measuring both factors.
3.1 Theoretical Explanation
Maximizing marginal likelihood is an important principle to leverage unlabeled data. Therefore, we rethink back translation from this principle because it makes use of target monolingual corpus  $\mathcal{M}$ . For each  $y\in \mathcal{M}$ , the marginal likelihood objective can be derived by the Bayesian Equation (5), Jansen Inequality (6), and importance sampling (7) as follows:
$$
\begin{array}{l} \log p (y; \theta) = \log \sum_ {x} p (x) p (y | x; \theta) (5) \\ \geq \sum_ {x} p (x) \log p (y | x; \theta) (6) \\ = \sum_ {x} p (x | y) \frac {p (x)}{p (x | y)} \log p (y | x; \theta) \\ = \mathbb {E} _ {\hat {x} \sim p (\cdot | y)} \left\{\frac {p (\hat {x})}{p (\hat {x} | y)} \log p (y | \hat {x}; \theta) \right\} \\ \approx \frac {p (\hat {x})}{p (\hat {x} | y)} \log p (y | \hat {x}; \theta) (7) \\ \end{array}
$$
where  $p(x)$  is a language model on source language  $\mathcal{X}$ ,  $p(x|y)$  is a backward translation model from  $\mathcal{V}$  to  $\mathcal{X}$  which serves as the proposal distribution for importance sampling, and  $\hat{x}$  is sampled from  $p(x|y)$ . If  $p(x|y)$  is set as the backward model  $p(x|y; \pi)$  optimized on  $\mathcal{B}$ , the last term in Equation 7 is the same as the second term in BT loss (i.e.,  $\log p(y|\hat{x})$  in Eq. 2), and the unique difference is the multiplicative term called importance weight:
$$
\operatorname {I m p} (\hat {x}; y) = \frac {p (\hat {x})}{p (\hat {x} | y)} \tag {8}
$$
The denominator is the candidate conditional probability to target, and the numerator is the candidate distribution on source language distribution. Since  $\mathrm{Imp}(\hat{x};y)$  is constant with respect to the parameter  $\theta$ , maximizing  $\log p(y|\hat{x};\theta)$  in BT loss implicitly maximizes  $\mathrm{Imp}(\hat{x};y)\log p(y|\hat{x})$ , which indicates that back translation aims to implicitly maximize the marginal likelihood objective. More importantly, according to Equation 7 we can find that the following two factors are critical to influence the marginal likelihood  $\log p(y;\theta)$ :
- Factor 1: The quality of  $\hat{x}$  as a translation of  $y$  corresponding to the  $\log p(y|\hat{x};\theta)$  in Eq. 7.
- Factor 2: The importance of  $\hat{x}$  as a translation of  $y$  corresponding to  $\operatorname{Imp}(\hat{x}; y)$  in Eq. 7.
Theoretically, if  $\hat{x}$  is of higher quality and contains more semantic information in  $y$ ,  $p(y|\hat{x};\theta)$  would be higher and thus it would lead to a higher  $\log p(y;\theta)$ , which is well acknowledged by prior work (Sennrich et al., 2016a; Wang et al., 2019). In particular, if  $\hat{x}$  is with higher importance weight, maximizing  $\log p(y|\hat{x};\theta)$  is more helpful to maximize  $\log p(y;\theta)$ . On the contrary, if  $\mathrm{Imp}(\hat{x};y)$  is very small, it needs to avoid such a sample  $\hat{x}$  from  $p(x|y)$ , which is essentially the rejection control strategy in importance sampling theory (Liu et al., 1998; Liu and Liu, 2001).
Unfortunately, in practice, both factors are mutually exclusive to some extent: if  $\hat{x}$  is with high quality,  $p(\hat{x} | y; \theta)$  would be higher as well leading to lower importance weight. This fact can explain the contradictory observation in Sec 2 that BT with high-quality synthetic data sometimes leads to better testing performance, while it may deliver worse performance at other times, which will be later justified in Sec 3.2.
Estimating Two Factors To measure the quality of  $\hat{x}$  for each  $y$ , it is natural to use the evaluation metric such as BLEU if the reference translation  $x$  of  $y$  is available. Otherwise, as a surrogate, we use the log likelihood  $\log p(\hat{x} | y; \pi)$  of the backward translation model  $\pi$  which is trained on the authentic data  $\mathcal{B}$ . Similarly, in order to estimate the importance of  $\hat{x}$ , we train an additional language model  $p(x; \omega)$  with GPT (Radford et al., 2018) on a large monolingual corpus for  $\mathcal{X}$ . In this way, the importance weight is estimated by
$$
\operatorname {I m p} (\hat {x}) \approx \frac {p (\hat {x} ; \omega)}{p (\hat {x} | y ; \pi)}
$$
Systems	BLEU(x̂)	log p(x̂|y, π)	Imp.	Test BLEU
beam	27.20	-15.65	-95.13	32.7
sampling	7.70	-157.62	-41.86	34.1
beam*	18.50	-26.66	-95.07	31.6
* The checkpoint of the backward model for generating synthetic corpus are only trained for 1 epoch. However, its log  $p(\hat{x} | y, \pi)$  is still measured by a standard backward model  $\pi$ .
Table 1: Testing BLEU (on test set), quality (measured by both BLEU and  $\log p(\hat{x} |y;\pi)$ ) and importance (Imp.) estimation of synthetic data (on dev set) with beam search or random sampling on WMT14 DE-EN task.
Systems	BLEU(x̂)	log p(x̂|y, π)	Imp.	Test BLEU
en-de(en)_beam	31.90	-15.29	-91.07	29.7
en-de(en)_sampling	10.90	-139.71	-46.88	30.0
ru-en(ru)_beam	33.10	-15.49	-89.71	35.9
ru-en(ru)_sampling	9.50	-155.82	-47.47	35.6
Table 2: Testing BLEU (on test set), quality (measured by both BLEU and  $\log p(\hat{x} |y;\pi)$ ) and importance (Imp.) estimation of synthetic data (on development set) with beam search or random sampling on WMT14 EN-DE and RU-EN tasks.
3.2 Empirical Justification
In this subsection, we aim to justify the following statements: 1) encouraging the quality of synthetic corpus may to some extent hurt the performance of BT due to the decrease of importance; 2) judging the testing performance in terms of quality only may be dangerous while it would be meaningful to judge the testing performance by taking into account both factors rather than either factor. To this end, we run some quick experiments on WMT14 datasets whose settings will be shown in Sec 5 later.
We set up two back translation systems with two different options (i.e., beam search and sampling) to generate synthetic corpus by using the best checkpoint of  $p(\hat{x} | y; \pi)$  tuned on the development set. Both beam search and sampling based BT systems are denoted by beam and sampling. In addition, we pick another checkpoint of  $p(\hat{x} | y; \pi)$  which is trained for only 1 epoch, and we use this weak checkpoint to set up another beam search based BT system, which is denoted as beam*. Table 1 shows BLEU on test dataset, the quality and importance on the development set according to three systems on WMT14 DE-EN task.
In Table 1, beam is better than sampling in the quality of synthetic corpus but its testing performance is worse. This is meaningful because the former relies on the synthetic corpus with lower importance weight according to our theoretical ex
planation. In addition, when comparing beam with beam*, we can find that beam delivers better testing performance because its quality is better meanwhile its importance weight is almost similar to that of beam*. Table 2 consistently demonstrates that it is meaningless to take into account quality only when evaluating BT. These facts justify our statements and provide an answer to the fundamental question in section 2.
4 Improving Synthetic Data for BT
As shown in the previous section, both importance and quality of synthetic corpus are beneficial to the overall testing performance of back translation. It is a natural idea to promote both factors when generating synthetic corpus such that running BT on such corpus leads to better testing performance. However, this is difficult because both factors are mutually exclusive as discussed in Section 3. In this section, we instead propose two methods (namely data manipulation and gamma score) to trade off both factors in the hope to yield better BT performance.
4.1 Data Manipulation
Since the synthetic data in sampling based BT is of high importance yet low quality whereas the case for the synthetic data in beam search based BT is opposite, we propose a data manipulation method to trade off importance and quality by combining both synthetic datasets. Through balancing the ratio between beam and sampling based synthetic corpora, we expect to find an optimized beam/sampling ratio to further improve NMT model performance.
Specifically, we randomly shuffle  $\mathcal{M}$  and divide it into two parts with the first part accounting for  $\gamma$ $(0 < \gamma < 1)$ ; then we generate translations for the first part with beam search while generating translations for the second part with sampling. Formally, we use the following corpus  $\mathcal{M}^c$  as the synthetic corpus for BT:
$$
\begin{array}{l} \mathcal {M} ^ {c} = \{(\hat {x} _ {i} ^ {b}, y _ {i}) _ {i = 0} ^ {k} \} \cup \{(\hat {x} _ {j} ^ {s}, y _ {j}) _ {j = k} ^ {| \mathcal {M} |} \} \\ k = \lfloor \gamma | \mathcal {M} | \rfloor \\ \end{array}
$$
Where  $\hat{x}^b$  denotes a translation of  $y$  generated by  $p(x|y;\pi)$  with beam search and  $\hat{x}^s$  is a translation with sampling,  $|\cdot |$  means the size of the corpus, and  $\gamma$  is the combination ratio for beam and sampling synthetic corpora. By tuning  $\gamma$  here, one can
modify the weightage for the number of beam and sampling sentences, to improve back-translation performance by training models on a combined synthetic corpus.
Although this method is easy to implement, its limitation is obvious. Since each  $\hat{x}$  is either from beam search or from sampling, the quality of  $\mathcal{M}^c$  is generally worse than that of beam search and its importance weight is generally worse than that of sampling. Consequently, we propose an alternative method in the next part of this section.
4.2 Gamma Score
The key idea to the alternative method is that it employs a score that balances both quality and importance to generate a translation  $\hat{x}$  for each  $y\in \mathcal{M}$ . A natural choice of such a score is defined by the interpolation score as follows:
$$
\gamma \log \mathrm {I m p} (\hat {x}; \omega , \pi) + (1 - \gamma) \log p (\hat {x} | y; \pi)
$$
where  $\gamma$  is used to trade off both factors as in corpus manipulation. With the help of this score, one may optimize the  $\hat{x}$  by beam search whose interpolation score is the best among all possible translations of  $y \in \mathcal{M}$ . Unfortunately, such an implementation leads to limited performance in our preliminary experiments, due to two major challenges.
On one hand, the estimations of quality and importance weight of  $\hat{x}$  are not well calibrated, and in particular, quality and importance are mutually exclusive as mentioned before. As a result, beam search with the interpolation score over the exponential space can not guarantee a desirable translation  $\hat{x}$  for each  $y$ . On the other hand, quality and importance weight of  $\hat{x}$  are not at the same scale for different  $y$ , it is difficult to balance both factors with a fixed  $\gamma$  in the interpolation score for different  $y$ .
To alleviate these issues, we propose a simple method as follows. Specifically, firstly, instead of beam search with the interpolation score, we simply utilize the backward translation  $p(x|y;\pi)$  to randomly sample a set of candidate translations which is denoted by  $A(y) = \{\hat{x}_i\}_i^N$  ( $N = 50$  in this paper as it works well). Then we pick a  $\hat{x}_j$  among  $A(y)$  according to the balancing score. Secondly, for each  $\hat{x}$ , we normalize the log values of importance and quality of each candidate by its
sequence length, then normalize these values with respect to all  $N$  candidates as follows:
$$
\tilde {\mathcal {F}} (\hat {x} _ {i}) = \frac {\log \left(\mathcal {F} (\hat {x} _ {i})\right) / \operatorname {l e n} (\hat {x} _ {i}) - \mu_ {\mathcal {F}}}{\sigma_ {\mathcal {F}}} \tag {9}
$$
where  $\mathcal{F}$  is either importance weight or quality estimations, and  $\mu_{\mathcal{F}} = \frac{1}{N}\sum_{i}\log \mathcal{F}(\hat{x}_i)$  and  $\sigma_{\mathcal{F}} = \frac{\sum_{i}(\log\mathcal{F}(\hat{x}_i) - \mu_{\mathcal{F}})^2}{N - 1}$  are mean and variance of  $N$  sampled candidates with length normalized. Finally, the Gamma score is defined on the normalized values of importance and quality as follows:
$$
\Gamma (\hat {x} _ {i}; \omega , \pi) =
$$
$$
\exp \left(\gamma \operatorname {I m p} (\hat {x} _ {i}; \omega , \pi) + (1 - \gamma) \tilde {p} (\hat {x} _ {i} | y, \pi)\right)
$$
$$
\overline {{\sum_ {j} \exp \left(\gamma \operatorname {I m p} (\hat {x} _ {j} ; \omega , \pi) + (1 - \gamma) \tilde {p} (\hat {x} _ {j} | y , \pi)\right)}} \tag {10}
$$
where  $\tilde{\mathrm{Imp}}$  and  $\tilde{p}$  are the normalized log value of importance weight and backward translation model  $p(\hat{x} |y,\pi)$  as defined in Equation 9.
Once the gamma score in Equation 10 is computed, there are two methods to select  $\hat{x}$  from  $A(y)$ , which are deterministic and stochastic methods. For deterministic selection, we simply select the candidates with maximum gamma score among  $N$  translation candidates; and for sampling, we sample a candidate according to its gamma score distribution. These two methods are called gamma selection and gamma sampling in our experiments.
We run all the experiments by using fairseq (Ott et al., 2019) framework. For dataset settings, since datasets WMT14 EN-DE and DE-EN are widely used (Li et al., 2019b; Zhu et al., 2020; Li et al., 2020; Fan et al., 2021; Le et al., 2021), we follow both standard benchmarks and additionally we employ WMT14 RU-EN as the third dataset to validate the effectiveness of the proposed methods. For back translation experiment, we use an equal scale monolingual corpus randomly sampled from Newscrawl 2020 (Barrault et al., 2019) comprising 4.5 million monolingual sentences for DE-EN language pair and 2.5 million for RU-EN direction, thus total 9 million sentences for DE-EN pair and 5 million for RU-EN direction are used. We tokenize the parallel corpus using Mose tokenizer (Koehn et al., 2007), and learn a source and target shared Byte-Pair-Encoding (BPE) (Sennrich et al., 2016b)
${}^{1}N$  -best decoding strategy with  $p\left( {x \mid  y;\pi }\right)$  to generate  $N$  candidates may be another solution which remains as future work.
Systems	DE-EN
w/o bitext	w bitext
Transformer	-	32.1
Beam BT	27.6	32.7
Sampling BT	29.2	34.1
DM	31.3	34.2
DM means the data manipulation method.
with 32K types. We develop on newstest2013 and report the results on newstest2014.