Venue: NAACL 2022

# On Synthetic Data for Back Translation+
Footnote †: This work was done during the internship of the first author at Tencent AI Lab. The code is available at [https://github.com/Jiahao004/Data-for-BT](https://github.com/Jiahao004/Data-for-BT)

 Jiahao Xu\({}^{1}\), Yubin Ruan\({}^{2}\), Wei Bi\({}^{2}\), Guoping Huang\({}^{2}\), Shuming Shi\({}^{2}\), Lihui Chen\({}^{1}\), Lemao Liu\({}^{2}\)

###### Abstract

Back translation (BT) is one of the most significant technologies in NMT research fields. Existing attempts on BT share a common characteristic: they employ either beam search or random sampling to generate synthetic data with a backward model but seldom work studies the role of synthetic data in the performance of BT. This motivates us to ask a fundamental question: _what kind of synthetic data contributes to BT performance?_ Through both theoretical and empirical studies, we identify two key factors on synthetic data controlling the back-translation NMT performance, which are quality and importance. Furthermore, based on our findings, we propose a simple yet effective method to generate synthetic data to better trade off both factors so as to yield a better performance for BT. We run extensive experiments on WMT14 DE-EN, EN-DE, and RU-EN benchmark tasks. By employing our proposed method to generate synthetic data, our BT model significantly outperforms the standard BT baselines (i.e., beam and sampling based methods for data generation), which proves the effectiveness of our proposed methods.

languageresource

thetic bilingual data and importance weight of its source. Since both elements are mutually exclusive to some extent, it may induce contradictory observation if one judges the BT performance according to a single element. In addition, such a theoretical explanation is supported by our empirical experiments. Furthermore, based on our findings, we propose a new heuristic approach to generate synthetic data whose both elements are better balanced so as to yield improvements over both sampling and beam search based methods. Extensive experiments on three WMT14 tasks show that our BT consistently outperforms the standard sampling and beam search based baselines by a significant margin.

Our contributions are three folds:

1. We point out that importance weight and quality of synthetic candidates are two key factors that affect the NMT performance.
2. We propose a simple yet effective method for synthetic corpus generation, which could better balance the quality and importance of synthetic data.
3. Our experiments prove the effectiveness of the aforementioned strategy, it outperforms beam or sampling decoding methods on three benchmark tasks.

## 2 Revisiting Back Translation

NMT builds a probabilistic model \(p(y|x;\theta)\) with neural networks parameterized by \(\theta\), which is used to translate a sentence \(x\) in source language \({\cal X}\) to a sentence \(y\) in target language \({\cal Y}\). The standard wisdom to train the model is to minimize the following objective function over a given bilingual corpus \({\cal B}=\{(x_{i},y_{i})\}\):

\[\ell({\cal B};\theta)=\sum_{(x_{i},y_{i})\in{\cal B}}\log p(y_{i}| x_{i};\theta) \tag{1}\]

Recently Sennrich et al. (2016a) propose a remarkable method called **Back Translation (BT)** to improve NMT by using a monolingual corpus \({\cal M}\) in target language \({\cal Y}\) besides \({\cal B}\) and back translation becomes one of the most successful techniques in NMT Fadaee and Monz (2018); Edunov et al. (2018). At a high level, back translation can be considered as a semi-supervised method because it leverages both labeled and unlabeled data. Suppose \(p(x|y;\pi)\) is the backward translation model whose parameter \(\pi\) is optimized over \({\cal B}\), the key idea of back translation can be summarized as the following two steps:

* **Synthetic Corpus Generation:** It firstly back-translates each target sentence \(y\in{\cal M}\) to \(\hat{x}\) obtain a synthetic bilingual corpus \(\{(\hat{x},y)~{}|~{}y\in{\cal M}\}\) by \(p(x|y;\pi)\).
* **Parameter Optimization:** It combines both authentic corpus \({\cal B}\) and the synthetic corpus and then optimizes the parameter \(\theta\) by minimizing the loss \[\ell({\cal B};\theta)+\sum_{y\in{\cal M}}\log p(y|\hat{x};\theta)\] (2) To make BT more efficient, the standard configuration is widely adopted: each sentence \(y\) is required to generate a single source \(\hat{x}\) and both two steps are performed for a single pass. We follow this standard in this paper for generality but our idea in this paper is straightforward to apply to other configurations such as Graca et al. (2019); Hoang et al. (2018); Nguyen et al. (2020).

In the first step, there are two main strategies to generate the synthetic corpus, i.e., deterministically decoding and randomly sampling with \(p(x|y;\pi)\). The first strategy aims to search the best candidate as follows,

\[\hat{x}^{b}=\arg\max p(\hat{x}|y;\pi) \tag{3}\]

The above optimization is achieved by the beam search decoding, which can be regarded as a degenerated shortest path problem with respect to the \(\log p(\hat{x}|y;\pi)\) with limited routing attempts. The alternative strategy is random sampling: it randomly samples a token with respect to the distribution estimated by a back-translation model at each decoding step. Such a process can be modelled by,

\[\hat{x}^{s}=\texttt{rand}\{p(\hat{x}|y;\pi)\} \tag{4}\]

#### 2.5.1 Research Question

Prior work points out Sennrich et al. (2016a) that the synthetic corpus with high quality is beneficial to the final performance of back translation. However, the recent studies Edunov et al. (2018) find that NMT models with unsatisfactory BLEU score corpus, for instance, the corpus generated by sampling based strategy, also establish the state-of-the-art (SOTA) achievement among back-translation NMT models 

This contradictory fact indicates that the quality of synthetic corpus is not the sole element for back translation. This motivates us to study a fundamental question for back translation: _what kind of synthetic corpus is beneficial to back translation?_

## 3 Understanding Synthetic Data by Two Factors

To answer the fundamental question presented in the previous section, we first start from the marginal likelihood objective defined on the target language \({\cal Y}\), and then we theoretically explain two factors (i.e., quality and importance) that are highly related to the training objective of back translation. Finally, we empirically explain why synthetic corpus with low quality may lead to better performance than synthetic corpus with high quality by measuring both factors.

### Theoretical Explanation

Maximizing marginal likelihood is an important principle to leverage unlabeled data. Therefore, we rethink back translation from this principle because it makes use of target monolingual corpus \({\cal M}\). For each \(y\in{\cal M}\), the marginal likelihood objective can be derived by the Bayesian Equation (5), Jansen Inequality (6), and importance sampling (7) as follows:

\[\log p(y;\theta) =\log\sum_{x}p(x)p(y|x;\theta) \tag{5}\] \[\geq\sum_{x}p(x)\log p(y|x;\theta)\] (6) \[=\sum_{x}p(x|y)\frac{p(x)}{p(x|y)}\log p(y|x;\theta)\] \[=\mathbb{E}_{\hat{x}\sim p(\cdot|y)}\Big{\{}\frac{p(\hat{x})}{p( \hat{x}|y)}\log p(y|\hat{x};\theta)\Big{\}}\] \[\approx\frac{p(\hat{x})}{p(\hat{x}|y)}\log p(y|\hat{x};\theta) \tag{7}\]

where \(p(x)\) is a language model on source language \({\cal X}\), \(p(x|y)\) is a backward translation model from \({\cal Y}\) to \({\cal X}\) which serves as the proposal distribution for importance sampling, and \(\hat{x}\) is sampled from \(p(x|y)\). If \(p(x|y)\) is set as the backward model \(p(x|y;\pi)\) optimized on \({\cal B}\), the last term in Equation 7 is the same as the second term in BT loss (i.e., \(\log p(y|\hat{x})\) in Eq. 2), and the unique difference is the multiplicative term called _importance_ weight:

\[\mathrm{Imp}(\hat{x};y)=\frac{p(\hat{x})}{p(\hat{x}|y)} \tag{8}\]

The denominator is the candidate conditional probability to target, and the numerator is the candidate distribution on source language distribution. Since \(\mathrm{Imp}(\hat{x};y)\) is constant with respect to the parameter \(\theta\), maximizing \(\log p(y|\hat{x};\theta)\) in BT loss implicitly maximizes \(\mathrm{Imp}(\hat{x};y)\log p(y|\hat{x})\), which indicates that back translation aims to implicitly maximize the marginal likelihood objective. More importantly, according to Equation 7 we can find that the following two factors are critical to influence the marginal likelihood \(\log p(y;\theta)\):

* **Factor 1:** The quality of \(\hat{x}\) as a translation of \(y\) corresponding to the \(\log p(y|\hat{x};\theta)\) in Eq. 7.
* **Factor 2:** The importance of \(\hat{x}\) as a translation of \(y\) corresponding to \(\mathrm{Imp}(\hat{x};y)\) in Eq. 7.

Theoretically, if \(\hat{x}\) is of higher quality and contains more semantic information in \(y\), \(p(y|\hat{x};\theta)\) would be higher and thus it would lead to a higher \(\log p(y;\theta)\), which is well acknowledged by prior work Sennrich et al. (2016a); Wang et al. (2019). In particular, if \(\hat{x}\) is with higher importance weight, maximizing \(\log p(y|\hat{x};\theta)\) is more helpful to maximize \(\log p(y;\theta)\). On the contrary, if \(\mathrm{Imp}(\hat{x};y)\) is very small, it needs to avoid such a sample \(\hat{x}\) from \(p(x|y)\), which is essentially the rejection control strategy in importance sampling theory Liu et al. (1998); Liu and Liu (2001).

Unfortunately, in practice, both factors are mutually exclusive to some extent: if \(\hat{x}\) is with high quality, \(p(\hat{x}|y;\theta)\) would be higher as well leading to lower importance weight. This fact can explain the contradictory observation in Sec 2 that BT with high-quality synthetic data sometimes leads to better testing performance, while it may deliver worse performance at other times, which will be later justified in Sec 3.2.

Estimating Two FactorsTo measure the quality of \(\hat{x}\) for each \(y\), it is natural to use the evaluation metric such as BLEU if the reference translation \(x\) of \(y\) is available. Otherwise, as a surrogate, we use the log likelihood \(\log p(\hat{x}|y;\pi)\) of the backward translation model \(\pi\) which is trained on the authentic data \({\cal B}\). Similarly, in order to estimate the importance of \(\hat{x}\), we train an additional language model \(p(x;\omega)\) with GPT Radford et al. (2018) on a large monolingual corpus for \({\cal X}\). In this way, the importance weight is estimated by

\[\mathrm{Imp}(\hat{x})\approx\frac{p(\hat{x};\omega)}{p(\hat{x}|y;\pi)}\] 



### Empirical Justification

In this subsection, we aim to justify the following statements: 1) encouraging the quality of synthetic corpus may to some extent hurt the performance of BT due to the decrease of importance; 2) judging the testing performance in terms of quality only may be dangerous while it would be meaningful to judge the testing performance by taking into account both factors rather than either factor. To this end, we run some quick experiments on WMT14 datasets whose settings will be shown in Sec 5 later.

We set up two back translation systems with two different options (i.e., beam search and sampling) to generate synthetic corpus by using the best checkpoint of \(p(\hat{x}|y;\pi)\) tuned on the development set. Both beam search and sampling based BT systems are denoted by beam and sampling. In addition, we pick another checkpoint of \(p(\hat{x}|y;\pi)\) which is trained for only 1 epoch, and we use this weak checkpoint to set up another beam search based BT system, which is denoted as beam*. Table 1 shows BLEU on test dataset, the quality and importance on the development set according to three systems on WMT14 DE-EN task.

In Table 1, beam is better than sampling in the quality of synthetic corpus but its testing performance is worse. This is meaningful because the former relies on the synthetic corpus with lower importance weight according to our theoretical explanation. In addition, when comparing beam with beam*, we can find that beam delivers better testing performance because its quality is better meanwhile its importance weight is almost similar to that of beam*. Table 2 consistently demonstrates that it is meaningless to take into account quality only when evaluating BT. These facts justify our statements and provide an answer to the fundamental question in section 2.

## 4 Improving Synthetic Data for BT

As shown in the previous section, both importance and quality of synthetic corpus are beneficial to the overall testing performance of back translation. It is a natural idea to promote both factors when generating synthetic corpus such that running BT on such corpus leads to better testing performance. However, this is difficult because both factors are mutually exclusive as discussed in Section 3. In this section, we instead propose two methods (namely data manipulation and gamma score) to trade off both factors in the hope to yield better BT performance.

### Data Manipulation

Since the synthetic data in sampling based BT is of high importance yet low quality whereas the case for the synthetic data in beam search based BT is opposite, we propose a data manipulation method to trade off importance and quality by combining both synthetic datasets. Through balancing the ratio between beam and sampling based synthetic corpora, we expect to find an optimized beam/sampling ratio to further improve NMT model performance.

Specifically, we randomly shuffle \(\mathcal{M}\) and divide it into two parts with the first part accounting for \(\gamma\) (\(0<\gamma<1\)); then we generate translations for the first part with beam search while generating translations for the second part with sampling. Formally, we use the following corpus \(\mathcal{M}^{c}\) as the synthetic corpus for BT:

\[\begin{split}\mathcal{M}^{c}=\{(\hat{x}_{i}^{b},y_{i})_{i=0}^{k} \}\cup\{(\hat{x}_{j}^{s},y_{j})_{j=k}^{|\mathcal{M}|}\}\\ k=&\lfloor\gamma|\mathcal{M}|\rfloor\end{split}\]

Where \(\hat{x}^{b}\) denotes a translation of \(y\) generated by \(p(x|y;\pi)\) with beam search and \(\hat{x}^{s}\) is a translation with sampling, \(|\cdot|\) means the size of the corpus, and \(\gamma\) is the combination ratio for beam and sampling synthetic corpora. By tuning \(\gamma\) here, one can

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Systems** & **BLEU**(\(\hat{x}\)) & \(\log p(\hat{x}|y;\pi)\) & **Imp.** & **Test BLEU** \\ \hline beam & **27.20** & **-15.65** & -95.13 & 32.7 \\ sampling & 7.70 & -157.62 & **-41.86** & 34.1 \\ beam* & **18.50** & **-26.66** & -95.07 & 31.6 \\ \hline \multicolumn{5}{r}{* The checkpoint of the backward model for generating synthetic corpus are only trained for 1 epoch. However, its \(\log p(\hat{x}|y,\pi)\) is still measured by a standard backward model \(\pi\).} \\ \end{tabular}
\end{table}
Table 1: Testing BLEU (on test set), quality (measured by both BLEU and \(\log p(\hat{x}|y;\pi)\)) and importance (Imp.) estimation of synthetic data (on dev set) with beam search or random sampling on WMT14 DE-EN task.

\begin{table}
\begin{tabular}{l l r r r} \hline \hline
**Systems** & **BLEU**(\(\hat{x}\)) & \(\log p(\hat{x}|y;\pi)\) & **Imp.** & **Test BLEU** \\ \hline en-det(en),beam & **31.90** & **-15.29** & -91.07 & 29.7 \\ en-det(en),sampling & 10.90 & -139.71 & **-46.88** & 30.0 \\ ru-en(ru),beam & **33.10** & **-15.49** & -89.71 & 35.9 \\ ru-en(ru),sampling & 9.50 & -155.82 & **-47.47** & 35.6 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Testing BLEU (on test set), quality (measured by both BLEU and \(\log p(\hat{x}|y;\pi)\)) and importance (Imp.) estimation of synthetic data (on development set) with beam search or random sampling on WMT14 EN-DE and RU-EN tasks.



modify the weightage for the number of beam and sampling sentences, to improve back-translation performance by training models on a combined synthetic corpus.

Although this method is easy to implement, its limitation is obvious. Since each \(\hat{x}\) is either from beam search or from sampling, the quality of \(\mathcal{M}^{c}\) is generally worse than that of beam search and its importance weight is generally worse than that of sampling. Consequently, we propose an alternative method in the next part of this section.

### Gamma Score

The key idea to the alternative method is that it employs a score that balances both quality and importance to generate a translation \(\hat{x}\) for each \(y\in\mathcal{M}\). A natural choice of such a score is defined by the interpolation score as follows:

\[\gamma\log\mathrm{Imp}(\hat{x};\omega,\pi)+(1-\gamma)\log p(\hat{ x}|y;\pi)\]

where \(\gamma\) is used to trade off both factors as in corpus manipulation. With the help of this score, one may optimize the \(\hat{x}\) by beam search whose interpolation score is the best among all possible translations of \(y\in\mathcal{M}\). Unfortunately, such an implementation leads to limited performance in our preliminary experiments, due to two major challenges.

On one hand, the estimations of quality and importance weight of \(\hat{x}\) are not well calibrated, and in particular, quality and importance are mutually exclusive as mentioned before. As a result, beam search with the interpolation score over the exponential space can not guarantee a desirable translation \(\hat{x}\) for each \(y\). On the other hand, quality and importance weight of \(\hat{x}\) are not at the same scale for different \(y\), it is difficult to balance both factors with a fixed \(\gamma\) in the interpolation score for different \(y\).

To alleviate these issues, we propose a simple method as follows. Specifically, firstly, instead of beam search with the interpolation score, we simply utilize the backward translation \(p(x|y;\pi)\) to randomly sample a set of candidate translations which is denoted by \(A(y)=\{\hat{x}_{i}\}_{i}^{N}\) (\(N=50\) in this paper as it works well). 1 Then we pick a \(\hat{x}_{j}\) among \(A(y)\) according to the balancing score. Secondly, for each \(\hat{x}\), we normalize the log values of importance and quality of each candidate by its sequence length, then normalize these values with respect to all \(N\) candidates as follows:

Footnote 1: \(N\)-best decoding strategy with \(p(x|y;\pi)\) to generate \(N\) candidates may be another solution which remains as future work.

\[\tilde{\mathcal{F}}(\hat{x}_{i})=\frac{\log\big{(}\mathcal{F}( \hat{x}_{i})\big{)}/\texttt{len}(\hat{x}_{i})-\mu_{\mathcal{F}}}{\sigma_{ \mathcal{F}}} \tag{9}\]

where \(\mathcal{F}\) is either importance weight or quality estimations, and \(\mu_{\mathcal{F}}=\frac{1}{N}\sum_{i}\log\mathcal{F}(\hat{x}_{i})\) and \(\sigma_{\mathcal{F}}=\frac{\sum_{i}(\log\mathcal{F}(\hat{x}_{i})-\mu_{ \mathcal{F}})^{2}}{N-1}\) are mean and variance of \(N\) sampled candidates with length normalized. Finally, the Gamma score is defined on the normalized values of importance and quality as follows:

\[\Gamma(\hat{x}_{i};\omega,\pi)=\\ \frac{\exp\left(\gamma\widetilde{\mathrm{Imp}}(\hat{x}_{i};\omega ,\pi)+(1-\gamma)\tilde{p}(\hat{x}_{i}|y,\pi)\right)}{\sum_{j}\exp\left(\gamma \widetilde{\mathrm{Imp}}(\hat{x}_{j};\omega,\pi)+(1-\gamma)\tilde{p}(\hat{x}_{ j}|y,\pi)\right)} \tag{10}\]

where \(\widetilde{\mathrm{Imp}}\) and \(\tilde{p}\) are the normalized log value of importance weight and backward translation model \(p(\hat{x}|y,\pi)\) as defined in Equation 9.

Once the gamma score in Equation 10 is computed, there are two methods to select \(\hat{x}\) from \(A(y)\), which are deterministic and stochastic methods. For deterministic selection, we simply select the candidates with maximum gamma score among \(N\) translation candidates; and for sampling, we sample a candidate according to its gamma score distribution. These two methods are called gamma selection and gamma sampling in our experiments.

## 5 Experiments

### Settings

We run all the experiments by using fairseq Ott et al. (2019) framework. For dataset settings, since datasets WMT14 EN-DE and DE-EN are widely used Li et al. (2019b); Zhu et al. (2020); Li et al. (2020); Fan et al. (2021); Le et al. (2021), we follow both standard benchmarks and additionally we employ WMT14 RU-EN as the third dataset to validate the effectiveness of the proposed methods. For back translation experiment, we use an equal scale monolingual corpus randomly sampled from Newscrawl 2020 Barrault et al. (2019) comprising 4.5 million monolingual sentences for DE-EN language pair and 2.5 million for RU-EN direction, thus total 9 million sentences for DE-EN pair and 5 million for RU-EN direction are used. We tokenize the parallel corpus using Mose tokenizer Koehn et al. (2007), and learn a source and target shared Byte-Pair-Encoding BPE Sennrich et al. (2016b) 

with 32K types. We develop on newstest2013 and report the results on newstest2014.

As for model architecture, we employ all the translation models using architecture transformer_wmt_en_de_big, which is a Big Transformer architecture with 6 blocks in the encoder and decoder, and is widely used as a standard backbone on various NMT research studies. We use the same hyperparameter settings across all the experiments, i.e., 1024 word representation size, 4096 inner dimensions of feed-forward layers, and dropout is set to 0.3 for all the experiments. In addition, for monolingual models, we apply transformer_lm_gpt architecture Radford et al. (2018) on source language side of the corpus without any extra corpus. 2 The detailed hyperparameters used for training translation and language models are shown in Appendix.

Footnote 2: Note that we do not use the pre-trained language models such as GPT-3 or T5 to exclude our gains from large scale monolingual data.

For baseline models, we train them for 400K updating steps, and train the models with backtranslation data for 1.6M updating steps. We save the checkpoints every 100k updating intervals, and only select the checkpoints with highest develop set performance. As for the back-translation data, we study beam decoding and sampling decoding as baselines since they are the common practice for BT research Roberts et al. (2020; Wang et al., 2019). We use baseline models' checkpoints at 400K updating steps to generate default beam5 decoding and sampling decoding synthetic corpus without any penalty. For monolingual models, we only select the checkpoints with the best develop set performance. When tuning \(\gamma\) on dev sets for data manipulation methods we select it from \(\{0,1/4,1/2,3/4,1\}\) and the optimal is \(\gamma=1/2\). For the Gamma Score method, \(\gamma\) is tuned among \(\{0.1,0.2,0.3,0.4,0.5\}\) and it is set \(\gamma=0.2\) for all three tasks.

All the experiments are conducted using 8 Nvidia V100-32GB graphic cards without any gradient accumulation or bitext upsampling, and the results in this paper are measured in case-sensitive detokenized BLEU with SacreBLEU3 by Post (2018).

Footnote 3: We use the fairseq default shell script sacrebleu.sh, with WMT14/full testsets to evaluate the model checkpoints. The sacrebleu output format is BLEU + case.mixed + lang.de-en + numrefs.1 + smooth.exp + test.wmt14/full + tok.13a + version.1.4.13.

### Main Results

#### 5.2.1 Results on DE-EN

Data ManipulationWe conduct two experiments to study the data manipulation for back-translation NMT model performance using aforementioned corpus with and without authentic corpus.

Table 3 show the data manipulation results compared with baseline. Firstly, for synthetic corpus experiment, we find that even if only monolingual corpus is used, the performance of back-translation NMT model can still be significantly improved to 31.3 from 29.2 by sampling or 27.6 by beam, and it is only 0.7 lower than bitext baseline by BLEU score measure. Secondly, for the experiments with bitext, the best performance by data manipulation only helps the back-translation NMT model achieves almost the same performance with sampling BT. This means data manipulation methods cannot achieve a higher BLEU score than sampling or beam.

Gamma ScoreIn this paragraph, we conduct the experiments based on gamma score method. We conduct both of the methods in this experiment: we select the candidate with highest gamma score for the deterministic method whereas sample the candi

\begin{table}
\begin{tabular}{l c} \hline \hline
* & **SacreBLEU*
* \\ \hline Transformer & 32.1 \\ Beam BT & 32.7 \\ Sampling BT & 34.1 \\ DM +bitext & 34.2 \\ Gamma sampling BT & **35.0**
* \\ Gamma selection BT & 34.7
* \\ \hline \hline \end{tabular}
\end{table}
Table 4: BLEU score on WMT14 DE-EN testset. Gamma criterion based method outperform beam search based and sampling based back-translation NMT models. The result marked with
* denotes that it is significantly better than sampling BT with \(p<0.0010\).

\begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Systems**} & \multicolumn{2}{c}{**DE-EN**} \\ \cline{2-3}  & **w/o bitext** & **w bitext** \\ \hline Transformer & - & 32.1 \\ Beam BT & 27.6 & 32.7 \\ Sampling BT & 29.2 & 34.1 \\ DM & **31.3** & **34.2** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Data manipulation achieves the almost the same BLEU score as sampling BT.

 

date by gamma score distribution for the stochastic method.

Once again, we use synthetic gamma corpus combined with bitext to train the back-translation NMT models on each corpus, the results are listed in 4. From the table, we can see that our proposed gamma sampling significantly outperforms the sampling based and beam search based back-translation baselines by 0.9 and 2.3 BLEU scores in terms of SacreBLEU. And our two proposed gamma score based methods outperform the data manipulation method as well.

In the rest of the experiments, we report results for both gamma selection and gamma sampling as the proposed methods and their hyperparameter \(\gamma\) for other tasks is fixed to \(0.2\).

### Results on other Datasets

We conduct the experiments on WMT14 EN-DE and RU-EN for both gamma selection and gamma sampling as well, and table 5 shows that our proposed gamma based methods significantly outperform beam and sampling based back-translation methods on both en-de and ru-en translation for almost 1 and 0.4 BLEU score respectively. Recently, Edunov et al. (2020) point out that BLEU might overlook the contributions from back translation since it poorly correlates with human evaluation on the data generated in back translation scenario. Follow their suggestions, to better reflect the scenario of back translation, we also evaluate our experiment using COMET metric suggested by Rei et al. (2020). The results are shown in table 6 and we can see that the proposed methods perform well in terms of COMET.

Discussion on EfficiencySince our method requires to run sampling with size of 50 to generate synthetic data, its efficiency is about 10x slower than that of beam BT with size of 5 and 50x slower than that of sampling BT with size 1. Luckily, because the bottleneck of BT is not the synthetic data generation but the parameter optimization on both synthetic and authentic data, our overall overhead is less than 0.5x slower than sampling BT. In addition, since decoding is very easy to be parallelized on GPU or CPU machines, the cost of decoding is not a serious issue for our method, which makes it possible to run our method on a large scale dataset.

### Analysis on Synthetic Corpus

In this subsection, we analyze the synthetic corpus of proposed gamma score methods on both sentence level and token level.

Sentence LevelWe evaluate the back-translation synthetic source sentences by their sentence representations. We use the baseline model to generate the hidden representations at the end-of-speech token as the sentence representation. Here, we compute the singular value spectrum of the representations for different back-translation corpora. 4

Footnote 4: Singular value spectrum analysis is a widely used method to measure the representation distribution. Gao et al. (2019) firstly introduces this method to measure the isotropy of representation, and Wang et al. (2019) directly employ spectrum control for better NMT performance. The idea is, representations of high linguistic variety usually are more isotropic, thus having a relatively uniform singular value distribution. We employ this method here to measure the variety of sentence-level information.

The spectrum is shown in figure 1(a). From the spectrum, sampling has a more uniform distribution whereas beam has the worst variety. Our proposed methods have moderate variety between sampling and beam, and gamma sampling consists of higher linguistic information richness compared with gamma selection.

Figure 1(b) shows the sequence length of the synthetic corpora of different generation methods. Beam generates the shortest synthetic sentences and gamma sampling generates the longest synthetic sentences on average. Between them, sampling and gamma selection generate almost the same sequence length, which means gamma selection candidates provide more learning signals than random sampling under the same length.

Token LevelFigure 1(c) is the token frequency histogram, which shows beam has higher probability to decode high frequency tokens while sampling

\begin{table}
\begin{tabular}{l c c} \hline \hline
* & **EN-DE*
* & **RU-EN*
* \\ \hline Transformer & 27.4 & 34.1 \\ Beam BT & 29.7 & 35.9 \\ Sampling BT & 30.0 & 35.6 \\ Gamma selection BT & **31.0**
* & 36.1
* \\ Gamma sampling BT & 30.9
* & **36.3**
* \\ \hline \hline \end{tabular}
\end{table}
Table 5: SacreBLEU score on WMT14 EN-DE and RU-EN testsets. Gamma criterion based methods outperform beam search based and sampling based back-translation NMT models. The result marked with
* denotes that it is significantly better than both sampling and beam based BT with \(p<0.001\).
 

prefers more low frequency tokens.

We also measure the vocabulary size, finding that the proposed gamma sampling shares the same vocabulary size as sampling method. This could be the reason that gamma sampling is based on random sampling for candidates generation.

## 6 Related Work

This section describes prior arts in back-translation for NMT, data augmentation, and semi-supervised machine translation.

Back-translation NMTBojar and Tamchyna (2011) firstly proposed back-translation, then Bertoldi and Federico (2009); Lambert et al. (2011) apply back translation to solve the domain adaptation problems in phrase-based NMT systems. Sennrich et al. (2016a) further extend the back translation for training NMT models integrally.

For understanding the back-translation synthetic corpus, Currey et al. (2017) use a copy of target as a pseudo source, and find that NMT model performance can still be improved under the low resource settings. Caswell et al. (2019) propose tagged back-translation to indicate to the model that the given source is synthetic. To further find an optimum back-translation corpus decoding method, Imamura et al. (2018) firstly use sampling based synthetic corpus and find such a stochastic decoding method outperforms beam search on boosting NMT model performance, and Edunov et al. (2018) broaden the investigation of a number of back-translation generation methods for synthetic source sentences. Their contribution shows that sampling or noisy synthetic data gives a much stronger training signal. Graca et al. (2019) reformulate back-translation in the context of optimization and clarifying to improve sampling based decoding method search space, thus proposing N best list sampling. Recently, Nguyen et al. (2020) diversify the training data by multiple forward and backward models translations and combine them with the original datasets.

Data Augmentation for NMTNMT researchers are the pioneers of data augmentation studies since back-translation is a natural type of data augmentation method. (Sennrich et al., 2016a; Norouzi et al., 2016; Zhang and Zong, 2016; Bi et al., 2021). To balance the token frequency in NMT corpus, Fadaee et al. (2017) create new sentences contain low-frequency words. However, as observed by Wang et al. (2018), the improvement across different translation tasks is not consistent, and they invent SwitchOut data augmentation policy. Recht et al. (2018, 2019); Werbachowski et al. (2019) also observe such an inconsistency of variance between training corpus and testing set as well as in the generation tasks. Recently, Li et al. (2019) try to understand data augmentation from input sensitivity and prediction margin, thus obtaining relatively low variance in generation.

Semi-supervised Machine TranslationHowever, as high quality bitext is always limited and costly to collect, Gulcehre et al. (2015) study methods for effectively leveraging monolingual data in

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**System** & **DE-EN** & **EN-DE** & **RU-EN** \\ \hline Transformer & 51.66 & 53.35 & 54.55 \\ Beam BT & 49.35 & 54.61 & 55.12 \\ Sampling BT & 52.71 & 56.01 & 54.34 \\ Gamma Selection BT & 53.83 & **58.22** & **57.03** \\ Gamma Sampling BT & **53.97** & 58.18 & 56.69 \\ \hline \hline \end{tabular}
\end{table}
Table 6: COMET metric evaluation results on WMT14 DE-EN, EN-DE and RU-EN datasets. The testset results are in accordance with BLEU metric.

Figure 1: Synthetic corpus analysis on singular value spectrum(a), sequence length histogram(b) and token frequency histogram(c).

 

NMT systems. He et al. (2016) develop a dual-learning mechanism, under such a learning objective, a NMT system is able to automatically learn from unlabeled data, thus improving NMT performance iteratively. Based on iterative learning, Lample et al. (2018) investigates how to learn NMT systems when only large monolingual corpora can be used in each language.

For supervision of models, Gulcehre et al. (2017) employ the target language model hidden states into NMT decoder to further improve performance. Edunov et al. (2020) show that back-translation improves translation quality of both naturally occurring text and translationese according to professional human translators. For supervision of learning corpus, Wu et al. (2019) study both the source-side and target-side monolingual data for NMT.

## 7 Conclusion

In this work, we answer a fundamental question about synthetic data for back translation. We theoretically and empirically show two key factors namely quality and importance weight of synthetic data play an important role in back translation, and then we propose a new method to generate synthetic data which better balances both factors so as to boost the back-translation performance. For future work, we think it would be of significance to apply our synthetic data generation method to other BT methods or even to more broad NLP tasks such as paraphrasing and style transfer.

## Acknowledgements

We would like to thank anonymous (meta) reviewers for suggestions on this paper. L. Liu is the corresponding author.

## References

* Artetxe et al. (2018) Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In _International Conference on Learning Representations_.
* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _arXiv preprint arXiv:1409.0473_.
* Barrault et al. (2019) Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Muller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT'19). In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 1-61, Florence, Italy. Association for Computational Linguistics.
* Bertoldi and Federico (2009) Nicola Bertoldi and Marcello Federico. 2009. Domain adaptation for statistical machine translation with monolingual resources. In _Proceedings of the fourth workshop on statistical machine translation_, pages 182-189.
* Bi et al. (2021) Wei Bi, Huayang Li, and Jiacheng Huang. 2021. Data augmentation for text generation without any augmented data. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 2223-2237, Online. Association for Computational Linguistics.
* Bojar and Tamchyna (2011) Ondrej Bojar and Ales Tamchyna. 2011. Improving translation model by monolingual data. In _Proceedings of the Sixth Workshop on Statistical Machine Translation_, pages 330-336.
* Caswell et al. (2019) Isaac Caswell, Ciprian Chelba, and David Grangier. 2019. Tagged back-translation. In _Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)_, pages 53-63.
* Currey et al. (2017) Anna Currey, Antonio Valerio Miceli Barone, and Kenneth Heafield. 2017. Copied monolingual data improves low-resource neural machine translation. In _Proceedings of the Second Conference on Machine Translation_, pages 148-156, Copenhagen, Denmark. Association for Computational Linguistics.
* Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 489-500.
* Edunov et al. (2020) Sergey Edunov, Myle Ott, Marc'Aurelio Ranzato, and Michael Auli. 2020. On the evaluation of machine translation systems trained with back-translation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2836-2846.
* Fadaee et al. (2017) Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 567-573.
* Fadaee and Monz (2018) Marzieh Fadaee and Christof Monz. 2018. Back-translation sampling by targeting difficult words in neural machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 436-446 

Zhihao Fan, Yeyun Gong, Dayiheng Liu, Zhongyu Wei, Siyuan Wang, Jian Jiao, Nan Duan, Ruofei Zhang, and Xuan-Jing Huang. 2021. Mask attention networks: Rethinking and strengthen transformer. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1692-1701.
* Gao et al. (2019) Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. Representation degeneration problem in training natural language generation models. In _International Conference on Learning Representations_.
* Graca et al. (2019) Miguel Graca, Yunsu Kim, Julian Schamper, Shahram Khadivi, and Hermann Ney. 2019. Generalizing back-translation in neural machine translation. In _Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)_, pages 45-52.
* Gulcehre et al. (2015) Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. _arXiv preprint arXiv:1503.03535_.
* Gulcehre et al. (2017) Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, and Yoshua Bengio. 2017. On integrating a language model into neural machine translation. _Computer Speech & Language_, 45:137-148.
* He et al. (2016) Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In _Advances in Neural Information Processing Systems_, volume 29. Curran Associates, Inc.
* Hoang et al. (2018) Vu Cong Duy Hoang, Philipp Koehn, Gholameza Haffari, and Trevor Cohn. 2018. Iterative back-translation for neural machine translation. In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_, pages 18-24.
* Imamura et al. (2018) Kenji Imamura, Atsushi Fujita, and Eiichiro Sumitta. 2018. Enhancement of encoder and attention using target monolingual corpora in neural machine translation. In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_, pages 55-63.
* Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _ICLR (Poster)_.
* Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions_, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
* Lambert et al. (2011) Patrik Lambert, Holger Schwenk, Christophe Servan, and Sadaf Abdul-Rauf. 2011. Investigations on translation model adaptation using monolingual data. In _Sixth Workshop on Statistical Machine Translation_, pages 284-293.
* Lample et al. (2018) Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 5039-5049.
* Le et al. (2021) Giang Le, Shinka Mori, and Lane Schwartz. 2021. Illinois Japanese \(\leftrightarrow\) English News Translation for WMT 2021. In _Proceedings of the Sixth Conference on Machine Translation_, pages 144-153, Online. Association for Computational Linguistics.
* Li et al. (2019a) Guanlin Li, Lemao Liu, Guoping Huang, Conghui Zhu, and Tiejun Zhao. 2019a. Understanding data augmentation in neural machine translation: Two perspectives towards generalization. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5689-5695.
* Li et al. (2020) Jierui Li, Lemao Liu, Huayang Li, Guanlin Li, Guoping Huang, and Shuming Shi. 2020. Evaluating explanation methods for neural machine translation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 365-375, Online. Association for Computational Linguistics.
* Li et al. (2019b) Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019b. On the word alignment from neural machine translation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1293-1303.
* Liu et al. (1998) Jun S Liu, Rong Chen, and Wing Hung Wong. 1998. Rejection control and sequential importance sampling. _Journal of the American Statistical Association_, 93(443):1022-1031.
* Liu and Liu (2001) Jun S Liu and Jun S Liu. 2001. _Monte Carlo strategies in scientific computing_, volume 10. Springer.
* Mallinson et al. (2017) Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017. Paraphrasing revisited with neural machine translation. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 881-893.
* Nguyen et al. (2020)Xuan-Phi Nguyen, Shafiq Joty, Wu Kui, and Ai Ti Aw. 2020. Data diversification: A simple strategy for neural machine translation.
* Norouzi et al. (2016) Mohammad Norouzi, Samy Bengio, zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. 2016. Reward augmented maximum likelihood for neural structured prediction. In _Advances in Neural Information Processing Systems_, volume 29. Curran Associates, Inc 

* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of NAACL-HLT 2019: Demonstrations_.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
* Prabhumoye et al. (2018) Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 866-876.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
* Recht et al. (2018) Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2018. Do cifar-10 classifiers generalize to cifar-10?
* Recht et al. (2019) Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. Do imagenet classifiers generalize to imagenet? In _International Conference on Machine Learning_, pages 5389-5400. PMLR.
* Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2685-2702.
* Roberts et al. (2020) Nicholas Roberts, Davis Liang, Graham Neubig, and Zachary C Lipton. 2020. Decoding and diversity in machine translation. _arXiv preprint arXiv:2011.13477_.
* Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 86-96, Berlin, Germany. Association for Computational Linguistics.
* Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1715-1725.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in neural information processing systems_, pages 3104-3112.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in neural information processing systems_, pages 5998-6008.
* Wang et al. (2019) Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun. 2019. Improving back-translation with uncertainty-based confidence estimation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 791-802.
* Wang et al. (2018) Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018. Switchout: an efficient data augmentation algorithm for neural machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 856-861.
* Werbachowski et al. (2019) Roman Werpachowski, Andras Gyorgy, and Csaba Szepesvari. 2019. Detecting overfitting via adversarial examples. _Advances in Neural Information Processing Systems_, 32.
* Wu et al. (2019) Lijun Wu, Yiren Wang, Yingce Xia, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2019. Exploiting monolingual data at scale for neural machine translation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4207-4216.
* Zhang and Zong (2016) Jiajun Zhang and Chengqing Zong. 2016. Exploiting source-side monolingual data in neural machine translation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1535-1545.
* Zhang et al. (2018) Zhirui Zhang, Shuo Ren, Shujie Liu, Jianying Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. 2018. Style transfer as unsupervised machine translation. _arXiv preprint arXiv:1808.07894_.
* Zhu et al. (2020) Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tieyan Liu. 2020. Incorporating bert into neural machine translation. In _International Conference on Learning Representations_.

## Appendix A Model Details

The models are optimized using Adam optimizer Kingma and Ba (2015), with \(\beta_{1}=0.9,\beta_{2}=0.98\). We use the same learning rate scheduler as Vaswani et al. (2017) with maximum learning rate \(7\times 10^{-4}\), and 4000 warmup updates. We use the fairseq 10.2 as the framework and the training command as well as the model hyperparameters are listed below,

fairseq-train \  --arch transformer_wmt_en_de_big  --share-all-embeddings  --dropout 0.3  --weight-decay 0.0  --criterion  label_smoothed_cross_entropy 

-label-smoothing 0.1  --optimizer adam --adam-betas '(0.9, 0.98)'  --clip-norm 0.0  --lr-scheduler inverse_sqrt --warmup-updates 4000  --max-tokens 4096  --max-update 1600000  --validate-interval-updates 10000  --save-interval-updates 100000  --lr 7e-4 --upsample-primary 1

And the GPT model we employ is only trained on source side of bitext corpus, without extra datasets. The training command line and core settings are listed below.

fairseq-train \ --task language_modeling --arch transformer_lm_gpt --share-decoder-input-output-embed --dropout 0.1 --optimizer adam --adam-betas '(0.9, 0.98)'  --weight-decay 0.01 --clip-norm 0.0 --lr 7e-5 --lr-schedulerinverse_sqrt --warmup-updates 8000  --tokens-per-sample 512 --sample-break-modenone --max-tokens 4096 --update-freq 1 --max-update 100000  --keep-last-epochs 5 --validate-interval-updates 10000  --save-interval-updates 10000