SlowGuess/NAACL_2022_OCR huggingface dataset:
As for model architecture, we employ all the translation models using architecture transformer_wmt_en_de/big, which is a Big Transformer architecture with 6 blocks in the encoder and decoder, and is widely used as a standard backbone on various NMT research studies. We use the same hyperparameter settings across all the experiments, i.e., 1024 word representation size, 4096 inner dimensions of feed-forward layers, and dropout is set to 0.3 for all the experiments. In addition, for monolingual models, we apply transformer_lm_gpt architecture (Radford et al., 2018) on source language side of the corpus without any extra corpus. The detailed hyperparameters used for training translation and language models are shown in Appendix.
For baseline models, we train them for 400K updating steps, and train the models with backtranslation data for 1.6M updating steps. We save the checkpoints every 100k updating intervals, and only select the checkpoints with highest develop set performance. As for the back-translation data, we study beam decoding and sampling decoding as baselines since they are the common practice for BT research (Roberts et al., 2020; Wang et al., 2019). We use baseline models' checkpoints at 400K updating steps to generate default beam5 decoding and sampling decoding synthetic corpus without any penalty. For monolingual models, we only select the checkpoints with the best develop set performance. When tuning  $\gamma$  on dev sets for data manipulation methods we select it from  $\{0,1/4,1/2,3/4,1\}$  and the optimal is  $\gamma = 1/2$ . For the Gamma Score method,  $\gamma$  is tuned among  $\{0.1,0.2,0.3,0.4,0.5\}$  and it is set  $\gamma = 0.2$  for all three tasks.
Table 3: Data manipulation achieves the almost the same BLEU score as sampling BT.
Systems	SacreBLEU
Transformer	32.1
Beam BT	32.7
Sampling BT	34.1
DM +bitext	34.2
Gamma sampling BT	35.0*
Gamma selection BT	34.7*
Table 4: BLEU score on WMT14 DE-EN testset. Gamma criterion based method outperform beam search based and sampling based back-translation NMT models. The result marked with * denotes that it is significantly better than sampling BT with  $p < 0.0010$ .
All the experiments are conducted using 8 Nvidia V100-32GB graphic cards without any gradient accumulation or bitext upsampling, and the results in this paper are measured in case-sensitive detokenized BLEU with SacreBLEU<sup>3</sup> by Post (2018).
5.2.1 Results on DE-EN
Data Manipulation We conduct two experiments to study the data manipulation for backtranslation NMT model performance using aforementioned corpus with and without authentic corpus.
Table 3 show the data manipulation results compared with baseline. Firstly, for synthetic corpus experiment, we find that even if only monolingual corpus is used, the performance of back-translation NMT model can still be significantly improved to 31.3 from 29.2 by sampling or 27.6 by beam, and it is only 0.7 lower than bitext baseline by BLEU score measure. Secondly, for the experiments with bitext, the best performance by data manipulation only helps the back-translation NMT model achieves almost the same performance with sampling BT. This means data manipulation methods cannot achieve a higher BLEU score than sampling or beam.
Gamma Score In this paragraph, we conduct the experiments based on gamma score method. We conduct both of the methods in this experiment: we select the candidate with highest gamma score for the deterministic method whereas sample the candi
3We use the faireq default shell script sacrebleu.sh, with WMT14/full testsets to evaluate the model checkpoints. The sacrebleu output format is BLEU  $^+$  case.mixed  $^+$  lang.deen  $^+$  numrefs.1  $^+$  smooth.exp  $^+$  test.wmt14/full  $^+$  tok.13a  $^+$  version.1.4.13.
<sup>2</sup>Note that we do not use the pre-trained language models such as GPT-3 or T5 to exclude our gains from large scale monolingual data.
System	EN-DE	RU-EN
Transformer	27.4	34.1
Beam BT	29.7	35.9
Sampling BT	30.0	35.6
Gamma selection BT	31.0*	36.1*
Gamma sampling BT	30.9*	36.3*
Table 5: SacreBLEU score on WMT14 EN-DE and RU-EN testsets. Gamma criterion based methods outperform beam search based and sampling based backtranslation NMT models. The result marked with * denotes that it is significantly better than both sampling and beam based BT with  $p < 0.001$ .
date by gamma score distribution for the stochastic method.
Once again, we use synthetic gamma corpus combined with bitext to train the back-translation NMT models on each corpus, the results are listed in 4. From the table, we can see that our proposed gamma sampling significantly outperforms the sampling based and beam search based back-translation baselines by 0.9 and 2.3 BLEU scores in terms of SacreBLEU. And our two proposed gamma score based methods outperform the data manipulation method as well.
In the rest of the experiments, we report results for both gamma selection and gamma sampling as the proposed methods and their hyperparameter  $\gamma$  for other tasks is fixed to 0.2.
5.3 Results on other Datasets
We conduct the experiments on WMT14 EN-DE and RU-EN for both gamma selection and gamma sampling as well, and table 5 shows that our proposed gamma based methods significantly outperform beam and sampling based back-translation methods on both en-de and ru-en translation for almost 1 and 0.4 BLEU score respectively. Recently, Edunov et al. (2020) point out that BLEU might overlook the contributions from back translation since it poorly correlates with human evaluation on the data generated in back translation scenario. Follow their suggestions, to better reflect the scenario of back translation, we also evaluate our experiment using COMET metric suggested by Rei et al. (2020). The results are shown in table 6 and we can see that the proposed methods perform well in terms of COMET.
Discussion on Efficiency Since our method requires to run sampling with size of 50 to generate
synthetic data, its efficiency is about  $10\mathrm{x}$  slower than that of beam BT with size of 5 and  $50\mathrm{x}$  slower than that of sampling BT with size 1. Luckily, because the bottleneck of BT is not the synthetic data generation but the parameter optimization on both synthetic and authentic data, our overall overhead is less than  $0.5\mathrm{x}$  slower than sampling BT. In addition, since decoding is very easy to be parallelized on GPU or CPU machines, the cost of decoding is not a serious issue for our method, which makes it possible to run our method on a large scale dataset.
5.4 Analysis on Synthetic Corpus
In this subsection, we analyze the synthetic corpus of proposed gamma score methods on both sentence level and token level.
Sentence Level We evaluate the back-translation synthetic source sentences by their sentence representations. We use the baseline model to generate the hidden representations at the end-of-speech token as the sentence representation. Here, we compute the singular value spectrum of the representations for different back-translation corpora.
The spectrum is shown in figure 1(a). From the spectrum, sampling has a more uniform distribution whereas beam has the worst variety. Our proposed methods have moderate variety between sampling and beam, and gamma sampling consists of higher linguistic information richness compared with gamma selection.
Figure 1(b) shows the sequence length of the synthetic corpora of different generation methods. Beam generates the shortest synthetic sentences and gamma sampling generates the longest synthetic sentences on average. Between them, sampling and gamma selection generate almost the same sequence length, which means gamma selection candidates provide more learning signals than random sampling under the same length.
Token Level Figure 1(c) is the token frequency histogram, which shows beam has higher probability to decode high frequency tokens while sampling
<sup>4</sup>Singular value spectrum analysis is a widely used method to measure the representation distribution. Gao et al. (2019) firstly introduces this method to measure the isotropy of representation, and Wang et al. (2019) directly employ spectrum control for better NMT performance. The idea is, representations of high linguistic variety usually are more isotropic, thus having a relatively uniform singular value distribution. We employ this method here to measure the variety of sentence-level information.
(a) Spectrum
(b) Sequence Length
Figure 1: Synthetic corpus analysis on singular value spectrum(a), sequence length histogram(b) and token frequency histogram(c).
(c) Token frequency
System	DE-EN	EN-DE	RU-EN
Transformer	51.66	53.35	54.55
Beam BT	49.35	54.61	55.12
Sampling BT	52.71	56.01	54.34
Gamma Selection BT	53.83	58.22	57.03
Gamma Sampling BT	53.97	58.18	56.69
Table 6: COMET metric evaluation results on WMT14 DE-EN, EN-DE and RU-EN datasets. The testset results are in accordance with BLEU metric.
prefers more low frequency tokens.
We also measure the vocabulary size, finding that the proposed gamma sampling shares the same vocabulary size as sampling method. This could be the reason that gamma sampling is based on random sampling for candidates generation.
This section describes prior arts in back-translation for NMT, data augmentation, and semi-supervised machine translation.
Back-translation NMT Bojar and Tamchyna (2011) firstly proposed back-translation, then Bertoldi and Federico (2009); Lambert et al. (2011) apply back translation to solve the domain adaptation problems in phrase-based NMT systems. Sennrich et al. (2016a) further extend the back translation for training NMT models integrally.
For understanding the back-translation synthetic corpus, Currey et al. (2017) use a copy of target as a pseudo source, and find that NMT model performance can still be improved under the low resource settings. Caswell et al. (2019) propose tagged back-translation to indicate to the model that the given source is synthetic. To further find an optimum back-translation corpus decoding method, Imamura et al. (2018) firstly use sampling based
synthetic corpus and find such a stochastic decoding method outperforms beam search on boosting NMT model performance, and Edunov et al. (2018) broaden the investigation of a number of backtranslation generation methods for synthetic source sentences. Their contribution shows that sampling or noisy synthetic data gives a much stronger training signal. Graça et al. (2019) reformulate backtranslation in the context of optimization and clarifying to improve sampling based decoding method search space, thus proposing N best list sampling. Recently, Nguyen et al. (2020) diversify the training data by multiple forward and backward models translations and combine them with the original datasets.
Data Augmentation for NMT NMT researchers are the pioneers of data augmentation studies since back-translation is a natural type of data augmentation method. (Sennrich et al., 2016a; Norouzi et al., 2016; Zhang and Zong, 2016; Bi et al., 2021). To balance the token frequency in NMT corpus, Fadaee et al. (2017) create new sentences contain low-frequency words. However, as observed by Wang et al. (2018), the improvement across different translation tasks is not consistent, and they invent SwitchOut data augmentation policy. Recht et al. (2018, 2019); Werpachowski et al. (2019) also observe such an inconsistency of variance between training corpus and testing set as well as in the generation tasks. Recently, Li et al. (2019a) try to understand data augmentation from input sensitivity and prediction margin, thus obtaining relatively low variance in generation.
Semi-supervised Machine Translation However, as high quality bitext is always limited and costly to collect, Gulcehre et al. (2015) study methods for effectively leveraging monolingual data in
NMT systems. He et al. (2016) develop a dual-learning mechanism, under such a learning objective, a NMT system is able to automatically learn from unlabeled data, thus improving NMT performance iteratively. Based on iterative learning, Lample et al. (2018) investigates how to learn NMT systems when only large monolingual corpora can be used in each language.
For supervision of models, Gulcehre et al. (2017) employ the target language model hidden states into NMT decoder to further improve performance. Edunov et al. (2020) show that back-translation improves translation quality of both naturally occurring text and translationese according to professional human translators. For supervision of learning corpus, Wu et al. (2019) study both the source-side and target-side monolingual data for NMT.
In this work, we answer a fundamental question about synthetic data for back translation. We theoretically and empirically show two key factors namely quality and importance weight of synthetic data play an important role in back translation, and then we propose a new method to generate synthetic data which better balances both factors so as to boost the back-translation performance. For future work, we think it would be of significance to apply our synthetic data generation method to other BT methods or even to more broad NLP tasks such as paraphrasing and style transfer.
We would like to thank anonymous (meta) reviewers for suggestions on this paper. L. Liu is the corresponding author.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In International Conference on Learning Representations.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Muller,
Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy. Association for Computational Linguistics.
Nicola Bertoldi and Marcello Federico. 2009. Domain adaptation for statistical machine translation with monolingual resources. In Proceedings of the fourth workshop on statistical machine translation, pages 182-189.
Wei Bi, Huayang Li, and Jiacheng Huang. 2021. Data augmentation for text generation without any augmented data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2223-2237, Online. Association for Computational Linguistics.
Ondrej Bojar and Ales Tamchyna. 2011. Improving translation model by monolingual data. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 330-336.
Isaac Caswell, Ciprian Chelba, and David Grangier. 2019. Tagged back-translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 53-63.
Anna Currey, Antonio Valerio Miceli Barone, and Kenneth Heafield. 2017. Copied monolingual data improves low-resource neural machine translation. In Proceedings of the Second Conference on Machine Translation, pages 148-156, Copenhagen, Denmark. Association for Computational Linguistics.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489-500.
Sergey Edunov, Myle Ott, Marc'Aurelio Ranzato, and Michael Auli. 2020. On the evaluation of machine translation systems trained with back-translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2836-2846.
Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 567-573.
Marzieh Fadaee and Christof Monz. 2018. Back-translation sampling by targeting difficult words in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 436-446.
Zhihao Fan, Yeyun Gong, Dayiheng Liu, Zhongyu Wei, Siyuan Wang, Jian Jiao, Nan Duan, Ruofei Zhang, and Xuan-Jing Huang. 2021. Mask attention networks: Rethinking and strengthen transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1692-1701.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. Representation degeneration problem in training natural language generation models. In International Conference on Learning Representations.
Miguel Graça, Yunsu Kim, Julian Schamper, Shahram Khadivi, and Hermann Ney. 2019. Generalizing back-translation in neural machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 45-52.
Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535.
Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, and Yoshua Bengio. 2017. On integrating a language model into neural machine translation. Computer Speech & Language, 45:137-148.
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018. Iterative backtranslation for neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 18-24.
Kenji Imamura, Atsushi Fujita, and Eiichiro Sumita. 2018. Enhancement of encoder and attention using target monolingual corpora in neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 55-63.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In *ICLR (Poster)*.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Patrik Lambert, Holger Schwenk, Christophe Servan, and Sadaf Abdul-Rauf. 2011. Investigations on translation model adaptation using monolingual data. In Sixth Workshop on Statistical Machine Translation, pages 284-293.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039-5049.
Giang Le, Shinka Mori, and Lane Schwartz. 2021. Illinois Japanese  $\leftrightarrow$  English News Translation for WMT 2021. In Proceedings of the Sixth Conference on Machine Translation, pages 144-153, Online. Association for Computational Linguistics.
Guanlin Li, Lemao Liu, Guoping Huang, Conghui Zhu, and Tiejun Zhao. 2019a. Understanding data augmentation in neural machine translation: Two perspectives towards generalization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5689-5695.
Jierui Li, Lemao Liu, Huayang Li, Guanlin Li, Guoping Huang, and Shuming Shi. 2020. Evaluating explanation methods for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 365-375, Online. Association for Computational Linguistics.
Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019b. On the word alignment from neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1293-1303.
Jun S Liu, Rong Chen, and Wing Hung Wong. 1998. Rejection control and sequential importance sampling. Journal of the American Statistical Association, 93(443):1022-1031.
Jun S Liu and Jun S Liu. 2001. Monte Carlo strategies in scientific computing, volume 10. Springer.
Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017. Paraphrasing revisited with neural machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 881-893.
Xuan-Phi Nguyen, Shafiq Joty, Wu Kui, and Ai Ti Aw. 2020. Data diversification: A simple strategy for neural machine translation.
Mohammad Norouzi, Samy Bengio, zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. 2016. Reward augmented maximum likelihood for neural structured prediction. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairoseq: A fast, extensible toolkit for sequence modeling*. In *Proceedings of NAACL-HLT* 2019: Demonstrations.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 866-876.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2018. Do CIFar-10 classifiers generalize to CIFar-10?
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. DoImagenet classifiers generalize toImagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685-2702.
Nicholas Roberts, Davis Liang, Graham Neubig, and Zachary C Lipton. 2020. Decoding and diversity in machine translation. arXiv preprint arXiv:2011.13477.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.