SlowGuess/NAACL_2021_OCR huggingface dataset:
We use the monolingual training data, including News Commentary and News Crawl 2010, provided by WMT'18, for Eq. 2, while the training set of each language pair is used for Eq. 3. The statistics of the monolingual and parallel corpora are summarized in Table 1 and 2, respectively.
To reduce memory overhead, we train a shared language model for Eq. 2 and 3, i.e.  $\theta_{1}$  and  $\theta_{2}$  are tied. A language model is trained for each language pair to avoid performing multilingual NMT for a fair comparison with baselines, as jointly training a single model for several language pairs has been shown to be effective for both low-resource and high-resource languages (Aharoni et al., 2019). Therefore, we pre-train four models for En-Tr, En-De, En-Vi and En-Fr, respectively.
The encoder and decoder are composed of six layers. The encoder is initialized with XLM (Lample and Conneau, 2019) pre-trained with the masked language model, while the decoder is randomly initialized. The input-output embeddings are tied for reducing the size of the model (Press and Wolf, 2016). To achieve faster convergence, we apply PreNorm (Nguyen and Salazar, 2019) for getting rid of the warm-up stage of Transformer. The learning rate is set to 1e-5 and is linearly decayed with more training steps. The hidden size  $o$  is set to 1024. Same as BERT, the maximum sequence size is set to 512. We use LAMB (You et al., 2019) as the optimizer. GELU (Hendrycks and Gimpel, 2016) is used as the activation function. 16 sequences are used at each pre-training step. We train the masked language model for  $50\%$  of the time and the left time is used for training the translation language model.
After pre-training, we use the pre-trained models to perform data augmentation on training data. Then, the augmented data are combined with training data for training NMT models.
Figure 3: BLEU scores on the development set of the WMT'18 English  $\rightarrow$  Turkish task with different pretraining steps.
Figure 4: Learning curves of training language models for the WMT'18 English  $\rightarrow$  Turkish task with either the random initialization or the XLM initialization.
5.2 NMT Model Details
We use  $\text{fairoseq}^2$  to implement the NMT models. The vocabulary size is 37K. Six encoder and decoder layers are applied. The hidden size is set to 1024. 16 self-attention heads are employed. We use Adam as the optimizer. The learning rate is initially set to 1e-7 and is gradually increased to 5e-4 with 4K warm-up steps, before applying linear decay. Dropout is set to 0.3. Label smoothing with the smoothing factor 0.1 is used. For decoding, we use beam search, and the beam size is set to 12. SacreBLEU (Post, 2018) is used as the metric.
5.3 Sensitivity Study
5.3.1 Pre-training Steps
We study the effect of pre-training steps on machine translation quality. We use the language models at different pre-training steps and evaluate these models on the development set of the WMT'18 English  $\rightarrow$  Turkish task. The results are shown in Figure 3. The BLEU score improves with more pre-training steps and peaks at around 110K steps. We do not observe better performance with more pre-training steps, as the models become more overfitted on the training sets.
Figure 5: BLEU scores on the development set of the WMT'18 English  $\rightarrow$  Turkish task with different sampling probabilities.
5.3.2 Effect of the XLM Initialization
We plot the learning curves of the language model for En-Tr with/without XLM initialization. As shown in Figure 4, the model with the XLM initialization converges faster and better compared to the model with the random initialization. As XLM is trained using the masked language model objective on large-scale monolingual data, we draw the conclusion that large-scale pre-training can improve downstream language model pre-training tasks. We further evaluate the models on the development set of the WMT'18 English  $\rightarrow$  Turkish task. The model with the XLM initialization also performs better (17.49 BLEU) compared to its counterpart (16.68 BLEU). Thus, the model with the XLM initialization can also generate better data for improving NMT.
5.3.3 Sampling Probability
As shown in Figure 5, we vary the sampling probability in Algorithm 2 and evaluate on the development set of the WMT'18 English  $\rightarrow$  Turkish task. We observe that the BLEU score is maximized with sampling probability 0.2. The BLEU scores decrease with larger sampling probabilities.
Method	En → Tr
TCWR	17.49
-Source	16.47
-Target	16.29
-Alignment	16.59
-Gumbel	16.85
Table 3: The BLEU scores on the development set of the WMT'18 English  $\rightarrow$  Turkish task with source context, target context, alignment, and Gumbel ablation.
We ablate the source context, target context and alignment to validate the effectiveness of these
Method	En → Tr	En → De	En → Vi
Baseline	15.35	27.54	31.66
+WordDropout	15.4	27.81	31.81
+SwitchOut	15.52	27.92	31.83
+SCDA	15.72	28.05	31.72
+TDA	15.69	28.16	31.79
+BPEDropout	15.95	28.29	33.59
+DADA	16.14	29.03	32.15
+TCWR	17.38	29.37	33.76
components. We randomly choose a phrase with uniform distribution to replace the original phrase for ablating source and target contexts. For ablating alignment, we randomly choose a position in the target sequence instead of following the alignment given by pialign. We also study removing  $\hat{g}_k$  in Eq. 1. Since  $\hat{g}_k$  comes from the abduction step, which encodes the information from the original pair  $(\mathcal{X},\mathcal{Y})$ , Eq. 1 encourages the model to sample a new pair that is similar to the original pair. Therefore, the model without  $\hat{g}_k$  collapses to a probabilistic approach that directly samples phrases from the translation language model, disregarding the information from the original pair.
The results are shown in Table 3. We observe that ablating the source context, target context and alignment are negative for translation quality, demonstrating the necessity of considering all these components for data augmentation. The result of ablating  $\hat{g}_k$  shows the effectiveness of incorporating the original information from  $(\mathcal{X},\mathcal{Y})$ .
5.5 Translation Result
We evaluate the algorithms on WMT'17 English  $\rightarrow$  German (En-De), WMT'18 English  $\rightarrow$  Turkish (En-Tr) and IWSLT'15 English  $\rightarrow$  Vietnamese (En-Vi). As shown in Table 2, En-Tr and En-Vi are two low-resource language pairs, while En-De is a high-resource language pair.
For En-Tr, we use newstest17 for validation and newstest18 for testing. For En-De, we use newstest16 for validation and newstest17 for testing. For En-Vi, we use the TED tst2012 for validation and the TED tst2013 for testing.
We compare TCWR with six baselines, WordDropout, BPEDropout, SwitchOut, SCDA, TDA and DADA. For WordDropout and BPEDropout, we perform a range search on its dropout probability from 0 to 1 and select the best one on de
Table 4: The BLEU scores on the testing sets of En-Tr, En-De and En-Vi. The baseline method denotes training without any data augmentation.
Method	En → Tr	En → De	En → Vi
Baseline	15.35	27.54	31.66
+TCWR	17.38	29.37	33.76
+BT	19.24	29.19	33.38
+BT +TCWR	20.19	30.26	35.72
Table 5: The BLEU scores on the testing sets with back-translation and TCWR.
velopment sets. Similarly, we choose the temperature with the highest score on development sets for SwitchOut. For SCDA, we search the replacing probability and set it to 0.15. We follow the official implementation<sup>3</sup> of TDA. We reuse the hyperparameters from Cheng et al. (2019) for DADA.
The results on three language pairs are shown in Table 4. Compared to the baseline with no data augmentation, TCWR yields improvements of 2.03, 1.63 and 1.79 BLEU for En-Tr, En-De and En-Vi, respectively. TCWR also outperforms the other augmentation methods, which further confirms the effectiveness of considering source context, target context, and alignment for NMT data augmentation. Besides, these results demonstrate that TCWR brings consistent improvements to both low-resource and high-resource language pairs.
5.6 Backtranslation Result
As backtranslation is a widely-used data augmentation method by utilizing monolingual data to generate new parallel pairs, we show how TCWR can be used with backtranslation. To perform backtranslation, we use the monolingual sequences from News Crawl 2017, News Crawl 2010 and VNTC<sup>4</sup> for En-Tr, En-De and En-Vi, respectively. Then we perform data augmentation on both training data and backtranslated data. As shown in Table 5, TCWR improves upon backtranslation, demonstrating that TCWR and backtranslation are not mutually exclusive, and TCWR can enhance the performance of backtranslation.
5.7 Machine Translation Robustness
Noisy or non-standard input text (e.g. text with spelling errors and code switching) can cause significant degradation in most NMT systems. We use the WMT'19 English  $\rightarrow$  French robustness dataset for evaluating translation robustness. As the parallel pairs are scarce for this task, we com
<sup>3</sup>https://github.com/marziehf/DataAugmentationNMT
<sup>4</sup>https://github.com/duyvuleo/VNTC
En:	Kosovo is taking a hard look at its privatisation process in light of recurring [complaints / problems].
Tr:	Kosova, tekrar eden [şikayetler / sorunlar] icosğinnda özelleşirme SSEcini incelemeye alyor.
En:	A decade later, we see that the [economy / system] is terribly unstructured.
Tr:	Onytic sonra, [ekonominin / sistemin] yapıninçok kütü bozulduguṇu görtüyoruz.
En:	Report : most [SEE / independent] countries advance in economic freedom.
Tr:	Rapor : [GDA / bagmsiz] ülkerelinin cognu ekonomik özgürlükte ilerliyor.
Table 6: A case study on TCWR, where augmented positions are marked as [original / substituted].
Method	En → Fr
Baseline	26.0
+WordDropout	26.52
+SwitchOut	26.61
+SCDA	26.85
+BPEDropout	27.08
+TDA	27.11
+DADA	28.14
+TCWR	28.92
Table 7: The BLEU scores on the WMT'19 English  $\rightarrow$  French robustness task.
bine its training data with the English  $\rightarrow$  French pairs from Europarl-v7. The models are validated on the development set of the MTNT dataset and tested on the released test set of the WMT'19 robustness task. The results are shown in Table 7. We observe that TCWR outperforms the baseline without any data augmentation or with the other methods. If we regard the task as adapting from the source dataset with clean text (Europarl-v7) to the target dataset with noisy text (WMT'19 robustness dataset), TCWR helps this adaptation via enlarging training examples with language models trained using noisy and non-standard text. We thereby conclude that TCWR can improve NMT robustness.
5.8 Case Study
As shown in Table 6, we perform a case study of TCWR. We observe that TCWR can reasonably substitute words in source sequences based on contexts and modify corresponding target words, which demonstrates the benefits of considering both context and alignment for augmentation.
We proposed a data augmentation method for NMT, which introduces a causal inductive bias that takes both context and alignment into account. The method was shown to improve the performance of translation, backtranslation and translation robustness on four NMT benchmarks.
We would like to thank the anonymous reviewers for their insightful comments. We also thank Chris Dyer and Jiatao Gu for helpful discussions.
Roeaharoni, Melvin Johnson, and Orhan First. 2019. Massively multilingual neural machine translation. arXiv preprint arXiv:1903.00089.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Ondrej Bojar and Ales Tamchyna. 2011. Improving translation model by monolingual data. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 330-336. Association for Computational Linguistics.
Peter F Brown, Vincent J Della Pietra, Stephen A Della Pietra, and Robert L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2):263-311.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709.
Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019. Robust neural machine translation with doubly adversarial inputs. arXiv preprint arXiv:1906.02443.
Yong Cheng, Lu Jiang, Wolfgang Macherey, and Jacob Eisenstein. 2020. Advaug: Robust adversarial augmentation for neural machine translation. arXiv preprint arXiv:2006.11834.
Anna Currey, Antonio Valerio Miceli-Barone, and Kenneth Heafield. 2017. Copied monolingual data improves low-resource neural machine translation. In Proceedings of the Second Conference on Machine Translation, pages 148-156.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. arXiv preprint arXiv:1808.09381.
Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. arXiv preprint arXiv:1705.00440.
Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou, and Tie-Yan Liu. 2019. Soft contextual data augmentation for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5539-5544.
Cyril Goutte, Nicola Cancedda, Marc Dymetman, and George Foster. 2009. Learning machine translation. Massachusetts Institute of Technology.
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In Advances in neural information processing systems, pages 820-828.
Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700-1709.
Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. arXiv preprint arXiv:1805.06201.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. arXiv preprint arXiv:1901.07291.
Chris J Maddison, Daniel Tarlow, and Tom Minka. 2014a. A* sampling. In Advances in Neural Information Processing Systems, pages 3086-3094.
Chris J Maddison, Daniel Tarlow, and Tom Minka. 2014b. A* sampling. NIPS, pages 3086-3094.
CJ Maddison and D Tarlow. 2017. Gumbel machinery.
Paul Michel and Graham Neubig. 2018. Mntt: A testbed for machine translation of noisy text. arXiv preprint arXiv:1809.00388.
Graham Neubig, Taro Watanabe, Eiichiro Sumita, Shin-suke Mori, and Tatsuya Kawahara. 2011. An unsupervised model for joint phrase alignment and extraction. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 632-641.
Toan Q Nguyen and Julian Salazar. 2019. Transformers without tears: Improving the normalization of self-attention. arXiv preprint arXiv:1910.05895.
Mohammad Norouzi, Samy Bengio, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans, et al. 2016. Reward augmented maximum likelihood for neural structured prediction. In Advances In Neural Information Processing Systems, pages 1723-1731.
Michael Oberst and David Sontag. 2019. Counterfactual off-policy evaluation with gumbel-max structural causal models. In International Conference on Machine Learning, pages 4881-4890. PMLR.
Judea Pearl. 2003. Causality: Models, reasoning, and inference. *Econometric Theory*, 19:675-685.
Judea Pearl, Madelyn Glymour, and Nicholas P Jewell. 2016. Causal inference in statistics: A primer. John Wiley & Sons.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
Ofir Press and Lior Wolf. 2016. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859.
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020. Bpe-dropout: Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1882-1892. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Keisuke Sakaguchi, Kevin Duh, Matt Post, and Benjamin Van Durme. 2017. Robsut wrod reocginiton via semi-character recurrent neural network. In Thirty-First AAAI Conference on Artificial Intelligence.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation systems for wmt 16. arXiv preprint arXiv:1606.02891.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Stephan Vogel, Hermann Ney, and Christoph Tillmann. 1996. Hmm-based word alignment in statistical translation. In Proceedings of the 16th conference on Computational linguistics-Volume 2, pages 836-841. Association for Computational Linguistics.
Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018. Switchout: an efficient data augmentation algorithm for neural machine translation. arXiv preprint arXiv:1808.07512.
Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos, and Graham Neubig. 2019. Generalized data augmentation for low-resource translation. arXiv preprint arXiv:1906.03785.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2019. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems, pages 9051-9062.
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. arXiv preprint arXiv:1604.02201.