SlowGuess/ACL_2020_OCR huggingface dataset:
Finally, the training objective in our AdvAug is a combination of the two losses:
$$
\boldsymbol {\theta} ^ {*} = \underset {\boldsymbol {\theta}} {\operatorname {a r g m i n}} \left\{\mathcal {L} _ {a u t} (\boldsymbol {\theta}) + \mathcal {L} _ {a d v} (\boldsymbol {\theta}) \right\}. \tag {17}
$$
Here, we omit two bidirectional language model losses for simplicity, which are used to recommend word candidates to maintain semantic similarities (Cheng et al., 2019).
In practice, we need to compute the loss via minibatch training. For the  $P_{\text{out}}$ , we follow the pair sampling inside each mini-batch in mixup. It can avoid padding too much tokens because sentences of similar lengths are grouped within a mini-batch (Vaswani et al., 2017). For the  $P_{\text{adv}}$ , we sample a pair of examples from  $A_{(\mathbf{x},\mathbf{y})}$  for each  $(\mathbf{x},\mathbf{y})$  and cover the distribution over multiple training epochs. The entire procedure to calculate the translation losses,  $\mathcal{L}_{\text{adv}}(\boldsymbol{\theta})$  and  $\mathcal{L}_{\text{out}}(\boldsymbol{\theta})$ , is presented in Algorithm 1. In a nutshell, for each batch of training examples, we firstly sample virtual examples from  $P_{\text{adv}}$  and  $P_{\text{out}}$  by interpolating the embeddings of the adversarial or authentic training examples. Then we calculate the translation loss using their interpolated embeddings.
Method	Loss Config.	MT06	MT02	MT03	MT04	MT05	MT08
Vaswani et al. (2017)	\( \mathcal{L}_{clean} \)	44.57	45.49	44.55	46.20	44.96	35.11
Miyato et al. (2017)	-	45.28	45.95	44.68	45.99	45.32	35.84
Sato et al. (2019)	-	45.75	46.37	45.02	46.49	45.88	35.90
Cheng et al. (2019)	-	46.95	47.06	46.48	47.39	46.58	37.38
Sennrich et al. (2016a)*	-	46.39	47.31	47.10	47.81	45.69	36.43
Edunov et al. (2018)*	-	46.20	47.78	46.93	47.80	46.81	36.79
Ours	\( \mathcal{L}_{mixup} \)	45.12	46.32	44.81	46.61	46.08	36.00
\( \mathcal{L}_{aut} \)	46.73	46.79	46.13	47.54	46.88	37.21
\( \mathcal{L}_{clean} + \mathcal{L}_{adv} \)	47.89	48.53	48.73	48.60	48.76	39.03
\( \mathcal{L}_{aut} + \mathcal{L}_{adv} \)	49.26	49.03	47.96	48.86	49.88	39.63
Ours*	\( \mathcal{L}_{aut} + \mathcal{L}_{adv} \)	49.98	50.34	49.81	50.61	50.72	40.45
Table 1: Baseline comparison on NIST Chinese-English translation. * indicates the model uses extra corpora and - means not elaborating on its training loss.
We verify our approach on translation tasks for three language pairs: Chinese-English, English-French, and English-German. The performance is evaluated with the 4-gram BLEU score (Papineni et al., 2002) calculated by the multi-bleu.perl script. We report case-sensitive tokenized BLEU scores for English-French and English-German, and case-insensitive tokenized BLEU scores for Chinese-English. Note that all reported BLEU scores in our approach are from a single model rather than averaging multiple models (Vaswani et al., 2017).
For the Chinese-English translation task, the training set is the LDC corpus consisting of 1.2M sentence pairs. The NIST 2006 dataset is used as the validation set, and NIST 02, 03, 04, 05, 08 are used as the test sets. We apply byte-pair encoding (BPE) (Sennrich et al., 2016b) with 60K merge operations to build two vocabularies comprising 46K Chinese sub-words and 30K English sub-words. We use the IWSLT 2016 corpus for English-French translation. The training corpus with 0.23M sentence pairs is preprocessed with the BPE script with 20K joint operations. The validation set is test2012 and the test sets are test2013 and test2014. For English-German translation, we use the WMT14 corpus consisting of 4.5M sentence pairs. The validation set is newtest2013 whereas the test set is newtest2014. We build a shared vocabulary of 32K sub-words using the BPE script.
We implement our approach on top of the Transformer (Vaswani et al., 2017). The size of the hidden unit is 512 and the other hyperparameters
are set following their default settings. There are three important hyperparameters in our approach,  $\alpha$  in the Beta distribution and the word replacement ratio of  $\gamma_{src} \in \xi_{src}$ , and  $\gamma_{tgt} \in \xi_{tgt}$  detailed in Eq. (4). Note that  $\gamma_{src}$  and  $\gamma_{tgt}$  are not new hyperparameters but inherited from (Cheng et al., 2019). We tune these hyperameters on the validation set via a grid search, i.e.  $\alpha \in \{0.2, 0.4, 4, 8, 32\}$ ,  $\gamma_{src} \in \{0.10, 0.15, 0.25\}$  and  $\gamma_{tgt} \in \{0.10, 0.15, 0.30, 0.5\}$ . For the mixup loss  $\mathcal{L}_{\text{mixup}}$ ,  $\alpha$  is fixed to 0.2. For the loss  $\mathcal{L}_{\text{aut}}$  and  $L_{\text{adv}}$ , the optimal value of  $\alpha$  is 8.0. The optimal values of  $(\gamma_{src}, \gamma_{tgt})$  are found to be (0.25, 0.50), (0.15, 0.30) and (0.15, 0.15) for Chinese-English, English-French and English-German, respectively, while it is set to (0.10, 0.10) only for back-translated sentence pairs.  $\beta$  in Eq. (14) is set to 250K, 100K, 1M for Chinese-English, English-French and English-German. Unlike Cheng et al. (2019), we remove the learning of target language models to speed up the training. For each training batch, we introduce a batch of augmented adversarial examples and a batch of augmented authentic examples, which costs twice the vanilla training. For constructing adversarial examples, we solely compute the gradients for word embeddings which takes little time. After summing up the time of all steps, our total training time is about 3.3 times the vanilla training.
Chinese-English Translation. Table 1 shows results on the Chinese-English translation task, in comparison with the following six baseline methods. For a fair comparison, we implement all these
Method	Loss Config.	English-French	English-German
test2013	test2014	newstest13	newstest14
Vaswani et al. (2017)	\( \mathcal{L}_{clean} \)	40.78	37.57	25.80	27.30
Sato et al. (2019)	-	41.67	38.72	25.97	27.46
Cheng et al. (2019)	-	41.76	39.46	26.34	28.34
Ours	\( \mathcal{L}_{mixup} \)	40.78	38.11	26.28	28.08
\( \mathcal{L}_{aut} \)	41.49	38.74	26.33	28.58
\( \mathcal{L}_{aut} + \mathcal{L}_{adv} \)	43.03	40.91	27.20	29.57
Table 2: Results on IWSLT16 English-French and WMT14 English-German translation.
methods using the Transformer backbone or report results from those papers on the same corpora.
1. The seminal Transformer model for NMT (Vaswani et al., 2017).
2. Following Miyato et al. (2017), we use adversarial learning to add continuous gradient-based perturbations to source word embeddings and extend it to the Transformer model.
3. Sato et al. (2019) leverage Miyato et al. (2017)'s idea into NMT by incorporating gradient-based perturbations to both source and target word embeddings and optimize the model with adversarial training.
4. Cheng et al. (2019) generate discrete adversarial examples guided by the gradients of word embeddings. Adversarial examples are used to both attack and defend the NMT model.
5. Sennrich et al. (2016a) translate monolingual corpora using an inverse NMT model and then augment the training data with them.
6. Based on Sennrich et al. (2016a), Edunov et al. (2018) propose three improved methods to generate back-translated data, which are sampling, top10 and beam+noise. Among those, we choose beam+noise as our baseline method, which can be regarded as an approach to incorporating noise into data.
We first verify the importance of different translation losses in our approach. We find that both  $\mathcal{L}_{\text{out}}$  and  $\mathcal{L}_{\text{adv}}$  are useful in improving the Transformer model.  $\mathcal{L}_{\text{adv}}$  is more important and yields a significant improvement when combined with the standard empirical loss  $\mathcal{L}_{\text{clean}}$  (cf. Eq. (1)). These results validate the effectiveness of augmenting with virtual adversarial examples. When we use both  $\mathcal{L}_{\text{out}}$  and  $\mathcal{L}_{\text{adv}}$  to train the model, we obtain the best performance (up to 4.92 BLEU points on MT05). We also compare with the mixup loss. However,  $\mathcal{L}_{\text{mixup}}$  is only slightly better than the
standard empirical loss  $\mathcal{L}_{\text{clean}}$ .
Compared with the baseline methods without using extra corpora, our approach shows significant improvements over the state-of-the-art models. In particular, the superiority of  $\mathcal{L}_{clean} + \mathcal{L}_{adv}$  over both Cheng et al. (2019) and Sato et al. (2019) verifies that we propose a more effective method to address adversarial examples in NMT. We also directly incorporate two adversarial examples to NMT models without interpolating their embeddings, but we do not observe any further gain over Cheng et al. (2019). This substantiates the superior performance of our approach on the standard data sets.
To compare with the approaches using extra monolingual corpora, we sample 1.25M English sentences from the Xinhua portion of the GIGA-WORD corpus and list our performance in the last row of Table 1. When the back-translated corpus is incorporated, our approach yields further improvements, suggesting our approach complements the back-translation approaches.
English-French and English-German Translation. Table 2 shows the comparison with the Transformer model (Vaswani et al., 2017), Sato et al. (2019) and Cheng et al. (2019) on English-French and English-German translation tasks. Our approach consistently outperforms all three baseline methods, yielding significant 3.34 and 2.27 BLEU point gains over the Transformer on the English-French and English-German translation tasks, respectively. We also conduct similar ablation studies on the translation loss. We still find that the combination of  $\mathcal{L}_{adv}$  abd  $\mathcal{L}_{aut}$  performs the best, which is consistent with the findings in the Chinese-English translation task. The substantial gains on these two translation tasks suggest the potential applicability of our approach to more language pairs.
Input	但（但是）协议执行过程一波三折，致使和平进程一再受挫
Reference	however, implementation of the deals has witnessed ups and downs, resulting in continuous setbacks in the peace process
Vaswani et al. on Input	however, the process of implementing the agreement was full of twists and turns, with the result that the peace process suffered setbacks again and again.
on Noisy Input	the process of the agreement has caused repeated setbacks to the peace process.
Ours on Input	however, the process of implementing the agreement experienced twists and turns, resulting in repeated setbacks in the peace process.
on Noisy Input	however, the process of implementing the agreement experienced twists and turns, resulting in repeated setbacks in the peace process.
Table 3: Translation Examples of Transformer and our model for an input and its adversarial input.
Loss	α =
0.2	0.4	4	8	32
Lmixup	45.28	45.38	45.64	45.09	-
Laut	45.95	45.92	46.70	46.73	46.54
Lclean+Ladv	47.06	46.88	47.60	47.89	47.81
Table 4: Effect of  $\alpha$  on the Chinese-English validation set. “-” indicates that the model fails to converge.
Method	0.00	0.05	0.10	0.15
Vaswani et al.	44.59	41.54	38.84	35.71
Miyato et al.	45.11	42.11	39.39	36.44
Sato et al.	45.75	44.04	41.25	38.78
Cheng et al.	46.95	44.20	41.71	39.89
Ours	49.26	47.53	44.71	41.76
Table 5: Results on artificial noisy inputs. The column lists results for different noise fractions.
4.3 Effect of  $\alpha$
The hyperparameter  $\alpha$  controls the shape of the Beta distribution over interpolation weights. We study its effect on the validation set in Table 4. Notable differences occur when  $\alpha < 1$  and  $\alpha > 1$ , this is because the Beta distribution show two different shapes with  $\alpha = 1$  as a critical point. As we see, both  $\mathcal{L}_{\text{out}}$  and  $\mathcal{L}_{\text{adv}}$  prefer a large  $\alpha$  and perform better when  $\alpha = 8$ . Recall that when  $\alpha$  is large,  $m_{\lambda}$  behaves similarly to a simple average function. In  $\mathcal{L}_{\text{mixup}}$ ,  $\alpha = 4$  performs slightly better, and a large  $\alpha = 32$  will fail the model training. Although the result with  $\alpha = 4$  appears to be slightly better, it consumes more iterations to train the model to reach the convergence, i.e.,  $90\mathrm{K}$  for  $\alpha = 4$  vs.  $20\mathrm{K}$  for  $\alpha = 0.2$ . These indicate the differences between the proposed vicinity distributions and the one used in mixup.
Figure 2: BLEU scores over iterations on the Chinese-English validation set.
4.4 Robustness to Noisy Inputs and Overfitting
To test robustness on noisy inputs, we follow Cheng et al. (2019) to construct a noisy data set by randomly replacing a word in each sentence of the standard validation set with a relevant alternative. The relevance between words is measured by the similarity of word embeddings. 100 noisy sentences are generated for each of them and then re-scored to pick the best one with a bidirectional language model. Table 5 shows the results on artificial noisy inputs with different noise levels. Our approach shows higher robustness over all baseline methods across all noise levels.
Figure 2 shows the evolution of BLEU scores during training. For  $\mathcal{L}_{clean}$ , the BLEU score reaches its peak at about 20K iterations, and then the model starts overfitting. In comparison, all of the training losses proposed in this paper are capable of resisting overfitting: in fact, even after 100K iterations, no significant regression is observed (not
shown in this figure). At the same iteration, our results are consistently higher than both the empirical risk  $(\mathcal{L}_{clean})$  and mixup  $(\mathcal{L}_{mixup})$ .
As shown in Table 3, the baseline yields an incorrect translation possibly because the word "danshi(但是)" seldom occurs in this context in our training data. In contrast, our model incorporates embeddings of virtual sentences that contain "danshi(但是)" or its synonym "dan(但)". This encourages our model to learn to push their embeddings closer during training, and make our model more robust to small perturbations in real sentences.
Data Augmentation. Data augmentation is an effective method to improve machine translation performance. Existing methods in NMT may be divided into two categories, based upon extra corpora (Sennrich et al., 2016a; Cheng et al., 2016; Zhang and Zong, 2016; Edunov et al., 2018) or original parallel corpora (Fadaee et al., 2017; Wang et al., 2018; Cheng et al., 2019). Recently, mixup (Zhang et al., 2018) has become a popular data augmentation technique for semi-supervised learning (Berthelot et al., 2019) and overcoming real-world noisy data (Jiang et al., 2019). Unlike prior works, we introduce a new method to augment the representations of the adversarial examples in sequence-to-sequence training of the NMT model. Even without extra monolingual corpora, our approach substantially outperforms the widely-used back-translation methods (Sennrich et al., 2016a; Edunov et al., 2018). Furthermore, we can obtain even better performance by including additional monolingual corpora.
Robust Neural Machine Translation. It is well known that neural networks are sensitive to noisy inputs (Szegedy et al., 2014; Goodfellow et al., 2014), and neural machine translation is no exception. Thus improving the robustness of NMT models has become a popular research topic (e.g., Belinkov and Bisk, 2018; Sperber et al., 2017; Ebrahimi et al., 2018; Cheng et al., 2018, 2019; Karpukhin et al., 2019; Li et al., 2019). Many of these studies focus on augmenting the training data to improve robustness, especially with adversarial examples (Ebrahimi et al., 2018; Cheng et al., 2019; Karpukhin et al., 2019; Michel et al., 2019). Others also tried to deal with this issue by finding better input representations (Durrani et al., 2019), adding adversarial regularization (Sato et al., 2019)
and so on. In contrast to those studies, we propose the vicinity distribution defined in a smooth space by interpolating discrete adversarial examples. Experimental results show substantial improvements on both clean and noisy inputs.
We have presented an approach to augment the training data of NMT models by introducing a new vicinity distribution defined over the interpolated embeddings of adversarial examples. To further improve the translation quality, we also incorporate an existing vicinity distribution, similar to mixup for observed examples in the training set. We design an augmentation algorithm over the virtual sentences sampled from both of the vicinity distributions in sequence-to-sequence NMT model training. Experimental results on Chinese-English, English-French and English-German translation tasks demonstrate the capability of our approach to improving both translation performance and robustness.
Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations.
Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. 2019. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint arXiv:1905.02249.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. 2001. Vicinal risk minimization. In Advances in neural information processing systems, pages 416-422.
Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019. Robust neural machine translation with doubly adversarial inputs. In Association for Computational Linguistics.
Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. 2018. Towards robust neural machine translation. In Association for Computational Linguistics.
Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Semi-supervised learning for neural machine translation. In Association for Computational Linguistics.
Nadir Durrani, Fahim Dalvi, Hassan Sajjad, Yonatan Belinkov, and Preslav Nakov. 2019. One size does not fit all: Comparing nmt representations of different granularities. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Javid Ebrahimi, Daniel Lowd, and Dejing Dou. 2018. On adversarial examples for character-level neural machine translation. In Proceedings of COLING.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In *Empirical Methods in Natural Language Processing*.
Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Association for Computational Linguistics.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In International Conference on Machine Learning.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems.
Lu Jiang, Di Huang, and Weilong Yang. 2019. Synthetic vs real: Deep learning on controlled noise. arXiv preprint arXiv:1911.09781.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. 2018. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning.
Vladimir Karpukhin, Omer Levy, Jacob Eisenstein, and Marjan Ghazvininejad. 2019. Training on synthetic noise improves robustness to natural noise in machine translation. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019).
Xian Li, Paul Michel, Antonios Anastasopoulos, Yonatan Belinkov, Nadir Durrani, Orhan Firat, Philipp Koehn, Graham Neubig, Juan Pino, and Hassan Sajjad. 2019. Findings of the first shared task on machine translation robustness. arXiv preprint arXiv:1906.11943.
Paul Michel, Xian Li, Graham Neubig, and Juan Pino. 2019. On evaluation of adversarial perturbations for sequence-to-sequence models. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. In International Conference on Learning Representations.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. 2016. Distributional smoothing with virtual adversarial training. In International Conference on Learning Representations.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Association for Computational Linguistics.
Motoki Sato, Jun Suzuki, and Shun Kiyono. 2019. Effective adversarial regularization for neural machine translation. In Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Association for Computational Linguistics.
Matthias Sperber, Jan Niehues, and Alex Waibel. 2017. Toward robust neural machine translation for noisy input sequences. In International Workshop on Spoken Language Translation.
Christian Szegedy, Wojciech Zaremba, Sutskever Ilya, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In International Conference on Machine Learning.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems.
Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018. Switchout: an efficient data augmentation algorithm for neural machine translation. In Empirical Methods in Natural Language Processing.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations.
Jiajun Zhang and Chengqing Zong. 2016. Exploiting source-side monolingual data in neural machine translation. In Empirical Methods in Natural Language Processing.