SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 6: SimCut is more robust to small perturbations in an authentic context. SimCut captures the translation of "in spielen" under the noisy input while the vanilla Transformer ignores the translation of "inspielen" due to the replacement of "denken" with "festgelegten".
Method	de→en
Transformer	32.4
mBART	38.5
mBART with SimCut	39.3
Table 7: SimCut achieves better performance on IWSLT14 de→en translation task compared with the standard finetuning approach based on mBART.
et al., 2020) as the backbone model, which is a sequence-to-sequence denoising auto-encoder pretrained on CC25 Corpus<sup>3</sup>. We conduct experiments on IWSLT14 de→en dataset and only remove the duplicated sentence pairs following mBART50 (Tang et al., 2021) in the data preprocessing step. The source and target sentences are jointly tokenized into sub-word units with the 250K Sentence-Piece (Kudo and Richardson, 2018) vocabulary of mBART. We use case-sensitive sacreBLEU (Post, 2018) to evaluate the translation quality, and the methods applied in the experiments are as follows:
- Transformer: The Transformer model is randomly initialized and trained from scratch. We utilize the same model and training configurations discussed in Section 3.
- mBART: The Transformer model is directly finetuned from mBART. We utilize the default training configurations of mBART.
- mBART with SimCut: The Transformer model is finetuned from mBART with SimCut regularization. We utilize the default training configurations of mBART.
From Table 7 we can see that SimCut could further improve the translation performance of mBART,
<sup>3</sup>https://github.com/pytorch/fairseq/tree/main/examples/msbart
Method	en→de	de→en
Transformer	28.70	34.99
Bi-Pretrain	28.94	35.64
+ Finetune	28.82	35.66
Bi-R-Drop Pretrain	30.30	37.01
+ R-Drop Finetune	30.85	37.55
Bi-SimCut Pretrain	30.57	37.70
+ SimCut Finetune	31.16	38.37
which again shows the effectiveness and universality of our method.
4.3 Training Strategy: Bidirectional Pretrain and Unidirectional Finetune
Bidirectional pretraining is shown to be very effective to improve the translation performance of the unidirectional NMT system (Ding et al., 2021; Xu et al., 2021). The main idea is to pretrain a bidirectional NMT model at first and use it as the initialization to finetune a unidirectional NMT model. Assume we want to train an NMT model for "English  $\rightarrow$  German", we first reconstruct the training sentence pairs to "English  $+$  German  $\rightarrow$  German  $+$  English", where the training dataset is doubled. We then firstly train a bidirectional NMT model with the new training sentence pairs:
$$
\underset {(\mathbf {x}, \mathbf {y}) \in \mathcal {S}} {\mathbb {E}} [ \ell (f (\mathbf {x}, \mathbf {y}; \theta), \ddot {\mathbf {y}}) + \ell (f (\mathbf {y}, \mathbf {x}; \theta), \ddot {\mathbf {x}}) ], \tag {11}
$$
and finetune the model with "English  $\rightarrow$  German" direction. We follow the same training strategy in Ding et al. (2021) and apply SimCut regularization to both pretraining and finetuning procedures. Table 8 shows that bidirectional pretraining and unidirectional finetuning strategy with SimCut regularization could achieve superior performance compared with strong baseline such as R-Drop.
Comparison with Existing Methods We summarize the recent results of several existing works on IWSLT14 en←de benchmark in Table 9. The existing methods vary from different aspects, including Virtual Adversarial Training (Sato et al., 2019), Mixed Tokenization for NMT (Wu et al., 2020), Unified Dropout for the Transformer model (Wu et al., 2021), Regularized Dropout (Liang et al.,
Table 8: Bidirectional pretrain and unidirectional fine-tune results on IWSLT14 en  $\leftrightarrow$  de datasets. Note that the results of bidirectional pretrain are from one model for dual-directional translations.
Method	en→de	de→en	Average
Transformer	28.70	34.99	31.85
VAT	29.45	35.52	32.49
Mixed Rep†	29.93	36.41	33.17
UniDrop†	29.99	36.88	33.44
R-Drop	30.73	37.30	34.02
BiBERT†	30.45	38.61	34.53
Bi-SimCut	31.16	38.37	34.77
Table 9: Our method achieves the superior performance over the existing methods on the IWSLT14 en  $\leftrightarrow$  de translation benchmark. † denotes the numbers are reported from the corresponding papers, others are based on our runs.
2021), and BiBERT (Xu et al., 2021). We can see that our approach achieves an improvement of 2.92 BLEU score over Vaswani et al. (2017) and surpass the current SOTA method BiBERT that incorporates large-scale pretrained model, stochastic layer selection, and bidirectional pretraining. Given the simplicity of Bi-SimCut, we believe it could be considered as a strong baseline for the NMT task.
5 Standard Resource Scenario
We here investigate the performance of Bi-SimCut on the larger translation benchmark compared with the IWSLT14 benchmark.
5.1 Dataset Description and Model Configuration
For the standard resource scenario, we evaluate NMT models on the WMT14 English-German dataset, which contains 4.5M parallel sentence pairs. We combine newstest2012 and newstest2013 as the validation set and use newstest2014 as the test set. We collect the pre-processed data from Xu et al. (2021)'s release<sup>4</sup>, where a shared dictionary with 52K BPE types is built. We apply a standard Transformer Big model with 6 encoder and decoder layers, 16 attention heads, embedding size 1024, and FFN layer dimension 4096. We apply cross-entropy loss with label smoothing rate 0.1 and set max tokens per batch to be 4096. We use Adam optimizer with Beta (0.9, 0.98), 4000 warmup updates, and inverse square root learning rate scheduler with initial learning rates  $1e^{-3}$ . We decrease the learning rate to  $5e^{-4}$  in the finetuning stage. We select the dropout rate from 0.3, 0.2, and 0.1 based on the validation performance. We
4https://github.com/fe1ixxu/BiBERT
Method	en→de	de→en	Average
Transformer + Large Batch† (Ott et al., 2018)	29.30	-	-
Evolved Transformer† (So et al., 2019)	29.80	-	-
BERT Initialization (12 layers)† (Rothe et al., 2020)	30.60	33.60	32.10
BERT-Fuse† (Zhu et al., 2020)	30.75	-	-
R-Drop (Liang et al., 2021)	30.13	34.54	32.34
BiBERT† (Xu et al., 2021)	31.26	34.94	33.10
SimCut	30.56	34.86	32.71
Bi-SimCut Pretrain	30.10	34.42	32.26
+ SimCut Finetune	30.78	35.15	32.97
use beam search decoding with beam size 4 and length penalty 0.6. We train all models until convergence on 8 NVIDIA Tesla V100 GPUs. All reported BLEU scores are from a single model.
We report test BLEU scores of all comparison methods and our approach on the WMT14 dataset in Table 10. With Bi-SimCut bidirectional pretraining and unidirectional finetuning procedures, our NMT model achieves strong or SOTA BLEU scores on en→de and de→en translation benchmarks. During the NMT training process, we fix  $p_{\mathrm{cut}}$  to be 0.05 and tune the hyper-parameter  $\alpha$  in both R-Drop and SimCut based on the performance on the validation set. Note that the BLEU scores of R-Drop are lower than that reported in Liang et al. (2021). Such gap might be due to the different prepossessing steps used in Liang et al. (2021) and Xu et al. (2021). It is worth mentioning that Bi-SimCut outperforms BiBERT on de→en direction even though BiBERT incorporates bidirectional pretraining, large-scale pretrained contextualized embeddings, and stochastic layer selection mechanism.
6 High Resource Scenario
To investigate the performance of Bi-SimCut on the distant language pairs which naturally do not share dictionaries, we here discuss the effectiveness of Bi-SimCut on the Chinese-English translation task.
6.1 Dataset Description and Model Configuration
For the high resource scenario, we evaluate NMT models on the WMT17 Chinese-English dataset, which consists of 20.2M training sentence pairs,
Table 10: Our method achieves the superior or comparable performance over the existing methods on the WMT14 en←de translation benchmark. † denotes the numbers are reported from Xu et al. (2021), others are based on our runs.
Method	share	zh→en
Transformer	x	25.53
Transformer	✓	25.31
SimCut	x	26.86
SimCut	✓	26.74
Bi-SimCut Pretrain	✓	26.13
+ SimCut Finetune	✓	27.17
Table 11: Our method achieves strong performance on the WMT17 zh→en translation benchmark. share denotes whether a shared dictionary is applied.
and we use newsdev2017 as the validation set and newstest2017 as the test set. We firstly build the source and target vocabularies with 32K BPE types separately and treat them as separated or joined dictionaries in our experiments. We apply the same Transformer Big model and training configurations used in the WMT14 experiments. We use beam search decoding with beam size 5 and length penalty 1. We train all models until convergence on 8 NVIDIA Tesla V100 GPUs. All reported BLEU scores are from a single model.
We report test BLEU scores of the baselines and our approach on the WMT17 dataset in Table 11. Note that share means the embedding matrices for encoder input, decoder input and decoder output are all shared. The NMT models with separated dictionaries perform slightly better than those with the shared dictionary. We can see that our approach significantly improves the translation performance. In particular, Bi-SimCut achieves more than 1.6 BLEU score improvement over Vaswani et al. (2017), showing the effectiveness and univer
sality of our approach on the distant language pair in the NMT task.
Adversarial Perturbation It is well known that neural networks are sensitive to noisy inputs, and adversarial perturbations are firstly discussed in the filed of image processing (Szegedy et al., 2014; Goodfellow et al., 2015). SimCut could be regarded as a perturbation-based method for the robustness research. In the field of natural language processing, Miyato et al. (2017) consider adversarial perturbations in the embedding space and show its effectiveness on the text classification tasks. For the NMT tasks, Sato et al. (2019) and Wang et al. (2019) apply adversarial perturbations in the embedding space during training of the encoder-decoder NMT model. Cheng et al. (2019) leverage adversarial perturbations and generate adversarial examples by replacing words in both source and target sentences. They introduce two additional language models for both sides and a candidate word selection mechanism for replacing words in the sentence pairs. Takase and Kiyono (2021) compare perturbations for the NMT model in view of computational time and show that simple perturbations are sufficiently effective compared with complicated adversarial perturbations.
Consistency Training Besides perturbation-based methods, our approach also highly relates to a few works of model-level and data-level consistency training in the NMT field. Among them, the most representative methods are R-Drop (Liang et al., 2021) and Cutoff (Shen et al., 2020). R-Drop studies the intrinsic randomness in the NMT model and regularizes the NMT model by utilizing the output consistency between two dropout sub-models with the same inputs. Cutoff considers consistency training from a data perspective by regularizing the inconsistency between the original sentence pair and the augmented samples with part of the information within the input sentence pair being dropped. Note that Cutoff takes the dropout sub-models into account during the training procedure as well. We want to emphasize that SimCut is not a new method, but a version of Cutoff simplified and adapted for the NMT tasks.
In this paper, we propose Bi-SimCut: a simple but effective two-stage training strategy to improve
NMT performance. Bi-SimCut consists of bidirectional pretraining and unidirectional finetuning procedures equipped with SimCut regularization for improving the generality of the NMT model. Experiments on low (IWSLT14 en  $\leftrightarrow$  de), standard (WMT14 en  $\leftrightarrow$  de), and high (WMT17 zh  $\rightarrow$  en) resource translation benchmarks demonstrate Bi-SimCut and SimCut's capabilities to improve translation performance and robustness. Given the universality and simplicity of Bi-SimCut and SimCut, we believe: 1) SimCut could be regarded as a perturbation-based method, and it could be used as a strong baseline for the robustness research. 2) Bi-SimCut outperforms many complicated methods which incorporate large-scaled pretrained models or sophisticated mechanisms, and it could be used as a strong baseline for future NMT research. We hope researchers of perturbations and NMT could use SimCut and Bi-SimCut as strong baselines to make the usefulness and effectiveness of their proposed methods clear. For future work, we will explore the effectiveness of SimCut and Bi-SimCut on more sequence learning tasks, such as multilingual machine translation, domain adaptation, text classification, natural language understanding, etc.
We would like to thank the anonymous reviewers for their insightful comments.
Philip Bachman, Ouais Alsharif, and Doina Precup. 2014. Learning with pseudo-ensembles. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3365-3373.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1171-1179.
Guandan Chen, Kai Fan, Kaibo Zhang, Boxing Chen, and Zhongqiang Huang. 2021. Manifold adversarial augmentation for neural machine translation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3184-3189, Online. Association for Computational Linguistics.
Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019. Robust neural machine translation with
doubly adversarial inputs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4324-4333, Florence, Italy. Association for Computational Linguistics.
Liang Ding, Di Wu, and Dacheng Tao. 2021. Improving neural machine translation by bidirectional training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3278-3284, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 1019-1027.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Xiaobo Liang, Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, and TieYan Liu. 2021. R-drop: Regularized dropout for neural networks. In Advances in Neural Information Processing Systems, volume 34, pages 10890-10905. Curran Associates, Inc.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pretraining for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Takeru Miyato, Andrew M. Dai, and Ian J. Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 1-9, Brussels, Belgium. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
Leslie Rice, Eric Wong, and Zico Kolter. 2020. Overfitting in adversarially robust deep learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 8093-8104. PMLR.
Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics, 8:264-280.
Motoki Sato, Jun Suzuki, and Shun Kiyono. 2019. Effective adversarial regularization for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 204-210, Florence, Italy. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. 2020. A simple but tough-to-beat data augmentation approach for natural language understanding and generation. arXiv preprint arXiv:2009.13818.
David R. So, Quoc V. Le, and Chen Liang. 2019. The evolved transformer. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5877-5886. PMLR.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. MASS: masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5926-5936. PMLR.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Sho Takase and Shun Kiyono. 2021. Rethinking perturbations in encoder-decoders for fast training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5767-5780, Online. Association for Computational Linguistics.
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2021. Multilingual translation from de-noising pre-training. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3450-3466, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Dilin Wang, ChengYue Gong, and Qiang Liu. 2019. Improving neural language modeling via adversarial training. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6555-6565. PMLR.
Lijun Wu, Shufang Xie, Yingce Xia, Yang Fan, JianHuang Lai, Tao Qin, and Tie-Yan Liu. 2020. Sequence generation with mixed representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 10388-10398. PMLR.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Zhen Wu, Lijun Wu, Qi Meng, Yingce Xia, Shufang Xie, Tao Qin, Xinyu Dai, and Tie-Yan Liu. 2021. UniDrop: A simple yet effective technique to improve transformer without extra cost. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3865-3878, Online. Association for Computational Linguistics.
Haoran Xu, Benjamin Van Durme, and Kenton Murray. 2021. BERT, mBERT, or BiBERT? a study on contextualized embeddings for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6663-6675, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, Online. Association for Computational Linguistics.
Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2020. Incorporating BERT into neural machine translation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.