SlowGuess/NAACL_2021_OCR huggingface dataset:
values and BLEU scores on the validation set for each training time when we applied each perturbation to Transformer (big). In addition, Figure 2 (c) shows the time required to achieve the BLEU score of Transformer w/o perturbation on the validation set (26.60, as described in Table 3). These figures show that ADV requires twice as much time or more relative to other methods to achieve performance comparable to others. In NLL curves, REP(UNI), REP(SIM), and WDROP achieved better values than those of Transformer w/o perturbation in the early stage. In addition, WDROP was the fastest to achieve better NLL value. Figure 2 (c) indicates that REP(UNI), REP(SIM), and WDROP achieved 26.60 BLEU score with smaller training time than that of Transformer w/o perturbation.
These results indicate that we can quickly improve the performance of Transformer with REP(UNI), REP(SIM), and WDROP. In particu
lar, when we prepare a large number of parameters for Transformer in machine translation, it is better to use these methods (and their combinations) as perturbations. We conduct more experiments to investigate whether these methods are also superior in other configurations.
4.2 High Resource
We compare each perturbation in the case where we have a large amount of training data.
Datasets We add synthetic parallel data generated from the German monolingual corpus using back-translation (Sennrich et al., 2016a) to the training data used in Section 4.1. The origin of the German monolingual corpus is NewsCrawl 2015-2018<sup>5</sup>. We randomly sampled 5M sentences from each NewsCrawl corpus, and thus, obtained 20M sentences in total. We back-translated the corpus
5data.statmt.org/news-crawl/de/
Method	Positions	2010	2011	2012	2013	2014	2015	2016	Average
w/o perturbation	-	25.63	23.62	24.54	28.39	31.50	32.96	36.47	29.02
REP(UNI)	both	26.36	24.18	25.14	28.54	32.35	33.80	37.73	29.73
REP(SIM)	both	26.04	23.79	25.01	28.43	32.06	33.28	37.40	29.43
WDROP	both	26.65	24.34	25.18	28.66	32.25	33.75	37.65	29.78
REP(UNI)+WDROP	both	26.45	24.07	25.09	28.72	32.21	33.42	37.68	29.66
REP(SIM)+WDROP	both	26.55	24.20	25.19	28.55	31.92	33.64	37.96	29.72
REP(SS)	dec	25.81	23.64	24.73	28.46	31.84	33.29	36.59	29.19
ADV	both	25.79	24.07	24.92	28.64	32.04	33.35	37.20	29.43
with the German-English translation model, which is identical to Transformer (big) (w/o perturbation) used in Section 4.1 except for the direction of translation. Finally, we pretended a special token  $\langle \mathrm{BT}\rangle$  to the beginning of the source (English) side of the synthetic data following (Caswell et al., 2019). In addition, we upsampled the original bitext to adjust the ratio of the original and synthetic bittexts to 1:1.
Methods In this setting, we increase the parameter size of Transformer from the (big) setting to take advantage of large training data. Specifically, we increased the internal layer size of the FFN part from 4096 to 8192, and used 8 layers for both the encoder and decoder. The other hyper-parameters are same as in Section 4.1.
Results Table 4 shows BLEU scores of each method when we used a large amount of training data. This table indicates that all perturbations outperformed Transformer w/o perturbation in all test sets. Moreover, the fast methods REP(UNI), REP(SIM), WDROP, and their combinations achieved the same or better averaged scores than REP(SS) and ADV. Thus, these methods are not only fast but also significantly improve the performance of Transformer. In particular, since Table 3 shows that REP(UNI) and WDROP barely have any negative effect on the computational time, we consider them as superior methods.
4.3 Low Resource
Datasets We also conduct an experiment on a low resource setting. We used IWSLT 2014 German-English training set which contains 160k sentence pairs. We followed the preprocessing described in fairseq $^{6}$  (Ott et al., 2019). We used dev2010, 2012, and tst2010-2012 as a test set.
Methods In this setting, we reduced the parameter size of Transformer from the (base) setting. We reduced the internal layer size of the FFN part from
Table 4: BLEU scores of each method trained with a large amount of data.
Method	Position	BLEU
w/o perturbation	-	35.22
REP(UNI)	both	35.53
REP(SIM)	both	35.49
WDROP	both	35.64
REP(SS)	dec	35.49
ADV	both	36.09
×2 training steps
REP(UNI)	both	35.81
REP(SIM)	both	35.96
WDROP	both	36.06
REP(UNI)+WDROP	both	36.20
REP(SIM)+WDROP	both	36.22
Table 5: BLEU scores in the low resource setting.
2048 to 1024. We used the same values for other hyper-parameters as in Section 4.1.
Results Table 5 shows BLEU scores of each method on the low resource setting. We trained three models with different random seeds for each method, and reported the averaged scores. In this table, we also report the results of REP(UNI), REP(SIM), WDROP, and their combinations trained with twice the number of updates (below  $\times 2$  training steps). This table shows that all perturbations also improved the performance from Transformer w/o perturbation. In contrast to Tables 3 and 4, ADV achieved the top score when each model was trained with the same number of updates.
However, as reported in Section 4.1, ADV requires twice or more as long as other perturbations for training. Thus, when we train Transformer with other perturbations with twice the number of updates, the training time is almost equal. In the comparison of (almost) equal training time, WDROP achieved a comparable score to ADV. Moreover,  $\mathrm{REP}(\mathrm{UNI}) + \mathrm{WDROP}$  and  $\mathrm{REP}(\mathrm{SIM}) + \mathrm{WDROP}^7$  outperformed ADV. Thus, in this low resource setting,  $\mathrm{REP}(\mathrm{UNI}) + \mathrm{WDROP}$  and  $\mathrm{REP}(\mathrm{SIM}) + \mathrm{WDROP}$  are slightly better than ADV in computational time.
In the low resource setting, we applied only WDROP to an encoder side for REP(UNI)  $^+$  WDROP and REP(SIM)  $^+$  WDROP because the configuration achieved better performance than applying both perturbations to both sides.
$^{6}$ github.com/pytorch/fairseq/tree/master/examples/translation
Method	0.00	0.01	0.05	0.10
w/o perturbation	26.88	25.81	21.94	17.80
REP(UNI)	27.57	26.58	23.14	18.93
REP(SIM)	27.37	26.95	25.34	23.13
WDROP	27.52	26.48	22.84	18.56
REP(UNI)+WDROP	27.51	26.55	23.18	19.05
REP(SIM)+WDROP	27.57	27.15	25.60	23.58
REP(SS)	26.99	25.89	22.08	17.93
ADV	27.34	26.32	22.43	18.08
4.4 Results on Perturbed Inputs
Recent studies have used perturbations, especially adversarial perturbations, to improve the robustness of encoder-decoders (Sato et al., 2019; Cheng et al., 2019; Wang et al., 2019). In particular, Cheng et al. (2019) analyzed the robustness of models trained with their adversarial perturbations over perturbed inputs. Following them, we also investigate the robustness of our trained Transformer (big) models.
We constructed perturbed inputs by replacing words in source sentences based on pre-defined ratio. If the ratio is 0.0, we use the original source sentences. In contrast, if the ratio is 1.0, we use the completely different sentences as source sentences. We set the ratio 0.01, 0.05, and 0.10. In this process, we replaced a randomly selected word with a word sampled from vocabulary based on uniform distribution. We applied this procedure to source sentences in newstest2010-2016.
Table 6 shows averaged BLEU scores of each method on perturbed newstest2010-2016. These BLEU scores are calculated against the original reference sentences. This table indicates that all perturbations improved the robustness of the Transformer (big) because their BLEU scores are better than one in the setting w/o perturbation. In comparison among perturbations, REP(SIM) (and REP(SIM)+WDROP) achieved significantly better scores than others on perturbed inputs. We emphasize that REP(SIM) surpassed ADV even though ADV is originally proposed to improve the robustness of models. This result implies that REP(SIM) is effective to construct robust models as well as to improve the performance.
Table 6: Averaged BLEU scores on newstest2010-2016 when we inject perturbations to a source sentence with each ratio.
Method	Pos	Valid	Test	CoNLL
w/o perturbation	-	47.25	64.74	61.62
REP(UNI)	both	47.77	64.67	62.22
REP(SIM)	both	47.58	64.51	62.29
WDROP	both	48.53	65.47	62.22
REP(UNI)+WDROP	both	48.58	65.94	62.33
REP(SIM)+WDROP	both	48.72	65.97	62.29
REP(SS)	dec	47.84	65.18	62.30
ADV	both	48.17	65.90	62.23
Kiyono et al. (2020)	-	-	65.0	62.2
Table 7:  $\mathrm{F}_{0.5}$  scores of each method. The row of Kiyono et al. (2020) represents the reported scores of the model trained with the same configuration.
5 Experiments on GEC
Datasets Following Kiyono et al. (2020), we used a publicly available dataset from the BEA shared task (Bryant et al., 2019). This dataset contains training, validation, and test splits. We also used the CoNLL-2014 test set (CoNLL) (Ng et al., 2014) as an additional test set. We report  $\mathrm{F}_{0.5}$  score measured by the ERRANT scorer (Bryant et al., 2017; Felice et al., 2016) for the BEA dataset and  $M^2$  scorer (Dahlmeier and Ng, 2012) for CoNLL.
Methods We used the same settings as Kiyono et al. (2020). Specifically, we trained Transformer (big) model w/o perturbation on the same parallel pseudo data provided by Kiyono et al. (2020), and then fine-tuned the model with perturbations. The hyper-parameters for perturbations are the same as those described in Section 4.1.
Results Table 7 shows the results of each method. This table reports the averaged score of five models trained with different random seeds. Moreover, we present the scores of Kiyono et al. (2020); our "w/o perturbation" model is a rerun of their work, that is, the experimental settings are identical<sup>9</sup>.
Table 7 shows that all perturbations improved the scores except for REP(UNI) and REP(SIM) in the BEA test set (Test). Similar to the machine translation results, the simple methods WDROP, REP(UNI)+WDROP, and REP(SIM)+WDROP achieved comparable scores to ADV. Thus, these faster methods are also effective for the GEC task.
Word Replacement The naive training method of neural encoder-decoders has a discrepancy between training and inference; we use the correct
8For more details, Tables 10, 11, and 12 in Appendix show BLEU scores on each perturbed input.
The scores of w/o perturbation are slightly worse than Kiyono et al. (2020). We consider that this is due to randomness in the training procedure.
tokens as inputs of the decoder in the training phase but use the token predicted at the previous time step as an input of the decoder in the inference phase. To address this discrepancy, Bengio et al. (2015) proposed the scheduled sampling that stochastically uses the token sampled from the output probability distribution of the decoder as an input instead of the correct token. Zhang et al. (2019) modified the sampling method to improve the performance. In addition, Duckworth et al. (2019) refined the algorithm to be suited to Transformer (Vaswani et al., 2017). Their method is faster than the original scheduled sampling but slower and slightly worse than more simple replacement methods such as REP(UNI) and REP(SIM) in our experiments. Xie et al. (2017) and Kobayashi (2018) used the unigram language model and neural language model respectively to sample tokens for word replacement. In this study, we ignored contexts to simplify the sampling process, and indicated that such simple methods are effective for sequence-to-sequence problems.
Word Dropout Gal and Ghahramani (2016) applied word dropout to a neural language model and it is a common technique in language modeling (Merit et al., 2018; Yang et al., 2018; Takase et al., 2018). Sennrich and Zhang (2019) reported that word dropout is also effective for low resource machine translation. However, word dropout has not been commonly used in the existing sequence-to-sequence systems. Experiments in this study show that word dropout is not only fast but also contributes to improvement of scores in various sequence-to-sequence problems.
Adversarial Perturbations Adversarial perturbations were first discussed in the field of image processing (Szegedy et al., 2014; Goodfellow et al., 2015). In the NLP field, Miyato et al. (2017) applied adversarial perturbations to an embedding space and reported its effectiveness on text classification tasks. In sequence-to-sequence problems, Wang et al. (2019) and Sato et al. (2019) applied adversarial perturbations to embedding spaces in neural encoder-decoders. Moreover, Sato et al. (2019) used virtual adversarial training (Miyato et al., 2016) in training of their neural encoder-decoders. Cheng et al. (2019) computed token-level adversarial perturbations in machine translation. In other words, they introduced the strategy of adversarial perturbations into word replacement.
Their method is also effective but requires more computational time than Wang et al. (2019) and Sato et al. (2019) because it runs language models to obtain candidate tokens for perturbations.
We compared perturbations for neural encoder-decoders in view of computational time. Experimental results show that simple techniques such as word dropout (Gal and Ghahramani, 2016) and random replacement of input tokens achieved comparable scores to sophisticated perturbations: the scheduled sampling (Bengio et al., 2015) and adversarial perturbations (Sato et al., 2019), even though those simple methods are faster. In the low resource setting in machine translation, adversarial perturbations achieved high BLEU score but those simple methods also achieved comparable scores to the adversarial perturbations when we spent almost the same time for training. For the robustness of trained models, REP(SIM) is superior to others. This study indicates that simple methods are sufficiently effective, and thus, we encourage using such simple perturbations as a first step. In addition, we hope for researchers of perturbations to use the simple perturbations as baselines to make the usefulness of their proposed method clear.
We thank Motoki Sato for sharing his code with us to compare adversarial perturbations. We thank Jun Suzuki and Sosuke Kobayashi for their valuable comments. We thank anonymous reviewers for their useful suggestions. This work was supported by JSPS KAKENHI Grant Number JP18K18119 and JST ACT-X Grant Number JPMJAX200I. The first author is supported by Microsoft Research Asia (MSRA) Collaborative Research Program.
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Muller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (WMT), pages 1-61.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances
in Neural Information Processing Systems 28 (NIPS), pages 1171-1179.
Christopher Bryant, Mariano Felice, Øistein E. Anderen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52-75.
Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 793-805.
Isaac Caswell, Ciprian Chelba, and David Grangier. 2019. Tagged back-translation. In Proceedings of the Fourth Conference on Machine Translation (WMT), pages 53-63.
Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019. Robust neural machine translation with doubly adversarial inputs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4324-4333.
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better Evaluation for Grammatical Error Correction. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 568-572.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems 32 (NeurIPS), pages 13063-13075.
Daniel Duckworth, Arvind Neelakantan, Ben Goodrich, Lukasz Kaiser, and Samy Bengio. 2019. Parallel scheduled sampling. CoRR, abs/1906.04331.
Mariano Felice, Christopher Bryant, and Ted Briscoe. 2016. Automatic Extraction of Learner Errors in ESL Sentences Using Linguistically Enhanced Alignments. In Proceedings of the 26th International Conference on Computational Linguistics (COLING), pages 825-835.
Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In Advances in Neural Information Processing Systems 29 (NIPS), pages 1019-1027.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of the 3rd International Conference on Learning Representations (ICLR).
Jianshu Ji, Qinlong Wang, Kristina Toutanova, Yongen Gong, Steven Truong, and Jianfeng Gao. 2017. A nested attention neural hybrid model for grammatical error correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 753-762.
Shun Kiyono, Jun Suzuki, Tomoya Mizumoto, and Kentaro Inui. 2020. Massive exploration of pseudo data for grammatical error correction. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2134-2145.
Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 452-457.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E. Gonzalez. 2020. Train large, then compress: Rethinking model size for efficient training and inference of transformers. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 11432-11442.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and Optimizing LSTM Language Models. In Proceedings of the 6th International Conference on Learning Representations (ICLR).
Takeru Miyato, Andrew M. Dai, and Ian Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. In Proceedings of the 5th International Conference on Learning Representations (ICLR).
Takeru Miyato, Shin ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. 2016. Distributional smoothing with virtual adversarial training. In Proceedings of the 4th International Conference on Learning Representations (ICLR).
Courtney Naples, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated Gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKEX), pages 95-100.
Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 Shared Task on Grammatical Error Correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning (CoNLL), pages 1-14.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairoseq: A fast, extensible toolkit for sequence modeling.* In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)*, pages 48-53.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation (WMT), pages 1-9.
Paul Over, Hoa Dang, and Donna Harman. 2007. Duc in context. Information Processing & Management, 43(6):1506-1520.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation (WMT), pages 186-191.
Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. ProphetNet: Predicting future n-gram for sequence-to-SequencePre-training. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 2401–2410.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A Neural Attention Model for Abstractive Sentence Summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 379-389.
Motoki Sato, Jun Suzuki, and Shun Kiyono. 2019. Effective adversarial regularization for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 204-210.
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2019. Green AI. CoRR, abs/1907.10597.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 86-96.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1715-1725.
Rico Sennrich and Biao Zhang. 2019. Revisiting low-resource neural machine translation: A case study. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 211-221.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pretraining for language generation. In International Conference on Machine Learning (ICML), pages 5926-5936.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 3645-3650.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems 27 (NIPS), pages 3104-3112.
Jun Suzuki and Masaaki Nagata. 2017. Cutting-off redundant repeating generations for neural abstractive summarization. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 291-297.