SlowGuess/NAACL_2021_OCR huggingface dataset:
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In Proceedings of the 2nd International Conference on Learning Representations (ICLR).
Sho Takase and Sosuke Kobayashi. 2020. All word embeddings from one embedding. In Advances in Neural Information Processing Systems 33 (NeurIPS), pages 3775-3785.
Sho Takase and Naoaki Okazaki. 2019. Positional encoding to control output sequence length. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 3999-4004.
Sho Takase, Jun Suzuki, and Masaaki Nagata. 2018. Direct output connection for a high-rank language model. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4599-4609.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30 (NIPS), pages 5998-6008.
Dilin Wang, Chengyue Gong, and Qiang Liu. 2019. Improving neural language modeling via adversarial training. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 6555-6565.
Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Levy, Aiming Nie, Dan Jurafsky, and Andrew Y. Ng. 2017. Data-noising as smoothing in neural network language models. In Proceedings of the 5th International Conference on Learning Representations (ICLR).
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018. Breaking the softmax bottleneck: A high-rank RNN language model. In Proceedings of the 6th International Conference on Learning Representations (ICLR).
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems 32 (NeurIPS), pages 9054-9065.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning (ICML).
Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. 2019. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4334-4343.
A Experiments on Summarization
We conduct experiments on two summarization datasets: Annotated English Gigaword (Napoles et al., 2012; Rush et al., 2015) and DUC 2004 task 1 (Over et al., 2007).
A.1 Annotated English Gigaword
Datasets We used sentence-summary pairs extracted from Annotated English Gigaword (Napoles et al., 2012; Rush et al., 2015) as the summarization dataset. This dataset contains 3.8M sentence-summary pairs as the training set and 1951 pairs as the test set. We extracted 3K pairs from the original validation set, which contains 190K pairs, for our validation set.
In summarization, most recent studies used large scale corpora to pre-train their neural encoder-decoder (Dong et al., 2019; Song et al., 2019; Zhang et al., 2020; Qi et al., 2020). Thus, we also augmented the training data. We extracted the first sentence and headline of a news article in REALNEWS (Zellers et al., 2019) and News Crawl (Barrault et al., 2019) as sentence-summary pairs. In total, we used 17.1M sentence-summary pairs as our training data.
We used BPE (Sennrich et al., 2016b) to construct a vocabulary set. We set the number of BPE merge operations at  $32\mathrm{K}$  and shared the vocabulary between both the encoder and decoder sides.
Methods We followed the configuration in Section 4.2 because it seems suitable for a large amount of training data. We used the same perturbations and hyper-parameters as in Section 4.2.
Results Table 8 shows the ROUGE  $\mathrm{F}_1$  scores of each method and scores reported in recent studies (Dong et al., 2019; Song et al., 2019; Zhang et al., 2020; Qi et al., 2020) In this experiment, we cannot report the result of ADV because the loss value of ADV exploded during training. We tried several random seeds for ADV, but all models failed to converge. Since we need a huge amount of budget to search more suitable hyper-parameters for ADV in this summarization dataset, we consider that it is impractical to report the result of ADV.
Table 8 indicates that all perturbations improved the ROUGE score. In addition, REP(UNI), REP(SIM), WDROP, and their combinations outperformed the scheduled sampling. Thus, these fast methods are also superior perturbations in the summarization task. Moreover, REP(UNI) and
Method	Position	R-1	R-2	R-L
w/o perturbation	-	39.20	19.84	36.21
REP(UNI)	both	39.81	20.40	36.93
REP(SIM)	both	39.70	20.14	36.77
WDROP	both	39.66	20.45	36.59
REP(UNI)+WDROP	both	39.36	20.13	36.62
REP(SIM)+WDROP	both	39.56	20.14	36.66
REP(SS)	dec	39.20	20.04	36.27
Dong et al. (2019)	-	38.45	19.45	35.75
Song et al. (2019)	-	38.73	19.71	35.96
Zhang et al. (2020)	-	39.12	19.86	36.24
Qi et al. (2020)	-	39.51	20.42	36.69
Table 8:  $F_1$  values of ROUGE-1, 2, and L (R-1, R-2, and R-L respectively) on the test set extracted from Annotated English Gigaword. The lower part represents the scores reported by recent studies.
WDROP outperformed the current top score (Qi et al., 2020) in ROUGE-1, L and ROUGE-2 respectively.
A.2 DUC 2004 Task 1
Datasets We used sentence-summary pairs extracted from Annotated English Gigaword (Napoles et al., 2012; Rush et al., 2015) and News Crawl (Barrault et al., 2019) as our training dataset, which contains 10.1M pairs. Following recent studies (Takase and Okazaki, 2019; Takase and Kobayashi, 2020), we used BPE to construct a vocabulary set for the encoder side and characters as vocabulary for the decoder side. We set the number of BPE merge operations at 16K.
For the test set, we used the DUC 2004 task 1 dataset (Over et al., 2007) which contains 500 source sentences and four kinds of manually constructed reference summaries. We truncated characters over 75 bytes in each generated summary based on the official configuration.
Methods We used the Transformer (big) setting in this experiment. In addition, we introduced the output length control method proposed by Takase and Okazaki (2019). We used the same perturbations and hyper-parameters as in Section 4.1.
Results Table 8 shows recall-based ROUGE scores of each method and scores reported in recent studies (Rush et al., 2015; Suzuki and Nagata, 2017; Takase and Okazaki, 2019; Takase and Kobayashi, 2020). We also cannot report the result of ADV for the same reason as described in Appendix A.1.
This table indicates that REP(SIM), WDROP, and their combination improved the ROUGE scores. In particular, REP(SIM) + WDROP outper
Method	Position	R-1	R-2	R-L
w/o perturbation	-	32.80	11.55	28.26
REP(UNI)	both	32.56	11.48	28.21
REP(SIM)	both	32.80	11.55	28.28
WDROP	both	33.06	11.45	28.51
REP(UNI)+WDROP	both	32.15	11.58	28.01
REP(SIM)+WDROP	both	32.80	11.73	28.46
REP(SS)	dec	32.83	11.41	28.14
Rush et al. (2015)	-	28.18	8.49	23.81
Suzuki and Nagata (2017)	-	32.28	10.54	27.80
Takase and Okazaki (2019)	-	32.29	11.49	28.03
Takase and Kobayashi (2020)	-	32.57	11.63	28.24
Table 9: Recall-based ROUGE-1, 2, and L (R-1, R-2, and R-L respectively) on DUC 2004 task 1 test set. The lower part represents the scores reported by recent studies.
formed the current top score in ROUGE-1, 2, and L. Moreover, WDROP achieved better ROUGE-1 and L scores than the current top score. In contrast, REP(UNI) slightly harmed the performance in this configuration. These results indicate that WDROP and REP(SIM) are also effective for summarization tasks.
Method	Positions	2010	2011	2012	2013	2014	2015	2016	Average
w/o perturbation	-	23.37	21.23	21.89	25.56	26.97	29.34	32.32	25.81
REP(UNI)	both	23.88	21.85	22.48	26.23	27.81	30.21	33.61	26.58
REP(SIM)	both	24.51	22.24	22.82	26.53	28.44	30.43	33.68	26.95
WDROP	both	24.01	21.90	22.48	26.24	27.60	29.71	33.44	26.48
REP(UNI)+WDROP	both	23.85	22.03	22.69	26.63	27.50	29.56	33.60	26.55
REP(SIM)+WDROP	both	24.47	22.61	23.15	26.88	28.24	30.14	34.53	27.15
REP(SS)	dec	23.49	21.18	21.82	25.79	27.17	29.39	32.38	25.89
ADV	both	23.94	21.70	22.46	25.99	27.71	29.28	33.13	26.32
Table 10: BLEU scores when we inject perturbations to a source sentence with 0.01.
Method	Positions	2010	2011	2012	2013	2014	2015	2016	Average
w/o perturbation	-	19.78	18.51	18.70	21.58	22.74	24.81	27.45	21.94
REP(UNI)	both	21.02	19.38	19.67	22.76	23.85	26.08	29.24	23.14
REP(SIM)	both	23.13	21.13	21.60	24.98	26.69	28.38	31.49	25.34
WDROP	both	20.94	19.24	19.44	22.41	23.67	25.42	28.74	22.84
REP(UNI)+WDROP	both	20.99	19.64	19.93	22.89	23.77	25.64	29.40	23.18
REP(SIM)+WDROP	both	23.20	21.55	21.87	25.53	26.50	28.49	32.05	25.60
REP(SS)	dec	20.06	18.58	18.90	21.92	23.01	24.59	27.51	22.08
ADV	both	20.45	18.91	19.10	22.02	23.50	24.97	28.05	22.43
Table 11: BLEU scores when we inject perturbations to a source sentence with 0.05.
Method	Positions	2010	2011	2012	2013	2014	2015	2016	Average
w/o perturbation	-	16.21	15.03	15.31	17.82	17.76	19.91	22.57	17.80
REP(UNI)	both	17.24	15.84	16.39	18.62	19.30	21.45	23.64	18.93
REP(SIM)	both	21.15	19.18	19.79	22.95	23.91	26.19	28.73	23.13
WDROP	both	16.79	15.54	16.06	18.35	18.68	20.57	23.96	18.56
REP(UNI)+WDROP	both	17.53	16.00	16.41	18.95	19.40	21.03	24.01	19.05
REP(SIM)+WDROP	both	21.58	19.86	20.10	23.50	24.22	26.27	29.55	23.58
REP(SS)	dec	16.31	15.21	15.18	18.01	18.11	20.00	22.69	17.93
ADV	both	16.47	15.24	15.50	18.01	18.07	19.84	23.44	18.08
Table 12: BLEU scores when we inject perturbations to a source sentence with 0.10.