SlowGuess/NAACL_2022_OCR huggingface dataset:
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In ACL.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2017. Hotflip: White-box adversarial examples for text classification. In ACL.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. arXiv preprint arXiv:1808.09381.
Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Edward Hovy. 2021. A survey of data augmentation approaches for nlp. In ACL-Findings.
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models' local decision boundaries via contrast sets. In EMNLP-Findings.
Siddhant Garg and Goutham Ramakrishnan. 2020. Bae: Bert-based adversarial examples for text classification. In EMNLP.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. In ICLR.
Hongyu Guo, Yongyi Mao, and Richong Zhang. 2019. Augmenting data with mixup for sentence classification: An empirical study. arXiv preprint arXiv:1905.08941.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In EMNLP.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. Smart: Robust and efficient fine-tuning for pretrained natural language models through principled regularized optimization. In ACL.
Yekyung Kim, Seohyeong Jeong, and Kyunghyun Cho. 2021. Linda: Unsupervised learning to interpolate in natural language processing. arXiv preprint arXiv:2112.13969.
Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In NAACL.
Samuli Laine and Timo Aila. 2017. Temporal ensembling for semi-supervised learning. In ICLR.
Dianqi Li, Yizhe Zhang, Hao Peng, Liquin Chen, Chris Brockett, Ming-Ting Sun, and Bill Dolan. 2021. Contextualized perturbation for textual adversarial attack. In ACL.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. Bert-attack: Adversarial attack against bert using bert. In EMNLP.
Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. 2019. A logic-driven framework for consistency of neural models. In EMNLP.
Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. 2020. Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations.
Pablo N Mendes, Max Jakob, and Christian Bizer. 2012. Dbpedia: A multilingual cross-domain knowledge base. In LREC. CiteSeer.
Paul Michel, Xian Li, Graham Neubig, and Juan Pino. 2019. On evaluation of adversarial perturbations for sequence-to-sequence models. In NAACL.
Pasquale Minervini and Sebastian Riedel. 2019. Adversarily regularising neural nli models to integrate logical background knowledge. In CONLL.
Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. *ICLR*.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8).
Nathan Ng, Kyunghyun Cho, and Marzyeh Ghassemi. 2020. Ssmba: Self-supervised manifold based data augmentation for improving out-of-domain robustness. In EMNLP.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language understanding. In ACL.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. 2018. Realistic evaluation of deep semi-supervised learning algorithms. In NeurIPS.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In *NAACL-HLT: Demonstrations*.
Jungsoo Park, Mujeen Sung, Jinhyuk Lee, and Jaewoo Kang. 2020. Adversarial subword regularization for robust neural machine translation. In EMNLP.
Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019. Are red roses red? evaluating consistency of question-answering models. In ACL.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In ACL.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tas-dizen. 2016. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In NeurIPS.
Motoki Sato, Jun Suzuki, and Shun Kiyono. 2019. Effective adversarial regularization for neural machine translation. In ACL.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In ACL.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In NAACL.
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. 2019. Interpolation consistency training for semi-supervised learning. IJI-CAI.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. In EMNLP.
Dilin Wang, Chengyue Gong, and Qiang Liu. 2019. Improving neural language modeling via adversarial training. In ICLR.
Jason Wei and Kai Zou. 2019. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In EMNLP.
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. In *NAACL*.
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. Conditional bert contextual augmentation. In International Conference on Computational Science. Springer.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmentation for consistency training. NeurIPS, 33.
Ziang Xie, Sida I Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y Ng. 2017. Data-noising as smoothing in neural network language models. In *ICLR*.
Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. G-daug: Generative data augmentation for commonsense reasoning. In EMNLP-Findings.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS.
Soyoung Yoon, Gyuwan Kim, and Kyumin Park. 2021. Ssmix: Saliency-based span mixup for text classification. In ACL-Findings, pages 3225-3234.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In *ICLR*.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NeurIPS.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Thomas Goldstein, and Jingjing Liu. 2020. Freelb: Enhanced adversarial training for language understanding. In ICLR.
Algorithm 1: VAT_D Module
Input :input sentence x, index to perturb I
Output: perturbed sentence  $\hat{\mathbf{x}}$
Function VAT_D(x, I):
$\hat{\mathbf{x}}\gets \mathbf{x}$
for  $m\in \mathbf{I}$  do
$$
g _ {x _ {m}} \gets \nabla_ {e (x _ {m})} \tilde {\mathcal {L}} (\mathbf {x}, \mathbf {x} ^ {\prime}) | _ {\mathbf {x} ^ {\prime} = \mathbf {x}}
$$
$$
\hat{x}_{m}\gets \operatorname *{argmax}_{x\in \operatorname {top}\_ k(x_{m},V)}\delta (x_{m},x)^{\top}\cdot g_{x_{m}}
$$
$$
\text {R e p l a c e} \hat {\mathbf {x}} \text {t o} \hat {\mathbf {x}} _ {m}
$$
return  $\hat{\mathbf{x}}$
Alg. 1 illustrates the procedure (VAT_D) to acquire virtual adversarial tokens with the modified consistency loss. We randomly select token indexes to perturb  $I$ , subject to the length of the sentence. Considering multiple substitutions, an exhaustive search over all possible combinations to find the optimal one is computationally intractable. For efficient generation during each training step, we replace multiple tokens simultaneously instead of greedy search or beam search, which has shown to work considerably well in previous works (Ebrahimi et al., 2017; Cheng et al., 2019). During training, the models are optimized with standard cross-entropy and consistency loss with an equal weight where we utilize KL-Divergence as the divergence  $D$ . Our method takes approximately 2.5 times the standard training whereas other baselines (e.g., EDA, Back-translation) take about 1.7 times the standard training. We utilize P40 for training the SSL experiments and V100 for the ANLI task.
In our preliminary experiment, utilizing the MLM with masking was worse than that without masking, similar to Li et al. (2020). While utilizing the MLM for filtering top-k candidates, we empirically verified that not applying masking operations to the sentence achieved better performance than doing so. We conjecture that the loss of information when applying masking operation has evoked the perturbed samples to significantly deviate from the original ones, resulting in a degradation in performance. The finding matches that of Li et al. (2020). Thus we do not apply masking operations throughout the experiments. Moreover, we do not fine-tune the off-the-shelf MLM on the training corpus but only the classification model, which is to ensure a fair comparison with other augmentation baselines.
Dataset	Genre	Class	Unlabel	Dev	Test
AG_NEWS	News	4	20k	20k	19k
YAHOO!	QA	10	50k	20k	60k
DBPedia	Wikipedia	14	70k	20k	50k
Table B.1: Data statistics for the topic classification datasets following the experimental setting from Chen et al. (2020a).
Dataset	Genre	Train	Dev	Test
A1	Wikipedia	17k	1k	1k
A2	Wikipedia	45k	1k	1k
A3	Various	100k	1.2k	1.2k
ANLI	Various	162k	3.2k	3.2k
MNLI	Various	392k	-	-
Fever	Wikipedia	208k	-	-
SNLI	Image Captions	549k	-	-
Table B.2: Data statistics for the ANLI with three rounds (A1-A3) and concerning NLI datasets for the training.
B Further Details on Data
The dataset statistics and split information regarding topic classification tasks and ANLI is presented in Table B.1 and Table B.2.
ANLI (Nie et al., 2020) is an NLI testbed recently introduced for evaluating the robustness of the models in natural language understanding. The dataset consists of three rounds (A1-A3), each consisting of a train-dev-test set with increasing difficulty, where the data is generated by human-and-model-in-the-loop fashion to fool the strong pretrained models (Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019).
C Further Details on Baselines
For the SSL setup, we use the following baselines:
BERT (Devlin et al., 2019) We use the pretrained BERT-base-uncased model and finetune it for the classification dataset using only standard cross-entropy loss.
EDA (Wei and Zou, 2019) EDA is a simple data augmentation strategy based on word unit operations such as synonym replacement or deletion. We perturb the unlabeled samples using  $\mathrm{EDA}^2$  and exploit them for consistency training.
UDA (Xie et al., 2020) UDA paraphrases the sentence using the back-translation. We employ the WMT-19 DE  $\leftrightarrow$  EN model from fairseq $^3$  (Ott et al.,
2https://github.com/jasonwei20/eda_nlp
<sup>3</sup>https://github.com/pytorch/fairseq
2019) to do the back-translation on unlabeled samples, and exploit them for consistency training.
VAT (Miyato et al., 2017, 2018) We re-implement VAT where we apply the consistency loss to the unlabeled samples.
D Augmentation Quality
We present some augmentation samples in Table D.1 from three topic-classification datasets. As presented in the table, the augmentation samples moderately modify some tokens from the original sentence following the original context.
However, since we are decoding multiple tokens at a same time, some samples are shown to be ungrammatical (e.g.,  $is \rightarrow will$  instead of will be). Moreover, if the chosen token to be modified are entities, the augmentation sample can sometimes change the information presented in the sentence (e.g., Patryk Dominik → Patryk Deinik). However, since we are solving the task of the closed-domain topic classification task, the problems didn't matter much in this setting. If we are to solve the knowledge-intensive task, we would have to consider other filtering modules for not changing the entities.
Source	Sample
AG_NEWS
Original	Turkey agonized over pressure to recognize cyprus in the last hurdle to an historic agreement
Augmentation	Turkey agonized over pressure to recognize cyprus in the final hurdle of an historic deal
Original	Rockets struck a baydah hotel housing foreign contractors and journalists late thursday
Augmentation	Rockets hit a baydah hotel housing visiting contractors and journalists late thursday
Original	Ten people were injured yesterday when a bomb exploded outside the Indonesian embassy in Paris
Augmentation	Ten civilians were injured yesterday when a bomb exploded outside the Jakarta embassy in Paris
Original	Pakistan authorities are putting the city of Karachi on ... for an al - qaida strike after its forces killed a top terror suspect
Augmentation	Pakistan governments are putting the city of karachi of ... for an al - qaida bomb after its members killed one top terrorism suspect
YAHOO!
Original	How can guests get sound security under wireless internet environment at hotel ?
Augmentation	How can visitors get sound security under wireless internet environment at hotel ...
Original	Can you find some ones screen name by using there real name ? ? ? yes
Augmentation	Could you find other ones screen name by using there real surname ? : ? yes
Original	What is the perfect gift for my girlfriends b - day ? ? ? (information in here about her) ? she loves to : ride your black sport bike
Augmentation	What will the perfect gift for my girlfriends b - day ? ? ? (information here here of her) ? she wants to : ride your black racing bike
Original	Purpose of administration and it department to a business ? i work in it ... without us , companies would be at a standstill
Augmentation	Purpose of administration and it department to a corporation ? i work in that ... without us of companies would be near a stands market
DBpedia
Original	Patryk Dominik Sztyber ( born 4 august 1979 in Opoczno ) stage name seth is a Polish heavy metal musician
Augmentation	Patryk Deinik Sztybor ( birth 8 august 1979 in Opoczno ) stage name seth is a Warsaw heavy metal musician
Original	Twill is a quarterly magazine published between Paris and Milan. It has an international readership
Augmentation	Twill is the quarterly magazine printed between Paris and Milan. It has an international readers range
Original	The pond creek station located east of Wallace Kansas ... is a two - story frame building that was a stagecoach station built 1865
Augmentation	The lake branch station built outside to Wallace Kansas ...is a two - story frame building that been a stagecoach station designed 1865
Original	Until we have wings is an album by randy stonehill released in 1990 on myrrh records
Augmentation	Until we have wings is an album by randy stonehill published mid 1989 on myrrh records
Table D.1: Generated augmentation examples from our method along with original samples