Venue: NAACL 2022

# Consistency Training with Virtual Adversarial Discrete Perturbation

Jungsoo Park\({}^{1,3}\) Gyuwan Kim\({}^{2}\)+  Jaewoo Kang\({}^{3}\)

\({}^{1}\)Clova AI, NAVER Corp.  \({}^{2}\) University of California, Santa Barbara   \({}^{3}\)Korea University

{jungsoo_park,kangjl}@korea.ac.kr

gyuwankim@ucsb.edu

Equal contributionWork done while working at NAVER Clova & AI Lab

###### Abstract

Consistency training regularizes a model by enforcing predictions of original and perturbed inputs to be similar. Previous studies have proposed various augmentation methods for the perturbation but are limited in that they are agnostic to the training model. Thus, the perturbed samples may not aid in regularization due to their ease of classification from the model. In this context, we propose an augmentation method of adding a discrete noise that would incur the highest divergence between predictions. This virtual adversarial discrete noise obtained by replacing a small portion of tokens while keeping original semantics as much as possible efficiently pushes a training model's decision boundary. Experimental results show that our proposed method outperforms other consistency training baselines with text editing, paraphrasing, or a continuous noise on semi-supervised text classification tasks and a robustness benchmark1.

Footnote 1: Code repo: [https://github.com/clovai/vat-d](https://github.com/clovai/vat-d)

## 1 Introduction

Building a natural language processing (NLP) system often requires an expensive process to collect a massive amount of labeled text data. Semi-supervised learning (SSL) Chapelle et al. (2009) mitigates the requirement for such labeled data by exploiting the structure of unlabeled data. Among the SSL methods, the consistency training framework Laine and Aila (2017); Sajladi et al. (2016) enforces a model to produce similar predictions of original and perturbed inputs. This method has several advantages over other training algorithms such as naively adding augmented samples into the training set Wei and Zou (2019); Ng et al. (2020) in that it provides a richer training signal than a one-hot label, and also applies to both labeled and unlabeled data Xie et al. (2020).

For perturbing a text while preserving its semantics, some approaches inject continuous noise to embedding vectors Xie et al. (2017); Miyato et al. (2018), and others modify text itself in discrete fashion by edit operations Kobayashi (2018); Wei and Zou (2019) or paraphrasing with back-translation Sennrich et al. (2016); Edunov et al. (2018); Xie et al. (2020). However, adding continuous noise might not strongly regularize the training model, compared to diverse discrete noise-based augmentation methods Ebrahimi et al. (2017); Cheng et al. (2019). Also, the augmentations with discrete noise are mostly black-box approaches based on simple rules or fixed models without access to the training model's internal states, having no control over output augmentations that would aid in the regularization of the training model. As seen in Fig. 1 (d), the

Figure 1: A simple illustration of the intuition behind our method is visualized in the two-dimensional space, where the augmented samples (triangle) would aid the training given a limited number of data (circle).



augmented samples with similar semantics but that are outside the training model's decision boundary (_i.e._ adversarial) are the ones that would effectively regularize the model to fit into the complex real data distribution.

To this end, we explore virtual adversarial training with discrete token replacements (_VAT-D_). Our framework (1) first perturbs a given input text by replacing a small subset of tokens to _maximize the divergence between the original and the perturbed samples' model predictions (i.e., virtual adversarial)_ while filtering tokens to replace for constraining the semantic similarity, and (2) train a model to minimize the divergence of the predictions of original and perturbed inputs.

VAT-D shares the advantages of virtual adversarial training (VAT) with continuous noise Miyato et al. (2018) in that the perturbation is model-dependent, changing over the training time to approximate the augmented samples that would effectively push the decision boundary. On the other hand, VAT-D differs from VAT in that the search space is discrete rather than continuous, thus not constrained by the pre-defined norm on the embedding space. Our method relies on the training model's predictions which do not require label information, hence being the first work to successfully apply the adversarial training with perturbation on discrete space to the SSL framework.

Our proposed method empirically outperforms previous state-of-the-art methods on topic classification datasets Chang et al. (2008); Mendes et al. (2012); Zhang et al. (2015) under various SSL scenarios. We additionally conduct experiments on ANLI robustness benchmark dataset Nie et al. (2020) for testing the robustness when only labeled samples are given where the method improves over the RoBERTa-Large Liu et al. (2019) by 8 points.

## 2 Background

We explain the concept of consistency training and VAT that our framework relies on.

Consistency TrainingConsistency training Laine and Aila (2017); Sajjadi et al. (2016) enforces models' predictions to be invariant when the input is perturbed. This regularization pushes the decision boundary to traverse a low-density region Verma et al. (2019). The consistency loss is formally defined as

\[{\mathcal{L}}({\mathbf{x}},{\mathbf{x^{\prime}}})=D[p(\cdot~{}|~{}{\mathbf{ x}}),p(\cdot~{}|~{}{\mathbf{x^{\prime}}})] \tag{1}\]

where \(D\) is a non-negative divergence metric between two probability distributions (_e.g._, KL-divergence), \({\mathbf{x^{\prime}}}\) is a perturbed sample from an input \({\mathbf{x}}\) by any transformation.

Virtual Adversarial TrainingVAT Miyato et al. (2017, 2018) is a consistency training method, which perturbs a given input with continuous noise to maximize the divergence from the model's prediction of the original input. Such virtually adversarial examples effectively smooth the decision boundary compared to the random perturbation Miyato et al. (2018). The formal definition of virtual adversarial samples is \(\hat{\mathbf{x}}=\underset{{\mathbf{x^{\prime}}}\in Neighbor({\mathbf{x}})}{ \operatorname*{argmax}}{\mathcal{L}}({\mathbf{x}},{\mathbf{x^{\prime}}})\) where the training objective is to minimize the \({\mathcal{L}}({\mathbf{x}},\hat{\mathbf{x}})\). Miyato et al. (2017) perturbs input by injecting noise to the embedding space, where the constraint of the perturbation is \(\epsilon\)-ball in \(L^{p}\) norm centered at \({\mathbf{x}}\), _i.e._\(Neighbor({\mathbf{x}})=\{{\mathbf{x^{\prime}}}\mid\lVert{\mathbf{x^{\prime}}}-{ \mathbf{x}}\rVert_{p}\leq\epsilon\}\).

## 3 Method

We aim to generate a perturbed sample by adding discrete noise that incurs the highest divergence of the model's prediction logits from the original one without significant changes in its semantics. Our augmentation is made _on-the-fly_ depending on the current model to push the decision boundary during training effectively.

Virtual Adversarial Discrete NoiseWe develop the consistency training framework by perturbing inputs with virtual adversarial discrete noise, called VAT-D. We want to perturb a given sentence \({\mathbf{x}}=(x_{1},\dots,x_{M})\in V^{M}\) of sequence length \(M\) into a new sentence \({\mathbf{x^{\prime}}}=(x^{\prime}_{1},\dots,x^{\prime}_{M})\in V^{M}\) of the same length, where \(V\) is the word vocabulary. In contrast with the continuous case, we constrain that \({\mathbf{x^{\prime}}}\) differs from \({\mathbf{x}}\) in only small portion of positions changing their surface forms, _i.e._\(Neighbor(x)=\{\lVert{\mathbf{x^{\prime}}}-{\mathbf{x}}\rVert_{H}/M\leq\tau\}\) where \(H\) denotes hamming distance in the token-level and \(\tau\) is the replacement ratio. In this work, we only focus on the replacement for simplicity.

Gradient InformationThe white-box approaches having an access to the training model's internal states, mostly rely on the gradient vectors of the loss function with respect to the input embeddings for finding adversarial discrete noise Ebrahimi et al. (2017). However, for acquiring such gradient information under the framework of 

consistency training as in Eq. 1, naively resorting to the linear approximation of the loss function with respect to the input embeddings like in previous works (Ebrahimi et al., 2017; Michel et al., 2019; Cheng et al., 2019) does not hold since the first-order term from Taylor expansion is zero when the label information is substituted to model's predictions (Miyato et al., 2018).

We bypass the obstacle by sharpening the distribution of original examples' predictions to enable the linear approximation. Sharpening the distribution makes high probabilities higher and lower probabilities lower while not changing their relative order. By sharpening the distribution of the original inputs' predictions, the first-order term does not result in zero, hence can be utilized for the approximation. This is because the modified divergence loss is not zero when \({\bf x}^{\prime}={\bf x}\) indicating the non-negative divergence is not necessarily minimum at \(r={\bf x}^{\prime}-{\bf x}=0\) (Note that the derivative of \(f(x)\) is zero when the \(f(x)\) is minimum at \(x\)). The optimizing objective of Eq. 1 is modified to

\[\tilde{\mathcal{L}}({\bf x},{\bf x}^{\prime})=D[p^{\textit{sharp}}(\cdot~{}|~ {}{\bf x})),p(\cdot~{}|~{}{\bf x}^{\prime})] \tag{2}\]

by sharpening the predicted distribution given an original input by the pre-defined temperature \(T\) as \(p^{\textit{sharp}}(\cdot~{}|~{}{\bf x})=p(\cdot|{\bf x})^{\frac{1}{T}}/\left\lVert p (\cdot~{}|~{}{\bf x})^{\frac{1}{T}}\right\rVert_{1}\).

Virtual Adversarial Token ReplacementConsequently, the optimization problem to find a virtual adversarial discrete perturbation changes to

\[\mathbf{\hat{x}}=\underset{{\bf x}^{\prime}\in Neighbor({\bf x})}{\operatorname {argmax}}\tilde{\mathcal{L}}({\bf x},{\bf x}^{\prime}).\]

Finally, we train the modified consistency loss function from Eq. 2 with obtained discrete perturbation. The replacement operation of \(m\)-th token \(x_{m}\) to the arbitrary token \(x\) can be written as \(\delta(x_{m},x):=e(x)-e(x_{m})\), where \(e(\cdot)\) denotes embedding look-up. We induce a virtual adversarial token by the following criteria (Ebrahimi et al., 2017; Michel et al., 2019; Cheng et al., 2019; Wallace et al., 2019; Park et al., 2020):

\[\hat{x}_{m}=\underset{x\in\text{top\_k}(x_{m},V)}{\operatorname {argmax}}\delta(x_{m},x)^{\top}\cdot g_{x_{m}} \tag{3}\] \[\quad\text{where $g_{x_{m}}=\nabla_{e(x_{m})}\tilde{\mathcal{ L}}({\bf x},{\bf x}^{\prime})|_{{\bf x}^{\prime}={\bf x}}$}\]

\(\mathbf{g}_{x_{m}}\) is the gradient vector of the sharpened consistency loss from Eq. 2 with respect to the \(m\)-th token. In brief, we replace the \(m\)-th original token \(x_{m}\) with one of the candidates \(x\) that approximately maximizes the consistency loss. We randomly select token indexes to perturb and replace them simultaneously. To bound the semantics similarity between the original sentence and the perturbed one, we use a masked language model (MLM) (Devlin et al., 2019; Liu et al., 2019) to restrict a set of possible candidates to replace \(x_{m}\). We filter top-k candidates (Cheng et al., 2019), denoted as \(\text{top\_k}(x_{m},V)\), from the vocabulary having the highest MLM probability at position \(m\) when an original sentence \(x\) is given to the MLM. More training details are in Appendix A.

## 4 Experimental Setup

### Dataset

We experiment on three topic classification datasets and Adversarial NLI (ANLI) (Nie et al., 2020). The former evaluate our method's effectiveness in SSL and the latter is for evaluating the robustness of the models under the standard supervised training framework. The three topic classification benchmarks consist of AG News (Zhang et al., 2015), DBpedia Mendes et al. (2012), and YAHOO! Answers Chang et al. (2008). We follow the experimental setting from Chen et al. (2020a), where we train with a limited number of labeled data in diverse settings, namely, 10, 200, 2500 per class. We randomly sample the labeled, unlabeled, and development set and report the performance on the official test set. For producing the confident results, we report the average of five different seeds' distinct runs.

As for ANLI, we train the model with two different settings, training with only the ANLI dataset or additionally training with other NLI datasets, including SNLI Bowman et al. (2015), MNLI Williams et al. (2017), and FEVER Thorne et al. (2018) following the original work Nie et al. (2020). Further details are in Appendix B.

### Baseline

We compare our method with various baselines of the perturbation methods including EDA Wei and Zou (2019), UDA Xie et al. (2020), VAT Miyato et al. (2017, 2018) for the topic classification SSL task. For the ANLI dataset, we compare with the baselines Devlin et al. (2019); Yang et al. (2019); Liu et al. (2019); Jiang et al. (2020) that have reported numbers on the official validation and test set. More details are in Appendix C 



### Training Details

We exploit the unlabeled data from the topic classification datasets and the labeled data from the ANLI for consistency loss. Throughout the experiments, we set the replacement ratio \(\tau\) as 0.25 and top-k as 10. We sharpen the predictions with \(T\) as 0.5 for topic classification datasets (including baselines) and 0.75 for the ANLI.

## 5 Experimental Results

### Semi Supervised Text Classification

Table 1 shows the experimental results on topic classification datasets under SSL setup. Our method outperforms the baselines by up to 7.1 points from the BERT model finetuned with standard cross-entropy loss and 2.9 points from other methods utilizing the consistency regularization loss. The accuracy gained from the proposed method from the baselines, especially when the number of labeled samples is limited. However, since all the methods have already achieved high accuracy in the DBpedia, the difference among methods is not significant.

Among the baselines, VAT Miyato et al. (2017, 2018) performs reasonably well. The finding supports the claim that a transformation during consistency training should be done with regard to the training model.

### Adversarial Natural Language Inference

Table 2 shows the experimental results on the ANLI dataset with different training settings: training with all the NLI datasets, or training with only the ANLI dataset. Our method improves over baselines, including RoBERTa-Large Liu et al. (2019) and SMART Jiang et al. (2020) in both settings. Specifically, our method improves on an average of 8.0 points in the test set from training with cross-entropy loss only. Compared to SMART, which combines smoothness regularization, i.e., a variation of VAT, and Bregman proximal point optimization for finetuning, our method outperforms it on an average of 1 point from the test set without using other techniques such as Bregman proximal point optimization.

## 6 Effectiveness of the White-box Search

Our central intuition behind the proposed method is to generate the augmented samples concerning the model, i.e., vulnerable to the model. This section further conducts an ablation study on whether such virtual adversarial search is crucial in discrete

\begin{table}
\begin{tabular}{l|c c c|c c c|c c c} \hline \multirow{2}{*}{**Method**} & \multicolumn{3}{c|}{**AG\_NEWS**} & \multicolumn{3}{c|}{**YAHOO!**} & \multicolumn{3}{c}{**DBpedia**} \\  & _10_ & _200_ & _2500_ & _10_ & _200_ & _2500_ & _10_ & _200_ & _2500_ \\ \hline BERT Devlin et al. (2019) & 79.4 & 88.6 & 91.6 & 58.2 & 70.1 & 73.9 & 97.8 & 98.8 & 99.1 \\ EDA Wei and Zou (2019) & 83.8 & 88.9 & 91.8 & 62.0 & 70.6 & 73.8 & 98.4 & 98.8 & 99.1 \\ UDA Xie et al. (2020) & 83.8 & 88.5 & 91.6 & 62.0 & 70.4 & 73.7 & 98.2 & 98.8 & 99.1 \\ VAT Miyato et al. (2017) & 82.3 & 88.9 & 91.8 & 62.4 & 70.7 & 74.1 & 98.4 & 98.8 & 99.1 \\ \hline VAT-D & **86.2** & **90.0** & **92.3** & **65.3** & **71.7** & 74.1 & 98.4 & **99.0** & **99.2** \\ \hline \end{tabular}
\end{table}
Table 1: Accuracy on topic classification datasets under the various SSL settings. 10, 200, 2500 denote the number of labeled samples per class used during training. We average five different runs with a differently indexed dataset to show the significance Dror et al. (2018). The numbers in the bold denote the best score.

\begin{table}
\begin{tabular}{l|c c c c|c c c} \hline \multirow{2}{*}{**Method**} & \multicolumn{3}{c|}{**Dev**} & \multicolumn{3}{c}{**Test**} \\  & A1 & A2 & A3 & ALL & A1 & A2 & A3 & ALL \\ \hline \multicolumn{8}{c}{**MNLI + SNLI + ANLI + FEVER**} \\ \hline BERTNie et al. (2020) & 57.4 & 48.3 & 43.5 & 49.3 & - & - & - & 44.2 \\ XLnet Nie et al. (2020) & 67.6 & 50.7 & 48.3 & 55.1 & - & - & - & 52.0 \\ ROBERTA Nie et al. (2020) & 73.8 & 48.9 & 44.4 & 53.7 & - & - & - & 49.7 \\ SMART Jiang et al. (2020) & **74.5** & 50.9 & 47.6 & 57.1 & **72.4** & 49.8 & **50.3** & 57.1 \\ VAT-D & **74.5** & **54.2** & **50.8** & **59.2** & **72.4** & **51.8** & 49.5 & **57.4** \\ \hline \multicolumn{8}{c}{**ANLI**} \\ \hline RoBERTa Nie et al. (2020) & 71.3 & 43.3 & 43.0 & 51.9 & - & - & - & - \\ SMART Jiang et al. (2020) & 74.2 & 49.5 & 49.2 & 57.1 & **72.4** & 50.3 & 49.5 & 56.9 \\ VAT-D & **74.8** & **52.1** & **51.1** & **58.8** & 72.1 & **51.4** & **51.7** & **57.9** \\ \hline \end{tabular}
\end{table}
Table 2: Accuracy on the ANLI benchmark. The numbers of the baselines are from the original papers Nie et al. (2020); Jiang et al. (2020). The upper section is for training with all the NLI datasets, and the bottom is for training with only the ANLI.



space. We select the token among the top-K candidates that would incur the highest divergence from the model's prediction. We compare with other sampling strategies among the top-K candidates, namely, uniform sampling (Uniform), selecting the token with maximum MLM probability (Argmax), and sampling from MLM probabilities (Sampling). We match other training details except for the sampling strategy for a fair comparison.

Table 3 illustrates the result of comparisons. The virtual adversarial search among candidates outperforms other search strategies in discrete space, especially when the number of labeled samples is limited. The result demonstrates that the virtual adversarial search is indeed the crucial component during perturbation. Furthermore, Fig. 2 shows the indexes that our method selected from top-k distribution (sampling from YAHOO! dataset) during training. The distribution of indexes selected from our method resembles the uniform distribution; however, as in the loss plot from Fig. 2, our method searches for the diverse yet adversarial candidates to the model, i.e., incurring high divergence.

## 7 Related Works

Consistency RegularizationConsistency regularizationLaine and Aila (2017); Sajjadi et al. (2016) has been mainly explored in the context of SSLChapelle et al. (2009); Oliver et al. (2018). A line of research in text-domain Miyato et al. (2017); Clark et al. (2018); Xie et al. (2020); Miyatto et al. (2018); Jiang et al. (2020); Asai and Hajishirzi (2020) explored the idea. Existing studies explored varying perturbation methods. Injecting norm-constrained continuous noise to the embedding space Miyato et al. (2017); Jiang et al. (2020); Liu et al. (2020); Chen et al. (2020); Sato et al. (2019) and directly perturbing the text Clark et al. (2018); Minervini and Riedel (2019); Li et al. (2019); Xie et al. (2020); Asai and Hajishirzi (2020) via discrete noise are the primary approaches for the perturbation. Our method perturbs the sentence by the discrete noise, yet the noise is generated concerning the training model.

Adversarial TrainingOur method extends the white-box-based adversarial training framework Goodfellow et al. (2014); Madry et al. (2018), which has recently been explored widely in NLP Miyato et al. (2017); Ebrahimi et al. (2017); Michel et al. (2019); Wang et al. (2019); Zhu et al. (2020); Jiang et al. (2020); Liu et al. (2020). Cheng et al. (2019) use adversarial training on machine translation by discrete word replacements relying on the label information, so not applicable to SSL different from ours. There are also black-box approaches for generating the adversarial attacks or test sets Jia and Liang (2017); Alzantot et al. (2018); Ribeiro et al. (2018, 2019); Gardner et al. (2020) to evaluate the vulnerability of the NLP models, unlike our method, which utilizes gradient information during training. Li et al. (2020); Garg and Ramakrishnan (2020); Li et al. (2021) perturb input using MLMs similar to ours but designed for an attack so inefficient for adversarial training.

Data AugmentationSynthetically generated training examples are utilized to augment an existing dataset Feng et al. (2021). Existing world-level augmentation methods Zhang et al. (2015); Xie et al. (2017); Wei and Zou (2019) are based on heuristics. Mixup-based methods Zhang et al. (2018) interpolate input texts in hidden embeddings Chen et al. (2020); Guo et al. (2019) or input-level Yoon et al. (2021); Kim et al. (2021). Other methods include utilizing back-translation models Sennrich et al. (2016); Xie et al. (2020), contextual language models Kobayashi (2018); Wu et al. (2019), or generative models Anaby-Tavor et al. (2020); Yang et al. (2020). Unlike previous works, our method is subject to the training model, thus approximating the augmented points, efficiently filling in gaps from the training data.

\begin{table}
\begin{tabular}{l|c c c|c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{3}{c}{**AG\_NEWS**} & \multicolumn{3}{c}{**YAHOO!**} \\  & _10_ & _200_ & _2500_ & _10_ & _200_ & _2500_ \\ \hline VAT-D & **86.2** & **89.8** & **92.3** & **65.3** & **71.7** & **74.1** \\ \hline Uniform & 83.8 & 89.3 & 91.8 & 63.2 & 70.8 & 73.8 \\ Argmax & 83.2 & 89.0 & 91.9 & 63.7 & 70.9 & 73.7 \\ Sampling & 84.8 & 89.3 & 91.8 & 63.5 & 70.9 & 73.8 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Accuracy according to different sampling strategies from top-K candidates.

Figure 2: Index distribution from the top-k candidates sorted by MLM scores (a) and consistency loss of different sampling strategies during training (b).

 



## Acknowledgements

We thank Jinhyuk Lee, Jaewook Kang, and Sungdong Kim for the discussion and feedback on the paper. We also thank the members of the Conversation team in Naver CLOVA for active discussion. This research was supported by National Research Foundation of Korea (NRF-2020R1A2C3010638) and the Ministry of Science and ICT, Korea, under the ICT Creative Consilience program (IITP-2022-2020-0-01819).

## References

* Alzantot et al. (2018) Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In _EMNLP_.
* Anaby-Tavor et al. (2020) Aeteret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020. Do not have enough data? deep learning to the rescue! In _AAAI_.
* Asai and Hajishirzi (2020) Akari Asai and Hannaneh Hajishirzi. 2020. Logic-guided data augmentation and regularization for consistent question answering. In _ACL_.
* Bowman et al. (2015) Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In _EMNLP_.
* Chang et al. (2008) Ming-Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic representation: Dalatess classification. In _AAAI_.
* Chapelle et al. (2009) Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. 2009. Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. _IEEE Transactions on Neural Networks_, 20(3).
* Chen et al. (2020a) Jiaao Chen, Zichao Yang, and Diyi Yang. 2020a. Mixtext: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In _ACL_.
* Chen et al. (2020b) Luoxin Chen, Weitong Ruan, Xinyue Liu, and Jianhua Lu. 2020b. Sequxt: Virtual adversarial training for semi-supervised sequence labeling. In _ACL_.
* Cheng et al. (2019) Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019. Robust neural machine translation with doubly adversarial inputs. In _ACL_.
* Clark et al. (2018) Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc Le. 2018. Semi-supervised sequence modeling with cross-view training. In _EMNLP_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _NAACL-HLT_.
* Dror et al. (2018) Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In _ACL_.
* Ebrahimi et al. (2017) Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2017. Hotflip: White-box adversarial examples for text classification. In _ACL_.
* Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. _arXiv preprint arXiv:1808.09381_.
* Feng et al. (2020) Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for nlp. In _ACL-Findings_.
* Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models' local decision boundaries via contrast sets. In _EMNLP-Findings_.
* Garg and Ramakrishnan (2020) Siddhant Garg and Southam Ramakrishnan. 2020. Bae: Bert-based adversarial examples for text classification. In _EMNLP_.
* Goodfellow et al. (2014) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. In _ICLR_.
* Guo et al. (2019) Hongyu Guo, Yongyi Mao, and Richong Zhang. 2019. Augmenting data with mixup for sentence classification: An empirical study. _arXiv preprint arXiv:1905.08941_.
* Jia and Liang (2017) Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In _EMNLP_.
* Jiang et al. (2020) Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. In _ACL_.
* Kim et al. (2021) Yekyang Kim, Seohyegong Jeong, and Kyunghyun Cho. 2021. Linda: Unsupervised learning to interpolate in natural language processing. _arXiv preprint arXiv:2112.13969_.
* Kobayashi (2018) Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In _NAACL_.
* Laine and Aila (2017) Samuli Laine and Timo Aila. 2017. Temporal ensembling for semi-supervised learning. In _ICLR_.



Dianqi Li, Yizhe Zhang, Hao Peng, Liqun Chen, Chris Brockett, Ming-Ting Sun, and Bill Dolan. 2021. Contextualized perturbation for textual adversarial attack. In _ACL_.
* Li et al. (2020) Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. Bert-attack: Adversarial attack against bert using bert. In _EMNLP_.
* Li et al. (2019) Tao Li, Vivek Gupta, Muitrey Mehta, and Vivek Srikumar. 2019. A logic-driven framework for consistency of neural models. In _EMNLP_.
* Liu et al. (2020) Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. 2020. Adversarial training for large neural language models. _arXiv preprint arXiv:2004.08994_.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Madry et al. (2018) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In _International Conference on Learning Representations_.
* Mendes et al. (2012) Pablo N Mendes, Max Jakob, and Christian Bizer. 2012. Dbpedia: A multilingual cross-domain knowledge base. In _IREC_. Citeseer.
* Michel et al. (2019) Paul Michel, Xian Li, Graham Neubig, and Juan Pino. 2019. On evaluation of adversarial perturbations for sequence-to-sequence models. In _NAACL_.
* Minervini and Riedel (2019) Pasquale Minervini and Sebastian Riedel. 2019. Adversarially regularising neural nli models to integrate logical background knowledge. In _CONLL_.
* Miyato et al. (2017) Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. _ICLR_.
* Miyato et al. (2018) Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. _IEEE Transactions on Pattern Analysis and Machine Intelligence_, 41(8).
* Ng et al. (2020) Nathan Ng, Kyunghyun Cho, and Marzyeh Ghassemi. 2020. Ssmba: Self-supervised manifold based data augmentation for improving out-of-domain robustness. In _EMNLP_.
* Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language understanding. In _ACL_.
* Oliver et al. (2018) Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. 2018. Realistic evaluation of deep semi-supervised learning algorithms. In _NeurIPS_.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _NAACL-HLT: Demonstrations_.
* Park et al. (2020) Jungsoo Park, Mujeen Sung, Jinhyuk Lee, and Jaewoo Kang. 2020. Adversarial subword regularization for robust neural machine translation. In _EMNLP_.
* Ribeiro et al. (2019) Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019. Are red roses red? evaluating consistency of question-answering models. In _ACL_.
* Ribeiro et al. (2018) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In _ACL_.
* Sajjadi et al. (2016) Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. 2016. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In _NeurIPS_.
* Sato et al. (2019) Motoki Sato, Jun Suzuki, and Shun Kiyono. 2019. Effective adversarial regularization for neural machine translation. In _ACL_.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In _ACL_.
* Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In _NAACL_.
* Verma et al. (2019) Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. 2019. Interpolation consistency training for semi-supervised learning. _IJCAI_.
* Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. In _EMNLP_.
* Wang et al. (2019) Dilim Wang, Chengyue Gong, and Qiang Liu. 2019. Improving neural language modeling via adversarial training. In _ICLR_.
* Wei and Zou (2019) Jason Wei and Kai Zou. 2019. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In _EMNLP_.
* Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. In _NAACL_.
* Wu et al. (2019) Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. Conditional bert contextual augmentation. In _International Conference on Computational Science_. Springer.
* Xie et al. (2020) Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmentation for consistency training. _NeurIPS_, 33 

Ziang Xie, Sida I Wang, Jiwei Li, Daniel Levy, Aiming Nie, Dan Jurafsky, and Andrew Y Ng. 2017. Data noising as smoothing in neural network language models. In _ICLR_.
* Yang et al. (2020)Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. G-daug: Generative data augmentation for commonsense reasoning. In _EMNLP-Findings_.
* Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In _NeurIPS_.
* Yoon et al. (2021) Soyoung Yoon, Gyuwan Kim, and Kyumin Park. 2021. Ssmix: Saliency-based span mixup for text classification. In _ACL-Findings_, pages 3225-3234.
* Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In _ICLR_.
* Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In _NeurIPS_.
* Zhu et al. (2020) Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Thomas Goldstein, and Jingjing Liu. 2020. Freelb: Enhanced adversarial training for language understanding. In _ICLR_.

 



## Appendix A Training Details

Alg. 1 illustrates the procedure (VAT_D) to acquire virtual adversarial tokens with the modified consistency loss. We randomly select token indexes to perturb \(I\), subject to the length of the sentence. Considering multiple substitutions, an exhaustive search over all possible combinations to find the optimal one is computationally intractable. For efficient generation during each training step, we replace multiple tokens simultaneously instead of greedy search or beam search, which has shown to work considerably well in previous works Ebrahimi et al. (2017); Cheng et al. (2019). During training, the models are optimized with standard cross-entropy and consistency loss with an equal weight where we utilize KL-Divergence as the divergence \(D\). Our method takes approximately 2.5 times the standard training whereas other baselines (e.g., EDA, Back-translation) take about 1.7 times the standard training. We utilize P40 for training the SSL experiments and V100 for the ANLI task.

In our preliminary experiment, utilizing the MLM with masking was worse than that without masking, similar to Li et al. (2020). While utilizing the MLM for filtering top-k candidates, we empirically verified that not applying masking operations to the sentence achieved better performance than doing so. We conjecture that the loss of information when applying masking operation has evoked the perturbed samples to significantly deviate from the original ones, resulting in a degradation in performance. The finding matches that of Li et al. (2020). Thus we do not apply masking operations throughout the experiments. Moreover, we do not fine-tune the _off-the-shelf_ MLM on the training corpus but only the classification model, which is to ensure a fair comparison with other augmentation baselines.

```
Input :input sentence \({\bf x}\), index to perturb \({\bf I}\) Output :perturbed sentence \(\hat{\bf x}\) FunctionVAT_D(\({\bf x}\), \({\bf I}\)): \(\hat{\bf x}\leftarrow{\bf x}\)for\(m\in{\bf I}\)do \(g_{x_{m}}\leftarrow\nabla_{e(x_{m})}\tilde{\mathcal{L}}({\bf x},{\bf x^{ \prime}})|_{{\bf x^{\prime}}={\bf x}}\) \(\hat{x}_{m}\leftarrow\underset{x\in\text{top.b}({\bf x},X(x_{m},V)}{\operatorname {argmax}}\delta(x_{m},x)^{\top}\cdot g_{x_{m}}\) Replace \(m\)-th token of \(\hat{\bf x}\) to \(\hat{x}_{m}\) return\(\hat{\bf x}\)
```

**Algorithm 1**VAT_D Module

## Appendix B Further Details on Data

The dataset statistics and split information regarding topic classification tasks and ANLI is presented in Table B.1 and Table B.2.

ANLI Nie et al. (2020) is an NLI testbed recently introduced for evaluating the robustness of the models in natural language understanding. The dataset consists of three rounds (A1-A3), each consisting of a train-dev-test set with increasing difficulty, where the data is generated by human-and-model-in-the-loop fashion to fool the strong pre-trained models Devlin et al. (2019); Yang et al. (2019); Liu et al. (2019).

## Appendix C Further Details on Baselines

For the SSL setup, we use the following baselines:

BERT Devlin et al. (2019)We use the pretrained BERT-base-uncased model and finetune it for the classification dataset using only standard cross-entropy loss.

EDA Wei and Zou (2019)EDA is a simple data augmentation strategy based on word unit operations such as synonym replacement or deletion. We perturb the unlabeled samples using EDA2 and exploit them for consistency training.

Footnote 2: [https://github.com/jasonwei20/eda_nlp](https://github.com/jasonwei20/eda_nlp)

UDA Xie et al. (2020)UDA paraphrases the sentence using the back-translation. We employ the WMT-19 DE\(\leftrightarrow\)EN model from fairseq3Ott et al. (

2019) to do the back-translation on unlabeled samples, and exploit them for consistency training.

VAT (**Miyato et al., 2017, 2018**)We re-implement VAT where we apply the consistency loss to the unlabeled samples.

## Appendix D Augmentation Quality

We present some augmentation samples in Table D.1 from three topic-classification datasets. As presented in the table, the augmentation samples moderately modify some tokens from the original sentence following the original context.

However, since we are decoding multiple tokens at a same time, some samples are shown to be ungrammatical (e.g., _is_\(\rightarrow\)_will_ instead of _will__be_). Moreover, if the chosen token to be modified are entities, the augmentation sample can sometimes change the information presented in the sentence (e.g., _Patryk Dominik_\(\rightarrow\)_Patryk Deinik_). However, since we are solving the task of the closed-domain topic classification task, the problems didn't matter much in this setting. If we are to solve the knowledge-intensive task, we would have to consider other filtering modules for not changing the entities 



\begin{table}
\begin{tabular}{l|p{284.5pt}} \hline Source & Sample \\ \hline AG\_NEWS & \\ Original & Turkey agonized over pressure to recognize cyprus in the list hurdle to an historic agreement \\ Augmentation & Turkey agonized over pressure to recognize cyprus in the final hurdle of an historic deal \\ Original & Rockets struck a baghdad hotel housing foreign contractors and journalists late thursday \\ Augmentation & Rockets hit a baghdad hotel housing visiting contractors and journalists late thursday \\ Original & Ten people were injured yesterday when a bomb exploded outside the Indonesian emphasys in Paris \\ Augmentation & Ten civilians were injured yesterday when a bomb exploded outside the Jakarét embassy in Paris \\ Original & Pakistan authorities are putting the city of Karachi on – for an al - qaida strike after its forces killed l top terro suspect \\ Augmentation & Pakistan government are putting the city of karachi off – for an al - qaida bond after its members killed gone top terrorism suspect \\ \hline YAHOO! & \\ Original & How can guests get sound security under wireless internet environment at hotel \\ Augmentation & How can visitors got sound security under wireless internet environment at hotel \\ Original & Cai you find some ones screen name by using there real manm???yes \\ Augmentation & Could you find other ones screen name by using three real surname??yes : yes \\ Original & What is the perfect gift for my girlfriends b day???(information here about her)?she ow to ride your black sport bike \\ Augmentation & What will the perfect gift for my girlfriends b- day???(information here here of her)?she wants to : ride your black racing bike \\ Original & Purpose of administration and it department to a business?iwork in §...without us, companies would be at a standstill \\ Augmentation & Purpose of administration and it department to a coprotation?iworkin that without us or companies would be near a stands market \\ \hline DBpedia & \\ Original & Patryk Dominant Szyrber ( born i august 1979 in Opoczo) stage name seth is a Polish heavy metal musician \\ Augmentation & Patryk Denink Szyrbor ( birth i august 1979 in Opoczno) stage name senth is a Warsaw heavy metal musician \\ Original & Twill is a quarterly magazine published between Paris and Milan. It has an international leadership \\ Augmentation & Twil is the quarterly magazine printed between Paris and Milan. It has an international readers range \\ Original & The pond crack station located east of Wallace Kansas – is a two - story frame building that was a stagecoach station built 1865 \\ Augmentation & The lake branch station built outside to Wallace Kansas —is a two - story frame building that been a staggered station designed 1865 \\ Original & Until we have wings is an album by randy stonehill released in 1990 on myrin records \\ Augmentation & Until we have wings is an album by randy stonehill published mid 1989 on myrin records \\ \hline \end{tabular}
\end{table}
Table D.1: Generated augmentation examples from our method along with original samples