SlowGuess/NAACL_2021_OCR huggingface dataset:
As shown in Table 3, CLARE has a significant advantage over TextFooler: in terms of similarity  $56\%$  responses prefer CLARE, while  $16\%$  prefer TextFooler. The trend is similar for fluency & grammaticality (42% vs.  $9\%$ ). This observation is consistent with results from automatic metrics. On label consistency, CLARE slightly underperforms TextFooler at  $68\%$  with a  $95\%$  confidence interval (CI)  $(66\%, 70\%)$ , versus  $70\%$  with a  $95\%$  CI  $(68\%, 73\%)$ . We attribute this to an inherent overlap of some categories in the AG News dataset, e.g., Science & Technology and Business, as evidenced by a  $71\%$  label consistency for original inputs.
Closing this section, Table 4 compares the adversarial examples generated by TextFooler and CLARE. More samples are listed in Appendix A.4.
This section first conducts an ablation study (§4.1). We then explore CLARE's potential to be used to improve downstream models' robustness and accuracy in §4.3. In §4.2, we empirically observe that CLARE tends to attack noun and noun phrases.
We ablate each component of CLARE to study its effectiveness. We evaluate on the 1,000 randomly selected AG news instances (§3.2). The results are summarized in Table 5.
We first investigate the performance of three perturbations when applied individually. Among three editing strategies, using INSERTONLY achieves the best performance, with REPLACEONLY coming a close second. MERGEONLY underperforms the other two, partly because the attacks are restricted to bigram noun phrases (§3.1). Combining all three perturbations, CLARE achieves the best performance with the least modifications.
Merge perturbation can only merge noun phrases, extracted by the NLTK toolkit(https://www.nltk.org/). We find that this helps produce more grammatical outputs.
AG
(Sci&Tech)	Sprint Corp. is in talks with Qualcomm Inc. about using a network the chipmaker is building to deliver live television to Sprint mobile phone customers.
TextFooer
(Business)	Sprint Corps. is in talks with Qualcomm Inc. about operated a network the chipmaker is consolidation to doing viva television to Sprint mobile phone customers.
CLARE
(Business)	Sprint Corp. is in talks with Qualcomm Inc. about using a network Qualcomm is building to deliver cable television to Sprint mobile phone customers.
MNLI
(Neutral)	Premise: Let me try it. She began snapping her fingers and saying the word eagerly, but nothing happened.
Hypothesis: She became frustrated when the spell didn’t work.
TextFooer
(Contra-
diction)	Premise: Authorisation me attempting it. She triggered flapping her pinkies and said the word eagerly, but nothing arisen.
Hypothesis: She became frustrated when the spell didn’t work.
CLARE
(Contra-
diction)	Premise: Let me try it. She began snapping her fingers and saying the word eagerly, but nothing unexpected happened.
Hypothesis: She became frustrated when the spell didn’t work.
To examine the efficiency of attacking order, we compare REPLACEONLY against BERTAttack. Notably, REPLACEONLY outperforms BERTAttack across the board. This is presumably because BERTAttack does not take into account the tokens to be inflated when selecting the attack positions.
We now turn to the two constraints imposed when constructing the candidate token set. Perhaps not surprisingly, ablating the textual similarity constraint  $(w / o\sin >l)$  decreases textual similarity performance, but increases other aspects. Ablating the masked language model yields a better success rate, but much worse perplexity, grammaticality, and textual similarity.
Finally, we compare CLARE implemented with different masked language models. Table 6 summarizes the results. Overall, distilled RoBERTa achieves the fastest speed without losing performance. Since the victim model is based on BERT, we conjecture that it is less efficient to attack a model using its own information.
Table 4: Adversarial examples produced by different models. The gold label of the original is shown below the (bolded) dataset name. Replace, Insert and Merge are highlighted in italic red, bold blue and sans serif yellow, respectively. (Best viewed in color).
Module	A-rate↑	Mod↓	PPL↓	GErr↓	Sim↑
CLARE	79.1	6.1	86.0	0.17	0.76
MERGEONLY8	47.2	6.2	95.3	0.08	0.79
INSERTONLY	68.1	7.2	93.1	0.23	0.74
REPLACEONLY	66.7	7.7	85.6	0.10	0.72
BERTAttack	63.4	7.9	90.6	0.25	0.71
w/o sim > ℓ	82.4	6.9	86.8	0.13	0.70
w/o pMLM > k	95.7	6.8	162.8	0.71	0.61
Table 5: Ablation study results. "w/o  $\sin >\ell$  " ablates the textual similarity constraint when constructing the candidate sets, while "w/o  ${p}_{\mathrm{{MLM}}} > k$  " ablates the masked language model probability constraint.
MLM	A-rate↑	Mod↓	PPL↓	Sim↑	Speed↑
RoBERTa distill	79.1	6.1	86.0	0.76	0.14
RoBERTa base	79.3	6.3	88.9	0.75	0.07
BERTbase	78.4	8.3	95.2	0.71	0.06
Table 6: Results of CLARE implemented with different masked language models (MLM). Speed is measured by number of processed samples per second.
4.2 Perturbations by Part-of-speech Tags
In this section, we break down the adversarial attacks by part-of-speech (POS) tags in AG News dataset. We find that most of the adversarial attacks happen to nouns or noun phrases. Presumably, in many topic classification datasets, the prediction heavily relies on some characteristic noun words/phrases. As shown in Table 7,  $64\%$  of the Replace actions are applied to nouns. Insert actions tend to insert tokens into noun phrase bigram: two of the most frequent POS bigrams are noun phrases. In fact, around  $48\%$  of the Insert actions are applied to noun phrases. This also justifies our choice of only applying Merge to noun phrases.
4.3 Adversarial Training
This section explores CLARE's potential in improving downstream models' accuracy and robustness. Following Tsipras et al. (2018), we use CLARE to generate adversarial examples for AG news training instances, and include them as additional training data. We consider two settings: training with (1) full training data and full adversarial data and (2)  $10\%$  randomly-sampled training data and its adversarial data, to simulate the low-resource scenario. For both settings, we compare a BERT-based MLP classifier and a TextCNN (Kim, 2014) classifier without any pretrained embedding.
Whether adversarial examples, as data augmentation, can help achieve better test accuracy? As
Replace	Insert	Merge
NOUN: 64%	(NOUN, NOUN): 12%	ADJ-NOUN: 31%
ADJ: 17%	(ADJ, NOUN): 10%	NOUN-NOUN: 22%
VERB: 7%	(NOUN, VERB): 9%	DT-NOUN: 12%
Context: ... Amit Yoran, the government's cybersecurity chief, abruptly resigned yesterday after a year ...
Replace: cybersecurity ← {security, surveillance, cryptography, intelligence, encryption ...}
Insert: cybersecurity ____ chief ← {technology, defense, intelligence, program, project ...}
Merge: cybersecurity chief ← {chief, consultant, administrator, scientist, secretary ...}
shown in Table 8, when the full training data is available, adversarial training slightly decreases the test accuracy by  $0.2\%$  and  $0.5\%$  respectively. This aligns with previous observations (Jia et al., 2019). Interestingly, in the low-data scenario with adversarial training, the BERT-based classifier has no accuracy drop, and TextCNN achieves a  $2.0\%$  absolute improvement. This suggests that a model with less capacity can benefit more from silver data.
Does adversarial training help the models defend against adversarial attacks? To evaluate this, we use CLARE to attack classifiers trained with and without adversarial examples. A higher success rate and fewer modifications indicate a victim classifier is more vulnerable to adversarial attacks. As shown in Table 8, in 3 out of the 4 cases, adversarial training helps to decrease the attack success rate by more than  $10.3\%$ , and to increase the number of modifications needed by more than 0.8. The only exception is the TextCNN model trained with  $10\%$  data. A possible reason can be that it is trained with little data and thus generalizes less well.
These results suggest that CLARE can be used to improve downstream models' robustness, with a negligible accuracy drop.
Textual adversarial attack. An increasing amount of effort is being devoted to generating better textual adversarial examples with various
Table 7: Top: Top-3 POS tags (or POS tag bigrams) and their percentages for each perturbation type.  $(a,b)$ : insert a token between  $a$  and  $b$ .  $a-b$ : merge  $a$  and  $b$  into a token. Bottom: An AG news sample, where CLARE perturbs token "cybersecurity." TextFooer is unable to attack this token since it is out of its vocabularies.
Victim Model	Acc↑	A-rate↓	Mod↑
BERT (100% data)	95.0	79.1	6.1
+ 100% adversarial	-0.2	-18.0	+5.1
TextCNN (100% data)	91.2	92.7	5.0
+ 100% adversarial	-0.5	-10.3	+0.8
BERT (10% data)	92.5	96.1	5.4
+ 10% adversarial	+0.0	-12.3	+7.6
TextCNN (10% data)	83.6	99.0	5.6
+ 10% adversarial	+2.0	-3.5	+0.3
Table 8: Adversarial training results on AG news test set. "Acc" indicates accuracy.
attack models. Character-based models (Liang et al., 2019; Ebrahimi et al., 2018; Li et al., 2018; Gao et al., 2018, inter alia) use misspellings to attack the victim systems; however, these attacks can often be defended by a spell checker (Pruthi et al., 2019; Zhou et al., 2019b; Jones et al., 2020). Many sentence-level models (Iyyer et al., 2018; Wang et al., 2020; Zou et al., 2020, inter alia) have been developed to introduce more sophisticated token/phrase perturbations. These, however, generally have difficulty maintaining semantic similarity with original inputs (Zhang et al., 2020a). Recent word-level models explore synonym substitution rules to enhance semantic meaning preservation (Alzantot et al., 2018; Jin et al., 2020; Ren et al., 2019; Zhang et al., 2019; Zang et al., 2020, inter alia). Our work differs in that CLARE uses three contextualized perturbations that produces more fluent and grammatical outputs.
Text generation with BERT. Generation with masked language models has been widely studied in various natural language tasks, ranging from lexical substitution (Wu et al., 2019a; Zhou et al., 2019a; Qiang et al., 2020; Wu et al., 2019b, inter alia) to non-autoregressive generation (Gu et al., 2018; Lee et al., 2018; Ghazvininejad et al., 2019; Wang and Cho, 2019; Ma et al., 2019; Sun et al., 2019; Ren et al., 2020; Zhang et al., 2020b, inter alia).
We have presented CLARE, a contextualized adversarial example generation model for text. It uses contextualized knowledge from pretrained masked language models, and can generate adversarial examples that are natural, fluent and grammatical. With three contextualized perturbation patterns, Replace, Insert and Merge in our
In preliminary experiments, we found that it is more difficult to use other models to attack a victim model trained with the adversarial examples generated by CLARE, than to use CLARE itself.
arsenal, CLARE can produce outputs of varied lengths and achieves a higher attack success rate than baselines and with fewer edits. Human evaluation shows significant advantages of CLARE in terms of textual similarity, fluency and grammaticality. We release our code and models at https://github.com/cookielee77/CLARE.
We would like to thank the reviewers for their constructive comments. We thank NVIDIA Corporation for the donation of the GPU used for this research. We also thank Tongshuang Wu, Guoyin Wang and Shuhuai Ren for their helpful discussions and feedback.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proc. of EMNLP.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175.
Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019. Robust neural machine translation with doubly adversarial inputs. In Proc. of ACL.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL.
William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. Hotflip: White-box adversarial examples for text classification. In Proc. of ACL.
Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. In IEEE Security and Privacy Workshops (SPW).
Siddhant Garg and Goutham Ramakrishnan. 2020. Bae: Bert-based adversarial examples for text classification. In Proc. of EMNLP.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proc. of EMNLP.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proc. of ICLR.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In Proc. of ICLR.
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proc. of NAACL.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proc. of EMNLP.
Robin Jia, Aditi Raghunathan, Kerem Goksel, and Percy Liang. 2019. Certified robustness to adversarial word substitutions. In Proc. of EMNLP.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? natural language attack on text classification and entailment. In Proc. of AAAI.
Erik Jones, Robin Jia, Aditi Raghunathan, and Percy Liang. 2020. Robust encodings: A framework for combating adversarial typos. In Proc. of ACL.
Katharina Kann, Sascha Rothe, and Katja Filippova. 2018. Sentence-level fluency evaluation: References help, but can be spared! In Proc. of CNLP, pages 313-323.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proc. of EMNLP.
Keita Kurita, Paul Michel, and Graham Neubig. 2020. Weight poisoning attacks on pre-trained models. In Proc. of ACL.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proc. of EMNLP.
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2018. Textbugger: Generating adversarial text against real-world applications. arXiv preprint arXiv:1812.05271.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. Bert-attack: Adversarial attack against bert using bert. In Proc. of EMNLP.
Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2019. Deep text classification can be fooled. In Proc. of IJCAI.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. 2019. Flowseq: Nonautoregressive conditional sequence generation with generative flow. In Proc. of EMNLP.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020a. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proc. of EMNLP: System Demonstrations.
John X Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. 2020b. Reevaluating adversarial examples in natural language. In Proc. of EMNLP Findings.
Nikola Mrksic, Diarmuid Řeaghdha, Blaise Thomson, Milica Gasic, Lina M Rojas Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In Proc. of NAACL.
Daniel Naber et al. 2003. A rule-based style and grammar checker. CiteSeer.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Proc. of NeurIPS.
Danish Pruthi, Bhuwan Dhingra, and Zachary C Lipton. 2019. Combating adversarial misspellings with robust word recognition. In Proc. of ACL.
Jipeng Qiang, Yun Li, Yi Zhu, and Yunhao Yuan. 2020. A simple bert-based approach for lexical simplification. In Proc. of AAAI.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text. In Proc. of EMNLP.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In Proc. of ACL.
Yi Ren, Jinglin Liu, Xu Tan, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2020. A study of non-autoregressive model for sequence generation. In Proc. of ACL.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In Proc. of ACL.
Suranjana Samanta and Sameep Mehta. 2017. Towards crafting text adversarial samples. arXiv preprint arXiv:1707.02812.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In *The 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing*, NeurIPS.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. of EMNLP.
Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and Zhihong Deng. 2019. Fast structured decoding for sequence models. In Proc. of NeurIPS.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2018. Robustness may be at odds with accuracy. In Proc. of ICLR.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. In Proc. of EMNLP.
Alex Wang and Kyunghyun Cho. 2019. Bert has a mouth, and it must speak: Bert as a markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proc. of ICLR.
Boxin Wang, Hengzhi Pei, Boyuan Pan, Qian Chen, Shuohang Wang, and Bo Li. 2020. T3: Treeautoencoder regularized adversarial text generation for targeted attack. In Proc. of EMNLP.
Xiaosen Wang, Hao Jin, and Kun He. 2019b. Natural language adversarial attacks and defenses in word level. arXiv preprint arXiv:1909.06723.
Adina Williams, Nikita Nangia, and Samuel Bowman 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proc. of NAACL.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019a. Conditional bert contextual augmentation. In Proc. of ICCS.
Xing Wu, Tao Zhang, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019b. "mask and infill": Applying masked language model to sentiment transfer. In Proc. of IJCAI.
Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020. Word-level textual adversarial attacking as combinatorial optimization. In Proc. of ACL.
Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li. 2019. Generating fluent adversarial examples for natural languages. In Proc. of ACL.
Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. 2020a. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology (TIST), 11(3):1-41.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proc. of NeurIPS.
Yizhe Zhang, Guoyin Wang, Chunyuan Li, Zhe Gan, Chris Brockett, and Bill Dolan. 2020b. Pointer: Constrained text generation via insertion-based generative pre-training. In Proc. of ACL.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples. In Proc. of ICLR.
Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, and Ming Zhou. 2019a. Bert-based lexical substitution. In Proc. of ACL.
Yichao Zhou, Jyun-Yu Jiang, Kai-Wei Chang, and Wei Wang. 2019b. Learning to discriminate perturbations for blocking adversarial attacks in text classification. In Proc. of EMNLP.
Wei Zou, Shujian Huang, Jun Xie, Xinyu Dai, and Jia-jun Chen. 2020. A reinforced generation of adversarial samples for neural machine translation. In Proc. of ACL.
A.1 Additional Experiment Details
Model Implementation. All pretrained models and victim models based on RoBERTa and  $\mathrm{BERT}_{\mathrm{base}}$  are implemented with Hugging Face transformers[10] (Wolf et al., 2019) based on PyTorch (Paszke et al., 2019). RoBERTa distill, RoBERTa base and uncase  $\mathrm{BERT}_{\mathrm{base}}$  models have 82M, 125M and 110M parameters, respectively. We use RoBERTa distill as our main backbone for fast inference purpose. TextFooler[11] and BERTAttack[12] are built with their open source implementation provided by the authors. In the implementation of TextFooler+LM, we use small sized GPT-2 language model (Radford et al., 2019) to further select those candidate tokens that have top  $20\%$  perplexity in the candidate token set. In the adversarial training (\$4.3), the small TextCNN victim model (Kim, 2014) has 128 embedding size and 100 filters for 3, 4, 5 window size with 0.5 dropout, resulting in 7M parameters.
During the implementation of  $w / o p_{\mathrm{MLM}} > k$  in the ablation study (§4.1), we randomly sample 200 tokens and then apply the similarity constraint to construct candidate set, as exhausting the vocabulary is computationally expensive.
Evaluation Metric. The similarity function sim builds on the universal sentence encoder (USE; Cer et al., 2018) to measure a local similarity at the perturbation position with window size 15 between the original input and its adversary. All baselines are equipped this sim when constructing the candidate vocabulary. The evaluation metric Sim uses USE to calculate a global similarity between two texts. These procedures are typically following Jin et al. (2020). We mostly rely on human evaluation (§3.3) to conclude the significant advantage of preserving textual similarity on CLARE compared with TextFooler.
Data Processing. When processing the data, we keep all punctuation in texts for both victim model training and attacking. This differs the preprocessing setting in TextFooler (Jin et al., 2020) as we empirically found that removing punctuation makes the victim model vulnerable. Since GLUE
Dataset	Avg. Length	# Classes	Train	Test	Acc
SST-2	10	2	67K	0.9K	92.3%
DBpedia	55	14	560K	70K	99.3%
QQP	13/13	2	363K	40K	91.4%
MRPC	23/23	2	3.6K	1.7K	81.4%
Table 9: Some statistics of datasets. The last column indicates the victim model's accuracy on the original test set without adversarial attack.
benchmark (Wang et al., 2019a) does not provide the label for test set, we instead use its dev set as the the test set for the included datasets (MNLI, QNLI, QQP, MRPC, SST-2) in the evaluation. For the sentence-pair tasks (e.g., MNLI, QNLI, QQP, MRPC), we attack the longer one excluding the tokens appearing in both sentences. This is because inference tasks usually require entailed data to have the same keywords, e.g., numbers, name entities, etc. All experiments are conducted on one Nvidia GTX 1080Ti GPU.