SlowGuess/NAACL_2022_OCR huggingface dataset:
Classifier	Sample Strategy	Orig. Acc.	TextFollower	Bert-Attack	TextBugger	PWWS
Acc.	ASR	Acc.	ASR	Acc.	ASR	Acc.	ASR
IMDB	LSTM	No Shielding	91	0	100	0	100	0	100	0	100
Shielding-Word	94	89	5	87	7	89	5	89	5
CNN	No Shielding	86	0	100	0	100	0	100	0	100
Shielding-Word	89	88	1	88	1	89	0	86	3
BERT	No Shielding	90	1	99	4	96	6	93	2	98
Shielding-Word	85	80	6	80	6	84	1	82	4
Yelp	LSTM	No Shielding	95	0	100	0	100	6	94	0	100
Shielding-Word	87	81	7	79	9	78	10	74	15
CNN	No Shielding	96	0	100	0	100	5	95	3	97
Shielding-Word	88	85	3	81	8	81	8	83	6
BERT	No Shielding	100	3	97	10	90	13	87	7	93
Shielding-Word	92	90	2	88	4	91	1	85	8
AG News	LSTM	No Shielding	93	1	99	0	100	16	83	13	86
Shielding-Word	87	78	10	84	3	78	10	84	3
CNN	No Shielding	92	1	99	0	100	7	92	3	97
Shielding-Word	87	81	7	87	0	84	3	83	5
BERT	No Shielding	99	20	78	11	89	60	39	15	85
Shielding-Word	88	81	8	82	7	83	6	85	3
Table 3: Results where attacker knows about Sample Shielding. Shielding settings:  $k = 30$ ,  $p = 0.3$ , majority voting. Acc: accuracy, ASR: success rate of attack (\%), Orig. Acc: accuracy on original texts.
Figure 4: Accuracy with various p values for LSTM on IMDB. Note that  $k$  is fixed to 100.
These results show the power of Sample Shielding as even with knowledge of both the classifier and Sample Shielding, attacks struggle to perturb the text in a manner that causes  $W$  to fail. Furthermore, the attacks do worse with feedback from Sample Shielding. This shows the misleading nature of feedback from Sample Shielding, and unreliability when guiding attacks.
5 Additional Analysis
5.1 Parameter search
Increasing  $p$  raises the risk of samples containing increased amounts of perturbed text. Decreasing  $k$  raises the risk of not covering enough of the unperturbed portions of the original text. While our settings of  $p = 0.3$  and  $k = 100$  for our main results are reasonable values (Table 1, Table 2) they are not necessarily optimal.
Optimal  $p$ . Figure 4 shows the results for all com
Figure 5: Accuracy with various p values for LSTM on AG News. Note that  $k$  is fixed to 100.
binations of attacks against LSTM on IMDB with word shielding as the defense,  $k$  fixed at 100. As we increase  $p$ , we see a continued drop in accuracy which is consistent with the idea that a higher  $p$  is more likely to capture perturbed text. The optimal value range appears to be in 0.2 - 0.4 range, although we do not see large drops until 0.6 onward. We also examined the same combination on AG News (Figure 5) since it's texts are considerably shorter and found consistent results.
Optimal  $k$ . Figure 6 shows results for all attacks against LSTM on IMDB with word sampling as the defense,  $p$  fixed at 0.3. The optimal  $k$  is not as clear as  $p$ . We see clear increases after 30 samples, but then the optimal  $k$  varies depending on attack. However, we see a leveling off around 90 samples, which gives some credence to our chosen  $k$  of 100. We also found similar results when examining the same combination on AG News (Figure 7), however,  $k$  stabilized lower (about 50).
Figure 6: Accuracy with various k values for LSTM on IMDB. Note that  $p$  is fixed to 0.3.
Figure 7: Accuracy with various k values for LSTM on AG News. Note that  $p$  is fixed to 0.3.
5.2 Reliability of Sample Shielding
Due to the randomness of samples, there may be concern over the consistency of Sample Shielding. To address this, we ran Sample Shielding 100 times on the IMDB attacked texts from Table 3 against BERT classifier. Each time 30 random samples were used to vote. As can be observed from Figure 8, Sample Shielding consistently protects against attacks. Median accuracies are above  $80\%$  dropping only to  $75\%$  in the worst case. This points to Sample Shielding as a consistent, reliable defense.
5.3 Comparison with other SOTA Defenses
Comparisons are limited as threat models differ. As noted earlier, other defenses assume a weaker threat model where the attacker queries the web
Figure 8: Boxplots of accuracies when Sample Shielding is applied 100 times to attacked IMDB texts with BERT as classifier. Red lines: accuracies reported in Table 3.
site's shielded  $W$  directly. To make ours equivalent we compare SOTA results with our accuracies obtained by the attacker using  $W'$  alone (with  $W = W'$ ). We calculate accuracies right after the final perturbed text is generated using  $W'$  eliminating a followup round of  $W$  with Sample Shielding. Table 4 provides our full results against this weaker threat model.
With BERT as base classifier for AG News, FreeLB++, an adversarial training technique (Li et al., 2021b) report accuracies of 51, 56, and 42 against TextFooler, TextBugger, and Bert-Attack respectively. RanMask (Zeng et al., 2021), which uses random masking of words report accuracies of 38, 45, and 49. In comparison, Sample Shielding achieves 48, 55, and 38 respectively outperforming RanMask in 2 out of 3, while only a few points behind FreeLB++. For IMDB, FreeLB++ reports 45, 43, and 40 and RanMask reports 22, 18, and 36 respectively. Equivalently, Sample Shielding achieves 18, 34, and 31. With some wins and some losses, Sample Shielding is in the mix with current SOTA defenses in this weaker threat model. However, when deployed as designed for the realistic threat model, it wins over these other defenses by large margins (see Table 3). While we do not know how FreeLB++, RanMask, and similar defenses would perform with our threat model any deterministic shield would give the exact same results when the classifier is applied once again by the website.
5.4 Limitations/Future work
First, in future work we will add in direct comparisons to the two closest methods to Sample Shielding (Zeng et al., 2021; Wang et al., 2021a). They are similar in spirit as they also work off samples though these are generated differently. We have not compared with them because these two papers appeared very recently, one last revised in July (Zeng et al., 2021) and the other appeared in arXiv in September 2021 (Wang et al., 2021a). Second, the neural net summarizer leverages a simple linear layer. Other networks, e.g., LSTM, maybe better at finding patterns in sequential data. In future work we will also explore layering Sample Shielding onto other defense strategies.
Another limitation of our current method is that we do not measure Sample Shielding's effectiveness on other common text tasks including Natural Language Understanding. Additionally, datasets which contain the shortest texts (e.g. SST2) are
Classifier	Sample Strategy	Orig. Acc.	TextFooler	Bert-Attack	TextBugger	PWWS
Acc.	ASR	Acc.	ASR	Acc.	ASR	Acc.	ASR
IMDB	LSTM	Local (W') Word	87	11	87	31	64	28	68	22	75
CNN	Local (W') Word	91	22	76	22	76	32	65	29	68
BERT	Local (W') Word	81	18	78	31	62	34	58	26	68
BERT	RanMask*	92	22	75	36	58	18	79	-	-
BERT	FreeLB++*	93	45	51	40	57	43	54	-	-
AG	LSTM	Local (W') Word	88	42	52	31	65	38	57	55	38
CNN	Local (W') Word	86	45	48	28	67	36	58	54	37
BERT	Local (W') Word	88	48	45	38	57	55	38	64	27
BERT	RanMask*	92	38	59	49	46	45	51	-	-
BERT	FreeLB++*	95	52	46	42	56	56	41	-	-
Table 4: Results of attack against local model with knowledge of Sample Shielding. For all shielding cases,  $k = 30$ ,  $p = 0.3$ , and majority voting is used. Acc. is accuracy, and ASR is success rate of attack (\%) and Orig. Acc. is accuracy on the original text. Note that the examples used by RanMask and FreeLB++ is not the set of dataset samples as our paper.
not currently tested in our experiments. Since sample shielding removes texts, it's performance could drop for these tasks and short texts. Thus, future work will include these comparisons.
Defenses using voting. The most similar methods to our own are RanMask and RS&V both appearing within the last five months. RanMask (Zeng et al., 2021) randomly masks tokens in input texts. This random masking occurs  $n$  times generating  $n$  inputs to be fed to a classifier. RS&V (Wang et al., 2021a) randomly replaces words in the input with synonyms. This it does  $k$  times to produce  $k$  samples which are then voted on. If the samples vote for a different label than the label produced by the unsampled input, then the text is labeled as an adversarial text. Our method is advantageous since it does not rely on specific models (i.e. Masked Language Model) or synonym sources.
Adversarial training. Classifiers train on perturbed data, learning to identify modified versions of the original input (Wang and Wang, 2020; Wang et al., 2021b; Zhu et al., 2020; Li et al., 2021b). As an example, Gil et al. (2019) propose HotFlip which uses white-box knowledge to generate adversarial attacks to train on. Specifically, they flip tokens based on the gradients of the one-hot input vectors. However, adversarial defenses are limited to known attackers. In contrast, Sample Shielding is 'plug-and-play' as it is a pre-processing step.
Other defenses. Several other shielding methods exist (Keller et al., 2021; Eger et al., 2019; Zhu et al., 2021). For example, Rodriguez and Galeano (2018) defend Perspective (Google's toxicity classification model) by neutralizing adversarial inputs via a negated predicates list. Again, these defenses
are restricted to contexts where specific lists may be identified, this is not so with Sample Shielding.
Sample Shielding, an intuitively designed defense which is attacker and classifier agnostic, protects effectively; reducing ASR from  $90 - 100\%$  down to  $14 - 34\%$  with minimal accuracy loss  $(3\%)$  in original texts. The randomness (through sampling) provides unreliable feedback for attackers, thus it even thwarts attackers who have query access to classifiers protected with Sample Shielding. Attack strategies will need to increase the amount of perturbation to make sure a majority of samples fail at classification. However, this will risk semantic integrity. Thus, we expect Sample Shielding to cause ripples in future adversarial attack strategies while providing text classifiers with a definite advantage.
Steffen Eger, Gözde Gül Şahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. 2019. Text processing like humans do: Visually attacking and shielding NLP systems. In Proceedings of the 2019 Conference of the North American Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1634-1647, Minneapolis, Minnesota. Association for Computational Linguistics.
Siddhant Garg and Goutham Ramakrishnan. 2020. Bae: Bert-based adversarial examples for text classification.
Yotam Gil, Yoav Chai, Or Gorodissky, and Jonathan Berant. 2019. White-to-black: Efficient distillation of black-box adversarial attacks. In Proceedings of
the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1373-1379, Minneapolis, Minnesota. Association for Computational Linguistics.
Akhil Goel, Akshay Agarwal, Mayank Vatsa, Richa Singh, and Nalini K. Ratha. 2020. Dndnet: Reconfiguring cnn for adversarial robustness. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 103-110.
Robin Jia, Aditi Raghunathan, Kerem Goksel, and Percy Liang. 2019. Certified robustness to adversarial word substitutions.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8018-8025.
Yannik Keller, Jan Mackensen, and Steffen Eger. 2021. BERT-defense: A probabilistic model based on BERT to combat cognitively inspired orthographic adversarial attacks. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 1616-1629, Online. Association for Computational Linguistics.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Linguistics.
Dianqi Li, Yizhe Zhang, Hao Peng, Liquin Chen, Chris Brockett, Ming-Ting Sun, and Bill Dolan. 2021a. Contextualized perturbation for textual adversarial attack. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5053-5069, Online. Association for Computational Linguistics.
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2019. Textbugger: Generating adversarial text against real-world applications. Proceedings 2019 Network and Distributed System Security Symposium.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. BERT-ATTACK: Adversarial attack against BERT using BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193–6202, Online. Association for Computational Linguistics.
Zongyi Li, Jianhan Xu, Jiehang Zeng, Linyang Li, Xiaoqing Zheng, Qi Zhang, Kai-Wei Chang, and Cho-Jui Hsieh. 2021b. Searching for an effective defender: Benchmarking defense against adversarial word substitution. In EMNLP.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. TextAttack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 119-126, Online. Association for Computational Linguistics.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1085-1097, Florence, Italy. Association for Computational Linguistics.
Nestor Rodriguez and Sergio Rojas Galeano. 2018. Shielding google's language toxicity model against adversarial attacks. CoRR, abs/1801.01828.
Xiaosen Wang, Yifeng Xiong, and Kun He. 2021a. Randomized substitution and vote for textual adversarial example detection. ArXiv, abs/2109.05698.
Xiaosen Wang, Yichen Yang, Yihe Deng, and Kun He. 2021b. Adversarial training with fast gradient projection method against synonym substitution based text attacks. In AAAI.
Zhaoyang Wang and Hongtao Wang. 2020. Defense of word-level adversarial attacks via random substitution encoding. In KSEM.
Jin Yong Yoo and Yanjun Qi. 2021. Towards improving adversarial training of nlp models.
Jiehang Zeng, Xiaoqing Zheng, Jianhan Xu, Linyang Li, Liping Yuan, and Xuanjing Huang. 2021. Certified robustness to text adversarial attacks by randomized [mask]. ArXiv, abs/2105.03743.
Bin Zhu, Zhaoquan Gu, Le Wang, and Zhihong Tian. 2021. Treated: Towards universal defense against textual adversarial attacks. ArXiv, abs/2109.06176.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2020. Freelb: Enhanced adversarial training for natural language understanding. In International Conference on Learning Representations.