SlowGuess/NAACL_2022_OCR huggingface dataset:
Don’t sweat the small stuff, classify the rest: Sample Shielding to protect text classifiers against adversarial attacks
Jonathan Rusert, Padmini Srinivasan
{jonathan-rusert,epadmini-srinivasan}@uiowa.edu
Deep learning (DL) is being used extensively for text classification. However, researchers have demonstrated the vulnerability of such classifiers to adversarial attacks. Attackers modify the text in a way which misleads the classifier while keeping the original meaning close to intact. State-of-the-art (SOTA) attack algorithms follow the general principle of making minimal changes to the text so as to not jeopardize semantics. Taking advantage of this we propose a novel and intuitive defense strategy called Sample Shielding. It is attacker and classifier agnostic, does not require any reconfiguration of the classifier or external resources and is simple to implement. Essentially, we sample subsets of the input text, classify them and summarize these into a final decision. We shield three popular DL text classifiers with Sample Shielding, test their resilience against four SOTA attackers across three datasets in a realistic threat setting. Even when given the advantage of knowing about our shielding strategy the adversary's attack success rate is  $<= 10\%$  with only one exception and often  $< 5\%$ . Additionally, Sample Shielding maintains near original accuracy when applied to original texts. Crucially, we show that the 'make minimal changes' approach of SOTA attackers leads to critical vulnerabilities that can be defended against with an intuitive sampling strategy. $^{1}$
Text classifiers have become ubiquitous. Unfortunately, they are subject to attacks from adversaries, typically executed using machine learning methods. Attackers work by making small modifications to the text that mislead the classifier. Adversarial attackers are now a growing part of the ecosystem.
Like classifiers, attack algorithms have achieved strong success due to advances in machine learning/deep learning. Current text attackers, like
TextFooler (Jin et al., 2020) and Bert-Attack (Li et al., 2020), are able to reduce near perfect classification accuracy down to  $5\%$ . Additionally, these attackers achieve this while perturbing (changing) only a small amount of the original text. This helps preserve the original meaning so that humans are able to understand the original message even though classifiers are duped.
As a counter, classifier shielding techniques are being explored. One such approach is adversarial training where the classifier, assumed to have access to the attacker, uses it to generate perturbed texts - these are added to the classifier's training data. While this leads to model resilience against that attacker it leaves the classifier open to attacks by new attackers. Other defenses involve modifying classifier structure to reduce the information an attacker can glean from it (Goel et al., 2020). However, this type of reconfiguration will not be possible if a third party classifier (e.g. Google Perspective) is leveraged. Even other approaches involve modifying the input text during classification time, but are currently limited to classifiers built from specific masked language models (Zeng et al., 2021) or rely on external synonym datasets (Wang et al., 2021a). We propose a shielding technique which is attacker-agnostic, does not require additional training/reconfiguration to the classifier, can shield any classifier, does not require an external data source, and can be used in a more realistic threat setting. We refer to this as Sample Shielding.
Sample Shielding takes advantage of current constraints in SOTA attacks. Mainly, to preserve original meaning, these make the minimal changes needed to deceive the classifier. For example, BERT-Attack (Li et al., 2020) only perturbs up to  $16\%$  of text, and often far less (e.g.  $1.1\%$  ) for some datasets. Thus, if we would look at the  $84\%$  to  $99\%$  of text that is untouched our model would be more likely to classify correctly. Hence, in Sample Shielding we take many samples of the input
$^{1}$ Our code and data are available at: https://github.com/JonRusert/SampleShielding
Human Language Technologies, pages 2716 - 2725
Figure 1: Threat model - Attacker modifies text with feedback from its local classifier  $W'$ . Dashed box included in path when attacker knows about Sample Shielding employed by website. When box excluded knowledge of Sample Shielding is unavailable.
text, classify these individually and combine their decisions as an ensemble to classify the text. Our contributions are as follows:
1. We propose a new, intuitive shielding technique called Sample Shielding for text classifiers.
2. We assess Sample Shielding under a realistic threat model where the attacker cannot query a website's classifier hundreds of times since that pattern is easily detectable by the website. We run experiments under two conditions, when the attacker has knowledge of Sample Shielding and when it does not. In both cases the attacker uses a local copy of the websites' classifier. This is an optimistic assumption favouring the attacker and thus provides a lower bound to our results.
3. We test against 4 SOTA text attack algorithms, 3 text datasets and 3 classifiers. When the attacker does not have knowledge of Sample Shielding, our defense reduces attack success rate from near total decimation  $90 - 100\%$  down to  $13 - 36\%$ , while still maintaining accuracy on original texts. When the attacker has knowledge of Sample Shielding, our defense performs even better, reducing attacks down to  $1 - 10\%$  success rate. This is partially due to Sample Shielding's random nature providing unreliable feedback to attackers.
Our success with Sample Shielding is good news for classifiers – and it raises the bar significantly for the next generation attackers. We share code and our perturbed text collections for future research.
2.1 Threat model
The typical attack strategy perturbing texts with word synonyms or character substitutions assumes to have query access to the target web site's classifier  $(W)$  (Yoo and Qi, 2021; Li et al., 2021a; Ren et al., 2019; Jin et al., 2020; Li et al., 2020; Garg
and Ramakrishnan, 2020; Jia et al., 2019; Li et al., 2019). The text is modified by querying  $W$  hundreds or thousands of times, each time with a text version differing only slightly from the previous - even by just a single word (Li et al., 2020; Jin et al., 2020). Such a querying pattern can be easily identified as adversarial by the website and countered. Thus, practically the only way in which such an attack can take place is when the attacker owns a local classifier  $W'$  which is either an exact copy of  $W$  or a close enough approximation. We adopt this more realistic threat model, shown in Figure 1.
In our threat model the attacker uses feedback from its local  $W'$  to generate a final perturbed version that defeats  $W'$  or is close enough to do so. The attacker submits only this final version to the website, expecting  $W$  to make the same error. However, the website defends  $W$  using Sample Shielding: sample based pre-processing on the input text, prior to applying  $W$ . The attacker may or may not be aware of this fact. Keeping  $W = W'$  which is consistent with other defenses, we evaluate our defense under two conditions:
1) The attacker does not know that the website employs Sample Shielding pre-processing when classifying text using  $W$ .
2) The Sample Shielding step is leaked and the attacker incorporates it locally when using  $W'$  to generate the final perturbed text.
We present results from experiments exploring both of these attack conditions.
2.2 Sample Shielding approach
Intuition. Current adversarial attackers have two goals: fool the classifier and maintain the original meaning. Since they make minimal changes, the extent of perturbation is in fact one of the reported statistics. For example, (Li et al., 2020) note that their  $10\%$  perturbation rate is far less than in previous attacks. (Li et al., 2019) also focus on minimal changes  $(4\%)$  needed in support of their attack success rate. Our defense approach capitalizes on this drive to make minimal changes. Specifically, in Sample Shielding, we take  $k$  samples each composed of  $p\%$  of the text. We choose a  $p$  which minimizes the chance of a sample including attacked (modified) words, while maximizing the content available for the classifier to make a correct classification. We choose a  $k$  which is large enough to cover key information but small enough to reduce redundancy. We classify each sample and combine
Figure 2: Proposed shielding method. Sentences or words are sampled  $k$  times at a rate of  $p$  percent (of the input text), the  $k$  samples are classified. The probabilities are used in a majority vote for the final prediction (solid box), or are sorted and given to a Neural Net Summarizer (NN or NN-BB) to make the final prediction (dotted box).
their decisions for the final classification. We explore two sampling and three decision combining methods.
2.2.1 Sampling methods
Random Sampling. We randomly sample  $p$  portions of the text. We explore both sentences and words as sampled units. A visualization of random sampling is in Figure 2.
Shifting Sampling. We sample the text using a moving window of length  $p \times$  length_of_text. The first starts at the beginning of the text. The next window starts right after the previous window ends. If there is insufficient text for the last window, then it wraps back to include the beginning text.
2.2.2 Decision strategy
Majority voting. This is a simple majority vote across the k samples (Figure 2).
Classifier trained on sample scores from original texts (NN). We train a neural network summarizer to make a final class prediction based on the  $k$  sample probabilities. Since sample ID does not carry any information, the input to the neural network is a sorted list of sample probabilities. The intent is to see if the neural network picks up on latent patterns in the probabilities that are not captured by majority voting (see Figure 2). It should be emphasized that the neural network summarizer is trained only on probabilities generated from original texts and does not consider probabilities from attacker modified texts. We use a simple feed forward neural net composed of 2 linear layers (size 500 and 300) as classification summarizer.
Classifier trained on sample scores from original and attacked texts (NN-BB). This is similar to the previous strategy except that the training data
includes scores from original texts and texts that have been modified by the attacker. Because this assumes more knowledge of the attacker we expect NN-BB to perform better than NN. The ground truth label for these modified texts is the original correct class label.
We examine three standard datasets in our experiments. Two have binary class labels (Yelp, IMDB) and the third has multi class labels (AG News). These have been used in adversarial generation and defense research (Zeng et al., 2021; Li et al., 2020). All datasets can be found via huggingface $^2$ .
1. IMDB - Movie review dataset for binary sentiment classification. 25k examples are provided for training and testing respectively.
2. Yelp - Yelp dataset for binary sentiment classification on reviews of businesses extracted from the Yelp Dataset Challenge<sup>3</sup>. 560k examples are provided for training and 38k for testing.
3. AG News - News articles from over 2000 news sources annotated by type of news: Sports, World, Business, and Science/Tech. 120k training and 7k test sets are provided.
Following previous research, (Li et al., 2020; Jin et al., 2020) we use all training data, and evaluate our method on random 1k samples of each dataset for the case where the local classifier does not employ Sample Shielding. Due to the high amount of queries used by the adversaries, we test on a subset of 100 samples for the case where the attacker's
huggingface.co/datasets
3www.yelp.com/dataset/challenge/winners
local classifier employs Sample Shielding.4
3.2 Adversarial models
We test our text classifier shielding strategy against 4 state-of-the-art (SOTA) text classifier attack algorithms. These algorithms have shown excellent performance in causing misclassifications while still producing readable texts. We defend against 3 word based attacks: TextFooler (Jin et al., 2020), Bert-Attack (Li et al., 2020), PWWS (Ren et al., 2019). TextFooler leverages word embeddings for word replacements, Bert-Attack leverages BERT itself by masking words and using BERT suggestions, PWWS selects and weights word replacements from WordNet. All three use some form of greedy selection for determining which words to replace. We also defend against a character based attack algorithm, TextBugger (Li et al., 2019).
3.3 Victim classifier models
We test our shielding approach against 3 standard classifiers<sup>5</sup> used in previous research, e.g. (Li et al., 2021a; Jin et al., 2020; Li et al., 2020):
1. CNN - A word based CNN (Kim, 2014), with three window sizes (3,4,5), 100 filters per window with dropout of 0.3 and Glove embeddings.
2. LSTM - A word based bidirectional LSTM with 150 hidden units. As with the CNN a dropout of 0.3 is used and Glove embeddings are leveraged.
3. BERT - The 12 layer BERT base model which has been fine-tuned on the corresponding dataset. These are provided by textattack via huggingface $^6$ .
3.4 Experimental design
We run experiments on the combination of the three victim classification models, three datasets, and four attack algorithms. These combinations are run on both threat model conditions (attacker is aware/ not aware of SampleShielding). This leads to 72 shielding experiments. For all attacks, we leverage TextAttack framework which provides classification algorithms and adversarial text generation algorithms implemented as specified in respective papers (Morris et al., 2020). In all experiments where the attacker does not use Sample Shielding
Figure 3: Average % of perturbed words for each attack. Percentages estimated by comparing words in original and perturbed texts. Since TextBugger adds whitespace in words skewing its percentage it is excluded.
we set  $k = 100$  and  $p = 0.3$ . While better performance was achieved with other values in preliminary experiments, we chose to go with a single combination of  $p$  and  $k$  for simplicity. In experiments where the attacker uses Sample Shielding pre-processing we reduce  $k$  to 30 for efficiency. Except where otherwise noted, majority voting is used to generate results. Additionally, shifting sampling (Section 2.2.1) shielding typically achieved 10-20 points lower accuracy compared to random, thus we do not include it in the results.
3.5 Evaluation measures
We examine accuracy and Attack Success Rate:
$$
a c c u r a c y = \frac {\# e x a m p l e s \_ c l a s s i f i e d \_ c o r r e c t l y}{\# t o t a l \_ e x a m p l e s} \tag {1}
$$
$$
A S R = \frac {\text {O r i g i n a l} _ {A c c .} - \text {A t t a c k e d} _ {A c c .}}{\text {O r i g i n a l} _ {A c c .}} \tag {2}
$$
We first present results for the condition where the attacker is not aware of Sample Shielding based pre-processing and then the results for when the attacker also employs Sample Shielding.
4.1 Condition 1: Attacker does not know about Sample Shielding
Results are in Table 1. BERT is the strongest classifier achieving  $91 - 100\%$  accuracy on the original datasets. Attacks are highly successful against unshielded texts. TextFooler and Bert-Attack are the most successful, dropping accuracies to  $0 - 5\%$  generally. Attacks were able to achieve strong drops with minimal amount of text perturbed (about  $10\%$ ). Figure 3 shows that the average percent of words perturbed across datasets for each attack are about
<sup>4</sup>We share the original and perturbed texts for replicability. We note that replicability of previous defenses are limited because the identity of their randomly sampled test instances are not provided.
<sup>5</sup>We calibrated classifier accuracies against previous research (Li et al., 2020; Jin et al., 2020)
huggingface.co/textattack
7textattack.readthedocs.io/en/latest/index.html
equal in the mid regions of the plots. For AG News, attacks are less successful against BERT; accuracy drops to  $19\%$  in the strongest attack (TextFooer), and only to  $49\%$  in the weakest (TextBugger). In general, TextBugger, the character-based attacker, is the least effective attacker.
Sample Shielding greatly reduces effectiveness of attacks while maintaining accuracy on original texts. The shielded classifier  $W$  maintains accuracy on original texts to within  $7\%$  of the original accuracy. Crucially, for attacked texts we see accuracy improve to between 60 and  $80\%$  (from post attack range of  $0 - 5\%$  generally). For example, TextFollower causes BERT's accuracy to drop from  $91\%$  to  $1\%$  for IMDB, however, Sample Shielding returns accuracy to  $78\%$ . In other words, the effectiveness of the attack is reduced from  $99\%$  effective to  $14\%$  effective. Additionally, accuracy on the original texts is maintained (91.3 to 91.5). This pattern is seen in the other attack classifier models and dataset combinations as well. For Yelp, LSTM drops from 92.5 to 0.7 when attacked by BERT-Attack, however, Word sampling brings it back up to 66.7, while achieving an original accuracy of 87.8. Overall, accuracy after shielding ranges from 60 to  $80\%$  (avg: 70), which corresponds to a 13 - 36 (avg: 25) attack success rate.
Sample Shielding effective against both word based and character based attacks. The results show effectiveness regardless of type of attack (word or character based). For example, all 4 attacks bring the original accuracy of LSTM from 88.3 down to  $\sim 0$  for IMDB. However, word sampling brings the accuracy back up to  $\sim 66$ . This is a great reduction in attack effectiveness. Again, similar trends are seen for the other classifiers, CNN is reduced from 94.1 to  $\leq 5.5$  for Yelp, but word sampling brings it back up to  $60 - 70\%$ .
Word sampling outperforms sentence sampling for LSTM, CNN, sentence sampling better for BERT. For example, for CNN on IMDB, word sampling increases accuracy more than 15 points over sentence sampling (69.8 vs 53.3). Similar trends hold for LSTM. However, the opposite is seen for BERT classifiers. For BERT on IMDB, we see an average of 6.5 higher points for sentence sampling over word sampling. These results are not surprising as LSTM and CNN leverage word embeddings for classification, while BERT leverages the context of the entire sentence.
Word sampling is more appropriate for short
texts. With AG news, we see a large drop in effectiveness of sentence sampling. The average length of AG News is 43 words compared to 157 and 215 words of Yelp and IMDB respectively (Li et al., 2020). This shorter length makes it more difficult to sample enough sentences. For Textfooler - CNN, sentence sampling is only able to increase accuracy from the attacked value of 0.4 to 13.2. However, word sampling is much more effective, increasing accuracy to 77.3. Text length may be crucial when choosing between the two strategies for a dataset.
Neural Network summarizer shows some improvements over majority voting. Comparisons of majority voting and the two neural net-based decision strategies are in Table 2. We experimented on the two binary datasets<sup>8</sup>. Replacing majority voting with a simple neural net (NN) gave somewhat disappointing results - accuracies stay the same or decrease slightly in all cases except for LSTM on the Yelp dataset (increases). However, when the neural nets are trained on perturbed texts (NN-BB), we see increases. For example, CNN vs TextFooler on Yelp, the neural net increases accuracy from 64.9 to 72.2, reducing attack success rate from 31 to 23. Possibly a more sophisticated neural net, such as a sequence aware LSTM, might better exploit patterns in the sorted probabilities.
4.2 Condition 2: Attacker knows about Sample Shielding
Results are in Table 3. As in the previous condition, classifiers perform well on original texts (Table 1) with BERT often achieving the highest accuracies. In this setting, every query by an attacker requires  $k$  samples to be processed, which greatly increases attack time. Thus, we reduce  $k$  to 30 for these experiments.
Sample Shielding repels attacks even when attacker uses Sample Shielding. We see that shielding is extremely successful in almost completely removing the negative effects of the attacks. For example, on the IMDB - TextFooer combination, attack success rate drops from 100 to 5 for LSTM, 100 to 1 for CNN, and 99 to 6 against BERT. The largest protection provided by Sample Shielding (100%) is for TextBugger vs CNN in IMDB. The smallest is for 85% (PWWS vs LSTM). On average the protection is 88.8%. The recovered accuracies are only 13 to 0 percent away from the originals.
<sup>8</sup>AG News was not included due to the complexity of translating multiple probabilities to a single input.
Classifier	Sample Shielding	Orig. Acc.	TextFollower	Bert-Attack	TextBugger	PWWS
Acc.	ASR	Acc.	ASR	Acc.	ASR	Acc.	ASR
IMDB	LSTM	No Shielding	88.3	0	100	0	100	0.3	100	0.1	100
Shielding-Sentence	85.1	61.4	30	62.0	30	60.3	32	56.2	36
Shielding-Word	85.1	66.0	25	67.0	24	66.0	25	65.7	26
CNN	No Shielding	86.2	0.1	100	0	100	0.3	100	0	100
Shielding-Sentence	84.5	55.3	36	55.2	36	53.6	38	48.9	43
Shielding-Word	84.7	69.8	19	66.7	23	71.6	17	67.8	21
BERT	No Shielding	91.3	1	99	3.7	96	9.2	90	0.7	99
Shielding-Sentence	91.5	78.1	14	79.2	13	80.1	12	78.0	15
Shielding-Word	86.8	74.4	19	71.5	22	78.8	14	63.4	31
Yelp	LSTM	No Shielding	92.5	0.3	100	0.7	99	5	95	1.5	98
Shielding-Sentence	90.0	62.3	33	61.1	34	60.5	35	58	37
Shielding-Word	87.8	65.5	29	66.7	28	68.5	26	61.9	33
CNN	No Shielding	94.1	0.8	99	0.4	100	5.5	94	2.4	97
Shielding-Sentence	91.7	58.5	38	54.1	43	57.1	39	50	47
Shielding-Word	88.1	64.9	31	62.2	34	70.4	25	60.2	36
BERT	No Shielding	100	5.9	94	8.3	92	15.5	85	4.9	95
Shielding-Sentence	98.6	74.8	25	72.6	27	79.3	21	68.5	32
Shielding-Word	93.5	69.9	30	75.1	25	78.7	21	71.1	29
AG News	LSTM	No Shielding	91.6	1.2	99	0.9	99	16.7	82	15.6	83
Shielding-Sentence	88.8	16.5	82	12.9	86	27.3	70	25.2	72
Shielding-Word	85.1	60.8	34	60.9	34	60.5	34	63.7	30
CNN	No Shielding	91.5	0.4	100	0.3	100	5.2	94	6.3	93
Shielding-Sentence	89.4	13.2	86	13.0	86	17.2	81	15.7	83
Shielding-Word	87.8	77.3	16	67.7	26	74.2	19	80	13
BERT	No Shielding	99.6	18.7	81	22.5	77	49.4	50	38.5	61
Shielding-Sentence	96.4	29.6	70	37.9	62	54.2	46	47.1	53
Shielding-Word	94.5	75.5	24	72.0	28	78.1	22	70.5	29
Table 1: Results where attacker does not know about Sample Shielding. Shielding settings:  $k = 100$ ,  $p = 0.3$ , majority voting. Acc: accuracy, ASR: success rate of attack (\%), Orig. Acc.: accuracy on original texts.
Classifier	Sampling Strategy	Orig. Acc.	TextFollower	Bert-Attack	TextBugger	PWWS
Acc.	SR	Acc.	SR	Acc.	SR	Acc.	SR
IMDB	LSTM	No Shielding	88.3	0	100	0	100	0.3	100	0.1	100
Maj. Vot.	85.1	66.0	25	67.0	24	66.0	25	65.7	26
NN	85.3	62.5	29	62.1	30	65.4	26	62.4	29
NN-BB	85.3	65.2	26	68.2	23	66.5	25	67.3	24
CNN	No Shielding	86.2	0.1	100	0	100	0.3	100	0	100
Maj. Vot.	84.7	69.8	19	66.7	23	71.6	17	67.8	21
NN	84.8	61.7	28	59.6	31	66.7	23	60.0	30
NN-BB	84.8	69.3	20	67.9	21	72.3	16	69.6	19
Yelp	LSTM	No Shielding	92.5	0.3	100	0.7	99	5	95	1.5	98
Maj. Vot.	87.8	65.5	29	66.7	28	68.5	26	61.9	33
NN	89.0	68.7	26	68.1	26	73.5	21	63.6	31
NN-BB	89	69.7	25	70.0	24	73.5	21	64.9	30
CNN	No Shielding	94.1	0.8	99	0.4	100	5.5	94	2.4	97
Maj. Vot.	88.1	64.9	31	62.2	34	70.4	25	60.2	36
NN	89.9	63.2	33	57.6	39	69.9	26	57.4	39
NN-BB	89.9	72.2	23	69.7	26	72.9	23	67.6	28
Table 2: Comparing vote summarizers. Settings:  $k = {100},p = {0.3}$  ,word sampling. Maj. Vot: majority voting, NN: neural network trained on original texts, NN-BB: neural network trained on original + perturbed texts.