SlowGuess/NAACL_2021_OCR huggingface dataset:
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge Belongie. 2018. Generative adversarial perturbations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4422-4431.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing.
Congzheng Song, Alexander M Rush, and Vitaly Shmatikov. 2020. Adversarial semantic collisions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4198-4210.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. In Empirical Methods in Natural Language Processing.
Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. 2018. Generating adversarial examples with adversarial networks. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence.
Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M Rush, and Yann LeCun. 2018a. Adversarily regularized autoencoders. In International Conference on Machine Learning (ICML).
Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018b. Generating natural adversarial examples. In International Conference on Learning Representations (ICLR).
Hyperparameter search For our gradient-based attack approach (Equation (1) in the main paper), there are three hyperparameters: the  $l_{2}$  norm budget  $\epsilon$  of the adversarial perturbation, the number of attack steps  $T$ , and the step size  $\eta$  in each attack step. Among them,  $\epsilon$  is super critical for our attacks. A too small  $\epsilon$  limits the search space over the ARAE (Zhao et al., 2018a) noise input, thus leads to a low attack success. A too large  $\epsilon$  changes the noise input significantly, thus leads to unnatural trigger generations. In our experiments, we use grid search to manually try different settings of these hyperparameter values:  $\epsilon$  is selected from  $\{2,5,10,20,50\}$ ;  $T$  is selected from  $\{500,1000,2000,5000\}$ ; and  $\eta$  is selected from  $\{10,100,1000,10000\}$ . Based on the attack success and the naturalness of generated triggers, we finally set  $\epsilon = 10$ ,  $T = 1000$ , and  $\eta = 1000$ .
Dataset and attack details We perform all the attack experiments on a single NVIDIA Tesla P100 GPU. For the sentiment analysis task, we use the binary Stanford Sentiment Treebank (SST) (Socher et al., 2013), which has 6,920 examples in the training set, 872 examples in the development set, and 1,821 examples in the test set. The SST classifier uses a two-layer LSTM (Hochreiter and Schmidhuber, 1997) followed by a linear layer for sentiment prediction, with 8.7 million parameters in total. When attacking the SST classifier, it takes around 2 minutes to generate the final trigger.
For the natural language inference task, we use the Stanford Natural Language Inference (SNLI) dataset (Bowman et al., 2015), which has 549, 367 examples in the training set, 9,842 examples in the development set, and 9,824 examples in the test set. The SNLI classifier is a pretrained Enhanced Sequential Inference Model (Chen et al., 2017) provided by AllenNLP (Gardner et al., 2018). It has 14.5 million parameters in total. When attacking the SNLI classifier, it takes around 27 minutes to generate the final trigger.
B.1 Attack Results with Different Trigger Lengths
Table 5 provides examples of attacks with varying lengths, along with their corresponding classifier accuracies (lower numbers indicate more successful attacks).
B.2 Attack Results on the Development Set and the Test Set
In our experiments, the attack trigger is first generated by increasing the target classifier's loss on the development set, and then applied on the test set to measure its success. Here, we present both the development accuracy and the test accuracy under the same attack triggers in Table 6. We can see that although generated by only attacking the development set, the trigger also works well on the test set: it causes similar accuracy drop on both the development set and the test set.
B.3 Naive Attacks with Random Triggers
In this section, we check how difficult it is to attack a certain task by implementing two naive attacks without gradient information. In the first attack method ("Random ARAE"), we randomly collect the candidate triggers generated by the ARAE model (Zhao et al., 2018a), compute the classifier accuracy for each trigger, and finally select the attack trigger as the one with lowest classifier accuracy. We can consider this attack as a simplified version of our attack (NUTS) by removing the gradient information. The second attack method ("Random outputs") is similar as the first one, except that we do not enforce the naturalness of the triggers: we select the attack trigger with the lowest classifier accuracy from many random word sequences. We can also consider this attack as a much simplified version of the baseline attack (Wallace et al., 2019). For both naive attacks, following our gradient-based attack, we select the final trigger from 256 candidates triggers for a fair comparison.
Table 7 shows all the attack results. First, we observe that these two naive attacks ("Random ARAE" and "Random outputs") are quite successful in attacking entailment and neutral examples in the SNLI task: they successfully decrease the classifier accuracy to  $0\%$  and  $3.45\%$ . This indicates that those examples are quite easy to be attacked. Second, for both positive and negative examples in the SST task and the contradiction examples in the SNLI task, the success of these two naive attacks is quite limited. We also observe a significant improvement on the attack success with these two gradient-based attacks correspondingly.
Task	Trigger length	Test data	NUTS (our attack)	Baseline (Wallace et al., 2019)
Trigger	Classifier accuracy	Trigger	Classifier accuracy
SST	No trigger	+	-	89.00%	-	89.00%
-	-	82.57%	-	82.57%
3	+	but neither the combined energy efficiency	43.01%	drown soggy timeout	18.92%
-	-	19.96%	vividly riveting soar	9.10%
5	+	a flat explosion empty over they can deeply restore our	25.85%	drown soggy mixes soggy timeout	10.67%
-	-	17.11%	captures stamina lifetime without prevents	5.26%
8	+	the accident forced the empty windows shut down will deliver a deeply affected children from parents	26.95%	collapses soggy timeout energy energy freshness intellect genitals sunny vitality blessed lifetime lifetime counterparts without pitfalls	15.51%
-	-	8.55%	counterparts without pitfalls	2.85%
SNLI	No trigger	+	-	89.76%	-	89.76%
0	-	86.52%	-	86.52%
3	+	-	79.83%	-	79.83%
0	he was jailed there is no he could leave	0.06%	alien spacecraft naked spaceship cats zombies humans possesses energies	0.00% 0.06%
-	-	2.52%	alien spacecraft nothing eat no cats running indoors destroy no mammals tall beings interact near	47.20% 0.00%
5	+	a man stabbed his son there is no one or he likes to inspire creativity	0.03% 2.27%	alien spacecraft nothing eat no cats running indoors destroy no mammals tall beings interact near	13.44% 0.00%
0	-	40.07%	mall destruction alien whatsoever shark pasture picnic no cats rounds murder pandas in alien spacecraft mars human humans initiate accomplishment energies near objects near	0.00% 0.00%
8	+	some black women taking the photo last month	0.00%	-	-
0	the man drowned in hospital and died in	3.26%	cats rounds murder pandas in alien spacecraft mars	0.00%
-	they are helping for training achievement for a	26.78%	human humans initiate accomplishment energies near objects near	23.02%
Table 5: Attack results on SST and SNLI: Compared to the baseline (Wallace et al., 2019), our attacks are slightly less successful at reducing test accuracy but generate more natural triggers. For SST, “+”=positive, “-”=negative sentiment. For SNLI, “+”=entailment, “0”=neutral, and “-”=contradiction. Lower numbers are better. ‘No trigger’=classifier accuracy without any attack.
B.4 Attack Results without GPT-2 Based Reranking
The GPT-2 based reranking is used to balance attack success and trigger naturalness. Without GPT-2 based reranking, the selected trigger will have a slightly higher attack success, however with significantly larger GPT-2 loss. For SST, without reranking, our attack triggers decrease accuracy to  $26.84\%$  and  $7.68\%$  on positive and negative data, but GPT-2 losses increase from 6.85 (or 6.65) to 8.80 (or 8.88) for positive (or negative) data.
B.5 Variance over Candidate Triggers
For our attacks against negative SST data with trigger length of 8, among all 256 candidate triggers, the average classifier accuracy after attack is 0.23 with a standard deviation of 0.10; and the average GPT-2 loss is 7.93 with a standard deviation of 0.85. There is no inherent tradeoff between naturalness and attack success: some triggers have both low classifier accuracy and low GPT-2 loss, and the Pearson correlation is -0.08.
C Human-Subject Study Details
We perform the human-subject study on Amazon Mechanical Turk. Crowdworkers were required to have a  $98\%$  HIT acceptance rate and a minimum of 5000 HITs. Workers were asked to spend a maximum of 5 minutes on each assignment (i.e., comparing the naturalness of a pair of our trigger vs baseline trigger, or evaluating the naturalness of a piece of text), and paid $0.01 for each assignment.
Task	Trigger length	Data	NUTS (our attack)	Baseline
Accuracy (dev set)	Accuracy (test set)	Accuracy (dev set)	Accuracy (test set)
SST	No trigger	+	88.29%	89.00%	88.29%	89.00%
-	82.94%	82.57%	82.94%	82.57%
3	+	40.54%	43.01%	20.27%	18.92%
-	21.26%	19.96%	10.51%	9.10%
5	+	26.35%	25.85%	12.39%	10.67%
-	18.46%	17.11%	6.31%	5.26%
8	+	27.25%	26.95%	17.79%	15.51%
-	10.05%	8.55%	1.87%	2.85%
SNLI	No trigger	+	90.96%	89.76%	90.96%	89.76%
0	88.07%	86.52%	88.07%	86.52%
3	+	79.53%	79.83%	79.53%	79.83%
0	0.03%	0.06%	0.00%	0.00%
0	2.53%	2.52%	0.00%	0.06%
5	+	54.58%	54.56%	46.55%	47.20%
0	0.00%	0.03%	0.00%	0.00%
0	1.82%	2.27%	0.00%	0.00%
8	+	39.48%	40.07%	13.24%	13.44%
0	0.00%	0.00%	0.00%	0.00%
0	3.74%	3.36%	0.00%	0.00%
-	25.90%	26.78%	22.76%	23.02%
Table 6: Universal attack results on both the development (dev) set and the test set for the Stanford Sentiment Treebank (SST) classifier and the Stanford Natural Language Inference (SNLI) classifier. For SST, “+”=positive, “-”=negative sentiment. For SNLI, “+”=entailment, “0”=neutral, and “-”=contradiction. We first generate the attack trigger by increasing the classifier's loss on the dev set, and then apply the same trigger on the test set. ‘No trigger' refers to classifier accuracy without any attack. We can observe that the same triggers achieve similar attack success in both the development set and the test set.
Attack method	Trigger length	SST	SNLI
Positive	Negative	Entail-ment	Neutral	Contrad-iction
No attack	-	89.00%	82.57%	89.76%	86.52%	79.83%
NUTS (Our attack)	3	43.01%	19.96%	0.06%	2.52%	54.56%
5	25.85%	17.11%	0.03%	2.27%	40.07%
8	26.95%	8.55%	0.00%	3.26%	26.78%
Random ARAE	3	54.46%	66.78%	0.09%	11.59%	58.02%
5	50.28%	43.75%	0.00%	13.36%	55.51%
8	43.23%	39.69%	0.03%	8.01%	42.79%
Baseline attack	3	18.92%	9.10%	0.00%	0.06%	47.20%
5	10.67%	5.26%	0.00%	0.00%	13.44%
8	15.51%	2.85%	0.00%	0.00%	23.02%
Random outputs	3	49.17%	32.79%	0.36%	17.43%	61.48%
5	47.19%	23.90%	0.00%	3.45%	53.35%
8	41.58%	20.07%	0.00%	7.80%	50.82%
Table 7: Universal attack results on both the Stanford Sentiment Treebank (SST) classifier and the Stanford Natural Language Inference (SNLI) classifier. Besides gradient-based attacks including our attack (NUTS) and the baseline attack (Wallace et al., 2019), we further implement two naive attacks without gradient-guided search: "Random ARAE" means we select the best attack trigger from random natural ARAE outputs; "Random outputs" represents we select the best attack trigger from random unnatural word sequences.