SlowGuess/NAACL_2021_OCR huggingface dataset:
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019a. Universal adversarial triggers for attacking and analyzing nlp. In EMNLP/IJCNLP.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019b. Universal adversarial triggers for attacking and analyzing NLP. In EMNLP.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface's transformers: State-of-the-art natural language processing.
Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai-Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. 2020. Automatic perturbation analysis for scalable certified robustness and beyond.
Puyudi Yang, Jianbo Chen, Cho-Jui Hsieh, Jane-Ling Wang, and Michael I. Jordan. 2020. Greedy attack and gumbel attack: Generating adversarial examples for discrete data. Journal of Machine Learning Research, 21(43):1-36.
Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020. Word-level textual adversarial attacking as combinatorial optimization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Huan Zhang, Hongge Chen, Zhao Song, Duane Boning, inderjit dhillon, and Cho-Jui Hsieh. 2019. The limitations of adversarial training and the blind-spot attack. In International Conference on Learning Representations.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Coterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In NAACL.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018a. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018b. Generating natural adversarial examples. In International Conference on Learning Representations.
Yi Zhou, Xiaqing Zheng, Cho-Jui Hsieh, Kai-Wei Chang, and Xuanjing Huang. 2020. Defense against adversarial attacks in nlp via dirichlet neighborhood ensemble. arXiv preprint arXiv:2006.11627.
A.1 Random Baseline
To validate the effectiveness of minimizing Eq. (4), we also experiment on a second-order baseline that constructs vulnerable examples by randomly replacing up to 6 words. We use the same masked language model and threshold as SO-Beam such that they share a similar neighborhood. We perform the attack on the same models as Table 2, and the attack success rates on robustly trained BoW, CNN, LSTM, and Transformers are  $18.8\%$ ,  $22.3\%$ ,  $15.2\%$ , and  $25.1\%$ , respectively. Despite being a second-order attack, the random baseline has low attack success rates thus demonstrates the effectiveness of SO-Beam.
A.2 Human Evaluation
We randomly select 100 successful attacks from SO-Beam and consider four types of examples (for a total of 400 examples): The original examples with and without synonym substitution  $\pmb{p}$ , and the vulnerable examples with and without synonym substitution  $\pmb{p}$ . For each example, we annotate the naturalness and sentiment separately as described below.
Naturalness of vulnerable examples. We ask the annotators to score the likelihood of being an original example (i.e., not altered by computer) based on grammar correctness and naturalness, with a Likert scale of 1-5: (1) Sure adversarial example. (2) Likely an adversarial example. (3) Neutral. (4) Likely an original example. (5) Sure original example.
Semantic similarity after the synonym substitution. We first ask the annotators to predict the sentiment on a Likert scale of 1-5, and then map the prediction to three categories: negative, neutral, and positive. We consider two examples to have the same semantic meaning if and only if they are both positive or negative.
A.3 Running Time
We experiment with the validation split on a single RTX 3090, and measure the average running time per example. As shown in Table 6, SO-Beam runs faster than SO-Enum since it utilizes the probability output. The running time may increase if the model has improved second-order robustness.
Running Time (seconds)
Genetic	BAE	SO-Enum	SO-Beam
Base Models:
BoW	31.6	0.9	6.2	1.8
CNN	28.8	1.0	5.9	1.7
LSTM	31.9	1.1	7.0	1.9
Transformer	51.9	0.5	6.5	2.5
BERT-base	65.6	1.1	35.4	7.1
Robust Models:
BoW	103.9	1.0	8.0	3.5
CNN	129.4	1.0	6.7	2.6
LSTM	116.4	1.1	10.7	5.3
Transformer	66.4	0.5	5.9	2.6
A.4 Additional Results on Protected Tokens
Fig. 7 presents the experimental results with additional protected tokens such as nationality, religion, and sexual orientation (from Ribeiro et al. (2020)). We use the same base LSTM as described in §4.2. One interesting observation is when  $p =$  (gay, straight) where the bias is negative, indicating that the sentiment classifier tends to give more negative prediction when substituting gay → straight in the input. This phenomenon is opposite to the behavior of toxicity classifiers (Dixon et al., 2018), and we hypothesize that it may be caused by the different distribution of training data. To verify the hypothesis, we count the number of training examples containing each word, and observe that we have far more negative examples than positive examples among those containing straight (Table 7). After looking into the training set, it turns out that straight to video is a common phrase to criticize a film, thus the classifier incorrectly correlates straight with negative sentiment. This also reveals the limitation of our method on polysemous words.
Table 6: The average running time over 872 examples (100 for Genetic due to long running time).
# Negative	# Positive
gay	37	20
straight	71	18
Table 7: Number of negative and positive examples containing gay and straight in the training set.
In Fig. 8, we measure the bias on  $\mathcal{X}_{\mathrm{test}}$  and observe positive bias on most tokens for both  $k = 0$  and  $k = 3$ , which indicates that the model "tends" to make more positive predictions for examples containing certain female pronouns than male pro
Figure 7: Additional counterfactual token bias measured on the original validation split with base LSTM.
nouns. Notice that even though gender swap mitigates the bias to some extent, it is still difficult to fully eliminate the bias. This is probably caused by tuples like (him, his, her) which cannot be swapped perfectly, and requires additional processing such as part-of-speech resolving (Zhao et al., 2018a).
Figure 8: Full results for gendered tokens measured on the original validation split.
To help evaluate the naturalness of our constructed examples used in §4, we provide sample sentences in Table 9 and Table 10. Bold words are the corresponding patch words  $\pmb{p}$ , taken from the predefined list of gendered pronouns.
A.5 Additional Results on Robustness
Table 8 provides the quality metrics for first-order attacks, where we measure the GPT-2 perplexity and  $\ell_0$  norm distance between the input and the adversarial example. For BAE we evaluate on 872 validation examples, and for Genetic we evaluate on 100 validation examples due to the long running time.
Table 11 shows additional attack results from
Genetic	BAE
Original PPL	Perturb PPL	\( \ell_0 \)	Original PPL	Perturb PPL	\( \ell_0 \)
Base Models:
BoW	145	258	3.3	192	268	1.6
CNN	146	282	3.0	186	254	1.5
LSTM	131	238	2.9	190	263	1.6
Transformer	137	232	2.8	185	254	1.4
BERT-base	201	342	3.4	189	277	1.6
Robust Models:
BoW	132	177	2.4	214	269	1.5
CNN	136	236	2.7	211	279	1.5
LSTM	163	267	2.5	220	302	1.6
Transformer	118	200	2.8	196	261	1.4
Table 8: The quality metrics for first-order attacks from successful attacks. We compare median perplexities (PPL) and average  $\ell_0$  norm distances.
SO-Beam on base LSTM, and Table 12 shows additional attack results from SO-Beam on robust CNN. Bold words are the corresponding patch words  $\pmb{p}$ , taken from the predefined list of counter-fitted synonyms.
Type	Predictions	Text
Original	95% Negative	94% Negative	it's hampered by a lifetime-channel kind of plot and a lead actor (actress) who is out of their depth.
Distance k=1	97% Negative	(97% Negative)	it's hampered by a lifetime-channel kind of plot and lone lead actor (actress) who is out of their depth.
56% Negative	(55% Positive)	it's hampered by a lifetime-channel kind of plot and a lead actor (actress) who is out of creative depth.
89% Negative	(84% Negative)	it's hampered by a lifetime-channel kind of plot and a lead actor (actress) who talks out of their depth.
98% Negative	(98% Negative)	it's hampered by a lifetime-channel kind of plot and a lead actor (actress) who is out of production depth.
96% Negative	(96% Negative)	it's hampered by a lifetime-channel kind of plot and a lead actor (actress) that is out of their depth.
Distance k=2	88% Negative	(87% Negative)	it's hampered by a lifetime-channel cast of stars and a lead actor (actress) who is out of their depth.
96% Negative	(95% Negative)	it's hampered by a simple set of plot and a lead actor (actress) who is out of their depth.
54% Negative	(54% Negative)	it's framed about a lifetime-channel kind of plot and a lead actor (actress) who is out of their depth.
90% Negative	(88% Negative)	it's hampered by a lifetime-channel mix between plot and a lead actor (actress) who is out of their depth.
78% Negative	(68% Negative)	it's hampered by a lifetime-channel kind of plot and a lead actor (actress) who storms out of their mind.
Distance k=3	52% Positive	(64% Positive)	it's characterized by a lifetime-channel combination comedy plot and a lead actor (actress) who is out of their depth.
93% Negative	(93% Negative)	it's hampered by a lifetime-channel kind of star and a lead actor (actress) who falls out of their depth.
58% Negative	(57% Negative)	it's hampered by a tough kind of singer and a lead actor (actress) who is out of their teens.
70% Negative	(52% Negative)	it's hampered with a lifetime-channel kind of plot and a lead actor (actress) who operates regardless of their depth.
58% Negative	(53% Positive)	it's hampered with a lifetime-channel cast of plot and a lead actor (actress) who is out of creative depth.
Table 9: Additional counterfactual bias examples on base LSTM with  $\mathbf{p} =$  (actor,actress). We only present 5 examples per  $k$  due to space constrain.
Type	Predictions	Text
Original	55% Positive	(67% Positive)	a hamfisted romantic comedy that makes our boy (girl) the hapless facilitator of an extended cheap shot across the mason-dixon line.
Distance k = 1	52% Positive	(66% Positive)	a hamfisted romantic comedy that makes our boy (girl) the hapless facilitator of an extended cheap shot from the mason-dixon line.
73% Positive	(79% Positive)	a hamfisted romantic comedy that makes our boy (girl) the hapless facilitator gives an extended cheap shot across the mason-dixon line.
56% Negative	(58% Positive)	a hamfisted romantic comedy that makes our boy (girl) the hapless facilitator of an extended cheap shot across the phone line.
75% Positive	(83% Positive)	a hamfisted romantic comedy that makes our boy (girl) the hapless facilitator of an extended chase shot across the mason-dixon line.
75% Positive	(81% Positive)	a hamfisted romantic comedy that makes our boy (girl) our hapless facilitator of an extended cheap shot across the mason-dixon line.
Distance k = 2	85% Positive	(85% Positive)	a hilarious romantic comedy that makes our boy (girl) the hapless facilitator of an emotionally cheap shot across the mason-dixon line.
81% Positive	(86% Positive)	a hamfisted romantic comedy romance makes our boy (girl) the hapless facilitator of an extended cheap delivery across the mason-dixon line.
84% Positive	(87% Positive)	a hamfisted romantic romance adventure makes our boy (girl) the hapless facilitator of an extended cheap shot across the mason-dixon line.
50% Negative	(62% Positive)	a hamfisted romantic comedy that makes our boy (girl) the hapless boss of an extended cheap shot behind the mason-dixon line.
77% Negative	(71% Negative)	a hamfisted lesbian comedy that makes our boy (girl) the hapless facilitator of an extended slap shot across the mason-dixon line.
Distance k = 3	97% Positive	(97% Positive)	a darkly romantic comedy romance makes our boy (girl) the hapless facilitator delivers an extended cheap shot across the mason-dixon line.
69% Positive	(74% Positive)	a hamfisted romantic comedy film makes our boy (girl) the hapless facilitator of an extended cheap shot across the production line.
87% Positive	(89% Positive)	a hamfisted romantic comedy that makes our boy (girl) the exclusive focus of an extended cheap shot across the mason-dixon line.
64% Positive	(76% Positive)	a hamfisted romantic comedy that makes our boy (girl) the hapless facilitator shoots an extended flash shot across the camera line.
99% Positive	(99% Positive)	a compelling romantic comedy that makes our boy (girl) the perfect facilitator of an extended story shot across the mason-dixon line.
Table 10: Additional counterfactual bias examples on base LSTM with  $\mathbf{p} =$  (boy,girl). We only present 5 examples per  $k$  due to space constrain.
Table 11: Additional sentiment classification results from SO-Beam on base LSTM.
Table 12: Additional sentiment classification results from SO-Beam on robust CNN.