SlowGuess/NAACL_2021_OCR huggingface dataset:
We perform human evaluation on the examples constructed by SO-Beam. Specifically, we randomly
5We report median due to the unreasonably large perplexity on certain sentences. e.g., 395 for that's a cheat. but 6740 for that proves perfect cheat.
$^{4}$ BAE is more effective on robust models as it may use replacement words outside the counter-fitted synonyms.
Naturalness (1-5)	Semantic Similarity (%)
Original	Perturb	Original	Perturb
3.87	3.63	85	71
Table 4: The quality metrics from human evaluation.
select 100 successful attacks and evaluate both the original examples and the vulnerable examples. To evaluate the naturalness of the constructed examples, we ask the annotators to score the likelihood (on a Likert scale of 1-5, 5 to be the most likely) of being an original example based on the grammar correctness. To evaluate the semantic similarity after applying the synonym substitution  $p$ , we ask the annotators to predict the sentiment of each example, and calculate the ratio of examples that maintain the same sentiment prediction after the synonym substitution. For both metrics, we take the median from 3 independent annotations. We use US-based annotators on Amazon's Mechanical Turk $^6$  and pay $0.03 per annotation, and expect each annotation to take 10 seconds on average (effectively, the hourly rate is about $11). See Appendix A.2 for more details.
As shown in Table 4, the naturalness score only drop slightly after the perturbation, indicating that our constructed vulnerable examples have similar naturalness as the original examples. As for the semantic similarity, we observe that  $85\%$  of the original examples maintain the same meaning after the synonym substitution, and the corresponding ratio is  $71\%$  for vulnerable examples. This indicates that the synonym substitution is an invariance transformation for most examples.
4 Evaluating Counterfactual Bias
In addition to evaluating second-order robustness, we further extend the double perturbation framework (§2) to evaluate counterfactual biases by setting  $\pmb{p}$  to pairs of protected tokens. We show that our method can reveal the hidden model bias.
4.1 Counterfactual Bias
In contrast to second-order robustness, where we consider the model vulnerable as long as there exists one vulnerable example, counterfactual bias focuses on the expected prediction, which is the average prediction among all examples within the neighborhood. We consider a model biased if the
Figure 4: An illustration of an unbiased model vs. a biased model. Green and gray indicate the probability of positive and negative predictions, respectively. Left: An unbiased model where the  $(x,x\oplus p)$  pair (yellowred dots) is relatively parallel to the decision boundary. Right: A biased model where the predictions for  $x\oplus p$  (red) are usually more negative (gray) than  $x$  (yellow).
expected predictions for protected groups are different (assuming the model is not intended to discriminate between these groups). For instance, a sentiment classifier is biased if the expected prediction for inputs containing woman is more positive (or negative) than inputs containing man. Such bias is harmful as they may make unfair decisions based on protected attributes, for example in situations such as hiring and college admission.
Counterfactual token bias. We study a narrow case of counterfactual bias, where counterfactual examples are constructed by substituting protected tokens in the input. A naive approach of measuring this bias is to construct counterfactual examples directly from the test set, however such evaluation may not be robust since test examples are only a small subset of natural sentences. Formally, let  $\pmb{p}$  be a pair of protected tokens such as (he, she) or (Asian, American),  $\mathcal{X}_{\mathrm{test}} \subseteq \mathcal{X}_p$  be a test set (as in §2.1), we define counterfactual token bias by:
$$
B _ {\boldsymbol {p}, k} := \underset {x \in \operatorname {N e i g h b o r} _ {k} \left(\mathcal {X} _ {\text {t e s t}}\right)} {\mathbb {E}} F _ {\text {s o f t}} (x; \boldsymbol {p}). \tag {5}
$$
We calculate Eq. (5) through an enumeration across all natural sentences within the neighborhood.7 Here  $\mathrm{Neighbor}_k(\mathcal{X}_{\mathrm{test}}) = \bigcup_{x\in \mathcal{X}_{\mathrm{test}}} \mathrm{Neighbor}_k(x)$  denotes the union of neighborhood examples (of distance  $k$ ) around the test set, and  $F_{\mathrm{soft}}(x;\pmb{p}) : \mathcal{X} \times \mathcal{V}^2 \to [-1,1]$  denotes the difference between probability outputs  $f_{\mathrm{soft}}$  (similar to Eq. (2)):
$$
F _ {\text {s o f t}} (x; \boldsymbol {p}) := f _ {\text {s o f t}} (x \oplus \boldsymbol {p}) - f _ {\text {s o f t}} (x). \tag {6}
$$
For gender bias, we employ a blacklist to avoid adding gendered tokens during the neighborhood construction. This is to avoid semantic shift when, for example,  $\pmb{p} = (\mathrm{he},\mathrm{she})$  such that it may refer to different tokens after the substitution.
$^{6}$ https://www.mturk.com
Patch Words	# Original	# Perturbed
he,she	5	325,401
his,her	4	255,245
him,her	4	233,803
men,women	3	192,504
man,woman	3	222,981
actor,actress	2	141,780
...
Total	34	2,317,635
Table 5: The number of original examples  $\left( {k = 0}\right)$  and the number of perturbed examples  $\left( {k = 3}\right)$  in  ${\mathcal{X}}_{\text{filter }}$  .
The model is unbiased on  $\pmb{p}$  if  $B_{\pmb{p},k} \approx 0$ , whereas a positive or negative  $B_{\pmb{p},k}$  indicates that the model shows preference or against to  $p^{(2)}$ , respectively. Fig. 4 illustrates the distribution of  $(x, x \oplus p)$  for both an unbiased model and a biased model.
The aforementioned neighborhood construction does not introduce additional bias. For instance, let  $x_0$  be a sentence containing  $\mathbf{he}$ , even though it is possible for  $\mathrm{Neighbor}_1(x_0)$  to contain many stereotyping sentences (e.g., contains tokens such as doctor and driving) that affect the distribution of  $f_{\mathrm{soft}}(x)$ , but it does not bias Eq. (6) as we only care about the prediction difference of replacing  $\mathbf{he} \rightarrow \mathbf{she}$ . The construction has no information about the model objective, thus it would be difficult to bias  $f_{\mathrm{soft}}(x)$  and  $f_{\mathrm{soft}}(x \oplus p)$  differently.
In this section, we use gender bias as a running example, and demonstrate the effectiveness of our method by revealing the hidden model bias. We provide additional results in Appendix A.4.
4.2.1 Setup
We evaluate counterfactual token bias on the SST-2 dataset with both the base and debiased models. We focus on binary gender bias and set  $\pmb{p}$  to pairs of gendered pronouns from Zhao et al. (2018a).
Base Model. We train a single layer LSTM with pre-trained GloVe embeddings and 75 hidden size (from TextAttack, Morris et al. 2020). The model has  $82.9\%$  accuracy similar to the baseline performance reported in GLUE.
Debiased Model. Data-augmentation with gender swapping has been shown effective in mitigating gender bias (Zhao et al., 2018a, 2019). We augment the training split by swapping all male entities with the corresponding female entities and vice-versa. We use the same setup as the base LSTM and attain  $82.45\%$  accuracy.
Figure 5: Our proposed  $B_{p,k}$  measured on  $\mathcal{X}_{\mathrm{filter}}$ . Here "original" is equivalent to  $k = 0$ , "perturbed" is equivalent to  $k = 3$ ,  $p$  is in the form of (male, female).
Metrics. We evaluate model bias through the proposed  $B_{p,k}$  for  $k = 0, \dots, 3$ . Here the bias for  $k = 0$  is effectively measured on the original test set, and the bias for  $k \geq 1$  is measured on our constructed neighborhood. We randomly sample a subset of constructed examples when  $k = 3$  due to the exponential complexity.
Filtered test set. To investigate whether our method is able to reveal model bias that was hidden in the test set, we construct a filtered test set on which the bias cannot be observed directly. Let  $\mathcal{X}_{\mathrm{test}}$  be the original validation split, we construct  $\mathcal{X}_{\mathrm{filter}}$  by the equation below and empirically set  $\epsilon = 0.005$ . We provide statistics in Table 5.
$$
\mathcal {X} _ {\text {f i l t e r}} := \{x \mid | F _ {\text {s o f t}} (x; \boldsymbol {p}) | <   \epsilon , x \in \mathcal {X} _ {\text {t e s t}} \}.
$$
Our method is able to reveal the hidden model bias on  $\mathcal{X}_{\mathrm{filter}}$ , which is not visible with naive measurements. In Fig. 5, the naive approach ( $k = 0$ ) observes very small biases on most tokens (as constructed). In contrast, when evaluated by our double perturbation framework ( $k = 3$ ), we are able to observe noticeable bias, where most  $p$  has a positive bias on the base model. This observed bias is in line with the measurements on the original  $\mathcal{X}_{\mathrm{test}}$  (Appendix A.4), indicating that we reveal the correct model bias. Furthermore, we observe mitigated biases in the debiased model, which demonstrates the effectiveness of data augmentation.
To demonstrate how our method reveals hidden bias, we conduct a case study with  $\pmb{p} =$  (actor, actress) and show the relationship between the bias  $B_{\pmb{p},k}$  and the neighborhood distance  $k$ . We present the histograms for  $F_{\mathrm{soft}}(x;\pmb{p})$  in Fig. 6 and plot the corresponding  $B_{\pmb{p},k}$  vs.  $k$  in the right-most panel. Surprisingly, for the base model, the bias is
Figure 6: Left and Middle: Histograms for  $F_{\mathrm{soft}}(x; \pmb{p})$  (x-axis) with  $\pmb{p} =$  (actor, actress). Right: The plot for the average  $F_{\mathrm{soft}}(x; \pmb{p})$  (i.e., counterfactual token bias) vs. neighborhood distance  $k$ . Results show that the counterfactual bias on  $\pmb{p}$  can be revealed when increasing  $k$ .
negative when  $k = 0$ , but becomes positive when  $k = 3$ . This is because the naive approach only has two test examples (Table 5) thus the measurement is not robust. In contrast, our method is able to construct 141,780 similar natural sentences when  $k = 3$  and shifts the distribution to the right (positive). As shown in the right-most panel, the bias is small when  $k = 1$ , and becomes more significant as  $k$  increases (larger neighborhood). As discussed in §4.1, the neighborhood construction does not introduce additional bias, and these results demonstrate the effectiveness of our method in revealing hidden model bias.
First-order robustness evaluation. A line of work has been proposed to study the vulnerability of natural language models, through transformations such as character-level perturbations (Ebrahimi et al., 2018), word-level perturbations (Jin et al., 2019; Ren et al., 2019; Yang et al., 2020; Hsieh et al., 2019; Cheng et al., 2020; Li et al., 2020), prepending or appending a sequence (Jia and Liang, 2017; Wallace et al., 2019a), and generative models (Zhao et al., 2018b). They focus on constructing adversarial examples from the test set that alter the prediction, whereas our methods focus on finding vulnerable examples beyond the test set whose prediction can be altered.
Robustness beyond the test set. Several works have studied model robustness beyond test sets but mostly focused on computer vision tasks. Zhang et al. (2019) demonstrate that a robustly trained model could still be vulnerable to small perturbations if the input comes from a distribution only slightly different than a normal test set (e.g., images with slightly different contrasts). Hendrycks and Dietterich (2019) study more sources of common corruptions such as brightness, motion blur and fog. Unlike in computer vision where simple
image transformations can be used, in our natural language setting, generating a valid example beyond test set is more challenging because language semantics and grammar must be maintained.
Counterfactual fairness. Kusner et al. (2017) propose counterfactual fairness and consider a model fair if changing the protected attributes does not affect the distribution of prediction. We follow the definition and focus on evaluating the counterfactual bias between pairs of protected tokens. Existing literature quantifies fairness on a test dataset or through templates (Feldman et al., 2015; Kiritchenko and Mohammad, 2018; May et al., 2019; Huang et al., 2020). For instance, Garg et al. (2019) quantify the absolute counterfactual token fairness gap on the test set; Prabhakaran et al. (2019) study perturbation sensitivity for named entities on a given set of corpus. Wallace et al. (2019b); Sheng et al. (2019, 2020) study how language generation models respond differently to prompt sentences containing mentions of different demographic groups. In contrast, our method quantifies the bias on the constructed neighborhood.
This work proposes the double perturbation framework to identify model weaknesses beyond the test dataset, and study a stronger notion of robustness and counterfactual bias. We hope that our work can stimulate the research on further improving the robustness and fairness of natural language models.
We thank anonymous reviewers for their helpful feedback. We thank UCLA-NLP group for the valuable discussions and comments. The research is supported NSF #1927554, #1901527, #2008173 and #2048280 and an Amazon Research Award.
Intended use. One primary goal of NLP models is the generalization to real-world inputs. However, existing test datasets and templates are often not comprehensive, and thus it is difficult to evaluate real-world performance (Recht et al., 2019; Ribeiro et al., 2020). Our work sheds a light on quantifying performance for inputs beyond the test dataset and help uncover model weaknesses prior to the real-world deployment.
Misuse potential. Similar to other existing adversarial attack methods (Ebrahimi et al., 2018; Jin et al., 2019; Zhao et al., 2018b), our second-order attacks can be used for finding vulnerable examples to a NLP system. Therefore, it is essential to study how to improve the robustness of NLP models against second-order attacks.
Limitations. While the core idea about the double perturbation framework is general, in §4, we consider only binary gender in the analysis of counterfactual fairness due to the restriction of the English corpus we used, which only have words associated with binary gender such as he/she, waiter/waitress, etc.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics.
Minhao Cheng, Jinfeng Yi, Pin-Yu Chen, Huan Zhang, and Cho-Jui Hsieh. 2020. Seq2sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):3601-3608.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES '18, page 67-73, New York, NY, USA. Association for Computing Machinery.
Krishnamurthy Dvijotham, Sven Gowal, Robert Stanforth, Relja Arandjelovic, Brendan O'Donoghue, Jonathan Uesato, and Pushmeet Kohli. 2018. Training verified learners with learned verifiers.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In Proceedings of the
56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31-36, Melbourne, Australia. Association for Computational Linguistics.
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 259-268.
Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H. Chi, and Alex Beutel. 2019. Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES '19, page 219-226, New York, NY, USA. Association for Computing Machinery.
Siddhant Garg and Goutham Ramakrishnan. 2020. BAE: BERT-based adversarial examples for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6174-6181, Online. Association for Computational Linguistics.
Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations.
Yu-Lun Hsieh, Minhao Cheng, Da-Cheng Juan, Wei Wei, Wen-Lian Hsu, and Cho-Jui Hsieh. 2019. On the robustness of self-attentive models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1520-1529.
Kuan-Hao Huang and Kai-Wei Chang. 2021. Generating syntactically controlled paraphrases without using annotated parallel pairs. In EACL.
Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Krishnamurthy Dvijotham, and Pushmeet Kohli. 2019. Achieving verified robustness to symbol substitutions via interval bound propagation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2020. Reducing sentiment bias in language models via counterfactual evaluation. Findings in EMNLP.
Mohit Iyyer, J. Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. ArXiv, abs/1804.06059.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on
Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 2021-2031. Association for Computational Linguistics.
Robin Jia, Aditi Raghunathan, Kerem Goksel, and Percy Liang. 2019. Certified robustness to adversarial word substitutions. In EMNLP/IJCNLP.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2019. Is bert really robust? a strong baseline for natural language attack on text classification and entailment.
Svetlana Kiritchenko and Saif Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 43-53, New Orleans, Louisiana. Association for Computational Linguistics.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual fairness. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4066-4076. Curran Associates, Inc.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. BERT-ATTACK: Adversarial attack against BERT using BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193–6202, Online. Association for Computational Linguistics.
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 622-628, Minneapolis, Minnesota. Association for Computational Linguistics.
Takeru Miyato, Andrew M. Dai, and Ian Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. ICLR.
John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp.
Nikola Mrkšić, Diarmuid Řeaghdha, Blaise Thomson, Milica Gašić, Lina M. Rojas-Barahona, PeiHao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 142-148, San Diego, California. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Vinodkumar Prabhakaran, Ben Hutchinson, and Margaret Mitchell. 2019. Perturbation sensitivity analysis to detect unintended model biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5740-5745, Hong Kong, China. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. Do ImageNet classifiers generalize to ImageNet? volume 97 of Proceedings of Machine Learning Research, pages 5389-5400, Long Beach, California, USA. PMLR.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1085-1097, Florence, Italy. Association for Computational Linguistics.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of nlp models with checklist. In *Association for Computational Linguistics (ACL)*.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In EMNLP.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2020. Towards controllable biases in language generation. In EMNLP-Finding.
Zhouxing Shi, Huan Zhang, Kai-Wei Chang, Minlie Huang, and Cho-Jui Hsieh. 2020. Robustness verification for transformers. In International Conference on Learning Representations.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.