SlowGuess/NAACL_2022_OCR huggingface dataset:
BBQ BBQ contains two formats: ambiguous (Fig. 2a) and disambiguated (Fig. 2b). We convert the same set of examples into NLI form as demonstrated in Fig. 2c-d, yielding BBQ→NLI. When the model is prompted in different ways, predictions for semantically identical examples yield vastly different distributions. Similar to the case of BBNLI, T0 fails to answer with neutral/unknown and points at one of the target or non-target options for the mentioned behavior (e.g. planting a bomb). In Table 4, when prompted in QA form using prompt templates that appeared in training, T0 often answers negative questions with the target answer and non-negative questions with the non-target answer, resulting in higher bias scores than NLI form, with approximately 44 and 37 (over 100) for gender and religion, respectively. While scores for NLI are also positive, they are much smaller in comparison. Moreover, bias scores for  $\mathrm{QA}^n$  are smaller than those of QA, but they are still significantly above NLI form. We speculate that the novelty of task has a greater effect on biased outputs than the novelty prompt templates.
In Table 5, we consider disambiguated examples for BBQ and provide bias scores. We also provide mean accuracies, in parentheses, for the complete set in Fig. 2d. Irrespective of biases, accuracy shows a model's ability in handling the task overall. We use the bias score formulae in Table 2 and Table 9 (in the appendix) for respective forms of the BBQ dataset. Note that a perfect accuracy in disambiguated examples yields a bias score of 0. In gender, QA achieves a near-perfect accuracy with  $99\%$  resulting in a smaller bias score. Religion exemplifies the case where accuracies for NLI and QA are fairly close, yet the predictions for the training task QA is more biased
2We provide additional results in the appendix (Table 8) for T0 checkpoint which agree with T0pp. In the text we often refer to T0pp as T0 throughout for brevity.
3 https://github.com/bigscience-workshop/promptsource
4 https://huggingface.co/inference-api
5 Inference API does not offer customization of decoding parameters for T0-suite as of this writing. Default parameters are available through the generate method's documentation for transformers.
Input Form	Gender	Race	Religion
Bias Score ↓
QA	5.13 (99%)	3.98 (86%)	14.51 (83%)
QAn	3.85 (99%)	6.68 (87%)	14.94 (83%)
NLI	10.26 (92%)	4.61 (87%)	4.41 (81%)
Accpro - Accanti ↓
QA	2.56 (99%)	3.19 (86%)	7.09 (83%)
QAn	2.91 (99%)	3.99 (87%)	7.47 (83%)
NLI	5.13 (92%)	2.30 (87%)	2.20 (81%)
Table 5: BBQ results of T0pp outputs where input is in question answering (QA), QA with novel prompts  $(\mathrm{QA}^n)$  and  $\mathrm{BBQ}\rightarrow \mathrm{NLI}$  (NLI). Context/premise are disambiguated. Mean accuracies for pro- and anti-stereotypical hypotheses are in (parentheses). Note that  $100\%$  mean accuracy results in a bias score of 0. We provide two different measure of bias: bias score and differences in accuracies  $\mathrm{Acc}_{pro} - \mathrm{Acc}_{anti}$ . Formulae for bias score is provided in Table 2. Differences between accuracies are computed when the disambiguated context is pro-stereotypical compared to when it is anti-stereotypical. This metric is an alternative indicator of biases exhibited by the model: it quantifies how much more successful the model is given a harmful stereotype in the context compared to an anti-stereotypical scenario.
than NLI.  $\mathrm{QA}^n$  is always higher than NLI form with no consistent advantage over QA. Table 5 also provides the differences in accuracies given a stereotypical example versus an anti-stereotypical example as in Fig. 2d. The model's ability to better handle pro-stereotypical scenarios, as opposed to anti-stereotypical, suggests another form of bias called allocational bias (Blodgett et al., 2020). Using this simple metric, we observe the same pattern as in bias scores where QA form results in more bias than NLI when accuracies are similar.
Is NLI less biased because it outputs random answers? In order to assess effectiveness of T0 to handle the premises in BBNLI, we use our test hypotheses in Table 6. We observe that model performs significantly better than chance in both forms and the accuracies are similar (NLI is slightly better)—suggesting that the model does not make random predictions, yet the predictions differ in their bias scores. We also consider differences given the pro- vs anti-stereotypical forms and find positive difference. For example in man_is_to_surgeon, pro-stereotypical premises suggest that women are less likely to become surgeons than men—which T0 is able to handle better
Figure 3: Bias scores for T0pp predictions using ambiguous contexts described in Table 7.
than the case when women surgeons are more likely.
What other factors contribute to biased answers? In Fig. 3, we observe that predictions are affected by (1) the order of names (e.g. "one Muslim man and one Christian man" vs "one Christian man and one Muslim man") as they appear in the input, also suggested by Parrish et al. (2021), (2) the length of the premise/context, and/or (3) details provided in the context/premise. First, we observe that in all three formats (examples shown in Table 7), but especially for ambiguous + short, the order in which target and non-target mentions appear is a significant predictor of model's answers hence the bias score. In QA, while addition of short causes a dip in bias score on average, it rises again given the additional information in long.
What causes a training input form to result in more bias than a novel form? It is highly likely that the question answering datasets (a few dozens were used in training T0) contain biases (Parrish et al., 2021) which makes it easy for T0 to exhibit stereotypical associations learned during the training when faced with this task form. In this familiar form, the model is also more likely to rely on spurious correlations when providing answers rather than generating a correct answer (e.g. "Unknown"). Within the scope of this paper, we argue that such associations cannot be consistently prevented by simply using novel prompt templates, however, more substantial changes such as the ones presented in BBNLI  $\rightarrow$  QA and BBQ  $\rightarrow$  NLI may be helpful.
Subtopic	BBNLI	BBNLI→QA
Accanti	Accpro	Accpro - Accanti	Accanti	Accpro	Accpro - Accanti
asian_men_to_feminine	0.48	0.57	0.09	0.47	0.56	0.09
black_is_to_criminal	0.64	0.64	0.00	0.44	0.53	0.08
black_is_to_impoverished	0.65	0.73	0.08	0.73	0.75	0.02
man_is_to-moneymanager*	0.68	0.65	0.02	0.60	0.55	0.05
man_is_to_breadwinner	0.38	0.40	0.02	0.40	0.38	-0.02
man_is_to Programmer	0.61	0.79	0.18	0.48	0.71	0.23
man_is_toSurgeon	0.53	0.61	0.08	0.43	0.56	0.13
catholic_woman_to_manykids*	0.75	0.75	0.00	0.75	0.71	0.04
muslim_man_to_oppressive	0.50	0.50	0.00	0.47	0.50	0.03
Average	0.58	0.63	0.05	0.53	0.58	0.07
Table 6: Difference (pro-anti) between test hypothesis/question accuracies for T0pp. We only list the subtopics whose (premise, test hypothesis) pairs are different for anti- and pro-stereotypical. * indicates that the pro-stereotypical premise is semantically in favor of the target group, hence the difference for the corresponding line is  $\mathrm{Acc}_{\text {anti }} - \mathrm{Acc}_{\text {pro }}$ . We compare BBNLI to BBNLI→QA and highlight highest difference.
Type	Context/Premise
ambiguous	A man and a woman are attending an academic conference.
ambiguous + short	A man and a woman are attending an academic conference, one was working at the front desk while the other was a researcher.
ambiguous + short + long	A man and a woman are attending an academic conference., one was working at the front desk while the other was a researcher. One of them is presenting a paper at a workshop while the other is working at the front desk making sure all the attendees get checked in.
Table 7: Alternative ambiguous contexts for BBQ. Negative and non-negative questions corresponding to this example are "Who is not involved in research?" and "Who is the researcher?", respectively. ambiguous is the original phrase provided in the BBQ dataset. In short, using abovementioned questions, we spell out the roles (i.e. research vs front-desk position) without indicating who is whom. Lastly in long, we use the disambiguating information that comes with BBQ, again obscuring who is responsible for which position.
How can the ideas presented in this study be bootstrapped from in debiasing efforts?
Within the scope of the problems and models considered in this work, we observe that presenting an input to a language model in a novel form results in less biased predictions. While we cannot control user-created queries in client-facing applications, we have control on the training data we use in developing our models. Hence for future work, one idea that is worth testing in multi-task learning is whether limiting the set of training tasks to those that are not immediately interesting to layperson and holding out "popular" tasks for testing would result in less biased predictions in popular tasks such as question answering.
In order to obtain a strong task-specific model to tackle various NLP tasks, the de facto practice has been to use a pretrained language model and fine-tune it on a downstream task (Alberti et al., 2019; Akyurek et al., 2020; Khashabi et al., 2020). We call these specific checkpoints of language models tailored for a particular downstream
task task-conditioned LMs and non-conditioned versions general-purpose LMs. Previous work established that both types of models exhibit social biases (Zhao et al., 2019; Schick et al., 2021). In the following parts, we discuss efforts aiming at systemically quantifying these biases in LMs.
Measuring Bias in Task-Conditioned Language Models Several benchmarks and metrics have been proposed to measure bias in coreference resolution (Zhao et al., 2018), text generation (Sheng et al., 2019; Kraft, 2021; Dhamala et al., 2021; Nozza et al., 2021)—or more specifically story completion (Lucy and Bamman, 2021), abusive language detection (Park et al., 2018), sentiment analysis (Kiritchenko and Mohammad, 2018) and for the tasks of interest to this work: question answering (Parrish et al., 2021; Li et al., 2020) and natural language inference (Dev et al., 2020; Dawkins, 2021; Sharma et al., 2021). These works take a step forward in bridging the gap between how biases are measured and what the model is actually been trained on and used for (Dev et al., 2020).
Measuring Bias in General-Purpose or Multi-task Language Models CrowS-Pairs (Nangia et al., 2020) is a collection pairs of sequences which differ only by a single word such that one sequence is stereotypical and the other anti-stereotypical. CrowS-Pairs can be used for measuring biases trained with the masked language modeling objective. Schick et al. (2021) presents an interesting self-diagnosis approach fit for both masked language modeling-style and autoregressive LMs. Techniques used for autoregressive LMs often intersect with those used in measuring bias in text generation, described above. Further, it is common to introduce a set of simple prompts such as "She works as" vs "He works as" and measure sentiment, regard (Sheng et al., 2019) or other metrics based on word occurrences (Nozza et al., 2021).
In this paper, we have tested whether the form in which a problem is encoded influences language model bias, independent of the content. Our results highlight that in the cases while performance is not affected, biases vary significantly across different forms of the semantically same input. Having demonstrated that it is extremely difficult for models like T0 to consistently escape logical fallacies and cognitive biases, alternative input formulations to those appeared in training may be used to alleviate biases without much sacrifice on performance.
Potential benefits Our conclusions show bias changes as a function of whether the form in which input is presented is different from that of training. Our results hint at how zero-shot generalization may provide some hopeful representation toward minimizing harm and bias in these large-scale language models. Further, our BBNLI dataset is designed to integrate detailed stereotypes and more complex logical statements that will be crucial to the accelerating advancements in natural language inference problem and measuring biases in multitask systems, more broadly.
Anticipated risks While this study is intended to shed a more nuanced and context-sensitive light toward various social biases in T0 as measured using two benchmarks, a potential risk lies in the models, tasks, prompt templates, domains and subtopics we were not able to exhaustively include. In BBNLI,
although we did our best to approach the top stereotypes and biases that appear in real-life, we were not able to include every ethnicity, gender, and religious point of view. Given these limitations, the risk of using our benchmark could be that the model will show biases in social-cultural categories we did not account for. Additionally, with the added complexity of skip-logic embedded within the premise and hypotheses, there may be some outputs that produce unexpected, unrelated biases that were not explicitly determined.
Moreover, the stereotypical hypotheses we devised are harmful social biases that have real-life consequences to certain groups of people. Further, out intent is to address these highly problematic statements as clearly as possible to understand model biases. However, when these hypotheses are taken out of context and interpreted at face-value, they can cause serious damage to what a model might output or create misunderstanding of our study's purpose.
Lastly, we acknowledge that as human researchers ourselves, we are prone to exuding biases that we have accumulated from our personal environments. As such, this work should be expanded upon by future works and more importantly, our bias dataset can be strengthened through increased collaborative efforts with scholars from the social sciences and humanities.
This work is supported in part by Google Research Scholar Award, DARPA HR001118S0044 (the LwLL program), and the U.S. NSF grant 1838193. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes. The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of Google, DARPA, or the U.S. Government.
Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Large language models associate muslims with violence. Nature Machine Intelligence, 3(6):461-463.
Afra Feyza Akyurek, Lei Guo, Randa Elanwar, Prakash Ishwar, Margrit Betke, and Derry Tanti Wijaya. 2020. Multi-label and multilingual news framing analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8614-
8624, Online. Association for Computational Linguistics.
Chris Alberti, Kenton Lee, and Michael Collins. 2019. A bert baseline for the natural questions. arXiv preprint arXiv:1901.08634.
Edwin Arnold and Marcia Pulich. 2003. Personality conflicts and objectivity in appraising performance. The Health Care Manager, 22(3):227-232.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of bias in nlp. arXiv preprint arXiv:2005.14050.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Neel Burton. 2012. Self-deception ii: Splitting | psychology today.
Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5925-5932, Hong Kong, China. Association for Computational Linguistics.
Hillary Dawkins. 2021. Marked attribute bias in natural language inference. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4214-4226, Online. Association for Computational Linguistics.
Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Srikumar. 2020. On measuring and mitigating biased inferences of word embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7659-7666.
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 862-872, New York, NY, USA. Association for Computing Machinery.
Klaus Fiedler. 2002. Frequency judgements and retrieval structures: splitting, zooming, and merging the units of the empirical world. *Etc. Frequency Processing and Cognition*, page 67-88.
Klaus Fiedler, Uli Hemmeter, and Carolin Hofmann. 1984. On the origin of illusory correlations. European Journal of Social Psychology, 14(2):191-201.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How Can We Know What Language Models Know? Transactions of the Association for Computational Linguistics, 8:423-438.
Ibram X Kendi. 2019. How to be an antiracist. One world.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1896-1907, Online. Association for Computational Linguistics.
Svetlana Kiritchenko and Saif Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 43-53, New Orleans, Louisiana. Association for Computational Linguistics.
Angelie Kraft. 2021. Triggering models: Measuring and mitigating bias in German language generation. Master's thesis, University of Hamburg.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications.
Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. 2020. UNQOVERing stereotyping biases via underspecified questions. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3475-3489, Online. Association for Computational Linguistics.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586.
Frederick G. Lopez. 2001. Adult attachment orientations, self-other boundary regulation, and splitting tendencies in a college sample. Journal of Counseling Psychology, 48(4):440-446.
Li Lucy and David Bamman. 2021. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding, pages 48-55, Virtual. Association for Computational Linguistics.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953-1967, Online. Association for Computational Linguistics.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885-4901, Online. Association for Computational Linguistics.
Debora Nozza, Federico Bianchi, and Dirk Hovy. 2021. HONEST: Measuring hurtful sentence completion in language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2398-2406, Online. Association for Computational Linguistics.
Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2799-2804, Brussels, Belgium. Association for Computational Linguistics.
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. 2021. BBQ: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Crystal Raypole. 2020. Horn effect: Definition, examples, and more.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207.
Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021. Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP. Transactions of the Association for Computational Linguistics, 9:1408-1424.
Shanya Sharma, Manan Dey, and Koustuv Sinha. 2021. Evaluating gender bias in natural language inference. arXiv preprint arXiv:2105.05541.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3407-3412, Hong Kong, China. Association for Computational Linguistics.
Amos Tversky and Daniel Kahneman. 1982. *Judgments of and by representativeness*, page 84–98. Cambridge University Press.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 629–634, Minneapolis, Minnesota. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics.
Throughout the main text we provide results using T0pp checkpoint from the T0-suite. In Table 8, we provide results using T0 checkpoint which reaffirms our conclusions that QA results in higher bias scores than NLI form.