SlowGuess/NAACL_2022_OCR huggingface dataset:
Before we answer the question of how language model stereotype scores align with human stereotypes across the measurements introduced in § 3, we first run a pilot experiment to select the best template(s) for each measurement-model pair from the set of templates in Table 4 (except for CEAT, which does not require templates). We randomly picked four social groups (Asian, Black, Hispanic, immigrant) and five annotations from each group
for the pilot. Since our goal is to inspect the alignment between human and model stereotypes, we take the averaged score of the five annotations as "ground truth" and select templates that give the correlation score according to Kendall  $\tau$ . We limit the selection to at most two templates to avoid overfitting on the pilot data, selected to maximize correlation for each measurement-model pair.
The selected templates and corresponding correlation scores are shown in appendix (Table 5); the score range for weak correlation is 0.10 - 0.19, moderate 0.20 - 0.29, and strong 0.30 and above (Botsch, 2011). For a fixed LM, the best templates tend to be similar across all measures: RoBERTa tends to achieve highest correlation with templates like "That [group] is [trait]." while for BERT the preferred templates tend to be "All [group] are [trait]." or "[Group] should be [trait]."
Given the best templates for each measurement-model pair, we measure to what degree language model stereotypes are aligned with human stereotypes with all annotations on 25 social groups. To quantify alignment, we both calculate the Kendall rank correlation coefficient (Kendall's  $\tau$ ) and the Precision at 3 ( $\mathrm{P@3}$ ). The former indicates the correlation between model and human scores on group-trait associations in terms of the number of swaps required to get the same order. The latter indicates the percentage of the model's top stereotypes which accord with human's judgements. For  $\mathrm{P@3}$ , we also calculate at both the group level and overall with all groups. For each group, we compute its  $\mathrm{P@3}$  score by taking the average of the  $\mathrm{P@3}$  scores with the top 3 traits (top at one polarity) and the score with the bottom 3 (top at the other polarity) because each trait has two polar adjectives and the group-trait score is calculated with the difference of the two polarities. To calculate the  $\mathrm{P@3}$  scores, we binarize the human group-trait scores at a threshold of 50. The overall  $\mathrm{P@3}$  score is the average of the groups' individual  $\mathrm{P@3}$  scores.
The overall scores are in Table 6. We see that in general that RoBERTa contains group-trait associations that are more similar to human judgements than does BERT. Additionally, we see that both ILPS\* and SeT have higher  $\mathbb{P}@\mathbb{3}$  scores than CEAT and ILPS. The RoBERTa model with the SeT measurement approach yields outputs are the most aligned with human's judgements, with RoBERTa/ILPS\* a close second. From its scores, we see that model's group-trait associations have
Measure	RoBERTa	BERT
τ	Template(s)	τ	Template(s)
ILPS	0.280	That [group] is [trait].	0.215	All [group] are [trait].[Group] should be [trait].
\(ILPS^{\star}\)	0.258	All [group] are [trait].[That [group] is [trait].	0.123	We expect that [group] are [trait].[Group] should be [trait].
SeT	0.253	That [group] is [trait].	0.214	All [group] are [trait].[Group] should be [trait].
Table 5: Best two templates for each measurement-model pair and corresponding correlations. Some have only one template because there is no combination of two templates that give higher correlation score than this one template.
CEAT	ILPS	ILPS*	SeT
RoBERTa	BERT	RoBERTa	BERT	RoBERTa	BERT	RoBERTa	BERT
Kendall's τ	0.019	0.111†	0.169†	0.094†	0.175†	0.015	0.199†	0.116
Precision at 3	0.500	0.587	0.620	0.533	0.653	0.560	0.653	0.613
Table 6: Overall alignment scores with human annotations. The highest scores are bold for each row. For correlation scores, we mark scores where the p-value is  $< 0.05$  with †.
moderate correlation with human's judgements. Moreover, in general, two out of the three top ranked group-trait associations from the model agree with human data. See Table A19 for the overall scores of test groups only, where the four pilot groups are excluded, and Appendix B for group level alignment scores.
5.2 Intersectional Groups in LMs
Background. Intersectionality is a core concept in Black feminism, introduced in the Combahee River Collective Statement in 1977 (1977; 1983), considering the ways in which feminist theory and antiracism need to combine: "Because the intersectional experience is greater than the sum of racism and sexism, any analysis that does not take intersectionality into account cannot sufficiently address the particular manner in which Black women are subordinated." The concept was applied in law by Crenshaw (1989) to analyze the ways in which U.S. antidiscrimination law fails Black women.
The concept of intersectionality has broadened and, while its boundaries remain contested (e.g., Browne and Misra, 2003), there are a number of core principles that are central (Steinbugler et al., 2006; Zinn and Dill, 1996): (1) social categories and hierarchies are historically contingent, (2) the experience at an intersection is more than the sum of its parts (Collins, 2002; King, 1988), (3) intersections create both oppression and opportunity (Bonilla-Silva, 1997), (4) individuals may experience both advantage and disadvantage as a result of intersectionality, and (5) these hierarchies impact social structure and social interaction.
Goals and Research Questions. We aim to understand whether we can measure evidence of intersectional behavior in language models with respect to stereotyping. In particular, we are interested in questions surrounding how language models stereotype people who simultaneously belong to multiple social groups. We will only use the term "intersectionality" when specifically considering cases where (per (3) above) the resulting experience (in this case, stereotyping) is more than the sum of its parts. For example, common U.S. stereotypes for Black women are as "welfare queens" (which may show up as low agency in our traits), while common stereotypes for Black men is as "criminal" (which may show up as low communion) (hooks, 1992; Collins, 2002). To limit our scope, we will only consider pairs of social groups (e.g., cis men), and will refer to the the groups that make up a pair as the component identities (e.g., cis, or men). We aim to answer the following research questions:
1. When presented with a paired identity, is the language model sensitive to the order in which the component identities appear?
2. When paired, do certain social categories dominate others in a language model's predictions?
3. Can the language model detect stereotypes that belong to an intersectional group (but not to either of the components that make up the pair)?
To answer these questions, we use the SeT measurement with the RoBERTa model (the best performing pair on the single-group experiments) to compute group-trait associations on our paired groups, which are combinations of all the single groups in Table 3. We manually omit the groups that do
not logically exist (e.g. "cis non-binary person", "teenage elderly person") or are grammatically awkward (e.g. "doctor elderly person", "immigrant blind person"). Note we include both orders of the single groups in the paired groups when possible (e.g. "Catholic teenager" and "teenage Catholic person"). We then conduct the analysis by computing the correlation between groups' list of trait scores with Kendall's  $\tau$ .
Q1: Identity Order. Given an paired group with two identities, the language model may not be able to capture both of the identities and may predict stereotypes based only on one of the components. In fact, the average correlation score between a paired group and the most correlated of its components is 0.56, which is moderately high. We thus calculate the correlation of trait scores between the paired group and both its first and second component identities (when both orders are possible). In addition, we calculate the correlation of paired groups with reversed identity order (e.g. "Asian teenager" and "teenage Asian person"). The average correlation score between a paired group and its first component is 0.43; the correlation score to its second component is 0.46, which are quite close. Further, the average correlation score of intersectional groups with reversed identity is 0.69, which is moderately high. Taken together, these results indicate that (a) many paired groups have similar group-trait association scores with one of their component identities alone; (b) the order does not matter significantly, but the language model tends to focus slightly more on the second component. The implication of this is that we can expect that the language model may be able to capture intersectional stereotypes.
Q2: Dominant Domains. Stryker (1980) suggests that people tend to identify themselves with their race/ethnicity identity before other identities, though this is contested and, in some cases, thought to be antithetical to the idea of intersectionality (e.g., Collins, 2002). Prompted by this debate, we ask if there is a hierarchy of the domains that language model picks up on for paired groups. To answer this question, for each identity domain pair, we compute the average correlation score between the paired groups with each of its two component identities, and take the difference of the averaged correlation scores of the two domains. For each domain, we count the domains it dominates (i.e.
has score difference  $\geq 0.1$  ) and is dominated by.
These results show that age and political stance are dominant domains, which is expected as identities within these two domains have strong characteristics that may overwhelm domains they are paired with. On the other end, race and nationality are, generally, dominated domains. It is surprising that the race domain is majorly dominated, contrasting documented literature in human behavior. The full results are shown in Appendix Table A8 as well as detailed scores Table A9.
Q3: Emergent Intersectional Stereotypes. Finally, we look into emergent stereotypes of paired groups, with the goal of finding intersectional behavior in the language model. To detect intersectional stereotypes, we need to operationalize the notion of the whole being greater than its parts. For a fixed paired group  $\mathsf{g} = (\mathsf{g}_1,\mathsf{g}_2)$  (e.g., "trans Democrats"), and a given trait t (e.g., warm), we compute  $S(\mathsf{g},\mathsf{t}) - \max \{S(\mathsf{g}_1,\mathsf{t}),S(\mathsf{g}_2,\mathsf{t})\}$ , where  $S$  is the score from the language model, capturing whether this trait is more associated with the paired group than the maximum of its association with the component identities. (We consider also the reverse, where we look for scores much less than the min.) We might hope to find some well attested intersectional identities from the literature, such as "Black women" have an attitude (low communion) and "White men" are privileged (high agency) (Ghavami and Peplau, 2013).
The top 50 emergent group-trait associations according to our measure are listed in Table A10. We also see some good examples are: the language model scores "Hispanic unemployed people" as more egotistic than people of the component identities, "Democrat teenagers" as more altruistic, "male doctors" as more benevolent, etc. However, there are also some unexpected patterns; for instance, almost all nationality identities combined with "mechanic" are trustworthy and likeable, and almost all nationality identities combined with "autistic" are egotistic. Looking into the scores themselves, we find that both "mechanic" and "autistic" have low scores on the corresponding traits, and combining them with nationalities raises to about average levels.
Aside from analyzing face validity—which is mixed—we compare the results of our model to the traits that Ghavami and Peplau (2013) found when conducting human studies of race/gender pairs. To do this, we categorize the traits from Ghavami and
Peplau (2013) to the ABC dimensions<sup>8</sup> and compare with our full list of emergent group-trait associations. Taking their group-trait matches as ground truth, our detection of traits for these race/gender intersectional groups achieves a precision 0.83 and recall 0.65—better than random guessing (precision 0.72, recall 0.50) but far from perfect.
There are several limitations to our work, which should be taken into account in the interpretation of our results.
First, our results are likely affected by reporting bias and by a defaulting effect where, when people annotate traits for "men", they may actually have in their head "cis straight white men", because the defaults go unremarked. This goes both for the human scores (how does a participant conceptualize "men"? and language model scores (what do sentences containing the word "man" assume given that most language a language model has been trained on likely exhibits defaulting?).
Second, our work only focuses on assessing stereotypes within language models and not in any deployed system. Though stereotypes from language models may impact the outputs of downstream systems which are built upon these language models, it is not clear how exactly the stereotypes transfer (Cao et al., 2022). Additionally, our work is limited to English and U.S. social stereotypes.
Third, although we followed and built on best practices from social psychology in developing the human study, it nevertheless has some shortcomings. In particular, even after many iterations on wording, it was difficult to phrase the survey questions to encourage people to reporting their true impressions. There is tension between asking a participant what they think—which risks a countounding potential social desirability bias (Latkin et al., 2017) (people's tendency to respond in socially acceptable ways)—and asking what they think others think—which led to comments from a few participants that they felt unqualified to speak for others. Asking these questions of participants and collecting the data also raises the possibility of this work inadvertently reinforcing stereotypes.
Finally, aggregating human judgements into a single number by averaging (or any other statistic)
to compare to model predictions risks collapsing a significant amount of information down to a single number. This number cannot distinguish between a weakly held but common stereotype and a strongly held but rare one. Nor can it distinguish between traits where half of annotators say 0 and the other half say 100, from traits where all annotators say 50. These average judgments should be interpreted as not what any single person would say, but an average over people. This limitation is exacerbated by the defaulting effect, where some people may imagine a different prototype for a given group, and other people may imagine another.
In this paper, we measured language model (LM) stereotypes by adopting the ABC stereotype model from social psychology. Comparing to previous work on detecting LM stereotypes, our approach is easy to extend to previously unconsidered groups, grounded in traits proven effective by social psychology, and exhaustively covering the space of possible stereotypes, at the cost of being more abstract than in other NLP work. This yields a different set of trade-offs than previous approaches to measuring stereotypes in LMs.
With the ABC model and data regarding human stereotypes from our human study, we assessed LM stereotypes using three different association measurements, including SeT, a metric we proposed. We showed that LM group-trait stereotypes in general have moderate correlation with human judgements, and that SeT provides correlations that better align with human's. Based on these results, we extended our analysis to intersectional groups. We found that the LM may be able to capture intersectional stereotypes but is not particularly good on identifying emergent intersectional stereotypes. Our results also show that that, in general, age and political stance are dominant domains in language models, whereas race and nationality are dominated domains. We hope that our work provides insights for future works on measuring and mitigating stereotypes in natural language processing systems, and that the grounding in theories from social psychology has benefits beyond just studying stereotypes.
This material is based upon work partially supported by the National Science Foundation under
$^{8}$ Ghavami and Peplau (2013) covers paired groups combined with race domain and binary genders. The traits they raised span the agency and communion dimensions.
Grant No. 2131508. The authors are also grateful to all the reviewers who have provided helpful suggestions to improve this work, and thank members of the CLIP lab at the University of Maryland for the support on this project. We are grateful to all those who participated in our human study, without whom this research would not have been possible.
Andrea Abele, Naomi Ellemers, Susan Fiske, Alex Koch, and Vincent Yzerbyt. 2020. Navigating the social world: Toward an integrated framework for evaluating self, individuals, and groups. *Psychological review*, 128.
Andrea Abele and Bogdan Wojciszke. 2014. Communal and agentic content a dual perspective model. Adv. Exp. Soc. Psychol., 50:198-255.
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610-623.
Alina Beygelzimer, Sanjoy Dasgupta, and John Langford. 2008. Importance weighted active learning. CoRR, abs/0812.4952.
Victor Bittorf, Benjamin Recht, Christopher Ré, and Joel Tropp. 2012. Factoring nonnegative matrices with linear programs. Advances in Neural Information Processing Systems, 2.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1004-1015, Online. Association for Computational Linguistics.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. NeurIPS, pages 4349-4357.
Eduardo Bonilla-Silva. 1997. Rethinking racism: Toward a structural interpretation. American sociological review, pages 465-480.
Robert E. Botsch. 2011. *Significance and Measures of Association*.
Irene Browne and Joya Misra. 2003. The intersection of gender and race in the labor market. Annual review of sociology, 29(1):487-513.
J.S. Bruner, Brunswik E, L. Festinger, F. Heider, K.F. Muenzinger, C.E. Osgood, and D. Rapaport. 1957. Going beyond the information given. Contemporary approaches to cognition, pages 41-67.
Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala, and Aram Galstyan. 2022. On the intrinsic and extrinsic fairness evaluation metrics for contextualized language representations.
Patricia Hill Collins. 2002. Black feminist thought: Knowledge, consciousness, and the politics of empowerment. routledge.
Combahee River Collective. 1977. A Black Feminist Statement. na.
Combahee River Collective. 1983. The combahee river collective statement. *Home girls: A Black feminist anthology*, 1:264-274.
Kimberlé Crenshaw. 1989. Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. u. Chi. Legal f., page 139.
Hal Daumé, III and Abhishek Kumar. 2017. Column squishing for multiclass updates (blog post).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT* (1).
Naomi Ellemers. 2017. Morality and the Regulation of Social Behavior: Groups as Moral Anchors.
Susan T. Fiske, Amy J. C. Cuddy, Peter Glick, and Jun Xu. 2002. A model of (often mixed) stereotype content: competence and warmth respectively follow from perceived status and competition. Journal of personality and social psychology, 82 6:878-902.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daume, and Kate Crawford. 2018. Datasheets for datasets.
Negin Ghavami and Letitia Anne Peplau. 2013. An intersectional analysis of gender and ethnic stereotypes: Testing three hypotheses. Psychology of Women Quarterly, 37(1):113-127.
Wei Guo and Aylin Caliskan. 2021. Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES '21, page 122-133, New York, NY, USA. Association for Computing Machinery.
Timothy J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, and Michael Golebiewski. 2020. On the social and technical challenges of web search autosuggestion moderation. arXiv:2007.05039 [cs]. ArXiv: 2007.05039.
bell hooks. 1992. Yearning: Race, gender, and cultural politics. Hypatia, 7(2).
Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 588-602, Online. Association for Computational Linguistics.
Lynne M. Jackson. 2011. The psychology of prejudice: From attitudes to social action. American Psychological Association.
Deborah K King. 1988. Multiple jeopardy, multiple consciousness: The context of a black feminist ideology. Signs: Journal of women in culture and society, 14(1):42-72.
Alex Koch, Roland Imhoff, Ron Dotsch, Christian Unkelbach, and Hans Alves. 2016. The abc of stereotypes about groups: Agency/socioeconomic success, conservative-progressive beliefs, and communion. Journal of personality and social psychology, 110:675-709.
Alex Koch, Vincent Yzerbyt, Andrea Abele, Naomi Ellemers, and Susan T. Fiske. 2021. Social evaluation: Comparing models across interpersonal, intra-group, intergroup, several-group, and many-group contexts, volume 63, page 1-68. Elsevier.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W. Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. CoRR, abs/1906.07337.
Carl A. Latkin, Catie Edwards, Melissa A. Davey-Rothwell, and Karin E. Tobin. 2017. The relationship between social desirability bias and self-reports of health, substance use, and social network factors among urban substance users in baltimore, maryland. Addictive Behaviors, 73:133-136.
Walter Lippmann. 1965. Public Opinion. New York: Free Press.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and