SlowGuess/NAACL_2022_OCR huggingface dataset:
Yang Trista Cao\*1, Anna Sotnikova\*1 Hal Daumé III\*2 Rachel Rudinger\*1 Linda Zou
<sup>1</sup>University of Maryland, College Park 2Microsoft Research
{ycao95, asotniko, hal3, rudinger, lxzou}@umd.edu
NLP models trained on text have been shown to reproduce human stereotypes, which can magnify harms to marginalized groups when systems are deployed at scale. We adapt the Agency-Belief-Communion (ABC) stereotype model of Koch et al. (2016) from social psychology as a framework for the systematic study and discovery of stereotypic group-trait associations in language models (LMs). We introduce the sensitivity test (SeT) for measuring stereotypical associations from language models. To evaluate SeT and other measures using the ABC model, we collect group-trait judgments from U.S.-based subjects to compare with English LM stereotypes. Finally, we extend this framework to measure LM stereotyping of intersectional identities.
Stereotypes are abstract and over-generalized pictures in people's minds that capture attributes about groups of people in the complex social world (Lippmann, 1965). They influence people's thoughts and behaviors, and allow people to make predictions beyond their personal experience or information given (Bruner et al., 1957; Wheeler and Petty, 2001). Stereotypes are also entwined with the production of prejudice, discrimination, and in-group favoritism (Stangor, 2014; Jackson, 2011). A long line of research in social psychology has established models of generic dimensions that estimate people's stereotypes of social groups (Koch et al., 2016; Fiske et al., 2002, i.a.). We build on the Agency Beliefs Commu-nion (ABC) model, which measures stereotypes toward a social group with respect to 16 traits in three dimensions: Agency/Socioeconomic Success, Conservative-Progressive Beliefs, and Com
Figure 1: Crowdsourced analysis of the social group "man" under the ABC model (Koch et al., 2016). Colors: purple=agency, red=belief, green=communion.
munion (§2); an analysis of the group "man" across 32 traits (16 opposing dyads) is shown in Figure 1.
Pre-trained language models (LMs) encode correlations between social groups and traits, like associating the group "Muslim" with the trait threatening, or "man" with confident (e.g., Bender et al., 2021; Nozza et al., 2021; Hovy and Yang, 2021). We conduct a systematic study of social stereotypes in contextualized English masked LMs, grounded in group-trait associations from the ABC model. To capture the group-trait associations in the LM, we first assess two previously proposed word association tests and also propose a new measurement: the sensitivity test (SeT) (§3).
To evaluate the degree to which two LMs—BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019)—align with human stereotype judgments, we design a human study for collecting group-trait judgments (§4). We show that our measure, SeT, best aligns with human judgements on group-trait associations and find that, in general, the association from language models have moderate alignment with human judgements.
Finally, with the best-aligned association measurement, we extend the ABC approach to study
Human Language Technologies, pages 1276 - 1295
Agency	powerless ↔ powerful
low status ↔ high status
dominated ↔ dominating
poor ↔ wealthy
unconfident ↔ confident
unassertive ↔ competitive	Beliefs	religious ↔ science-oriented
conventional ↔ alternative
conservative ↔ liberal
traditional ↔ modern	Communion	untrustworthy ↔ trustworthy
dishonest ↔ sincere
cold ↔ warm
benevolent ↔ threatening
repellent ↔ likable
egotistic ↔ altruistic
LM stereotypes on intersectional groups (§ 5.2). Due largely to the difficulty of extending current approaches for measuring stereotypes in LMs to large numbers of groups, most current approaches only study isolated groups, despite the fact that people's social identities are multifaceted (Ghavami and Peplau, 2013). Because our approach is generalizable to unstudied groups, we take a step towards exploring stereotypes of intersectional identities, finding some correspondence between model behavior and the literature on intersectional stereotypes.
People's impressions of the world and the actions they take are guided by their stereotypes. To systematize this observation, the field of social psychology has proposed models of stereotypes, including traits that can coordinate social behaviors to serve as fundamental dimensions of stereotyping. Some models are designed to focus on social evaluation towards individual persons (Abele and Wojciszke, 2014), ingroup members (Ellemers, 2017; Yzerbyt, 2018), or a small set of outgroups (Fiske et al., 2002); the Agency Beliefs Communion (ABC) model—whose traits are designed to distinguish groups—is suited for a larger set of U.S. social groups (Abele et al., 2020). The ABC model takes a data-driven strategy to select a set of traits by eliminating those that are less effective in capturing stereotypes. The list contains 16 pairs, where each pair represents two polarities (see Table 1), categorized into three dimensions: agency/socioeconomic success, conservative progressive beliefs, and communion/warmth.
Ours is far from the first work to assess stereotypes in language models, and has both advantages and disadvantages compared to previous approaches (see Table 2). Past work has generally taken one of two approaches. The first approach tests systems with hand-constructed templates like "The [group] is  $\square$ ", where [group] ranges over social groups (e.g., "woman" or "Hispanic"), and represents a "masked word" and ranges over occupations ("a professor" or "a nurse") (e.g., Boluk
Table 1: List of stereotype dimensions and corresponding traits in the ABC model (Koch et al., 2016).
Measurement	Generalizes	Grounded	Exhaustive	Natural	Specificity
Debiasing (Bolukbasi et al.)	✓	✓
CrowS-Pairs (Nangia et al.)	✓	✓	✓
Stereoset (Nadeem et al.)	✓	✓	✓
S. Bias Frames (Sap et al.)	✓	✓✓	✓
CEAT (Guo and Caliskan)	✓	✓	✓✓
This Work	✓	✓	✓✓
Table 2: Comparison with previous work: Generalizes denotes approaches that naturally extend to previously unconsidered groups; Grounded approaches are those that are grounded in social science theory; Exhaustiveness refers to how well the traits cover the space of possible stereotypes; Naturalness is the degree to which the text input to the LM is natural (we consider naturally occurring web scraped data as "very natural" and crowdsourced sentences as "somewhat natural"). Specificity indicates whether the stereotype is specific or abstract.
basi et al., 2016; May et al., 2019) or associations drawn from implicit association tests (IAT) (e.g., pleasant/unpleasant words or career/family-related words) (e.g., Caliskan et al., 2017; Guo and Caliskan, 2021). In Table 2 we refer to these as "unnatural" prompts. The second approach collects more natural sentences containing stereotypes, either by web crawling with crowdworkers annotations for social bias (Sap et al., 2019) or by having crowdworkers directly write stereotyping sentences (Nangia et al., 2020; Nadeem et al., 2020).
In our work, we take the first approach with traits from the ABC model, using prompts. The advantage of this approach is that the templates and the traits are completely controlled and are easy to extend to other social groups. The second approach is harder to control, which also leads to significant annotation challenges (Blodgett et al., 2021). Using natural sentences limits generalizability, as it requires a unique collection of prompts (and embedded traits) for each social group; in contrast, the prompt-based approach easily generalizes to any plausible group, especially when based on a theoretically grounded framework like ABC or IAT.
An advantage of our work is that the ABC traits
Domain	Groups
Gender/sexuality	man, woman, non-binary, trans, cis, gay, lesbian
Race/ethnicity	Black, White, Hispanic, Asian, Native American
Religion	Jewish, Muslim, Christian, Buddhist, Mormon, Catholic, Amish, Protestant, Atheist, Hindu
Socio-economic	wealthy, working class, immigrant, veteran, unemployed, refugee, doctor, mechanic
Age	teenager, elderly
Disability status	blind, autistic, neurodivergent, Deaf, person with a disability
Politics	Democrat, Republican
Nationality	Mexican, Chinese, Russian, Indian, Irish, Cuban, Italian, Japanese, German, French, British, Jamaican, American, Filipino
Table 3: Social groups domains and corresponding social groups used for the model experiments and human experiments. Single groups for human experiments are highlighted with italic font style.
are more exhaustive in stereotype coverage with verification from social psychological experiments. The ABC model covers three dimensions with 16 traits, which are consensual, spontaneous, and have been tested using expansive range of social groups (Koch et al., 2021). They used a carefully designed data-driven approach to gather people's fundamental dimensions of social perceptions with as little sampling bias as possible. Thus the resulted 16 traits cover most stereotypes.
Nevertheless, the main trade-off of our approach is that the testing data are not as natural and specific as other approaches. Although we carefully pick and adjust the templates and the form of the social group terms so that the testing sentences are grammatically correct, they are likely not representative of sentences seen in the real world or in the training data of the language models. Further, while our approach has the benefit of near-exhaustive coverage of potential stereotypes, this comes at a cost: the traits we consider are much more high level (e.g., "repellent") than more fine-grained stereotypes collected by other means (e.g., the angry Black woman stereotype (Collins, 2002))—this approach therefore trades coverage for specificity.
3 Measuring Stereotypes in LMs
Our goal is to measure stereotypes in (masked) LMs, and compare them to stereotypes elicited from people. In §4 we describe our approach for eliciting human judgments of group-trait affinities;
here we describe how we measure these in LMs. Previous work has proposed various ways to measure word associations in LMs, including increased log probability score (ILPS) and contextualized embedding association test (CEAT), both of which we summarize below. Finally, we present a new measurement which we call the Sensitivity Test (SeT), which adapts concepts from active learning to the task of measuring a LM's associations.
3.1 Measurements of Word Associations
Increased Log Probability Score (ILPS) quantifies word associations in language models through masked word probabilities. It calculates the association score with a pre-defined template, "[Group] are  $\square$ ." (Kurita et al., 2019), where  $\square$  is a masked token. For example, given a group "Asian" and a trait smart,  $P(\text{"Asian"}$ , smart) measures the probability of smart given "Asians" by filling in the template. Since this probability is affected by the prior probability of smart, ILPS normalizes this probability by the "prior" probability of the trait given a masked group, as below:
$$
\operatorname {I L P S} (\mathfrak {g}, \mathfrak {t}) = \log \frac {P (\square = \mathfrak {t} \mid \mathfrak {g} \text {a r e} \square .)}{P (\square_ {2} = \mathfrak {t} \mid \square_ {1} \text {a r e} \square_ {2 .})}
$$
Intuitively, ILPS measures how much each group raises the likelihood of a trait filling in the template. One can easily show that this equivalent to the weight of evidence of the trait in favor of the hypothesis that the group is the target:  $s(\mathsf{g},\mathsf{t}) = \mathrm{woe}(\mathsf{g}:\mathsf{t}\mid \mathsf{template})$  (Wod, 1985).
Contextualized Embedding Association Test (CEAT) estimates word associations with word embedding distances (Guo and Caliskan, 2021). Intuitively, CEAT measures whether some groups are closer to certain traits in a latent vector space. Given two sets of target words defining groups  $X, Y$  (e.g.  $X_{\text{male}} = \{\text{"man", "father", ...}\}$ ,  $Y_{\text{female}} = \{\text{"woman", "mother", ...}\}$  ) and two sets of polar traits  $A, B$  (e.g.  $A_{\text{pleasant}} = \{\text{love, peace, ...}\}$ ,  $B_{\text{pleasant}} = \{\text{evil, nasty, ...}\}$ ), CEAT computes the effect sizes of the difference between  $X$  and  $Y$  being closer to  $A$  than  $B$  and corresponding p-values. Since contextualized word representations are affected by the contexts around the word, for each word in the four word sets, CEAT randomly samples 1000 sentences from Reddit, in which the word appears, and uses these to approxi
2Both the code and the dataset, along with a datasheet (Gebru et al., 2018), are available under a MIT licence at: https://github.com/TristaCao/U.S_Stereotypes.
Singular
The/That/A [group] is □.	Plural
Most/Many/All [group] are □. / [Group] are □.
Declarative
[Group] are □.	Interrogative
Why are [group] □?
Non-adverbial
[Group] are □.	Adverbial
[Group] are very/so/mostly □.
Fact
[Group] are □.	Belief
I/We/Everyone/People believe/expect/think/know(s) that [group] are □.
Fact
[Group] are □.	Social Expectation
[Group] are supposed to be/should be/are seen as/ought to be/are expected to be □.
Group-first
[Group] are □.	Trait-first
The □ people are [group].
Non-comparative
[Group] are □.	Comparative
[Group] are more likely to be □ than others.
Table 4: Template Variations.
mate the true effect size as below:
$$
\begin{array}{l} \operatorname {CEAT}(A,B,X,Y) = \frac{\underset{\mathfrak{g}\sim X}{\hat{\mathbb{E}}}s(\mathfrak{g},A,B) - \underset {\mathfrak{g}\sim Y}{\hat{\mathbb{E}}}s(\mathfrak{g},A,B)}{\underset{\mathfrak{g}\sim X\cup Y}{\hat{\mathbb{S}}}s(\mathfrak{g},A,B)} \\ s (\mathfrak {g}, A, B) = \underset {\mathfrak {t} \sim A} {\hat {\mathbb {E}}} \cos (\vec {\mathfrak {g}}, \vec {\mathfrak {t}}) - \underset {\mathfrak {t} \sim B} {\hat {\mathbb {E}}} \cos (\vec {\mathfrak {g}}, \vec {\mathfrak {t}}) \\ \end{array}
$$
$\hat{\mathbb{E}}$  (resp.  $\hat{\mathbb{S}}$ ) is the empirical expectation (resp. standard deviation), and  $\vec{x}$  denotes the embedding of  $x$ .
In our setting, since we care about social bias among multiple groups rather than the difference between two groups, we modify the CEAT to calculate the effect size of the distance difference between  $g$  with  $A$  and  $B$  for each group as below:
$$
\operatorname {CEAT} (\mathfrak {g}, A, B) = \frac {\underset {\mathrm {t} \sim A} {\hat {\mathbb {E}}} \cos (\vec {\mathfrak {g}} , \vec {\mathfrak {t}}) - \underset {\mathrm {t} \sim B} {\hat {\mathbb {E}}} \cos (\vec {\mathfrak {g}} , \vec {\mathfrak {t}})}{\underset {\mathrm {t} \sim A \cup B} {\hat {\mathbb {S}}} \cos (\vec {\mathfrak {g}} , \vec {\mathfrak {t}})}
$$
Sensitivity Test (SeT) is a new approach we propose to measure word association for social bias in language models, inspired by ideas from active learning (Beygelzimer et al., 2008). The intuition of SeT is that even though a model assigns the same probability to two different words, the robustness of those two probabilities may be different. For example, both  $p(\text{competent} | \text{"Blind people are"})$  and  $p(\text{kind} | \text{"Men are"})$  might be low. However, the language model may well not have seen many examples with blind people, as opposed to the presumably very large number of examples of men. In this case, a small number of examples may be sufficient to alter the model's predictions about blind people, while a larger number would be required for men. SeT captures the model's confidence in a prediction by measuring how much the model weights would have to change in order to change that prediction. Specifically, SeT computes the minimal change to the last-layer of the language model
so that a given trait becomes the highest probability trait (over the full vocabulary).
For example, consider the template "The [group] is  $\square$ " with the group "woman" and the trait incompetent. Let  $\ell$  be the logits at  $\square$  when the input is "The woman is  $\square$ ," and let  $t$  be the index of incompetent in  $\ell$  (so that  $\ell_{t} = p(\text{incompetent | context})$ ). Let  $h$  be the last hidden layer before the logits, and let  $A$  be the matrix of the last linear layer so that  $\ell = A h$ . SetT computes the minimal distance between  $A$  and some other matrix  $A'$  so that  $t$  is the top word among the new logits  $\ell' = A'h$ . Formally:
$$
\operatorname {S e T} (g, t) = \log \frac {\Delta \left(\mathbf {A} , \mathbf {h} _ {g} , t\right)}{\Delta \left(\mathbf {A} , \mathbf {h} _ {\square} , t\right)}
$$
where  $\mathbf{h}_{\mathrm{g}}$  is the penultimate layer on input g A is the matrix before the logits
$$
\begin{array}{l} \Delta (\mathbf {A}, \mathbf {h}, t) = \min  _ {\mathbf {A} ^ {\prime}} \| \mathbf {A} ^ {\prime} - \mathbf {A} \| _ {2} ^ {2} \\ s. t. \left(\mathbf {A} ^ {\prime} \mathbf {h}\right) _ {t} \geq \left(\mathbf {A} ^ {\prime} \mathbf {h}\right) _ {t ^ {\prime}} + \gamma , \forall t ^ {\prime} \neq t \\ \end{array}
$$
for a fixed margin  $\gamma > 0$ , which we set to 1. SeT returns the negative distance as measure of the association between the corresponding group and trait, normalized by a prior akin to ILPS. This optimization problem does not (to our knowledge) admit a closed form solution; we solve it iteratively using the column squishing algorithm (Bittorf et al., 2012; Daumé and Kumar, 2017).
We test the above measurements on both BERT and RoBERTa pretrained large models from an open-source HuggingFace library.
Social groups. Table 3 lists all the individual social groups we cover in this work. We manually construct the list by combining and picking groups from the list of social groups from Sotnikova et al. (2021) and Koch et al. (2016) and also adding social groups we think are stereotyped in U.S. culture.
Traits. We use the 32 adjectives of the 16 traits from the ABC model (Table 1). For each traits, we calculate the score of its left-side adjective from its right-side adjective:  $S_{\text{powerless-powerful}}(\mathfrak{g}) = S(\mathfrak{g}, \text{powerful}) - S(\mathfrak{g}, \text{powerless})$ , where  $S$  is one of the scores from §3.1.
Templates. ILPS and SeT both require templates in calculating scores. We thus carefully construct a list of templates (Table 4) that covers multiple grammatical and semantic variations, inspired by work investigating harmful search automatic suggestions (Hazen et al., 2020). We find that different model structure requires different templates in order to bring up stereotypes that correlate with human data. See §5 for evidence.
Subwords. Due to the nature of BERT and RoBERTa's tokenizers, some of the adjectives are divided into multiple subwords. This is problematic because all the measurements compute their scores at token level. Neither ILPS nor CEAT deals with subwords directly: in their released implementations, they either take the first or the last sub-token of the word. To remedy this, we adjust the ILPS measurement (denoted as  $\text{ILPS}^{\star}$ ) to properly compute the probability of traits in context using the chain rule across subwords. For SeT, we calculate the sensitivity score for each subword individually and take the maximum SeT score as the SeT score for the word, which effectively computes a lower-bound on how much the model parameters would need to change. We did not modify CEAT's measurement as it is not clear what is the best way to compute comparable word embeddings for words that consist of multiple subwords.
4 Human Study
In the previous section, we describe how we compute associations between groups and traits in lan
guage models. $^{5}$  In this section, we assess stereotypes of social groups through groups-trait association, like in Figure 1. We adopt this approach because it is widely used to evaluate group stereotypes in social psychology field (Fiske et al., 2002; Koch et al., 2016). It also aligns with Lippmann (1965)'s theory of stereotypes that they are abstract pictures in people's head. We broadly follow procedures from previous social psychology papers to collect human evaluation on social groups.
Survey Design. We recruit participants from Prolific. Each participant is paid \(2.00 to rate 5 social groups on 16 pairs of traits and on average participants spend about 10 minutes on the survey. This results in a pay of \)12.00 per hour. Maryland's current minimum wage is \)12.20. First, participants read the consent form, and if they agree to participate in the study, they see the survey's instructions. For each social group, participants read "As viewed by American society, (while my own opinions may differ), how [e.g., powerless, dominant, poor] versus [e.g., powerful, dominated, wealthy] are <group>?" They then rate each trait with a 0-100 slider scale where two sides are the two dimensions of the trait (e.g. powerless and powerful). Each annotated group is shown on a separate page, and participants cannot go back to previous pages. To avoid social-desirability bias, we explicitly write in the instruction that "we are not interested in your personal beliefs, but rather how you think people in America view these groups."
Participant Demographics. At the end of the survey we collect participants' demographic information, including gender, race, age, education level, type of living area, etc. Our participants represent 26 states, with  $63.3\%$  from California, New York, Texas, or Florida; the gender breakdown is  $48.2\%$  male,  $49.6\%$  female, and  $2.2\%$  genderqueer, agender, or questioning; and skew young, with over  $96\%$  at most 40 years old; and with racial demographics that approximately match the U.S. census. For more details on demographics, see Appendix E.
Quality Assurance. Ensuring annotation quality in a highly subjective task is a challenge, and common approaches in NLP like having questions where we "know" the answer as tests, measuring
4 In preliminary experiments, when calculating the score for each adjective, we considered including 1-3 additional adjectives by averaging their scores to improve robustness and mitigate ambiguity. The full list is in Appendix Table A7. However, we found that this did not improve correlations, so we reverted to using the 32 adjectives from the ABC model.
Approved by our institutional IRB, #1724519-1.
<sup>6</sup>https://www.prolific.co/
<sup>7</sup>https://www.minimum-wage.org/maryland
interannotator agreement, and calibrating reviewers against each other (Paun et al., 2018) do not make sense here. Yet, it is still important to ensure the annotation quality. After much iteration, we include three test questions, and warn the participants at the beginning that there are test questions.
1. After the first group, participants must name the group they just scored.
2. After the second, participants must list one trait they just marked high and one marked low.
3. The fifth (final) group is a repetition of one of the four groups they previously scored.
We discard annotations with incorrect answers to either of the first two questions. For the third test, we compute intra-annotator (self) agreement and discard annotations with accuracy-to-self lower than  $80\%$ . For each group we collect 20 annotations that pass our quality threshold. In total, we collected annotations from 247 participants, with 133 passing the quality tests (suggesting that having such tests is important). The 114 annotations that did not pass tests were excluded from our dataset, but all 247 participants were paid.
Social groups and traits. The social groups we used for the human study are highlighted in Table 3. This table contains only single groups used for the model § 3 and human experiments. We collect annotations for 25 social groups within 5 domains, across all 16 pairs of traits.
In this section we present results on correlations between human and model stereotypes for individual groups, comparing across different measurements, including our proposed measurement, SeT (§5.1). Next, we analyze how model scores change for intersectional social groups. We consider several possible factors that may influence the score changes such as identity order, some domain domination, and consider emergent traits (§5.2).
5.1 Correlation on Individual Groups