SlowGuess/ACL_2020_OCR huggingface dataset:
Buffalo, NY, 14226
Jonathan H. Morgan
University of Applied Sciences
Fachhochschule Potsdam
morgan@fh-potsdam.de
Social biases are encoded in word embeddings. This presents a unique opportunity to study society historically and at scale, and a unique danger when embeddings are used in downstream applications. Here, we investigate the extent to which publicly-available word embeddings accurately reflect beliefs about certain kinds of people as measured via traditional survey methods. We find that biases found in word embeddings do, on average, closely mirror survey data across seventeen dimensions of social meaning. However, we also find that biases in embeddings are much more reflective of survey data for some dimensions of meaning (e.g. gender) than others (e.g. race), and that we can be highly confident that embedding-based measures reflect survey data only for the most salient biases.
In April of 2015, protests erupted over the murder of Freddie Gray, Jr. Questions about what to call those protesting quickly became the focus of a national debate. In an interview on CNN with Erin Burnett, $^{1}$  Baltimore City Councilman Carl Stokes admonished then-President Barack Obama and then-Mayor Stephanie Rawlings-Blake for using the word thugs to refer to the protesters. Burnett challenged Stokes' admonition, claiming the protesters were indeed thugs because "They know it's wrong to steal and burn." Stokes responded by stating the protesters were "...children who have been set aside [and] marginalized."
The argument between Burnett and Stokes is over the way we label people, the meanings of those labels, and the impacts these meanings can have. Councelman Stokes wants to avoid using the label "thug" because of its established, negative
Figure 1: Beliefs (x-axis) about four identities (separate plots) along four dimensions of social meaning (y-axis). Beliefs are displayed as distributions of survey responses. The scale is different for each dimension, e.g. for Evaluation, survey participants responded with their belief as to whether people who held the identity were more likely to be a bad (further left on the x-axis) or good (further right).
connotation towards black Americans (Dow, 2016). The survey data collected for this paper, a sample of which is shown in Figure 1, provides further evidence of this association between thugs and black Americans. Respondents to our survey, on average, expected thugs to be bad, and that approximately  $42.4\%$  of thugs would be black. Of the 57 identities we studied, the only identity perceived to be more black was criminal, at  $47.3\%$ .
The beliefs we have about people who hold particular identities (McCall and Simmons, 1978) are important, because they often determine the behaviors we take towards people who are labeled with those identities (Ridgeway and Smith-Lovin, 1999). For example, as Councelman Stokes knows,
Different kinds of beliefs about identities have different names. For example, contextualized beliefs are called impressions (Heise, 1987), and aggregations of beliefs across multiple dimensions of meaning are called stereotypes (Fiske et al., 2002). The beliefs we study here are typically called sentiments or associations. However, given the distinct meaning of these terms in NLP, we use the general term "belief" in this paper. This aligns roughly with the generic use of the terms "bias" and "stereotype" in related NLP literature.
<sup>1</sup>http://nymag.com/intelligencer/2015/04/carl-stokes-to-cnn-thug-is-racially-charged.html
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4392-4415 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
we do not behave the same way towards children as we do towards thugs. This is because, as reflected in Figure 1, people generally believe that children are weak and good, whereas thugs are bad and powerful. This leads us to want to do things like help children, versus wanting to attack thugs (Heise, 2007).
However, measuring beliefs is difficult. Traditionally, we have relied on surveys to collect these measurements. But there are tens of thousands of identities (Joseph et al., 2016; MacKinnon and Heise, 2010), and beliefs about them can form along many different dimensions of sociocultural meaning (e.g. gender, race, and others displayed in Figure 1). Measuring beliefs about many identities, on many dimensions, using traditional surveys can therefore be difficult. Further, measuring the evolution of beliefs is often impossible with surveys, because survey data is extremely sparse historically (Garg et al., 2018). Finally, measuring how these beliefs change with additional contextual information (e.g. beliefs about specific teachers, rather than teachers in general) is notoriously difficult with survey data (Heise, 2007).
Recognizing these difficulties, scholars have begun to develop NLP tools to measure beliefs about identities historically, at scale, and in context (Joseph et al., 2017; Hoyle et al., 2019; Fast et al., 2016; Garg et al., 2018; Field et al., 2019). Most recent methods derive these measures by manipulating word embeddings. Studying beliefs enmeshed in word embeddings is also critical because embeddings are widely used in downstream NLP models, which are themselves beginning to label people, for example, as job-worthy or not (De-Arteaga et al., 2019). Measuring beliefs about people using embeddings therefore serves the dual purpose of understanding human biases and of ensuring such biases are not propelled further along by algorithms.
However, work remains to understand when embedding-based measures of beliefs about identities accurately reflect more traditional survey measures, and why some beliefs may be reflected more accurately than others. The present work combines new and existing survey data with an extensive set of embedding-based measurement strategies to explore this at both the dimension level and the belief level. At the dimension level, for example, we ask, how well do embeddings capture beliefs about gender, relative to race? And if differences exist, why? Such issues have arisen in existing work, for
example, where Garg et al. (2018) see correlations of .65 between embedding-based and survey-based measures of beliefs about gender, but only .15 for ethnicity-based beliefs. At the beliefs-level, we ask, for example, how much more accurately do we capture beliefs about the Potency (strength) of thugs, relative to beliefs about the Potency of children? Accuracy at this level is critical for linking historical trends in social behavior to societal-level beliefs about particular identities.
Our primary contributions are as follows:
- We show that what we measure is more important than how we measure it in determining the correlation between embedding-based and survey-based measures of beliefs about people.
- At the dimension level, the beliefs we measure most accurately are also the most important for how we label others.
- At the belief level, assuming we can identify a good measurement model, embedding-based measures are significantly more accurate for more extreme, and more agreed upon, beliefs.
All code and data necessary to replicate the analyses in this article can be found at https://github. com/kennyjoseph/embedding_impressions.
Our work is grounded in literature on measuring beliefs about identities in social psychology in general and, more specifically, via word embeddings. We address these two literatures separately here.
2.1 Belief Measurement in Social Psychology
A common approach for measuring beliefs about specific identities is to assume a dimensional representation—that is, to assume a set of distinct dimensions of social meaning can be used to characterize how we think and feel about someone that holds a particular identity. From this dimensional perspective, two primary questions arise.
First, what are the dimensions along which beliefs form? Social psychologists have identified three classes of important dimensions: traits, affective meanings, and semantic associations. Traits represent visible—although also socioculturally defined—characteristics like age, gender, and race (Freeman and Ambady, 2011). Affective dimensions of social meaning represent how we feel about a given person and/or identity (Todorov et al., 2015; Fiske et al., 2002; Heise, 2007). Here,
we use the three affective dimensions proposed by Heise (2007) and that are popular in sociology (Rogers et al., 2013)—Evaluation (goodness/badness), Potency (strength/weakness), and Activity (active/passive). Finally, social psychologists often characterize beliefs about identities in terms of semantic associations to particular concepts (Freeman and Ambady, 2011) or institutions (MacKinnon and Heise, 2010). For example, people link the identities brother and sister together because they are both associated with the family institution. In the present work, we collect beliefs for seventeen different dimensions of social meaning, incorporating age, race, gender, evaluation, potency, activity, and six institutional associations.
Second, given a theorized dimension of meaning, how should we measure society-wide beliefs about where particular identities lie on that dimension? Here, we adopt perhaps the most common approach, which uses semantic differential scales on surveys (Osgood et al., 1975). The semantic differential technique asks respondents to place an identity on a sliding scale with two opposing concepts (e.g. weak and strong, see the example in Figure 2A). Finally, it is worth noting that here, like in most social psychology research, we assume that responses from survey participants generalize to American culture writ large. This assumption is built on the well-established culture-as-consensus paradigm in psychological anthropology (Karabatsos and Batchelder, 2003; Batchelder and Romney, 1988), and empirical work showing that people tend to agree on the vast majority of their beliefs about people (Heise, 2007). Nonetheless, many counterexamples exist (Berger et al., 1992; Smith-Lovin and Douglas, 1992). We leave questions about how to address these issues to future work.
2.2 Measuring beliefs with embeddings
Embedding-based approaches to measuring beliefs typically follow a three step process of corpus/embedding selection, dimension selection, and word position measurement.
Corpus/Embedding Selection Several recent works have argued that the corpus used can impact measures of beliefs about people derived from word embeddings (Lauscher and Glavaš, 2019; Mirzaev et al., 2019; Sweeney and Najafian, 2019). For example, Brunet et al. (2019) show how to reduce gender bias in embeddings by removing particular documents from a corpus. However, several oth
ers have shown that in their analyses, the corpus used does not significantly impact results (Spirling and Rodriguez, 2019; Garg et al., 2018; Kozlowski et al., 2019; Caliskan et al., 2017). Differences in the embedding model used have also been observed to impact measurements (Chaloner and Maldonado, 2019). Again, though, robustness checks from other studies suggest a limited effect beyond the somewhat general hyperparameters of window size and the number of dimensions estimated (Garg et al., 2018; Kozlowski et al., 2019).
Dimension Selection To measure beliefs, one first must select a dimension along which the belief is assumed to be held. Much of the literature has focused on dimensions related to gender or race. Others, however, have seen value in moving beyond these dimensions (Agarwal et al., 2019; Sweeney and Najafian, 2019). Most relevant is the work of Kozlowski et al. (2019), who study the association of 59 concepts across 20 different dimensions of sociocultural meaning, and that of An et al. (2018), who induce 732 different dimensions using WordNet to study contextual effects of linguistic meaning. While neither work focuses heavily on identities, these efforts compliment our goal of studying a broad range of dimensions of social meaning.
Scholars then identify a direction within the embedding that represents this dimension. To do so, an approach similar to the semantic differential idea is used. Terms are selected to represent the two ends of the dimension. For example, to identify the gender direction, words at one end might be he and him, and words at the other end, she and her. Scholarship varies on how these dimension-inducing word sets are selected. For example, several scholars have used demographically gendered and/or racialized names (Bolukbasi et al., 2016; Caliskan et al., 2017), while others have relied on careful extraction of concepts from dictionaries and thesauri (Kozlowski et al., 2019). Kozlowski et al. (2019) find that having more words at each end generally provides better measurements, and others have found a need to use frequently occurring terms (Ethayarajh et al., 2019; Brunet et al., 2019). Beyond these observations, however, scholars have generally found stable results as long as reasonable word sets are selected.
Word Position Measurement Finally, the position of each identity along this direction must be identified. Doing so entails two major deci
sions. First, how should one quantify the direction, given the dimension-inducing words? For example, Bolukbasi et al. (2016) identify the direction by taking the first dimension of a PCA on the full set of direction words. Second, how should one define the position of points along this line? For example, several works use the cosine similarity between the identified "bias direction" and the embedding of each identity. Scholars have also recently proposed supervised methods for word position measurement (Sweeney and Najafian, 2019; Agarwal et al., 2019). Such approaches are important, but assume the existence of some training data, which may or may not be available in certain measurement contexts. We therefore do not explore these methods further in the present work.
In sum, using embeddings to measure beliefs requires a series of decisions, the impacts of which are still debated. Below, we provide the most comprehensive study to date on the importance of these decisions on measurement quality.
3 Survey Data
We collect two new survey datasets for this paper. The first measures beliefs about the 57 identities<sup>3</sup> in Table 1 on seventeen dimensions of social meaning described below. The second measures the ways in which a set of survey respondents label people with these identities in hypothetical social situations.
We used a cluster-based approach to select the 57 identities we study. We study nine sets of six identities, where each set has been shown in prior work to be related in some way. Five of the sets are characterized by a salient association to a specific institution described by MacKinnon and Heise (2010). Three sets are characterized by salient trait similarities and differences on gender, age or race/ethnicity. And one set of identities is included where all identities have strong negative Evaluations. Finally, we include three random identities as a mechanism for comparison in other work not described here. For further details on the selection criteria, survey populations, and results, see the Appendix.
3.1 New Belief Measurement Data
We collected survey data on beliefs about identities from 247 respondents on Amazon's Mechanical Turk. Each survey respondent provided re
(a)
(b)
Figure 2: A) Example of a semantic differential question used to measure beliefs about identities (here, for the identity "thug" on the Evaluation dimension); B) Example of a hypothetical identity labeling question used to evaluate the importance of different dimensions.
sponses for four different, randomly selected identities. Each identity was given to a total of 15 different respondents. For each identity, we asked a set of seven questions, some of which had multiple subparts. Following prior work, beliefs for affective dimensions were solicited using a slider-based Likert scale. For the Evaluative dimension, the opposing ends of the Likert scale were labeled "bad, awful" and "good, nice". For the Potency dimension, "powerless, little" and "powerful, big", and for the Activity dimension, "slow, quiet, inactive" and "fast, noisy, active". See Heise (2010) for more details on the development of these questions. The fourth and fifth question used Likert scales to measure beliefs about age and gender, with ends representing "young" and "old" and "Always male" and "Always female," respectively.
The sixth question asked "Of all [some identity, e.g., bullies], what percentage of them do you think are..." and then provided one slider each for the following ethnic/racial categories drawn from the planned 2020 Census: White, Hispanic or Latino, Asian, Middle Eastern, and Black. The seventh question, modeled after the association-based measures from Hill et al. (2015), asked "To what extent does thinking about [some identity, e.g., bullies] lead you to think about..." and then provided a slider for the following institutional settings: family, politics, (criminal) justice, medicine, business, education, and religion. Each slider had qualitative labels ranging from "Not at all", to "Somewhat", to "Immediate response".
<sup>3</sup>Because not all embedding models account for bigrams, we removed "police officer" from all analyses in this paper. However, for future purposes, we include it in our description of the data here.
Dimension Identities
Politics	conservative, Democrat, liberal, Republican, politician, senator
Family	brother, sister, daughter, son, father, mother
Law	judge, criminal, lawyer, witness, cop, police officer
Medicine	doctor, physician, surgeon, nurse, patient, dentist
Business	executive, consultant, secretary, intern, banker, boss
Gender	woman, guy, girl, boy, man, lady
Age	teenager, kid, child, toddler, adult, minor
Race & Ethnicity	black, white, Hispanic, Asian, Arab, American
Negative Evaluation	thug, idiot, jerk, goon, punk, bully
Random	principal, scientist, coach
Table 1: The 57 identities we collect data on. Note that the dimensions used for sampling do not include all dimensions used in our belief measurement study.
3.2 Identity Labeling Data
We collect responses from 402 participants to a pair of identity labeling tasks. Note that these respondents are different than those who provided the belief measurements. Each participant answered a set of 40 hypothetical identity labeling questions. Questions could be either an IsA or a SeenWith question. An example of an IsA question is given in Figure 1B). SeenWith questions were formatted in the same way, except the question text instead says "Who would you say is most likely to be seen with a [mother]?"
Questions varied on both the identity provided in the text and the identities serving as potential answers. From the 57 identities we study, we create survey questions roughly<sup>5</sup> as follows: for a given identity, we generate 14 random sets of the 56 other identities; each set contains four identities. We then generate one IsA and one SeenWith question for each of these sets, where these four identities constitute the possible answers to the question, and the given identity is used in the question text. This process is then repeated ten times for each identity. This process generates ten questions for each of the 3,192 identity pairs for each type of question.
Variable Description
i	A social identity (e.g. doctor, author)
d	A dimension of meaning (e.g. gender)
r	A survey respondent
Sd,i,r	A matrix of survey responses to semantic differential measures on a given dimension d for identity i by respondent r. Each observation is in [0, 1], where 0 and 1 imply something unique for each dimension depending on the ends of the semantic differential scale.
Sid,i,*	The average belief of identity i on dimension d in the survey data
E	A matrix of word embeddings generated from a particular combination of corpus and embedding algorithm. We refer to this as an embedding model
dw	A dimension-inducing word set; that is, a set of words that define the ends of a particular dimension of meaning
wp	A word position measurement model, i.e., a method to place a given identity on a given dimension of social meaning.
mE,dw,wp(i)	An embedding-based measurement model. Defined by an embedding model E, a dimension-inducing word set dw, and a word position measurement model wp. Returns a position for i along the induced dimension
Table 2: Notation used in outlining our approach.
3.3 Belief Measures From Prior Work
To further substantiate our claims, we ensure our main results hold using three other datasets on beliefs about identities: beliefs about gender for 287 occupational identities from Bolukbasi et al. (2016), beliefs about 195 national and occupational identities on the Big Five Personality Traits from Agarwal et al. (2019), and beliefs about 654 identities on the Evaluation, Potency, and Activity dimensions by Smith-Lovin and Robinson (2015).
Our primary research question is, how accurately can we recover beliefs measured using surveys with word-embedding based measures? We study this first at the dimension level, i.e., how accurately do embedding-based measures reflect survey data across a set of identities on a given dimension of social meaning? We then study accuracy at the belief level, i.e., how accurately do embedding-based measures reflect survey data for specific identities on specific dimensions? Our approach is straightforward, but is best explained by introducing some additional notation, provided in Table 2.
4 These identity labeling questions are similar to, but distinct from, those used in our prior work focused on the impact of semantic associations and semantic similarity on identity labeling decisions (Joseph and Carley, 2016).
Due to a bug in Qualtrics, a small percentage of questions were not asked or asked more than once. See Appendix for details
4.1 Dimension-level analysis
At the dimension level, we consider first how different factors relating to the measurement itself impact accuracy. We then study why measurements are more accurate for some dimensions than others. We do so by connecting the degree of accuracy for a given dimension to how important that dimension is in how survey respondents select identities for others in our identity labeling task.
4.1.1 Impact of measurement strategy
As discussed above, the accuracy of embedding-based measurements may vary across properties of the dimension being measured, as well as the way in which the embedding-based measurement is constructed. We first study the relative effects of a) the dimension  $(d)$ , b) the embedding model  $(E)$ , c) the dimension-inducing wordset  $(dw)$ , and d) the word position measurement model  $(wp)$  on the accuracy of embedding-based measurements. As is standard in the literature, we use the Pearson correlation between the mean survey response and the output of the embedding-based measure as our definition of accuracy. That is, for a given dimension  $d$ , survey dataset  $S$ , embedding-based measure  $m_{E,dw,wp}$ , and set of identities of size  $I$ , we compute the accuracy of the embedding-based measure as the Pearson correlation between  $\{S_{d,i_0,*}, S_{d,i_1,*}, \ldots, S_{d,i_I,*}\}$  and  $\{m_{E,dw,wp}(i_0), m_{E,dw,wp}(i_0), \ldots, m_{E,dw,wp}(i_I)\}$ . We then run a linear regression to understand how accuracy varies across the factors considered.
Our analysis involves all dimensions of social meaning studied in the four survey datasets described above. For embedding models,  $E$ , we consider twelve different publicly available corpus/embedding combinations from prior work. To construct dimension-inducing wordsets,  $dw$ , we using one of three approaches. The first is to use the same terms as were placed on the semantic differential scale on the survey (e.g. powerless, powerful, little, big for Potency, as in Figure 2a). In certain cases, we also include a survey-augmented condition that extends this wordset using a thesaurus, after discussion amongst authors. Third, where applicable, we use direction-inducing wordsets from prior work. Finally, we consider several of the major established approaches in the literature for word position measurement  $wp$ . We use the approaches from Kozlowski et al. (2019), Swinger et al. (2019), Ethayarajh et al. (2019), Bolukbasi et al. (2016),