SlowGuess/ACL_2020_OCR huggingface dataset:
and Garg et al. (2018). In the Appendix, we provide full details on the different values of  $E$ ,  $dw$ , and  $wp$  that we consider.
4.1.2 Explaining variation across dimensions
As we will show, controlling for  $E$ ,  $dw$  and  $wp$ , there are large differences in accuracy across dimensions. To better understand these differences across dimension, we compute two measurements. First, Kozlowski et al. (2019) show that the variance of the survey data on a dimension, that is,  $\mathrm{Var}(\overline{S_{d,i_0,*}},\overline{S_{d,i_1,*}},\dots,\overline{S_{d,i_n,*}})$ , is strongly correlated with the accuracy of embedding-based measures. However, they also note that "high explained variance... reveals little about how these valences are deployed in social life" (pg. 930). Here, we therefore compute a second measure that connects variance of the survey data on a given dimension to a significant social outcome, how strongly people rely on that dimension when labeling other people.
To do so, we first construct a  $57x17$  matrix  $X$  of scaled-and-centered mean survey responses for each identity on each dimension in our survey data, i.e.  $X_{i_0,d_0} = \overline{S_{d_0,i_0,*}}$ . We then construct an observation with a binary outcome that pairs the identity in the question with each possible answer. The outcome is 1 if the answer was selected, and 0 otherwise. For example, in Figure 2B), the pairings created would be "mother, adult", "mother, sister", "mother, son", and "mother, lady". If the respondent answered "lady", then the outcomes would be 0, 0, 0, and 1, respectively. The  $40.3\%$  of questions where respondents answered "all are equally unlikely" were ignored. In total, we obtained 9,597 responses where the respondent did not answer "All are equally (un)likely," split roughly evenly between SeenWith and IsA questions.
We then train a logistic regression model for IsA and SeenWith questions separately, each with seventeen parameters. For a given observation, the parameters represent the absolute difference between each dimension, e.g. the first parameter is  $|X_{i_q,d_0} - X_{i_a,d_0}|$ , where  $i_q$  is "mother" in Figure 2B),  $i_a$  is, e.g., "adult", and  $d_0$  is, e.g., gender.
In the Appendix, we provide full results for these regressions. Intuitively, larger negative coefficients for a given dimension indicate that the further away two identities are on that dimension, the less likely the respondent is to select them as a pair. For example, we find that Evaluation has a strong negative correlation for IsA questions, indicating that respondents typically do not expect two identities to
be assigned to the same person if one identity is perceived to be for "good people" and the other for "bad people". Positive coefficients imply assortativity on the dimension. For example, for SeenWith questions, Potency has a positive coefficient, implying that we expect powerful identities to be seen with less powerful counterparts. The magnitude of these coefficients represent the importance given to that dimension by survey respondents. We use the maximum of the two coefficients across SeenWith and IsA questions as a measure of this importance.
4.2 Belief-level analysis
We are also interested in accuracy for specific beliefs. For example, how accurately do embedding-based measures reflect survey data on beliefs about the typical age of a boy? As an outcome for this belief-level analysis, we use a ranking task similar to prior work (Spirling and Rodriguez, 2019; Kozlowski et al., 2019). We describe this outcome by continuing with the example of beliefs about the age of boys. We first compute the set of identities  $N$ , for which  $\overline{S_{age,boy,*}} - se(S_{age,boy,*}) > \overline{S_{age,i,*}} + se(S_{age,i,*})$ , where  $se$  is the standard error function. That is,  $N$  represents all identities we are reasonably confident respondents believed to be younger than boys. We then determine the subset of  $N$ ,  $N_c$ , where boy is also ranked above those identities in the embedding measure. We do the same for identities survey respondents said were older than boys, adding these to  $N$ , and to  $N_c$  if they are correctly ranked in the embedding measure. Finally, we use  $\frac{N_c}{N}$  to study accuracy at the belief level.
We are interested both in overall levels of accuracy for belief-level measurements, as well as the factors that explain variation in accuracy. We consider four factors that might explain this variation (continuing with the age/boy example):  $sd(S_{age,boy,*})$ , the distance of  $\overline{S_{age,boy,*}}$  to the median over all identities on that dimension, the logged frequency of the identity in a large corpora, and the number of synsets for the identity in WordNet. To study the impact of these different factors, we use a generalized additive model with a binomial link function where  $\frac{N_c}{N}$  is the outcome and points are weighted by  $N$ . Finally, as opposed to considering results across all possible  $E$ ,  $dw$ , and  $wp$ , we first select those settings that maximize the Pearson correlation for each dimension.
Figure 3: Regression results for the dimension-level analysis. Coefficients for each factor are relevant to a baseline. For the embedding models, the baseline is the FastText 300 dimensional model trained on the Common CWEl. For the dimension-inducing wordset, it is the terms used to define gender by Bolukbasi et al. (2016). For word position measurement, it is the approach from Garg et al. (2018), and for dimension, association with Politics.
5.1 Dimension-level results
Across all conditions and survey datasets, the Pearson correlation between the embedding and survey measures is 0.32 [.31,.33]. However, considerable variation exists. Figure 3 presents results of a regression that attempts to explain the sources of this variance (x-axis) and the effects of each source (y-axis). Separate colors represent results from the four different survey datasets analyzed. In general, results are largely consistent across the different datasets, and thus we will not emphasize differences across datasets below.
Figure 3 shows that the embedding model used can decrease correlation by as much as .35. As others have found, this effect decreases when one focuses only on 300-dimensional embeddings. It is worth noting, however, that no embedding model is universally best. For example, nine of the twelve
$^\mathrm{6}$ according to (Speer et al., 2018)
embedding models studied are responsible for producing the highest observed correlation for at least one dimension.
Selection of the dimension-inducing words,  $dw$ , also has a limited effect. The one exception is when survey-matched words are used for the Gender dimension, where correlations drop by, on average, around 0.5 relative to the "he/she" baseline. The fact that using the same words as the semantic differential scale is a terrible choice, but for only one of the seventeen dimensions studied, reflects the fact that selection of  $dw$ , like elements of other forms of quantitative social science, remains a mix of art and science (Sterling and Taveter, 2009).
In contrast, even the most scientifically appealing approaches to word position measurement (Ethayarajh et al., 2019) provide marginal gains. The only consistent observation we draw is that approaches that normalize measurements across dimensions related to the same overarching concept (e.g. that normalize racialized beliefs across all perceived dimensions of race) perform slightly better. Results thus reflect that the details of measurement are less important than what is being measured.
Reflecting this same fact, the strongest impacts on correlation between the survey and embedding-based measures come from which dimension is being studied. Some of these results reflect the salience of these dimensions in social life. Associations to institutions, which are most accurately measured on average, are a primary tool we use to sort people into groups (MacKinnon and Heise, 2010). And stronger correlations between the embedding and survey-based measures for Evaluation and Potency, relative to Activity, reflects the increased importance in affective perceptions of these two dimensions (Rogers et al., 2013). However, scholars largely agree that trait-based beliefs on gender and race serve as "default characteristics" (Ridgeway and Smith-Lovin, 1999) along which we almost automatically categorize others (Todorov et al., 2015). Given their shared salience, why is gender the only trait that can be accurately measured?
Figure 4A) shows, as first identified by Kozlowski et al. (2019), that much of this is due to the variance of the survey data along that dimension; the correlation between variance and the coefficients in Figure 3 is 0.91. However, as discussed above, Kozlowski et al. (2019) study more general concepts on more general dimensions, and note that they have no easy way to connect their
Figure 4: In both A) and B), the y-axis gives the coefficient value for the regression presented in Figure 3. In A), the x-axis represents the variance in survey means along the dimension. In B), an estimate of the dimension's importance for identity labeling
Figure 5: Results for our belief-level outcome (y-axis) as a belief's distance from the dimension-level median (x-axis) varies for all dimensions (left) and only the "Black" dimension (right). Points on the right are colored by variance, with higher (lower) variance estimates colored blue (red)
observations to any critical social processes. In contrast, here, Figure 4B) shows a significant positive correlation between variance in the survey data along a dimension (and hence measurement accuracy) and that dimensions' importance in explaining patterns of labeling in our identity labeling task. Embedding-based measures of beliefs about identities, we therefore show, are most likely to reflect traditional survey measures particularly when those beliefs are salient for identity labeling.
Critically, then, results for biases in word embeddings are tied not only to the salience of dimensions in general social life, but also to the identities selected for measurement. Selecting only heavily racialized and non-gendered identities, for example, might well reverse the positions of racialized dimensions and gender in Figure 4. This makes it all the more critical to identify theoretically-driven concepts—salience in labeling, and variance in measurement—that move beyond measures of specific identities on specific dimensions to help us understand what is measurable and what is not,
particularly when survey data is not available.
5.2 Belief-level results
As with the dimension-level results, we find that embedding-based measures are generally accurate predictors of survey-based measures for specific beliefs. On average,  $74.9\%$  of the beliefs collected for this paper are correctly ranked, as are  $82.1\%$ ,  $72.0\%$ , and  $71.4\%$  of the beliefs from Bolukbasi et al. (2016), Smith-Lovin and Robinson (2015), and Agarwal et al. (2019), respectively. One caveat to keep in mind, however, is that we focus only on the single best embedding measurement approach for each source/dimension combination.
Regardless, as with the dimension-level results, there is considerable variance at the belief level. Some of this variance (approximately  $32\%$ , see the Appendix for full regression results) can be explained by the factors we consider. The strongest explanation we find to explain ranking accuracy, reflected in the left-hand plot in Figure 5, is the distance of the survey-based belief measure from the median on its dimension. At the extremes, ranking accuracy is almost perfect. Because extreme observations are also most likely to be low variance—for example, consider that beliefs at the most extreme values of a scale must have zero variance—a more general claim can be made: word embedding-based measures accurately capture our most extreme and agree-upon beliefs about people, but show significant unexplained (at least by us) variance for more neutral and/or less-agreed upon beliefs.
This variance is on display in the right-hand plot in Figure 5, which gives results for the blackness dimension. The embedding-based measure captures with perfect accuracy racialized perceptions of the identities thug and criminal, but not, e.g., liberal, which is similar along the other explanatory factors we consider here. As far as we are aware, it remains an open question as to why this is the case.
In this paper, we asked, can we trust measures of beliefs about people derived from word embeddings? We find the answer to be yes, at least on average. Depending on one's perspective, this could be good or bad. From a cultural studies/social psychological perspective, this positive correlation further validates efforts to use word embeddings to study perceptions of people historically, at scale, and in context. On the other hand, from the "bias" per
spective, this suggests that a vast array of social biases are encoded in embeddings.
However, we also find that some beliefs—specifically, extreme beliefs on salient dimensions—are easier to measure than others. More generally, across four datasets, we find that what we measure is more important than how we measure it. Again, two different perspectives on this are needed. With respect to the study of culture and human stereotypes, we may be safest in studying only the most extreme results from embedding models, as has been done by, e.g., Spirling and Rodriguez (2019).
From the bias perspective, given the rash of recent work on debiasing word embeddings, our results suggest that much more attention needs to be paid to how we are evaluating these approaches. Currently, upstream evaluations of debiasing are centered almost exclusively on occupational identities on gender, where some of the most salient social biases we know of exist (Ridgeway, 2011). Others have argued that removing these salient beliefs may not remove gender information from embeddings (Gonen and Goldberg, 2019). But Gonen and Goldberg's 2019 argument relies on a technical deficiency of existing approaches. We can make a similar critique by simply changing what is being measured. For example, the correlation between gender beliefs and the gender direction in the Hard-Debiased embeddings of Bolukbasi et al. (2016) is  $0.05(p = .84)$  using identities in their data, and  $0.4(p < .05)$  using the identities in our data.
Similarly, removing gender bias does not remove bias on other dimensions. For example, while Sweeney and Najafian (2019) show that the NumberBatch embeddings harbor the least gender bias, we find that they are the only embedding to show consistently high correlations with age, leading to the potential for ageism downstream. More generally, stereotypes exist along a network of beliefs (Freeman and Ambady, 2011) reflecting unwarranted correlations between many dimensions (Ridgeway, 2011); we must therefore be careful not to expect that removing meaning along one dimension will expel social biases from our models.
K.J. was supported by NSF IIS-1939579. This research was supported in part a SUNY Germination Space Grant. We thank Lynn Smith-Lovin, Lisa Friedland, Tobias Schroeder, and Yuhao Du for comments on earlier versions of this work.
Oshin Agarwal, Funda Durupinar, Norman I. Badler, and Ani Nenkova. 2019. Word embeddings (also) encode human personality stereotypes. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM* 2019), pages 205-211, Minneapolis, Minnesota. Association for Computational Linguistics.
Jisun An, Haewoon Kwak, and Yong-Yeol Ahn. 2018. Semaxis: A lightweight framework to characterize domain-specific word semantics beyond sentiment. In ACL.
William H. Batchelder and A. Kimball Romney. 1988. Test theory without an answer key. Psychometrika, 53(1):71-92.
Joseph Berger, Robert Z. Norman, James W. Balkwell, and Roy F. Smith. 1992. Status inconsistency in task situations: A test of four status processing principles. American Sociological Review, pages 843-855.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. arXiv preprint arXiv:1607.06520.
Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ashton Anderson, and Richard Zemel. 2019. Understanding the Origins of Bias in Word Embeddings. In International Conference on Machine Learning, pages 803-811.
Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
Kaytlin Chaloner and Alfredo Maldonado. 2019. Measuring Gender Bias in Word Embeddings across Domains and Discovering New Gender Bias Word Categories. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 25-32, Florence, Italy. Association for Computational Linguistics.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120-128.
Dawn Marie Dow. 2016. The deadly challenges of raising African American boys: Navigating the controlling image of the "thug". Gender & Society, 30(2):161-188.
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019. Understanding undesirable word embedding associations. In Proceedings of the 57th Annual
Meeting of the Association for Computational Linguistics, pages 1696-1705, Florence, Italy. Association for Computational Linguistics.
Ethan Fast, Tina Vachovsky, and Michael S. Bernstein. 2016. Shirtless and dangerous: Quantifying linguistic signals of gender bias in an online fiction writing community. In Tenth International AAAI Conference on Web and Social Media.
Anjalie Field, Gayatri Bhat, and Yulia Tsvetkov. 2019. Contextual affective analysis: A case study of people portrayals in online# metoo stories. In Proceedings of the International AAAI Conference on Web and Social Media, volume 13, pages 158-169.
Susan T. Fiske, Amy JC Cuddy, Peter Glick, and Jun Xu. 2002. A model of (often mixed) stereotype content: Competence and warmth respectively follow from perceived status and competition. Journal of personality and social psychology, 82(6):878.
Jonathan B. Freeman and Nalini Ambady. 2011. A dynamic interactive theory of person construal. *Psychological review*, 118(2):247.
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635-E3644.
Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of NAACL-HLT, pages 609-614.
David R. Heise. 1987. Affect control theory: Concepts and model. The Journal of Mathematical Sociology, 13(1-2):1-33.
David R. Heise. 2007. Expressive Order. Springer.
David R. Heise. 2010. Surveying Cultures: Discovering Shared Conceptions and Sentiments. John Wiley & Sons.
Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665-695.
Alexander Hoyle, Wolf-Sonkin, Hanna Wallach, Isabelle Augenstein, and Ryan Cotterell. 2019. Unsupervised discovery of gendered language through latent-variable modeling.
Kenneth Joseph and Kathleen M. Carley. 2016. Relating semantic similarity and semantic association to how humans label other people. NLP+ CSS 2016, page 1.
Kenneth Joseph, Wei Wei, and Kathleen M. Carley. 2016. Exploring patterns of identity usage in tweets: A new problem, solution and case study. In Proceedings of the 25th International Conference on World Wide Web, pages 401-412.
Kenneth Joseph, Wei Wei, and Kathleen M. Carley. 2017. Girls rule, boys drool: Extracting semantic and affective stereotypes from Twitter. In 2017 ACM Conference on Computer Supported Cooperative Work.(CSCW).
George Karabatsos and William H. Batchelder. 2003. Markov chain estimation for test theory without an answer key. Psychometrika, 68(3):373-389.
Austin C. Kozlowski, Matt Taddy, and James A. Evans. 2019. The Geometry of Culture: Analyzing the Meanings of Class through Word Embeddings. American Sociological Review, 84(5):905-949.
Anne Lauscher and Goran Glavaš. 2019. Are we consistently biased? multidimensional analysis of biases in distributional word vectors. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM* 2019), pages 85-91.
Neil J. MacKinnon and David R. Heise. 2010. Self, Identity, and Social Institutions. Palgrave Macmillan.
Otto H MacLin and Roy S Malpass. 2001. Racial categorization of faces: The ambiguous race face effect. Psychology, Public Policy, and Law, 7(1):98.
George J. McCall and Jerry L. Simmons. 1978. Identities and interactions. New York.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
Inom Mirzaev, Anthony Schulte, Michael Conover, and Sam Shah. 2019. Considerations for the interpretation of bias measures of word embeddings. arXiv preprint arXiv:1906.08379.
Charles Egerton Osgood, William H. May, Murray Samuel Miron, and Murray S. Miron. 1975. Cross-Cultural Universals of Affective Meaning, volume 1. University of Illinois Press.
Olutobi Owoputi, Brendan O'Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In Proceedings of NAACL.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.