Venue: NAACL 2019

# On Measuring Social Biases in Sentence Encoders

Chandler May\({}^{1}\) Alex Wang\({}^{2}\) Shikha Bordia\({}^{2}\)

Samuel R. Bowman\({}^{2}\) Rachel Rudinger\({}^{1}\)

###### Abstract

The _Word Embedding Association Test_ shows that GloVe and word2vec word embeddings exhibit human-like implicit biases based on gender, race, and other social constructs Caliskan et al. (2017). Meanwhile, research on learning reusable text representations has begun to explore sentence-level texts, with some sentence encoders seeing enthusiastic adoption. Accordingly, we extend the Word Embedding Association Test to measure bias in sentence encoders. We then test several sentence encoders, including state-of-the-art methods such as ELMo and BERT, for the social biases studied in prior work and two important biases that are difficult or impossible to test at the word level. We observe mixed results including suspicious patterns of sensitivity that suggest the test's assumptions may not hold in general. We conclude by proposing directions for future work on measuring bias in sentence encoders.

1Johns Hopkins University  2New York University

{cjmay,rudinger}@jhu.edu {alexwang,sb6416,bowman}@nyu.edu

## 1 Introduction

Word embeddings quickly achieved wide adoption in natural language processing (NLP), precipitating the development of efficient, word-level neural models of human language. However, prominent word embeddings such as word2vec Mikolov et al. (2013) and GloVe Pennington et al. (2014) encode systematic biases against women and black people Bolukbasi et al. (2016); Garg et al. (2018), i.a.), implicating many NLP systems in scaling up social injustice. We investigate whether sentence encoders, which extend the word embedding approach to sentences, are similarly biased.1

Footnote 1: While encoder training data may contain perspectives from outside the U.S., we focus on biases in U.S. contexts.

The previously developed Word Embedding Association Test (WEAT; Caliskan et al., 2017) measures bias in word embeddings by comparing two sets of target-concept words to two sets of attribute words. We propose a simple generalization of WEAT to phrases and sentences: the Sentence Encoder Association Test (SEAT). We apply SEAT to sentences generated by inserting individual words from Caliskan et al.'s tests into simple templates such as "This is a[n]<word>."

To demonstrate the new potential of a sentence-level approach and advance the discourse on bias in NLP, we also introduce tests of two biases that are less amenable to word-level representation: the _angry black woman_ stereotype Collins (2004); Madison (2009); Harris-Perry (2011); hooks (2015); Gillespie (2016) and a _double bind_ on women in professional settings Heilman et al. (2004).

The use of sentence-level contexts also facilitates testing the impact of different experimental designs. For example, several of Caliskan et al.'s tests rely on given names associated with European American and African American people or rely on terms referring to women and men as groups (such as "woman" and "man"). We explore the effect of using given names versus group terms by creating alternate versions of several bias tests that swap the two. This is not generally feasible with WEAT, as categories like _African Americans_ lack common single-word group terms.

We find varying evidence of human-like bias in sentence encoders using SEAT. Sentence-to-vector encoders largely exhibit the angry black woman stereotype and Caliskan biases, and to a lesser degree the double bind biases. Recent sentence encoders such as BERT Devlin et al. (2018) display limited evidence of the tested biases. However, while SEAT can confirm the existence of bias, negative results do not indicate the model is bias-free. Furthermore, discrepancies in the results suggest that the confirmed biases may not generalize beyond the specific words and sentences in our test data, and in particular that cosine similarity may not be a suitable measure of representational similarity in recent models, indicating a need for alternate bias detection techniques.





## 2 Methods

**The Word Embedding Association Test WEAT imitates the human implicit association test**Greenwald et al.****,** 1998**)** **for word embeddings, measuring the association between two sets of target concepts and two sets of attributes. Let** \(X\) **and** \(Y\) **be equal-size sets of target concept embeddings and let** \(A\) **and** \(B\) **be sets of attribute embeddings. The test statistic is a difference between sums over the respective target concepts,**

\[s(X,Y,A,B)=\begin{bmatrix}\sum_{x\in X}s(x,A,B)-\\ \sum_{y\in Y}s(y,A,B)\end{bmatrix},\]

**where each addend is the difference between mean cosine similarities of the respective attributes,**

\[\begin{array}{l}s(w,A,B)=\begin{bmatrix}\mathrm{mean}_{a\in A}\cos(w,a)- \\ \mathrm{mean}_{b\in B}\cos(w,b)\end{bmatrix}\end{array}\]

**A permutation test on** \(s(X,Y,A,B)\) **is used to compute the significance of the association between** \((A,B)\) **and** \((X,Y)\)**,**

\[p=\Pr\left[s(X_{i},Y_{i},A,B)>s(X,Y,A,B)\right],\]

**where the probability is computed over the space of partitions** \((X_{i},Y_{i})\) **of** \(X\cup Y\) **such that** \(X_{i}\) **and** \(Y_{i}\) **are of equal size, and a normalized difference of means of** \(s(w,A,B)\) **is used to measure the magnitude of the association (the effect size; Caliskan et al.****,** 2017**),**

\[d=\frac{\mathrm{mean}_{x\in X}s(x,A,B)-\mathrm{mean}_{y\in Y}s(y,A,B)}{ \mathrm{std}_{\text{dev}_{w\in X\cup Y}}s(w,A,B)}.\]

**Controlling for significance, a larger effect size reflects a more severe bias. We detail our implementations in the supplement.**

**The Sentence Encoder Association TestSEAT compares sets of sentences, rather than sets of words, by applying WEAT to the vector representation of a sentence. Because SEAT operates on fixed-sized vectors and some encoders produce variable-length vector sequences, we use pooling as needed to aggregate outputs into a fixed-sized vector. We can view WEAT as a special case of SEAT in which the sentence is a single word. In fact, the original WEAT tests have been run on the Universal Sentence Encoder**Cer et al.****,** 2018**.**

**To extend a word-level test to sentence contexts, we slot each word into each of several semantically bleached sentence templates such as "This is** \(<\)**word**\(>\)**.", "**\(<\)**word**\(>\) **is here.", "This will** \(<\)**word**\(>\)**.", and "**\(<\)**word**\(>\) **are things.". These templates make heavy use of deixis and are designed to convey little specific meaning beyond that of the terms inserted into them.**2 **For example, the word version of Caliskan Test 3 is illustrated in Table** 1 **and the sentence version is illustrated in Table** 2**. We choose this design to focus on the associations a sentence encoder makes with a given term rather than those it happens to make with the contexts of that term that are prevalent in the training data; a similar design was used in a recent sentiment analysis evaluation corpus stratified by race and gender**Kirtichenko and Mohammad****,** 2018**)**. To facilitate future work, we publicly release code for SEAT and all of our experiments.**3  Footnote 2: See the supplement for further details and examples.

Footnote 3: [http://github.com/W4ngatang/sent-bias](http://github.com/W4ngatang/sent-bias)

## 3Biases Tested

#### Caliskan Tests

We first test whether the sentence encoders reproduce the same biases that word embedding models exhibited in Caliskan et al.****(**2017**). These biases correspond to past social psychology studies of implicit associations in human subjects.4 We apply both the original

\begin{table}
\begin{tabular}{l l} \hline \hline
**Target Concepts** & **Attributes** \\ \hline _European American names_: Adam, Harry, Nancy, Ellen, Alan, Paul, Katie, ... & _Pleasant_: love, cheer, miracle, peace, friend, happy, ... \\ \hline _African American names_: Jamel, Lavar, Lavon, Tai, Latisha, Malika, ... & _Unpleasant_: ugly, evil, abuse, murder, assault, rotten, ... \\ \hline \hline \end{tabular}
\end{table}
Table 1: Subsets of target concepts and attributes from Caliskan Test 3. Concept and attribute names are in italics. The test compares the strength of association between the two target concepts and two attributes, where all four are represented as sets of words.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Target Concepts** & **Attributes** \\ \hline _European American names_: “This is Katie.”, ” This is Adam.” Adam is there.”, ... & _“Pleaseasant_: _“There is love.”, ” That is happy.”, ” This is a friend.”, .... \\ \hline _African American names_: “Jamel is here.”, ”That is Vai.”, ”They are evil..”, ”Tia is a person.”, ... & _“Doescan_: _“These are really?

word-level versions of these tests as well as our generated sentence-level versions.

Angry Black Woman StereotypeIn the _Sapphire_ or _angry black woman__(ABW)_ stereotype, black women are portrayed as loud, angry, and imposing Collins (2004); Madison (2009); Harris-Perry (2011); hooks (2015); Gillespie (2016). This stereotype contradicts common associations made with the ostensibly race-neutral (unmarked) category of _women_Bem (1974), suggesting that that category is implicitly white. _Intersectionality_ reveals that experiences considered common to women are not necessarily shared by black women, who are marginalized both among women and among black people Crenshaw (1989). Recently, intersectionality has been demonstrated in English Wikipedia using distributional semantic word representations Herbelot et al. (2012), and in the disparate error rates of machine learning technologies like face recognition Buolamwini and Gebru (2018).

To measure sentence encoders' reproduction of the angry black woman stereotype, we create a test whose target concepts are black-identifying and white-identifying female given names from Sweeney (2013, Table 1) and whose attributes are adjectives used in the discussion of the stereo-type in Collins (2004, pp. 87-90) and their antonyms. We also produce a version of the test with attributes consisting of terms describing black women and white women as groups, as well as sentence versions in which attribute and target concept terms are inserted in sentence templates.

Double BindsWomen face many _double binds_, contradictory or unsatisfiable expectations of femininity and masculin Stone Lovejoy (2004); Harris-Perry (2011); Mitchell (2012). If women clearly succeed in a male gender-typed job, they are perceived less likable and more hostile than men in similar positions; if success is ambiguous, they are perceived less competent and achievement-oriented than men. Both outcomes can interfere in performance evaluations Heilman et al. (2004), contributing to the _glass ceiling_ impeding women's career advancement.5

Footnote 5: See Schluter (2018) for a recent exposition of the glass ceiling in the NLP research community.

We test this double bind in sentence encoders by translating Heilman et al.'s experiment to two SEAT tests. In the first, we represent the two target concepts by names of women and men, respectively, in the single sentence template "\(<\)word\(>\) is an engineer with superior technical skills."; the attributes are _likable_ and _non-hostile_ terms, based on Heilman et al.'s design, in the sentence template "The engineer is \(<\)word\(>\)." In the second, we use the shortened target concept sentence template "\(<\)word\(>\) is an engineer" and fill the attribute templates from before with _competent_ and _achieverment-oriented_ terms based on Heilman et al.'s design.6 We refer to these tests as semantically _unbleached_ because the context contains important information about the bias. We produce two variations of these tests: word-level tests in which target concepts are names in isolation and attributes are adjectives in isolation, as well as corresponding semantically bleached sentence-level tests. These control conditions allow us to probe the extent to which observed associations are attributable to gender independent of context.

Footnote 6: We consider other formulations in the supplement.

## 4 Experiments and Results

We apply SEAT to seven sentence encoders (listed in Table 3) including simple bag-of-words encoders, sentence-to-vector models, and state-of-the-art sequence models.7 For all models, we use publicly available pretrained parameters.

Footnote 7: We provide further details and explore variations on these model configurations in the supplement.

Table 4 shows effect size and significance at 0.01 before and after applying the Holm-Bonferroni multiple testing correction Holm (1979) for a subset of tests and models; complete results are provided in the supplement.8

[FOOTNOTE:7][ENDFOOTNOT

\begin{table}
\begin{tabular}{l l c} \hline \hline
**Model** & **Agg.** & **Dim.** \\ \hline CBoW (GloVe), 840 billion token web corpus version Pennington et al. (2014) & mean & 300 \\ InferSent, AllNLI Conneau et al. (2017) & max & 4096 \\ GenSen, +STN +Fr +De +NLI +L +STP (Subramanian et al., 2018) & last & 4096 \\ Universal Sentence Encoder (USE), DAN version Cer et al. (2018) & N/A & 512 \\ ELMo Peters et al. (2018), sum over layers after mean-pooling over sequence GPT Radford et al. (2018) & mean & 1024 \\ BERT, large, cased Devlin et al. (2018) & [CLS] & 1024 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Models tested (disambiguated with notation from cited paper), aggregation functions applied across token representations, and representation dimensions.

 

Specifically, we select Caliskan Test 1 associating flowers/insects with pleasant/unpleasant, Test 3 associating European/African American names with pleasant/unpleasant, and Test 6 associating male/female names with career/family, as well as the angry black woman stereotype and the competent and likable double bind tests. We observe that tests based on given names more often find a significant association than those based on group terms; we only show the given-name results here.

Footnote 8: [http://www.cdsweb.org/en/2017/09/](http://www.cdsweb.org/en/2017/09/)

We find varying evidence of bias in sentence encoders according to these tests. Bleached sentence-level tests tend to elicit more significant associations than word-level tests, while the latter tend to have larger effect sizes. We find stronger evidence for the Caliskan and ABW stereotype tests than for the double bind. After the multiple testing correction, we only find evidence of the double bind in bleached, sentence-level _competent_ control tests; that is, we find women are associated with incompetence independent of context.9

Footnote 9: However, the double bind results differ across models; we show no significant associations for ELMo or GPT and only one each for USE and BERT.

Some patterns in the results cast doubt on the reasonableness of SEAT as an evaluation. For instance, Caliskan Test 7 (association between _math/art_ and _male/female_) and Test 8 (_science/art_ and _male/female_) elicit counterintuitive results from several models. These tests have the same sizes of target concept and attribute sets. For CBoW on the word versions of those tests, we see \(p\)-values of 0.016 and \(10^{-2}\), respectively. On the sentence versions, we see \(p\)-values of \(10^{-5}\) for both tests. Observing similar \(p\)-values agrees with intuition: The _math/art_ association should be similar to the _science/art_ association because they instantiate a disciplinary dichotomy between _math/science_ and _arts/language_[4]. However, for BERT on the sentence version, we see discrepant \(p\)-values of \(10^{-5}\) and 0.14; for GenSen, 0.12 and \(10^{-3}\); and for GPT, 0.89 and \(10^{-4}\).

Caliskan Tests 3, 4, and 5 elicit even more counterintuitive results from ELMo. These tests measure the association between _European American/African American_ and _pleasant/unpleasant_. Test 3 has larger attribute sets than Test 4, which has larger target concept sets than Test 5. Intuitively, we expect increasing \(p\)-values across Tests 3, 4, and 5, as well-designed target concepts and attributes of larger sizes should yield higher-power tests. Indeed, for CBoW, we find increasing \(p\)-values of \(10^{-5}\), \(10^{-5}\), and \(10^{-4}\) on the word versions of the tests and \(10^{-5}\), \(10^{-5}\), and \(10^{-2}\) on the sentence versions, respectively.10 However, for ELMo, we find _decreasing_\(p\)-values of \(0.95\), \(0.45\), and \(0.08\) on the word versions of the tests and \(1\), \(0.97\), and \(10^{-4}\) on the sentence versions. We interpret these results as ELMo producing substantially different representations for conceptually similar words. Thus, SEAT's assumption that the sentence representations of each target concept and attribute instantiate a coherent concept appears invalid.

\begin{table}
\begin{tabular}{l l c c c c c c} \hline \hline
**Test** & **Context** & **CBoW** & **InferSent** & **GenSen** & **USE** & **ELMo** & **GPT** & **BERT** \\ \hline C1: Flowers/Insects & word & \(1.50^{**}\) & \(1.56^{**}\) & \(1.24^{**}\) & \(1.38^{**}\) & \(-0.03\) & \(0.20\) & \(0.22\) \\ C1: Flowers/Insects & sent & \(1.56^{**}\) & \(1.65^{**}\) & \(1.22^{**}\) & \(1.38^{*}\) & \(0.42^{**}\) & \(0.81^{**}\) & \(0.62^{**}\) \\ C3: EA/AA Names & word & \(1.41^{**}\) & \(1.33^{**}\) & \(1.32^{**}\) & \(0.52\) & \(-0.40\) & \(0.60^{*}\) & \(-0.11\) \\ C3: EA/AA Names & sent & \(0.52^{**}\) & \(1.07^{**}\) & \(0.97^{**}\) & \(0.32^{*}\) & \(-0.38\) & \(0.19\) & \(0.05\) \\ C6: MF Names, Career & word & \(1.81^{*}\) & \(1.78^{*}\) & \(1.84^{*}\) & \(0.02\) & \(-0.45\) & \(0.22\) & \(0.21\) \\ C6: MF Names, Career & sent & \(1.74^{**}\) & \(1.69^{**}\) & \(1.63^{**}\) & \(0.83^{**}\) & \(-0.38\) & \(0.35\) & \(0.08\) \\ ABW Stereotype & word & \(1.10^{*}\) & \(1.18^{*}\) & \(1.57^{**}\) & \(-0.39\) & \(0.53\) & \(0.08\) & \(-0.32\) \\ ABW Stereotype & sent & \(0.62^{**}\) & \(0.98^{**}\) & \(1.05^{**}\) & \(-0.19\) & \(0.52^{*}\) & \(-0.07\) & \(-0.17\) \\ Double Bind: Competent & word & \(1.62^{*}\) & \(1.09\) & \(1.49^{*}\) & \(1.51^{*}\) & \(-0.35\) & \(-0.28\) & \(-0.81\) \\ Double Bind: Competent & sent & \(0.79^{**}\) & \(0.57^{*}\) & \(0.83^{**}\) & \(0.25\) & \(-0.15\) & \(0.10\) & \(0.39\) \\ Double Bind: Competent & sent (u) & \(0.84\) & \(1.42^{*}\) & \(1.03\) & \(0.71\) & \(0.20\) & \(0.71\) & \(1.17^{*}\) \\ Double Bind: Likable & word & \(1.29^{*}\) & \(0.65\) & \(1.31^{*}\) & \(0.16\) & \(-0.60\) & \(0.91\) & \(-0.55\) \\ Double Bind: Likable & sent & \(0.69^{*}\) & \(0.37\) & \(0.25\) & \(0.32\) & \(-0.45\) & \(-0.20\) & \(-0.35\) \\ Double Bind: Likable & sent (u) & \(0.51\) & \(1.33^{*}\) & \(0.05\) & \(0.48\) & \(-0.90\) & \(-0.87\) & \(0.99\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: SEAT effect sizes for select tests, including word-level (word), bleached sentence-level (sent), and unbleached sentence-level (sent (u)) versions. \(CN\): test from Caliskan et al. (2017, Table 1) row \(N\); ‘*’: significant at \(0.01\), **: significant at \(0.01\) after multiple testing correction.

 

Conclusion

At face value, our results suggest recent sentence encoders exhibit less bias than previous models do, at least when "bias" is considered from a U.S. perspective and measured using the specific tests we have designed. However, we strongly caution against interpreting the number of significant associations or the average significant effect size as an absolute measure of bias. Like WEAT, SEAT only has positive predictive ability: It can detect presence of bias, but not its absence. Considering that these representations are trained without explicit bias control mechanisms on naturally occurring text, we argue against interpreting a lack of evidence of bias as a lack of bias.

Moreover, the counterintuitive sensitivity of SEAT on some models and biases suggests that biases revealed by SEAT may not generalize beyond the specific words and sentences in our test data. That is, our results invalidate the assumption that each set of words or sentences in our tests represents a coherent concept/attribute (like _African American_ or _pleasant_) to the sentence encoders; hence, we do not assume the encoders will exhibit similar behavior on other potential elements of those concepts/attributes (other words or sentences representing, for example, _African American_ or _pleasant_).

One possible explanation of the observed sensitivity at the sentence level is that, from the sentence encoders' view, our sentence templates are not as _semantically bleached_ as we expect; small variations in their relative frequencies and interactions with the terms inserted into them may be undermining the coherence of the concepts/attributes they implement. Another possible explanation that also accounts for the sensitivity observed in the word-level tests is that cosine similarity is an inadequate measure of text similarity for sentence encoders. If this is the case, the biases revealed by SEAT may not translate to biases in downstream applications. Future work could measure bias at the application level instead, following Bailey and Deery (2018)'s recommendation based on the tension between descriptive and normative correctness in representations.

The angry black woman stereotype represents an _intersectional_ bias, a phenomenon not well anticipated by an additive model of racism and sexism Crenshaw (1989). Previous work has modeled biases at the intersection of race and gender in distributional semantic word representations Herbelot et al. (2012), natural language inference data Rudinger et al. (2017), and facial recognition systems Buolamwini and Gebru (2018), as well as at the intersection of dialect and gender in automatic speech recognition Tatman (2017). We advocate for further consideration of intersectionality in future work in order to avoid reproducing the erasure of multiple minorities who are most vulnerable to bias.

We have developed a simple sentence-level extension of an established word embedding bias instrument and used it to measure the degree to which pretrained sentence encoders capture a range of social biases, observing a large number of significant effects as well as idiosyncrasies suggesting limited external validity. This study is preliminary and leaves open to investigation several design choices that may impact the results; future work may consider revisiting choices like the use of semantically bleached sentence inputs, the aggregation applied to models that represent sentences with sequences of hidden states, and the use of cosine similarity between sentence representations. We challenge researchers of fairness and ethics in NLP to critically (re-)examine their methods; looking forward, we hope for a deeper consideration of the social contexts in which NLP systems are applied.

## Acknowledgments

We are grateful to Carolyn Rose, Jason Phang, Sebastien Jean, Thibault Fevry, Katharina Kann, and Benjamin Van Durme for helpful conversations concerning this work and to our reviewers for their thoughtful feedback. CM is funded by IARPA MATERIAL; RR is funded by DARPA AIDA; AW is funded by an NSF fellowship. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes. The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of IARPA, DARPA, NSF, or the U.S. Government.

## References

* Bailey and Deey (2018) Katherine Bailey and Oisin Deey. 2018. How (and how not!) to control for morally relevant bias in NLP. The Second ACL Workshop on Ethics in Nat 

ural Language Processing, New Orleans, LA. Invited Talk.
* Bem (1974) Sandra L. Bem. 1974. The measurement of psychological androgyny. _Journal of Consulting and Clinical Psychology_, 42(2):155-162.
* Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, _Advances in Neural Information Processing Systems 29_, pages 4349-4357. Curran Associates, Inc.
* Buolamwini and Gebru (2018) Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_, volume 81 of _Proceedings of Machine Learning Research_, pages 77-91, New York, NY, USA. PMLR.
* Caliskan et al. (2017) Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. _Science_, 356(6334):183-186.
* Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for english. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 169-174, Brussels, Belgium. Association for Computational Linguistics.
* Collins (2004) Patricia Hill Collins. 2004. _Black Sexual Politics: African Americans, Gender, and the New Racism_. Routledge, New York.
* Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 670-680, Copenhagen, Denmark. Association for Computational Linguistics.
* Crenshaw (1989) Kimberle Crenshaw. 1989. Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory, and antracist politics. _Feminist Legal Theory: Readings in Law and Gender_, 1989(1):57-80.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _ArXiv e-prints_.
* Garg et al. (2018) Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. _Proceedings of the National Academy of Sciences_.
* Gillespie (2016) Andra Gillespie. 2016. Race, perceptions of femininity, and the power of the first lady: A comparative analysis. In Nadia E. Brown and Sarah Allen Gershon, editors, _Distinct Identities: Minority Women in U.S. Politics_, chapter 15, pages 223-238. Routledge, New York.
* Greenwald et al. (1998) Anthony G. Greenwald, Debbie E. McGhee, and Jordan L. K. Schwartz. 1998. Measuring individual differences in implicit cognition: The implicit association test. _Journal of Personality and Social Psychology_, 74(6):1464-1480.
* Greenwald et al. (2009) Anthony G. Greenwald, T. Andrew Poehman, Eric Luis Ulhmann, and Mahzarin R. Banaji. 2009. Understanding and using the implicit association test: III. Meta-analysis of predictive validity. _Journal of Personality and Social Psychology_, 97(1):17-41.
* Harris-Perry (2011) Melissa V. Harris-Perry. 2011. _Sister Citizen: Shame, Stereotypes, and Black Women in America_. Yale University Press, New Haven, Connecticut.
* Heilman et al. (2004) Madeline E. Heilman, Aaron S. Wallen, Daniella Fuchs, and Melinda M. Tamkins. 2004. Penalties for success: Reactions to women who succeed at male gender-typed tasks. _Journal of Applied Psychology_, 89(3):416-427.
* Herbelot et al. (2012) Aurelie Herbelot, Eva von Redecker, and Johanna Muller. 2012. Distributional techniques for philosophical enquiry. In _Proceedings of the 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities_, pages 45-54, Avignon, France. Association for Computational Linguistics.
* Holm (1979) Sture Holm. 1979. A simple sequentially rejective multiple test procedure. _Scandinavian Journal of Statistics_, 6(2):65-70.
* hooks (2015) bell hooks. 2015. _Ain't I a Woman: Black Women and Feminism_, 2 edition. Routledge, Taylor & Francis Group, New York.
* Kiritchenko and Mohammad (2018) Svetlana Kiritchenko and Saif Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_, pages 43-53, New Orleans, Louisiana. Association for Computational Linguistics.
* Madison (2009) D. Soyini Madison. 2009. Crazy patriotism and angry (post)black women. _Communication and Critical/Cultural Studies_, 6(3):321-326.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 26_, pages 3111-3119. Curran Associates, Inc.

 

Kaye Mitchell. 2012. Raunch versus prude: contemporary sex blogs and exotic memoirs by women. Psychology & Sexuality, 3(1):12-25.
* Nosek et al. (2002) Brian A. Nosek, Mahzarin R. Banaji, and Anthony G. Greenwald. 2002. Math \(=\) male, me \(=\) female, therefore math \(\neq\) me. _Journal of Personality and Social Psychology_, 83(1):44-59.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
* Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Online; accessed November 28, 2018.
* Rudinger et al. (2017) Rachel Rudinger, Chandler May, and Benjamin Van Durme. 2017. Social bias in elicited natural language inferences. In _Proceedings of the First ACL Workshop on Ethics in Natural Language Processing_, pages 74-79, Valencia, Spain. Association for Computational Linguistics.
* Schluter (2018) Natalie Schluter. 2018. The glass ceiling in nlp. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2793-2798, Brussels, Belgium. Association for Computational Linguistics.
* Stone and Lovejoy (2004) Pamela Stone and Meg Lovejoy. 2004. Fast-track women and the "choice" to stay home. _The ANNALS of the American Academy of Political and Social Science_, 596(1):62-83.
* Subramanian et al. (2018) Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. In _International Conference on Learning Representations_, Vancouver, Canada.
* Sweeney (2013) Latanya Sweeney. 2013. Discrimination in online ad delivery. _Communications of the ACM_, 56(5):44-54.
* Tatman (2017)Rachael Tatman. 2017. Gender and dialect bias in youtube's automatic captions. In _Proceedings of the First ACL Workshop on Ethics in Natural Language Processing_, pages 53-59, Valencia, Spain. Association for Computational Linguistics.