Introduction
There is a classic riddle: A man and his son get into a terrible car crash. The father dies, and the boy is badly injured. In the hospital, the surgeon looks at the patient and exclaims, “I can't operate on this boy, he's my son!” How can this be?
That a majority of people are reportedly unable to solve this riddle is taken as evidence of underlying implicit gender bias BIBREF0 : many first-time listeners have difficulty assigning both the role of “mother” and “surgeon” to the same entity.
As the riddle reveals, the task of coreference resolution in English is tightly bound with questions of gender, for humans and automated systems alike (see Figure 1 ). As awareness grows of the ways in which data-driven AI technologies may acquire and amplify human-like biases BIBREF1 , BIBREF2 , BIBREF3 , this work investigates how gender biases manifest in coreference resolution systems.
There are many ways one could approach this question; here we focus on gender bias with respect to occupations, for which we have corresponding U.S. employment statistics. Our approach is to construct a challenge dataset in the style of Winograd schemas, wherein a pronoun must be resolved to one of two previously-mentioned entities in a sentence designed to be easy for humans to interpret, but challenging for data-driven systems BIBREF4 . In our setting, one of these mentions is a person referred to by their occupation; by varying only the pronoun's gender, we are able to test the impact of gender on resolution. With these “Winogender schemas,” we demonstrate the presence of systematic gender bias in multiple publicly-available coreference resolution systems, and that occupation-specific bias is correlated with employment statistics. We release these test sentences to the public.
In our experiments, we represent gender as a categorical variable with either two or three possible values: female, male, and (in some cases) neutral. These choices reflect limitations of the textual and real-world datasets we use.
Coreference Systems
In this work, we evaluate three publicly-available off-the-shelf coreference resolution systems, representing three different machine learning paradigms: rule-based systems, feature-driven statistical systems, and neural systems.
Winogender Schemas
Our intent is to reveal cases where coreference systems may be more or less likely to recognize a pronoun as coreferent with a particular occupation based on pronoun gender, as observed in Figure 1 . To this end, we create a specialized evaluation set consisting of 120 hand-written sentence templates, in the style of the Winograd Schemas BIBREF4 . Each sentence contains three referring expressions of interest:
We use a list of 60 one-word occupations obtained from Caliskan183 (see supplement), with corresponding gender percentages available from the U.S. Bureau of Labor Statistics. For each occupation, we wrote two similar sentence templates: one in which pronoun is coreferent with occupation, and one in which it is coreferent with participant (see Figure 2 ). For each sentence template, there are three pronoun instantiations (female, male, or neutral), and two participant instantiations (a specific participant, e.g., “the passenger,” and a generic paricipant, “someone.”) With the templates fully instantiated, the evaluation set contains 720 sentences: 60 occupations $\times $ 2 sentence templates per occupation $\times $ 2 participants $\times $ 3 pronoun genders.
Results and Discussion
We evaluate examples of each of the three coreference system architectures described in "Coreference Systems" : the BIBREF5 sieve system from the rule-based paradigm (referred to as RULE), BIBREF6 from the statistical paradigm (STAT), and the BIBREF11 deep reinforcement system from the neural paradigm (NEURAL).
By multiple measures, the Winogender schemas reveal varying degrees of gender bias in all three systems. First we observe that these systems do not behave in a gender-neutral fashion. That is to say, we have designed test sentences where correct pronoun resolution is not a function of gender (as validated by human annotators), but system predictions do exhibit sensitivity to pronoun gender: 68% of male-female minimal pair test sentences are resolved differently by the RULE system; 28% for STAT; and 13% for NEURAL.
Overall, male pronouns are also more likely to be resolved as occupation than female or neutral pronouns across all systems: for RULE, 72% male vs 29% female and 1% neutral; for STAT, 71% male vs 63% female and 50% neutral; and for NEURAL, 87% male vs 80% female and 36% neutral. Neutral pronouns are often resolved as neither occupation nor participant, possibly due to the number ambiguity of “they/their/them.”
When these systems' predictions diverge based on pronoun gender, they do so in ways that reinforce and magnify real-world occupational gender disparities. Figure 4 shows that systems' gender preferences for occupations correlate with real-world employment statistics (U.S. Bureau of Labor Statistics) and the gender statistics from text BIBREF14 which these systems access directly; correlation values are in Table 1 . We also identify so-called “gotcha” sentences in which pronoun gender does not match the occupation's majority gender (BLS) if occupation is the correct answer; all systems perform worse on these “gotchas.” (See Table 2 .)
Because coreference systems need to make discrete choices about which mentions are coreferent, percentage-wise differences in real-world statistics may translate into absolute differences in system predictions. For example, the occupation “manager” is 38.5% female in the U.S. according to real-world statistics (BLS); mentions of “manager” in text are only 5.18% female (B&L resource); and finally, as viewed through the behavior of the three coreference systems we tested, no managers are predicted to be female. This illustrates two related phenomena: first, that data-driven NLP pipelines are susceptible to sequential amplification of bias throughout a pipeline, and second, that although the gender statistics from B&L correlate with BLS employment statistics, they are systematically male-skewed (Figure 3 ).
Related Work
Here we give a brief (and non-exhaustive) overview of prior work on gender bias in NLP systems and datasets. A number of papers explore (gender) bias in English word embeddings: how they capture implicit human biases in modern BIBREF1 and historical BIBREF15 text, and methods for debiasing them BIBREF16 . Further work on debiasing models with adversarial learning is explored by DBLP:journals/corr/BeutelCZC17 and zhang2018mitigating.
Prior work also analyzes social and gender stereotyping in existing NLP and vision datasets BIBREF17 , BIBREF18 . tatman:2017:EthNLP investigates the impact of gender and dialect on deployed speech recognition systems, while zhao-EtAl:2017:EMNLP20173 introduce a method to reduce amplification effects on models trained with gender-biased datasets. koolen-vancranenburgh:2017:EthNLP examine the relationship between author gender and text attributes, noting the potential for researcher interpretation bias in such studies. Both larson:2017:EthNLP and koolen-vancranenburgh:2017:EthNLP offer guidelines to NLP researchers and computational social scientists who wish to predict gender as a variable. hovy-spruit:2016:P16-2 introduce a helpful set of terminology for identifying and categorizing types of bias that manifest in AI systems, including overgeneralization, which we observe in our work here.
Finally, we note independent but closely related work by zhao-wang:2018:N18-1, published concurrently with this paper. In their work, zhao-wang:2018:N18-1 also propose a Winograd schema-like test for gender bias in coreference resolution systems (called “WinoBias”). Though similar in appearance, these two efforts have notable differences in substance and emphasis. The contribution of this work is focused primarily on schema construction and validation, with extensive analysis of observed system bias, revealing its correlation with biases present in real-world and textual statistics; by contrast, zhao-wang:2018:N18-1 present methods of debiasing existing systems, showing that simple approaches such as augmenting training data with gender-swapped examples or directly editing noun phrase counts in the B&L resource are effective at reducing system bias, as measured by the schemas. Complementary differences exist between the two schema formulations: Winogender schemas (this work) include gender-neutral pronouns, are syntactically diverse, and are human-validated; WinoBias includes (and delineates) sentences resolvable from syntax alone; a Winogender schema has one occupational mention and one “other participant” mention; WinoBias has two occupational mentions. Due to these differences, we encourage future evaluations to make use of both datasets.
Conclusion and Future Work
We have introduced “Winogender schemas,” a pronoun resolution task in the style of Winograd schemas that enables us to uncover gender bias in coreference resolution systems. We evaluate three publicly-available, off-the-shelf systems and find systematic gender bias in each: for many occupations, systems strongly prefer to resolve pronouns of one gender over another. We demonstrate that this preferential behavior correlates both with real-world employment statistics and the text statistics that these systems use. We posit that these systems overgeneralize the attribute of gender, leading them to make errors that humans do not make on this evaluation. We hope that by drawing attention to this issue, future systems will be designed in ways that mitigate gender-based overgeneralization.
It is important to underscore the limitations of Winogender schemas. As a diagnostic test of gender bias, we view the schemas as having high positive predictive value and low negative predictive value; that is, they may demonstrate the presence of gender bias in a system, but not prove its absence. Here we have focused on examples of occupational gender bias, but Winogender schemas may be extended broadly to probe for other manifestations of gender bias. Though we have used human-validated schemas to demonstrate that existing NLP systems are comparatively more prone to gender-based overgeneralization, we do not presume that matching human judgment is the ultimate objective of this line of research. Rather, human judgements, which carry their own implicit biases, serve as a lower bound for equitability in automated systems.
Acknowledgments
The authors thank Rebecca Knowles and Chandler May for their valuable feedback on this work. This research was supported by the JHU HLTCOE, DARPA AIDA, and NSF-GRFP (1232825). The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes. The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA or the U.S. Government.

Question: Which coreference resolution systems are tested?
Answer: the BIBREF5 sieve system from the rule-based paradigm (referred to as RULE), BIBREF6 from the statistical paradigm (STAT), and the BIBREF11 deep reinforcement system from the neural paradigm (NEURAL).