SlowGuess/ACL_2020_OCR huggingface dataset:
Yang Trista Cao  
University of Maryland  
ycao95@cs.umd.edu
Hal Daumé III  
University of Maryland  
Microsoft Research  
me@hal3.name
Correctly resolving textual mentions of people fundamentally entails making inferences about those people. Such inferences raise the risk of systemic biases in coreference resolution systems, including biases that can harm binary and non-binary trans and cis stakeholders. To better understand such biases, we foreground nuanced conceptualizations of gender from sociology and sociolinguistics, and develop two new datasets for interrogating bias in crowd annotations and in existing coreference resolution systems. Through these studies, conducted on English text, we confirm that without acknowledging and building systems that recognize the complexity of gender, we build systems that lead to many potential harms.
Coreference resolution—the task of determining which textual references resolve to the same real-world entity—requires making inferences about those entities. Especially when those entities are people, coreference resolution systems run the risk of making unlicensed inferences, possibly resulting in harms either to individuals or groups of people. Embedded in coreference inferences are varied aspects of gender, both because gender can show up explicitly (e.g., pronouns in English, morphology in Arabic) and because societal expectations and stereotypes around gender roles may be explicitly or implicitly assumed by speakers or listeners. This can lead to significant biases in coreference resolution systems: cases where systems “systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others” (Friedman and Nissenbaum, 1996, p. 332).
Gender bias in coreference resolution can manifest in many ways; work by Rudinger et al. (2018), Zhao et al. (2018a), and Webster et al. (2018) focused largely on the case of binary gender dis
crimination in trained coreference systems, showing that current systems over-rely on social stereotypes when resolving HE and SHE pronouns<sup>1</sup> (see §2). Contemporaneously, critical work in Human-Computer Interaction has complicated discussions around gender in other fields, such as computer vision (Keyes, 2018; Hamidi et al., 2018).
Building on both lines of work, and inspired by Keyes's (2018) study of vision-based automatic gender recognition systems, we consider gender bias from a broader conceptual frame than the binary "folk" model. We investigate ways in which folk notions of gender—namely that there are two genders, assigned at birth, immutable, and in perfect correspondence to gendered linguistic forms—lead to the development of technology that is exclusionary and harmful of binary and non-binary trans and cis people. Addressing such issues is critical not just to improve the quality of our systems, but more pointedly to minimize the harms caused by our systems by reinforcing existing unjust social hierarchies (Lambert and Packer, 2019).
There are several stakeholder groups who may easily face harms when coreference systems is used (Blodgett et al., 2020). Those harms include several possible harms, both allocational and representation harms (Barocas et al., 2017), including quality of service, erasure, and stereotyping harms. Following Bender's (2019) taxonomy of stakehold
1Throughout, we avoid mapping pronouns to a "gender" label, preferring to use the pronoun directly, include (in English) SHE, HE, the non-binary use of singular THEY, and neopronouns (e.g., ZE/HIR, XEY/XEM), which have been in usage since at least the 1970s (Bustillos, 2011; Merriam-Webster, 2016; Bradley et al., 2019; Hord, 2016; Spivak, 1997).
$^{2}$ Following GLAAD (2007), transgender individuals are those whose gender differs from the sex they were assigned at birth. This is in opposition to cisgender individuals, whose assigned sex at birth happens to correspond to their gender. Transgender individuals can either be binary (those whose gender falls in the "male/female" dichotomy) or non-binary (those for which the relationship is more complex).
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568-4595 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ers and Barocas et al.'s (2017) taxonomy of harms, there are several ways in which trans exclusionary coreference resolution systems can cause harm:
$\diamond$  Indirect: subject of query. If a person is the subject of a web query, pages about xem may be missed if "multiple mentions of query" is a ranking feature, and the system cannot resolve xyr pronouns  $\Rightarrow$  quality of service, erasure.
$\diamond$  Direct: by choice. If a grammar checker uses coreference, it may insist that an author writing hir third-person autobiography is repeatedly making errors when referring to himself  $\Rightarrow$  quality of service, stereotyping, denigration.
$\diamond$  Direct: not by choice. If an information extraction system run on résumés relies on cisnormative assumptions, job experiences by a candidate who has transitioned and changed his pronouns may be missed  $\Rightarrow$  allocative, erasure.
$\diamond$  Many stakeholders. If a machine translation system uses discourse context to generate pronouns, then errors can results in directly misgendering subjects of the document being translated  $\Rightarrow$  quality of service, denigration, erasure.
To address such harms as well as understand where and how they arise, we need to complicate (a) what "gender" means and (b) how harms can enter into natural language processing (NLP) systems. Toward (a), we begin with a unifying analysis ( $\S 3$ ) of how gender is socially constructed, and how social conditions in the world impose expectations around people's gender. Of particular interest is how gender is reflected in language, and how that both matches and potentially mismatches the way people experience their gender in the world. Then, in order to understand social biases around gender, we find it necessary to consider the different ways in which gender can be realized linguistically, breaking down what previously have been considered "gendered words" in NLP papers into finer-grained categories that have been identified in the sociolinguistics literature of lexical, referential, grammatical, and social gender.
Toward (b), we focus on how bias can enter into two stages of machine learning systems: data annotation (§4) and model definition (§5). We construct two new datasets: (1) MAP (a similar dataset to GAP (Webster et al., 2018) but without binary gender constraints) on which we can perform counterfactual manipulations and (2) GICoref (a fully annotated coreference resolution dataset
written by and about trans people). In all cases, we focus largely on harms due to over- and under-representation (Kay et al., 2015), replicating stereotypes (Sweeney, 2013; Caliskan et al., 2017) (particular those that are cisnormative and/or heteronormative), and quality of service differentials (Buolamwini and Gebru, 2018).
The primary contributions of this paper are: (1) Connecting existing work on gender bias in NLP to sociological and sociolinguistic conceptions of gender to provide a scaffolding for future work on analyzing "gender bias in NLP" (§3). (2) Developing an ablation technique for measuring gender bias in coreference resolution annotations, focusing on the human bias that can enter into annotation tasks (§4). (3) Constructing a new dataset, the Gender Inclusive Coreference dataset (GICOREF), for testing performance of coreference resolution systems on texts that discuss non-binary and binary transgender people (§5).
There are four recent papers that consider gender bias in coreference resolution systems. Rudinger et al. (2018) evaluates coreference systems for evidence of occupational stereotyping, by constructing Winograd-esque (Levesque et al., 2012) test examples. They find that humans can reliably resolve these examples, but systems largely fail at them, typically in a gender-stereotypical way. In contemporaneous work, Zhao et al. (2018a) proposed a very similar, also Winograd-esque scheme, also for measuring gender-based occupational stereotypes. In addition to reaching similar conclusions to Rudinger et al. (2018), this work also used a similar "counterfactual" data process as we use in §4.1 in order to provide additional training data to a coreference resolution system. Webster et al. (2018) produced the GAP dataset for evaluating coreference systems, by specifically seeking examples where "gender" (left underspecified) could not be used to help coreference. They found that coreference systems struggle in these cases, also pointing to the fact that some success of current coreference systems is due to reliance on (binary) gender stereotypes. Finally, Ackerman (2019) presents an alternative breakdown of gender than we use (§3), and proposes matching criteria for model
3Both datasets are released under a BSD license at github.com/TristaCao/into_inclusivecoref with corresponding datasheets (Gebru et al., 2018).
ing coreference resolution linguistically, taking a trans-inclusive perspective on gender.
Gender bias in NLP has been considered more broadly than just in coreference resolution, including, natural language inference (Rudinger et al., 2017), word embeddings (e.g., Bolukbasi et al., 2016; Romanov et al., 2019; Gonen and Goldberg, 2019), sentiment analysis (Kiritchenko and Mohammad, 2018), machine translation (Font and Costa-jussa, 2019; Prates et al., 2019; Dryer, 2013; Frank et al., 2004; Wandruszka, 1969; Nissen, 2002; Doleschal and Schmid, 2001), among many others (Blodgett et al., 2020, inter alia). Gender is also an object of study in gender recognition systems (Hamidi et al., 2018). Much of this work has focused on gender bias with a (usually implicit) binary lens, an issue which was also called out recently by Larson (2017b) and May (2019).
3 Linguistic & Social Gender
The concept of gender is complex and contested, covering (at least) aspects of a person's internal experience, how they express this to the world, how social conditions in the world impose expectations on them (including expectations around their sexuality), and how they are perceived and accepted (or not). When this complex concept is realized in language, the situation becomes even more complex: linguistic categories of gender do not even remotely map one-to-one to social categories. As observed by Bucholtz (1999):
"Attempts to read linguistic structure directly for information about social gender are often misguided."
For instance, when working in a language like English which formally marks gender on pronouns, it is all too easy to equate "recognizing the pronoun that corefers with this name" with "recognizing the real-world gender of referent of that name."
Furthermore, despite the impossibility of a perfect alignment with linguistic gender, it is generally clear that an incorrectly gendered reference to a person (whether through pronominalization or otherwise) can be highly problematic (Johnson et al., 2019; McLemore, 2015). This process of misgendering is problematic for both trans and cis individuals to the extent that transgender historian Stryker (2008) writes:
"[o]ne's gender identity could perhaps best be described as how one feels about being referred to by a particular pronoun."
3.1 Sociological Gender
Many modern trans-inclusive models of gender recognize that gender encompasses many different aspects. These aspects include the experience that one has of gender (or lack thereof), the way that one expresses one's gender to the world, and the way that normative social conditions impose gender norms, typically as a dichotomy between masculine and feminine roles or traits (Kramarae and Treichler, 1985; West and Zimmerman, 1987; Butler, 1990; Risman, 2009; Serano, 2007). Gender self-determination, on the other hand, holds that each person is the "ultimate authority" on their own gender identity (Zimman, 2019; Stanley, 2014), with Zimman (2019) further arguing the importance of the role language plays in that determination.
Such trans-inclusive models deconflate anatomical and biological traits and the sex that a person had assigned to them at birth from one's gendered position in society; this includes intersex people, whose anatomical/biological factors do not match the usual designational criteria for either sex. Transinclusive views typically recognize that gender exists beyond the regressive "female"/"male" binary $^4$ ; additionally, one's gender may shift by time or context (often "genderfluid"), and some people do not experience gender at all (often "agender") (Kessler and McKenna, 1978; Schilt and Westbrook, 2009; Darwin, 2017; Richards et al., 2017). In  $\S 5$  we analyze the degree to which NLP papers make trans-inclusive or trans-exclusive assumptions.
Social gender refers to the imposition of gender roles or traits based on normative social conditions (Kramarae and Treichler, 1985), which often includes imposing a dichotomy between feminine and masculine (in behavior, dress, speech, occupation, societal roles, etc.). Ackerman (2019) highlights a highly overlapping concept, "bio-social gender", which consists of gender role, gender expression, and gender identity. Taking gender role as an example, upon learning that a nurse is coming to their hospital room, a patient may form expectations that this person is likely to be "female," and may generate expectations around how their face or body may look, how they are likely to be dressed, how and where hair may appear, how to refer to them, and so on. This process, often referred to as gendering (Serano, 2007) occurs both in real world
<sup>4</sup>Some authors use female/male for sex and woman/man for gender; we do not need this distinction (which is itself contestable) and use female/male for gender.
interactions, as well as in purely linguistic settings (e.g., reading a newspaper), in which readers may use social gender clues to assign gender(s) to the real world people being discussed.
3.2 Linguistic Gender
Our discussion of linguistic gender largely follows (Corbett, 1991; Ochs, 1992; Craig, 1994; Corbett, 2013; Hellinger and Motschenbacher, 2015; Fuertes-Olivera, 2007), departing from earlier characterizations that postulate a direct mapping from language to gender (Lakoff, 1975; Silverstein, 1979). Our taxonomy is related but not identical to (Ackerman, 2019), which we discuss in  $\S 2$ .
Grammatical gender, similarly defined in Ackerman (2019), is nothing more than a classification of nouns based on a principle of grammatical agreement. In "gender languages" there are typically two or three grammatical genders that have, for animate or personal references, considerable correspondence between a FEM (resp. MASC) grammatical gender and referents with female- (resp. male-) social gender. In comparison, "noun class languages" have no such correspondence, and typically many more classes. Some languages have no grammatical gender at all; English is generally seen as one (Nissen, 2002; Baron, 1971) (though this is contested (Bjorkman, 2017)).
Referential gender (similar, but not identical to Ackerman's (2019) "conceptual gender") relates linguistic expressions to extra-linguistic reality, typically identifying referents as "female," "male," or "gender-indefinite." Fundamentally, referential gender only exists when there is an entity being referred to, and their gender (or sex) is realized linguistically. The most obvious examples in English are gendered third person pronouns (SHE, HE), including neopronouns (ZE, EM) and singular  $\mathrm{THEY}^6$ , but also includes cases like "policeman" when the intended referent of this noun has social gender "male" (though not when "policeman" is used non-referentially, as in "every policeman needs to hold others accountable").
Lexical gender refers to an extra-linguistic properties of female-ness or male-ness in a non-referential way, as in terms like "mother" as well
as gendered terms of address like “Mrs.” Importantly, lexical gender is a property of the linguistic unit, not a property of its referent in the real world, which may or may not exist. For instance, in “Every son loves his parents”, there is no real world referent of “son” (and therefore no referential gender), yet it still (likely) takes HIS as a pronoun anaphor because “son” has lexical gender MASC.
3.3 Social and Linguistic Gender Interplays
The relationship between these aspects of gender is complex, and none is one-to-one. The referential gender of an individual (e.g., pronouns in English) may or may not match their social gender and this may change by context. This can happen in the case of people whose everyday life experience of their gender fluctuates over time (at any interval), as well as in the case of drag performers (e.g., some men who perform drag are addressed as SHE while performing, and HE when not (for Transgender Equality, 2017)). The other linguistic forms of gender (grammatical, lexical) also need not match each other, nor match referential gender (Hellinger and Motschenbacher, 2015).
Social gender (societal expectations, in particular) captures the observation that upon hearing "My cousin is a librarian", many speakers will infer "female" for "cousin", because of either an entailment of "librarian" or some sort of probabilistic inference (Lyons, 1977), but not based on either grammatical gender (which does not exist in English) or lexical gender. We focus on English, which has no grammatical gender, but does have lexical gender. English also marks referential gender on singular third person pronouns.
Below, we use this more nuanced notion of different types of gender to inspect how bias play out in coreference resolution systems. These biases may arise in the context of any of these notions of gender, and we encourage future work to extend care over and be explicit about what notions of gender are being utilized and when.
4 Bias in Human Annotation
A possible source of bias in coreference systems comes from human annotations on the data used to train them. Such biases can arise from a combination of (possibly) underspecified annotations guidelines and the positionality of annotators themselves. In this section, we study how different aspects of linguistic notions impact an annotator's
One difficulty in this discussion is that linguistic gender and social gender use the terms "feminine" and "masculine" differently; to avoid confusion, when referring to the linguistic properties, we use FEM and MASC.
6People's mental acceptability of singular THEY is still relatively low even with its increased usage (Prasad and Morris, 2020), and depends on context (Conrod, 2018).
Figure 1: Example of applying all ablation substitutions for an example context in the MAP corpus. Each substitution type is marked over the arrow and separately color-coded.
judgments of anaphora. This parallels Ackerman (2019) linguistic analysis, in which a Broad Matching Criterion is proposed, which posits that "matching gender requires at least one level of the mental representation of gender to be identical to the candidate antecedent in order to match."
Our study can be seen as evaluating which conceptual properties of gender are most salient in human judgments. We start with natural text in which we can cast the coreference task as a binary classification problem ("which of these two names does this pronoun refer to?") inspired by Webster et al. (2018). We then generate "counterfactual augmentations" of this dataset by ablating the various notions of linguistic gender described in §3.2, similar to Zmigrod et al. (2019). We finally evaluate the impact of these ablations on human annotation behavior to answer the question: which forms of linguistic knowledge are most essential for human annotators to make consistent judgments. See Appendix A for examples of how linguistic gender may be used to infer social gender.
4.1 Ablation Methodology
In order to determine which cues annotators are using and the degree to which they use them, we construct an ablation study in which we hide various aspects of gender and evaluate how this impacts annotators' judgments of anaphoricity. We construct binary classification examples taken from Wikipedia pages, in which a single pronoun is selected, and two possible antecedent names are given, and the annotator must select which one. We cannot use Webster et al.'s GAP dataset directly, because their data is constrained that the "gender" of the two possible antecedents is "the same"; for us, we are specifically interested in how annotators make decisions even when additional gender information is available. Thus, we construct a dataset called Maybe Ambiguous Pronoun (MAP) follow
ing Webster et al.'s approach, but we do not restrict the two names to match gender.
In ablating gender information, one challenge is that removing social gender cues (e.g., "nurse" tending female) is not possible because they can exist anywhere. Likewise, it is not possible to remove syntactic cues in a non-circular manner. For example in (1), syntactic structure strongly suggests the antecedent of "herself" is "Liang", making it less likely that "He" corefers with Liang later (though it is possible, and such cases exist in natural data due either to gender fluidity or misgendering).
(1) Liang saw herself in the mirror...He...
Fortunately, it is possible to enumerate a high coverage list of English terms that signal lexical gender: terms of address (Mrs., Mr.) and semantically gendered nouns (mother). We assembled a list by taking many online lists (mostly targeted at English language learners), merging them, and manual filtering. The assembling process and the final list is published with the MAP dataset and its datasheet.
To execute the "hiding" of various aspects of gender, we use the following substitutions:
(a)  $\neg \mathrm{PRO}$ : Replace third person pronouns with gender neutral variants (THEY, XEY, ZE).
(b)  $\neg$  NAME: Replace names by random names with only a first initial and last name.
(c)  $\neg$  SEM: Replace semantically gendered nouns with gender-indefinite variants.
(d)  $\neg$  ADDR: Remove terms of address.
See Figure 1 for an example of all substitutions.
We perform two sets of experiments, one following a "forward selection" type ablation (start with everything removed and add each back in one-at-a-time) and one following "backward selection" (remove each separately). Forward selection is necessary in order to de-conflate syntactic cues from
<sup>7</sup>It is unclear from the GAP dataset what notion of "gender" is used, nor how it was determined to be "the same."
These are, however, sometimes complex. For instance, "actress" signals lexical gender of female, while "actor" may signal social gender of male and, in certain varieties of English, may also signal lexical gender of male.
An alternative suggested by Cassidy Henry that we did not explore would be to replace all with Mx. or Dr.
stereotypes; while backward selection gives a sense of how much impact each type of gender cue has in the context of all the others.
We begin with ZERO, in which we apply all four substitutions. Since this also removes gender cues from the pronouns themselves, an annotator cannot substantially rely on social gender to perform these resolutions. We next consider adding back in the original pronouns (always HE or SHE here), yielding  $\neg$  NAME  $\neg$  SEM  $\neg$  ADDR. Any difference in annotation behavior between ZERO and  $\neg$  NAME  $\neg$  SEM  $\neg$  ADDR can only be due to social gender stereotypes. The next setting,  $\neg$  SEM  $\neg$  ADDR removes both forms of lexical gender (semantically gendered nouns and terms of address); differences between  $\neg$  SEM  $\neg$  ADDR and  $\neg$  NAME  $\neg$  SEM  $\neg$  ADDR show how much names are relied on for annotation. Similarly,  $\neg$  NAME  $\neg$  ADDR removes names and terms of address, showing the impact of semantically gendered nouns, and  $\neg$  NAME  $\neg$  SEM removes names and semantically gendered nouns, showing the impact of terms of address.