SlowGuess/ACL_2020_OCR huggingface dataset:
In the backward selection case, we begin with ORIG, which is the unmodified original text. To this, we can apply the pronoun filter to get  $\neg$  PRO; differences in annotation between ORIG and  $\neg$  PRO give a measure of how much any sort of gender-based inference is used. Similarly, we get  $\neg$  NAME by only removing names, which gives a measure of how much names are used (in the context of all other cues); we get  $\neg$  SEM by only removing semantically gendered words; and  $\neg$  ADDR by only removing terms of address.
4.2 Annotation Results
We construct examples using the methodology defined above. We then conduct annotation experiments using crowdworkers on Amazon Mechanical Turk following the methodology by which the original GAP corpus was created<sup>10</sup>. Because we wanted to also capture uncertainty, we ask the crowdworkers how sure they are in their choices, between "definitely" sure, "probably" sure and "unsure."
Figure 2 shows the human annotation results as binary classification accuracy for resolving the pronoun to the antecedent. We can see that removing pronouns leads to significant drop in accuracy. This indicates that gender-based inferences, especially social gender stereotypes, play the most significant
Figure 2: Human annotation results for the ablation study on MAP dataset. Each column is a different ablation, and the y-axis is the degree of accuracy with  $95\%$  significance intervals. Bottom bar plots are annotator certainties as how sure they are in their choices.
role when annotators resolve coreferences. This confirms the findings of Rudinger et al. (2018) and Zhao et al. (2018a) that human annotated data incorporates bias from stereotypes.
Moreover, if we compare ORIG with columns left to it, we see that name is another significant cue for annotator judgments, while lexical gender cues do not have significant impacts on human annotation accuracies. This is likely in part due to the low appearance frequency of lexical gender cues in our dataset. Every example has pronouns and names, whereas  $49\%$  of the examples have semantically gendered nouns but only  $3\%$  of the examples include terms of address. We also note that if we compare  $\neg$  NAME  $\neg$  SEM  $\neg$  ADDR to  $\neg$  SEM  $\neg$  ADDR and  $\neg$  NAME  $\neg$  ADDR, accuracy drops when removing gender cues. Though the differences are not statistically significant, we did not expect the accuracy drop.
Finally, we find annotators' certainty values follow the same trend as the accuracy: annotators have a reasonable sense of when they are unsure. We also note that accuracy score are essentially the same for ZERO and  $\neg$  PRO, which suggests that once explicit binary gender is gone from pronouns, the impact of any other form of linguistic gender in annotator's decisions is also removed.
5 Bias in Model Specifications
In addition to biases that can arise from the data that a system is trained on, as studied in the previ
10 Our study was approved by the Microsoft Research Ethics Board. Workers were paid $1 to annotate ten contexts (the average annotation time was seven minutes).
ous section, bias can also come from how models are structured. For instance, a system may fail to recognize anything other than a dictionary of fixed pronouns as possible referents to entities. Here, we analyze prior work in models for coreference resolution in three ways. First, we do a literature study to quantify how NLP papers discuss gender. Second, similar to Zhao et al. (2018a) and Rudinger et al. (2018), we evaluate five freely available systems on the ablated data from §4. Third, we evaluate these systems on the dataset we created: Gender Inclusive Coreference (GICOREF).
5.1 Cis-normativity in published NLP papers
In our first study, we adapt the approach Keynes (2018) took for analyzing the degree to which computer vision papers encoded trans-exclusive models of gender. In particular, we began with a random sample of  $\sim 150$  papers from the ACL anthology that mention the word "gender" and coded them according to the following questions:
- Does the paper discuss coreference resolution?
- Does the paper study English?
- L.G: Does the paper deal with linguistic gender (grammatical gender or gendered pronouns)?
- S.G: Does the paper deal with social gender?
- L.G≠S.G: (If yes to L.G and S.G:) Does the paper distinguish linguistic from social gender?
- S.G Binary: (If yes to S.G:) Does the paper explicitly or implicitly assume that social gender is binary?
- S.G immutable: (If yes to S.G:) Does the paper explicitly or implicitly assume social gender is immutable?
- They/Neo: (If yes to S.G and to English:) Does the paper explicitly consider uses of definite singular "they" or neopronouns?
The results of this coding are in Table 1 (the full annotation is in Appendix B). We see out of the 22 coreference papers analyzed, the vast majority conform to a "folk" theory of language:
Only  $5.5\%$  distinguish social from linguistic gender (despite it being relevant);
Only  $5.6\%$  explicitly model gender as inclusive of non-binary identities;
No papers treat gender as anything other than completely immutable;<sup>11</sup>
All Papers	Coref Papers
L.G?	52.6%	(of 150)	95.4%	(of 22)
S.G?	58.0%	(of 150)	86.3%	(of 22)
L.G≠S.G?	11.1%	(of 27)	5.5%	(of 18)
S.G Binary?	92.8%	(of 84)	94.4%	(of 18)
S.G Imperable?	94.5%	(of 74)	100.0%	(of 14)
They/Neo?	3.5%	(of 56)	7.1%	(of 14)
Table 1: Analysis of a corpus of 150 NLP papers that mention "gender" along the lines of what assumptions around gender are implicitly or explicitly made.
Only  $7.1\%$  (one paper!) considers neopronouns and/or specific singular THEY.
The situation for papers not specifically about coreference is similar (the majority of these papers are either purely linguistic papers about grammatical gender in languages other than English, or papers that do "gender recognition" of authors based on their writing; May (2019) discusses the (re)production of gender in automated gender recognition in NLP in much more detail). Overall, the situation more broadly is equally troubling, and generally also fails to escape from the folk theory of gender. In particular, none of the differences are significant at a  $p = 0.05$  level except for the first two questions, due to the small sample size (according to an  $n - 1$  chi-squared test). The result is that although we do not know exactly what decisions are baked in to all systems, the vast majority in our study (including two papers by one of the authors (Daumé and Marcu, 2005; Orita et al., 2015)) come with strong gender binary assumptions, and exist within a broader sphere of literature which erases non-binary and binary trans identities.
5.2 System performance on MAP
Next, we analyze the effect that our different ablation mechanisms have on existing coreference resolutions systems. In particular, we run five coreference resolution systems on our ablated data: the AI2 system (AI2; Gardner et al., 2017), hugging face (HF; Wolf, 2017), which is a neural system based on spacy, and the Stanford deterministic (SfdD; Raghunathan et al., 2010), statistical (SfdS; Clark and Manning, 2015) and neural (SfdN; Clark and Manning, 2016) systems. Figure 3 shows the results. We can see that the system accuracies mostly follow the same pattern as human accuracy scores, though all are significantly lower than human results. Accuracy scores for systems drop
11 The most common ways in which papers implicitly assume that social gender is immutable is either 1) by relying on external knowledge bases that map names to "gender"; or 2) by scraping a history of a user's social media posts or emails and assuming that their "gender" today matches the gender of
that historical record.
Figure 3: Coreference resolution systems results for the ablation study on MAP dataset. The y-axis is the degree of accuracy with  $95\%$  significance intervals.
dramatically when we ablate out referential gender in pronouns. This reveals that those coreference resolution systems reply heavily on gender-based inferences. In terms of each systems, HF and SfdN systems have similar results and outperform other systems in most cases. SfdD accuracy drops significantly once names are ablated.
These results echo and extend previous observations made by Zhao et al. (2018a), who focus on detecting stereotypes within occupations. They detect gender bias by checking if the system accuracies are the same for cases that can be resolved by syntactic cues and cases that cannot, with original data and reversed-gender data. Similarly, Rudinger et al. (2018) focus on detecting stereotypes within occupations as well. They construct dataset without any gender cues other than stereotypes, and check how systems perform with different pronouns – THEY, SHE, HE. Ideally, they should all perform the same because there is not any gender cues in the sentence. However, they find that systems do not work on “they” and perform better on “he” than “she”. Our analysis breaks this stereotyping down further to detect which aspects of gender signals are most leveraged by current systems.
5.3 System behavior on gender-inclusive data
Finally, in order to evaluate current coreference resolution models in gender inclusive contexts we introduce a new dataset, GICOREF. Here we focused on naturally occurring data, but sampled specifically to surface more gender-related phenomena than may be found in, say, the Wall Street Journal.
Our new GICOREF dataset consists of 95 doc
Precision	Recall	F1
AI2	40.4%	29.2%	33.9%
HF	68.8%	22.3%	33.6%
SfdD	50.8%	23.9%	32.5%
SfdS	59.8%	24.1%	34.3%
SfdN	59.4%	24.0%	34.2%
Table 2: LEA scores on GICOREF (incorrect reference excluded) with various coreference resolution systems. Rows are different systems while columns are precision, recall, and F1 scores. When evaluate, we only count exact matches of pronouns and name entities.
uments from three types of sources: articles from English Wikipedia about people with non-binary gender identities, articles from LGBTQ periodicals, and fan-fiction stories from Archive Of Our Own (with the respective author's permission) $^{12}$ . These documents were each annotated by both of the authors and adjudicated. $^{13}$  This data includes many examples of people who use pronouns other than SHE or HE (the dataset contains  $27\%$  HE,  $20\%$  SHE,  $35\%$  THEY, and  $18\%$  neopronouns, people who are genderfluid and whose names or pronouns change through the article, people who are misgendered, and people in relationships that are not heteronormative. In addition, incorrect references (misgendering and deadnaming $^{14}$ ) are explicitly annotated. $^{15}$  Two example annotated documents, one from Wikipedia, and one from Archive of Our Own, are provided in Appendix C and Appendix D.
We run the same systems as before on this dataset. Table 2 reports results according the standard coreference resolution evaluation metric LEA (Moosavi and Strube, 2016). Since no systems are implemented to explicitly mark incorrect references, and no current evaluation metrics address this case, we perform the same evaluation twice. One with incorrect references included as regular references in the ground truth; and other with incorrect references excluded. Due to the limited number of incorrect references in the dataset, the
12See https://archiveofourown.org; thanks to Os Keyes for this suggestion.
13We evaluate inter-annotator agreement by treating one annotation as gold standard and the other as system output and computing the LEA metric; the resulting F1-score is  $92\%$ . During the adjudication process we found that most of the disagreement are due to one of the authors missing/overlooking mentions, and rarely due to true "disagreement."
14 According to Clements (2017) deadnaming occurs when someone, intentionally or not, refers to a person who's transgender by the name they used before they transitioned.
<sup>15</sup>Thanks to an anonymous reader of a draft version of this paper for this suggestion.
difference of the results are not significant. Here we only report the latter.
The first observation is that there is still plenty room for coreference systems to improve; the best performing system achieves as F1 score of  $34\%$ , but the Stanford neural system's F1 score on CoNLL2012 test set reaches  $60\%$  (Moosavi, 2020). Additionally, we can see system precision dominates recall. This is likely partially due to poor recall of pronouns other than HE and SHE. To analyze this, we compute the recall of each system for finding referential pronouns at all, regardless of whether they are correctly linked to their antecedents. We find that all systems achieve a recall of at least  $95\%$  for binary pronouns, a recall of around  $90\%$  on average for THEY, and a recall of around a paltry  $13\%$  for neopronouns (two systems—Stanford deterministic and Stanford neural—never identify any neopronouns at all).
6 Discussion and Moving Forward
Our goal in this paper was to analyze how gender bias exist in coreference resolution annotations and models, with a particular focus on how it may fail to adequately process text involving binary and non-binary trans referents. We thus created two datasets: MAP and GICOREF. Both datasets show significant gaps in system performance, but perhaps moresco, show that taking crowdworker judgments as "gold standard" can be problematic. It may be the case that to truly build gender inclusive datasets and systems, we need to hire or consult experiential experts (Patton et al., 2019; Young et al., 2019).
Moreover, although we studied crowdworkers on Mechanical Turk (because they are often employed as annotators for NLP resources), if other populations are used for annotation, it becomes important to consider their positionality and how that may impact annotations. This echoes a related finding in annotation of hate-speech that annotator positionality matters (Olteanu et al., 2019). More broadly, we found that trans-exclusionary assumptions around gender in NLP papers is made commonly (and implicitly), a practice that we hope to see change in the future because it fundamentally limits the applicability of NLP systems.
The primary limitation of our study and analysis is that it is limited to English. This is particularly limiting because English lacks a grammatical gender system, and some extensions of our work to languages with grammatical gender are non-trivial.
We also emphasize that while we endeavored to be inclusive, our own positionality has undoubtedly led to other biases. One in particular is a largely Western bias, both in terms of what models of gender we use and also in terms of the data we annotated. We have attempted to partially compensate for this bias by intentionally including documents with non-Western non-binary expressions of gender in the GICoref dataset<sup>16</sup>, but the dataset nonetheless remains Western-dominant.
Additionally, our ability to collect naturally occurring data was limited because many sources simply do not yet permit (or have only recently permitted) the use of gender inclusive language in their articles. This led us to counterfactual text manipulation, which, while useful, is essentially impossible to do flawlessly. Moreover, our ability to evaluate coreference systems with data that includes incorrect references was limited as well, because current systems do not mark any forms of misgendering or deadnaming explicitly, and current metrics do not take this into account. Finally, because the social construct of gender is fundamentally contested, some of our results may apply only under some frameworks.
We hope this paper can serve as a roadmap for future studies. In particular, the gender taxonomy we presented, while not novel, is (to our knowledge) previously unattested in discussions around gender bias in NLP systems; we hope future work in this area can draw on these ideas. We also hope that developers of datasets or systems can use some of our analysis as inspiration for how one can attempt to measure—and then root out—different forms of bias in coreference resolution systems and NLP systems more broadly.
The authors are grateful to a number of people who have provided pointers, edits, suggestions, and annotation facilities to improve this work: Lauren Ackerman, Cassidy Henry, Os Keyes, Chandler May, Hanyu Wang, and Marion Zepf, all contributed to various aspects of this work, including suggestions for data sources for the GI Coref dataset. We also thank the CLIP lab at the University of Maryland for comments on previous drafts.
<sup>16</sup>We endeavored to represent some non-Western gender identities that do not fall into the male/female binary, including people who identify as hijra (Indian subcontinent), phuying (Thailand, sometimes referred to as kathoey), muxe (Oaxaca), two-spirit (Americas), fa'afafine (Samoa) and mahu (Hawaii).
Saleem Abuleil, Khalid Alsamara, and Martha Evens. 2002. Acquisition system for Arabic noun morphology. In Proceedings of the ACL-02 Workshop on Computational Approaches to Semitic Languages, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Lauren Ackerman. 2019. Syntactic and cognitive issues in investigating gendered coreference. *Glossa: a journal of general linguistics*, 4.
Apoory Agarwal, Jiehan Zheng, Shruti Kamath, Sriramkumar Balasubramanian, and Shirin Ann Dey. 2015. Key female characters in film have more to talk about besides men: Automating the Bechdel test. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 830-840, Denver, Colorado. Association for Computational Linguistics.
Tafseer Ahmed Khan. 2014. Automatic acquisition of Urdu nouns (along with gender and irregular plurals). In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 2846-2850, Reykjavik, Iceland. European Language Resources Association (ELRA).
Sarah Alkuhlani and Nizar Habash. 2011. A corpus for modeling morpho-syntactic agreement in Arabic: Gender, number and rationality. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 357-362, Portland, Oregon, USA. Association for Computational Linguistics.
Sarah Alkuhlani and Nizar Habash. 2012. Identifying broken plurals, irregular gender, and rationality in Arabic text. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 675-685, Avignon, France. Association for Computational Linguistics.
Tania Avgustinova and Hans Uszkoreit. 2000. An ontology of systematic relations for a shared grammar of Slavic. In COLING 2000 Volume 1: The 18th International Conference on Computational Linguistics.
Bogdan Babych, Jonathan Geiger, Mireia Ginesti Rosell, and Kurt Eberle. 2014. Deriving de/het gender classification for Dutch nouns for rule-based MT generation tasks. In Proceedings of the 3rd Workshop on Hybrid Approaches to Machine Translation (HyTra), pages 75-81, Gothenburg, Sweden. Association for Computational Linguistics.
Ibrahim Badr, Rabih Zbib, and James Glass. 2008. Segmentation for English-to-Arabic statistical machine translation. In Proceedings of ACL-08: HLT, Short
Papers, pages 153-156, Columbus, Ohio. Association for Computational Linguistics.
Ibrahim Badr, Rabih Zbib, and James Glass. 2009. Syntactic phrase reordering for English-to-Arabic statistical machine translation. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 86-93, Athens, Greece. Association for Computational Linguistics.
R. I. Bainbridge. 1985. Montagovian definite clause grammar. In Second Conference of the European Chapter of the Association for Computational Linguistics, Geneva, Switzerland. Association for Computational Linguistics.
Janet Baker, Larry Gillick, and Robert Roth. 1994. Research in large vocabulary continuous speech recognition. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
Murali Raghu Babu Balusu, Taha Merghani, and Jacob Eisenstein. 2018. Stylistic variation in social media part-of-speech tagging. In Proceedings of the Second Workshop on Stylistic Variation, pages 11-19, New Orleans. Association for Computational Linguistics.
Francesco Barbieri and Jose Camacho-Collados. 2018. How gender and skin tone modifiers affect emoji semantics in twitter. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 101-106, New Orleans, Louisiana. Association for Computational Linguistics.
Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The Problem With Bias: Allocative Versus Representational Harms in Machine Learning. In Proceedings of SIGCIS.
Naomi S. Baron. 1971. A reanalysis of english grammatical gender. Lingua, 27:113-140.
Emily M. Bender. 2019. A typology of ethical risks in language technology with an eye towards where transparent documentation can help. The Future of Artificial Intelligence: Language, Ethics, Technology.
Shane Bergsma and Dekang Lin. 2006. Bootstrapping path-based pronoun resolution. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 33-40, Sydney, Australia. Association for Computational Linguistics.
Shane Bergsma, Dekang Lin, and Randy Goebel. 2009. Glen, glenda or glendale: Unsupervised and semi-supervised learning of English noun gender. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 120–128, Boulder, Colorado. Association for Computational Linguistics.
Shane Bergsma, Matt Post, and David Yarowsky. 2012. Stylometric analysis of scientific articles. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 327-337, Montréal, Canada. Association for Computational Linguistics.
Bronwyn M. Bjorkman. 2017. Singular they and the syntactic representation of gender in english. Glossa: A Journal of General Linguistics, 2(1):80.
Su Lin Blodgett, Solon Barocas, Hal Daumé, III, and Hanna Wallach. 2020. Language (technology) is power: The need to be explicit about NLP harms. In Proceedings of the Conference of the Association for Computational Linguistics (ACL).
Ondrej Bojar, Rudolf Rosa, and Aleš Tamchyna. 2013. Chimera - three heads for English-to-Czech translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 92-98, Sofia, Bulgaria. Association for Computational Linguistics.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. In Proceedings of NeurIPS.
Constantinos Boulis and Mari Ostendorf. 2005. A quantitative analysis of lexical differences between genders in telephone conversations. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 435-442, Ann Arbor, Michigan. Association for Computational Linguistics.
Evan Bradley, Julia Salkind, Ally Moore, and Sofi Teitsort. 2019. Singular 'they' and novel pronouns: gender-neutral, nonbinary, or both? Proceedings of the Linguistic Society of America, 4(1):36-1-7.
Mary Bucholtz. 1999. Gender. Journal of Linguistic Anthropology. Special issue: Lexicon for the New Millennium, ed. Alessandro Duranti.
Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification.
John D. Burger, John Henderson, George Kim, and Guido Zarrella. 2011. Discriminating gender on twitter. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1301-1309, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Felix Burkhardt, Martin Eckert, Wiebke Johannsen, and Joachim Stegmann. 2010. A database of age and gender annotated telephone speech. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), Valletta, Malta. European Language Resources Association (ELRA).