SlowGuess/ACL_2020_OCR huggingface dataset:
Saif M. Mohammad  
National Research Council Canada  
Ottawa, Canada  
saif.mohammad@nrc-cnrc.gc.ca
Disparities in authorship and citations across gender can have substantial adverse consequences not just on the disadvantaged genders, but also on the field of study as a whole. Measuring gender gaps is a crucial step towards addressing them. In this work, we examine female first author percentages and the citations to their papers in Natural Language Processing (1965 to 2019). We determine aggregate-level statistics using an existing manually curated author-gender list as well as first names strongly associated with a gender. We find that only about  $29\%$  of first authors are female and only about  $25\%$  of last authors are female. Notably, this percentage has not improved since the mid 2000s. We also show that, on average, female first authors are cited less than male first authors, even when controlling for experience and area of research. Finally, we discuss the ethical considerations involved in automatic demographic analysis.
Gender gaps are quantitative measures of the disparities in social, political, intellectual, cultural, or economic success due to one's gender. Gender gaps can also refer to disparities in access to resources (such as healthcare and education), which in turn lead to disparities in success. We need to pay attention to gender gaps not only because they are inherently unfair but also because better gender balance leads to higher productivity, better health and well-being, greater economic benefits, better decision making, as well as political and economic stability (Skjelsboek and Smith, 2001; Woetzel et al., 2015; Hakura et al., 2016; Mehta et al., 2017; Gallego and Gutierrez, 2018).
Historically, gender has often been considered binary (male and female), immutable (cannot change), and physiological (mapped to biological
sex). However, those views have been discredited (Hyde et al., 2019; Richards et al., 2017; Darwin, 2017; Lindsey, 2015; Kessler and McKenna, 1978). Gender is complex, and does not necessarily fall into binary male or female categories (e.g. nonbinary people), and also does not necessarily correspond to one's assigned gender at birth.
Society has often viewed different gender groups differently, imposing unequal social and power structures (Lindsey, 2015). The World Economic Forum's 2018 Global Gender Gap Report (which examined data from more than 144 countries) highlighted the gender gap between men and women in Artificial Intelligence as particularly alarming (WEF, 2018). It indicated that only  $22\%$  of the professionals in AI are women and that this low representation in a transformative field requires urgent action—otherwise, the AI gap has the potential to widen other gender gaps. Other studies have identified substantial gender gaps in science (Håkanson, 2005; Larivière et al., 2013; King et al., 2017; Andersen and Nielsen, 2018).
Perez (2019) discusses, through numerous examples, how there is a considerable lack of disaggregated data for women and how that is directly leading to negative outcomes in all spheres of their lives, including health, income, safety, and the degree to which they succeed in their endeavors. This holds true even more for transgender people. Our work obtains disaggregated data for female Natural Language Processing (NLP) researchers and determines the degree of gender gap between female and male NLP researchers. (NLP is an interdisciplinary field that includes scholarly work on language and computation with influences from Artificial Intelligence, Computer Science, Linguistics, Psychology, and Social Sciences to name a few.) We hope future work will explore other gender gaps (e.g., between trans and cis people). Measuring gender gaps is a crucial step towards addressing them.
We examine tens of thousands of articles in the ACL Anthology (AA) (a digital repository of public domain NLP articles) for disparities in female authorship. We also conduct experiments to determine whether female first authors are cited more or less than male first authors, using citation counts extracted from Google Scholar (GS).
We extracted and aligned information from the ACL Anthology and Google Scholar to create a dataset of tens of thousands of NLP papers and their citations as part of a broader project on analyzing NLP Literature. We refer to this dataset as the NLP Scholar Dataset. We determined aggregate-level statistics for female and male researchers in the NLP Scholar dataset using an existing manually curated author-gender list as well as first names that are strongly associated with a gender.
Note that attempts to automatically infer gender of individuals can lead to harm (Hamidi et al., 2018). Our work does not aim to infer gender of individual authors. We use name-gender association information to determine aggregate-level statistics for male and female researchers. Further, one may not know most researchers they cite, other than from reading their work. Thus perceived gender (from the name) can lead to unconscious effects, e.g., Dion et al. (2018) show that all male and mixed author teams cite fewer papers by female authors than all female teams. Further, seeing only a small number of female authors cited can demoralize young researchers entering the field.
We do not explore the reasons behind gender gaps. However, we will note that the reasons are often complex, intersectional, and difficult to disentangle. We hope that this work will increase awareness of gender gaps and inspire concrete steps to improve inclusiveness and fairness in research.
It should also be noted that even though this paper focuses on female-male disparities, there are many aspects to demographic diversity including: representation from transgender people; representation from various nationalities and race; representation by people who speak a diverse set of languages;
diversity by income, age, physical abilities, etc. All of these factors impact the breadth of technologies we create, how useful they are, and whether they reach those that need it most.
Resources for the NLP Scholar project can be accessed through the project homepage.<sup>3</sup>
Pilcher (2017) shows that names function not only to identify individuals, but also to manage gender throughout one's life. There is a strong cultural norm in various parts of the world to assign a first name to newborns as per their category of sex (Pilcher, 2017; Barry III and Harper, 2014; Lieberson et al., 2000; Alford, 1987). Pilcher (2017) argues that, throughout their life, the first name plays a role in repeatedly categorizing a person as being male or female. People may change their name and appearance to manage their gender (Connell, 2010; Pilcher, 2017). People may choose a name that is not associated with male or female categorizations (Connell, 2010).
The strong normative tendency to use names to signal gender has led to a large body of work on automatically determining gender by one's first name, not just for scientometric analysis discussed below, but also for language studies, social sciences, public health, and commerce. However, this can also lead to misgendering, which can cause significant pain and harm (Hamidi et al., 2018). (Misgendering is when a person—or in this case, a machine—associates someone with a gender with which they do not identify.) Further, work that does not explicitly consider gender to be inclusive of trans people can reinforce stereotypes such as the dichotomy of gender.
We expect gender disparities to be different depending on the groups being compared: female-male, trans-cis, and so on. Our work does not aim to infer gender of individual authors. We obtain disaggregated statistics for women, specifically, to study the disparities between female and male NLP researchers. We discuss ethical considerations further in Section 6. See also Mihaljevic et al. (2019) for a discussion on ethical considerations in using author name to estimate gender statistics in the Gender Gap in Science Project—a large ongoing project tracking gender gaps in Mathematical and Natural Sciences. $^4$
<sup>1</sup>https://www.aclweb.org/anthology/
$^{2}$ Mohammad (2019) presents an overview of the many research directions pursued, using this data. Notably, Mohammad (2020a) explores questions such as: how well cited are papers of different types (journal articles, conference papers, demo papers, etc.)? how well cited are papers published in different time spans? how well cited are papers from different areas of research within NLP? etc. Mohammad (2020c) presents an interactive visualization tool that allows users to search for relevant related work in the ACL Anthology.
4https://gender-gap-in-science.org
Most studies on gender and authorship have found substantial gender disparities in favor of male researchers. They include work on  $\sim 1700$  articles from journals of library and information science (Håkanson, 2005), on  $\sim 12$  million articles from the Web of Science (for Sociology, Political Science, Economics, Cardiology and Chemistry) (Ghiasi et al., 2016; Andersen and Nielsen, 2018), on  $\sim 2$  million mathematics articles (Mihaljevic-Brandt et al., 2016), on  $\sim 1.6$  million articles from PubMed life science and biomedical research (Mishra et al., 2018), on  $\sim 1.5$  million articles from fifty disciplines published in JSTOR (King et al., 2017), and on  $\sim 0.5$  million publications from US research universities (Duch et al., 2012). There also exists some work that shows that in fields such as linguistics (LSA, 2017) and psychology (Willyard, 2011), female and male participation is either close to parity or tilted in favor of women.
In NLP research, Schluter (2018) showed that there are barriers in the paths of women researchers, delaying their attainment of mentorship status (as estimated through last author position in papers). Anderson et al. (2012) examine papers from 1980 to 2008 to track the ebb and flow of topics within NLP, and the influence of researchers from outside NLP on NLP. Vogel and Jurafsky (2012) examined about 13,000 papers from 1980 to 2008 to determine basic authorship statistics by women and men. Gender statistics were determined by a combination of automatic and manual means. The automatic method relied on lists of baby names from various languages. They found that female authorship has been steadily increasing from 1980 to 2008. Our work examines a much larger set of NLP papers (1965-2019), re-examines some of the questions raised in Vogel and Jurafsky (2012), and explores several new questions, especially on first author gender and disparities in citation.
We extracted and aligned information from the ACL Anthology (AA) and Google Scholar (GS) to create a dataset of tens of thousands of NLP papers and their citations. We aligned the information across AA and GS using the paper title, year of publication, and first author last name. Details about the dataset, as well as an analysis of the volume of research in NLP over the years, are available in Mohammad (2020b). We summarize key information below.
The ACL Anthology is available through its website and a github repository. We extracted paper title, names of authors, year of publication, and venue of publication from the repository.
As of June 2019, AA had  $\sim 50\mathrm{K}$  entries; however, this includes some entries that are not truly research publications (for example, forewords, prefixes, programs, schedules, indexes, invited talks, appendices, session information, newsletters, lists of proceedings, etc.). After discarding them, we are left with 44,894 papers.7
Inferring Aggregate Gender Statistics: The ACL Anthology does not record author demographic information. To infer aggregate statistics for male and female authors, we create two bins of authors: A-Mname (authors that have self-reported as males or with names commonly associated with males) and A-Fnames (authors that have self-reported as females or with names commonly associated with females). We made use of three resources to populate A-Mname and A-Fname:
1. A manually curated list of 11,932 AA authors and their genders provided by Vogel and Jurafsky (2012) (VJ-AA list) (3,359 female and 8,573 male).<sup>9</sup>
2. A list of 55,924 first names that are strongly associated with females and 30,982 first names that are strongly associated with males, that we generated from the US Social Security Administration's (USSA) published database of names and genders of newborns.[10]
3. A list of 26,847 first names that are strongly associated with females and 23,614 first names that are strongly associated with males, that we generated from a list of 9,300,182
<sup>5</sup>https://www.aclweb.org/anthology/ https://github.com/acl-org/acl-anthology
Multiple authors can have the same name and the same authors may use multiple variants of their names in papers. The AA volunteer team handles such ambiguities using both semi-automatic and manual approaches (fixing some instances on a case-by-case basis). Additionally, AA keeps a file that includes canonical forms of author names.
<sup>7</sup>We used simple keyword searches for terms such as foreword, invited talk, program, appendix and session in the title to pull out entries that were likely to not be research publications. These were then manually examined to verify that they did not contain any false positives.
Note that a person may have a name commonly associated with one gender but belong to a different gender.
<sup>9</sup>https://nlp.stanford.edu/projects/gender.shtml
<sup>10</sup>https://www.ssa.gov/oact/babynames/limits.html
firstname–gender association list	P	R	F
from USSA data	98.4	69.8	81.7
from PUBMED data	98.3	81.4	89.1
Table 1: Precision (P), Recall (R), and F-score (F) of how well the first name and gender association matches information in the VJ-AA list.
PUBMED authors and their genders (Torvik and Smalheiser, 2009; Smith et al., 2013).<sup>11</sup>
We acknowledge that despite a large expatriate population, the US census information is not representative of the names from around the world. Further, Chinese origin names tend not to be as strongly associated with gender as names from other parts of the world. However, it should be noted that Vogel and Jurafsky (2012) made special effort to include information from a large number of Asian AA authors in their list. The PUBMED list is also noted for having a substantial coverage of Asian names (Torvik and Smalheiser, 2009).
We determined first name-gender association, by calculating the percentages of first names corresponding to male and female genders as per each of the PUBMED and USSA fullname-gender lists. We consider a first name to be strongly associated with a gender if the percentage is  $\geq 95\%$ .<sup>12</sup> Table 1 shows how well the first name and gender association matches with the VJ-AA list.
Given the high precision (over  $98\%$ ) of the USSA and PUBMED lists of gender-associated first names, we use them (in addition to the VJ-AA list) to populate the M-names and F-names bins. Eventually, the A-Mname and A-Fname bins together had 28,682 ( $76\%$ ) of the 37,733 AA authors. Similarly, we created bins for Papers whose First Author is from A-Mname (P-FA-Mname), Papers whose First Author is from A-Fname (P-FA-Fname), Papers whose Last Author is from A-Mname (P-LA-Mname), and Papers whose Last Author is from A-Fname (P-LA-Fname) to estimate aggregate-level statistics for papers with male and female first and last authors. P-FA-Mname and P-FA-Fname together have 37,297 ( $83\%$ ) AA papers (we will refer to this subset as  $\mathrm{AA^{*}}$ ), P-LA-Mname and P-LA-Fname together have 39,368 ( $88\%$ ) AA papers (we will refer to this subset as  $\mathrm{AA^{**}}$ ).
NLP Academic Age as a Proxy for Experience in NLP: First author percentage may vary due to experience, area of research within NLP, venue of publication, etc. To gauge experience, we use the number of years one has been publishing in AA; we will refer to this as the NLP Academic Age. So if this is the first year one has published in AA, then their NLP academic age is 1. If one published their first AA paper in 2001 and their latest AA paper in 2018, then their academic age is 18. Note that NLP academic age is not always an accurate reflection of one's research experience. Also, one can publish NLP papers outside of AA.
Google Scholar allows researchers to create and edit public author profiles called Google Scholar Profiles. Authors can include their papers (along with their citation information) on this page.
We extracted citation information from Google Scholar profiles of authors who published at least three papers in the ACL Anthology. This yielded citation information for 1.1 million papers in total. We will refer to this dataset as the NLP Subset of the Google Scholar Dataset, or GScholar-NLP for short. Note that GScholar-NLP includes citation counts not just for NLP papers, but also for non-NLP papers published by authors who have at least three papers in AA.
GScholar-NLP includes 32,985 of the 44,894 papers in AA (about  $75\%$ ). We will refer to this subset of the ACL Anthology papers as  $\mathrm{AA^{\prime}}$ . The citation analyses presented in this paper are on  $\mathrm{AA^{\prime}}$ .
4 Gender Gap in Authorship
We use the datasets described in §3.1 (especially AA* and AA**) to answer a series of questions on female authorship. Since we do not have full self-reported information for all authors, these should be treated as estimates.
First author is a privileged position in the author list that is usually reserved for the researcher that has done the most work and writing. $^{14}$  In NLP, first authors are also often students. Thus we are especially interested in gender gaps that effect them. The last author position is often reserved for the most senior or mentoring researcher. We explore last author disparities only briefly (in Q1).
<sup>11</sup>https://experts.illinois.edu/en/datasets/genni-ethnea-forthe-author-ity-2009-dataset-2
12A choice of other percentages such as  $90\%$  or  $99\%$  would also have been reasonable.
This is allowed by GS's robots exclusion standard.
14 A small number of papers have more than one first author. This work did not track that.
Q1. What percentage of the authors in AA are female? What percentage of the AA papers have female first authors (FFA)? What percentage of the AA papers have female last authors (FLA)? How have these percentages changed since 1965?
A. Overall, we estimate that about  $29.7\%$  of the 28,682 authors are female; about  $29.2\%$  of the first authors in 37,297 AA* papers are female; and about  $25.5\%$  of the last authors in 39,368 AA** papers are female. Figure 1 shows how these percentages have changed over the years.
Discussion: Across the years, the percentage of female authors overall is close to the percentage of papers with female first authors. (These percentages are around  $28\%$  and  $29\%$ , respectively, in 2018.) However, the percentage of female last authors is markedly lower (hovering at about  $25\%$  in 2018). These numbers indicate that, as a community, we are far from obtaining male-female parity. A further striking (and concerning) observation is that the female author percentages have not improved since the year 2006.
To put these numbers in context, the percentage of female scientists worldwide (considering all areas of research) has been estimated to be around  $30\%$ . The reported percentages for many computer science sub-fields are much lower.[15] The percentages are much higher for certain other fields such as psychology (Willyard, 2011) and linguistics (LSA, 2017).
Q2. How does FFA vary by paper type and venue?
A. Figure 2 shows FFA percentages by paper type and venue.
Discussion: Observe that FFA percentages are lowest for CoNLL, EMNLP, IJCNLP, and system demonstration papers (21% to 24%). FFA percentages for journals, other top-tier conferences, SemEval, shared task papers, and tutorials are the next lowest (24% to 28%). The percentages are markedly higher for LREC, *Sem, and RANLP (33% to 36%), as well as for workshops (31.7%).
Q3. How does female first author percentage change with NLP academic age?
A. In order to determine these numbers, every paper in AA* was placed in a bin corresponding to NLP academic age: if the paper's first author had an academic age of 1 in the year when the
paper was published, then the paper is placed in bin 1; if the paper's first author had an academic age of 2 in the year when the paper was published, then the paper is placed in bin 2; and so on. The bins for later years contained fewer papers. This is expected as senior authors in NLP often work with students, and students are encouraged to be first authors. Thus, we combine some of the bins in later years: one bin for academic ages between 10 and 14; one for 15 to 19; one for 20 to 34; and one for 35 to 50. Once the papers are assigned to the bins, we calculate the percentage of papers in each bin that have a female first author. Figure 3 shows the results.
Discussion: Observe that, with the exception of the 35 to 50 academic age bin, FFA% is highest (30%) at age 1 (first year of publication). There is a period of decline in FFA% until year 6 (27.4%)—this difference is statistically significant (t-test, p < 0.01). This might be a potential indicator that graduate school has a progressively greater negative impact on the productivity of women than of men. (Academic age 1 to 6 often correspond to the period when the first author is in graduate school or in a temporary post-doctoral position.) After year 6, we see a recovery back to 29.4% by year 8, followed by a period of decline once again.
Q4. How does female first author percentage vary by area of research (within NLP)? Which areas have higher-than-average FFA%? Which areas have lower-than-average FFA%? How does FFA% correlate with popularity of an area—that is, does FFA% tend to be higher- or lower-than-average in areas where lots of authors are publishing?
A. We use word bigrams in the titles of papers to sample papers from various areas. $^{16}$  The title has a privileged position in a paper. Primarily, it conveys what the paper is about. For example, a paper with machine translation in the title is likely about machine translation. Figure 4 shows the list of top 66 bigrams that occur in the titles of more than  $100\mathrm{AA}^*$  papers (in decreasing order of the bigram frequency). For each bigram, the figure also shows the percentage of papers with a female first author. In order to determine whether there is a correlation between the number of papers corresponding to a bigram and FFA%, we calculated the Spearman's rank correlation between the rank of a bigram by
16 Other approaches such as clustering are also reasonable; however, results with those might not be easily reproducible.
<sup>15</sup>https://unesdoc.unesco.org/ark:/48223/pf0000235155
% authors by gender (ALL authors)
% of authors by gender each year (#authors shown in brackets)
% papers by gender of FIRST author
% of papers by gender of FIRST author each year (#papers shown in brackets)
% papers by gender of LAST author
% of papers by gender of LAST author each year (#papers shown in brackets)
Figure 1: Female authorship percentages in AA over the years: overall, as first author, and as last author.
Figure 2: FFA percentage by venue and paper type. The number of FFA papers is shown in parenthesis.
Figure 3: FFA percentage by academic age. The number of FFA papers is shown in parenthesis.
number of papers and the rank of a bigram by FFA%. This correlation was found to be only 0.16. This correlation is not statistically significant at p  $< 0.01$  (two-sided p-value  $= 0.2$ ). Experiments with lower thresholds (174 bigrams occurring in 50 or more papers and 1408 bigrams occurring in 10 or more papers) also resulted in very low and non-significant correlation numbers (0.11 and 0.03, respectively).