SlowGuess/ACL_2020_OCR huggingface dataset:
Discussion: Observe that FFA% varies substantially depending on the bigram. It is particularly low for title bigrams such as dependency parsing, language models, finite state, context free, and neural models; and markedly higher than average for domain specific, semantic relations, dialogue system, spoken dialogue, document summarization, and language resources. However, the rank correlation experiments show that there is no correlation between the popularity of an area (number of papers that have a bigram in the title) and the percentage of female first authors.
To obtain further insights, we also repeat some of the experiments described above for unigrams in paper titles. We found that FFA rates are relatively high in non-English European language research such as papers on Russian, Portuguese, French, and Italian. FFA rates are also relatively high for work on prosody, readability, discourse, dialogue, paraphrasing, and individual parts of speech such as adjectives and verbs. FFA rates are particularly low for papers on theoretical aspects of statistical modelling, and for terms such as WMT, parsing, markov, recurrent, and discriminative.
Figure 4: Top 66 bigrams in AA* titles and FFA% (30%: light grey; <30%: blue; >30%: green).
5 Gender Gap in Citations
Research articles can have impact in a number of ways—pushing the state of the art, answering crucial questions, finding practical solutions that directly help people, etc. However, individual measures of research impact are limited in scope—they measure only some kinds of contributions. The most commonly used metrics of research impact are derived from citations including: number of citations, average citations, h-index, and impact factor (Bornmann and Daniel, 2009). Despite their limitations, citation metrics have substantial impact on a researcher's scientific career; often through a combination of funding, the ability to attract talented students and collaborators, job prospects, and other opportunities in the wider research community. Thus, disparities in citations (citation gaps) across demographic attributes such as gender, race, and location have direct real-world adverse implications. This often also results in the demoralization of researchers and marginalization of their work—thus negatively impacting the whole field.
Therefore, we examine gender disparities in citations in NLP. We use a subset of the 32,985  $\mathrm{AA^{\prime}}$  papers (§3.2) that were published from 1965 to 2016 for the analysis (to allow for at least 2.5 years for the papers to collect citations). There are 26,949 such papers.
Q5. How well cited are women and men?
A. For all three classes (females, males, and gender unknown), Figure 5 shows: a bar graph of number of papers, a bar graph of total citations received, and box and whisker plots for citations received by individuals. The whiskers are at a distance of 1.5 times the inter-quartile length. Number of citations pertaining to key points such as 25th percentile, median, and 75th percentile are indicated on the left of the corresponding horizontal bars.
Discussion: On average, female first author papers have received markedly fewer citations than male first author papers (37.6 compared to 50.4). The difference in median is smaller (11 compared to 13). The difference in the distributions of males and females is statistically significant (Kolmogorov-Smirnov test,  $p < 0.01$ ).<sup>17</sup> The large difference in averages and smaller difference in medians suggests that there are markedly more
Figure 5: #papers, total citations, box plot of citations per paper: for female, male, gender-unknown first authors. The orange dashed lines mark averages.
very heavily cited male first-author papers than female first-author papers.
The differences in citations, or citation gap, across genders may itself vary: (1) by period of time; (2) due to confounding factors such as academic age and areas of research. We explore these next.
Q6. How has the citation gap across genders changed over the years?
A. Figure 6 (left side) shows the citation statistics across four time periods.
Discussion: Observe that female first authors have always been a minority in the history of ACL; however, on average, their papers from the early years (1965 to 1989) received a markedly higher number of citations than those of male first authors from the same period. We can see from the graph that this changed in the 1990s when male first-author papers obtained markedly more citations on average. The citation gap reduced considerably in the 2000s, and the 2010-2016 period saw a further reduction. It remains to be seen whether the citation gap for these 2010-2016 papers widens in the coming years.
It is also interesting to note that the gender-unknown category has almost bridged the gap with the male category in terms of average citations. Further, the proportion of the gender-unknown
<sup>17</sup>Kolmogorov-Smirnov (KS) test is a non-parametric test that can be applied to compare any two distributions without making assumptions about the nature of the distributions.
Figure 6: Citation gap across genders for papers: published in different time spans (left); by academic age (right).
authors has steadily increased over the years—arguably, an indication of better representation of authors from around the world in recent years.[18]
Q7. How have citations varied by gender and academic age? Is the citation gap a side effect of a greater proportion of new-to-NLP female first authors than new-to-NLP male first authors?
A. Figure 6 (right side) shows citation statistics broken down by gender and academic age.
Discussion: The graphs show that female first authors consistently receive fewer citations than male first authors across the spans of their academic age. (The gap is highest at academic age 4 and lowest at academic age 7.) Thus, the citation gap is likely due to factors beyond differences in average academic age between men and women.
Q8. How prevalent is the citation gap across areas of research within NLP? Is the gap simply because more women work in areas that receive low numbers of citations (regardless of gender)?
A. On average, male first authors are cited more than female first authors in 54 of the 66 areas (82% of the areas) discussed earlier in Q4 and Figure 4. Female first authors are cited more in the sets of papers whose titles have: word sense, sentiment analysis, information extraction, neural networks, neural network, semeval 2016, language model, document summarization, multi document, spoken dialogue, dialogue systems, and speech tagging.
If women chose to work in areas that happen to attract less citations by virtue of the area, then we would not expect to see citation gaps in so many areas. Recall also that we already showed that FFA% is not correlated with rank of popularity of an area (Q4). Thus it is unlikely that the choice of area of research is behind the gender gap.
<sup>18</sup>Our method is expected to have a lower coverage of names from outside North America and Europe because USSA and PUBMED databases historically have had fewer names from outside North America and Europe.
Q9. What are the limitations and ethical considerations involved with this work?
A. Data is often a representation of people (Zook et al., 2017). This is certainly the case here and we acknowledge that the use of such data has the potential to harm individuals. Further, while the methods used are not new, their use merits reflection.
Analysis focused on women and men leaves out non-binary people.19 Additionally, not disaggregating cis and trans people means that the statistics are largely reflective of the more populous cis class. We hope future work will obtain disaggregated information for various genders. However, careful attention must be paid as some gender classes might include too few NLP researchers to ensure anonymity even with aggregate-level analysis.
The use of female- and male-gender associated names to infer population level statistics for women and men excludes people that do not have such names and people from some cultures where names are not as strongly associated with gender.
Names are not immutable (they can be changed to indicate or not indicate gender) and people can choose to keep their birth name or change it (providing autonomy). However, changing names can be quite difficult. Also, names do not capture gender fluidity or contextual gender.
A more inclusive way of obtaining gender information is through self-reported surveys. However, challenges persist in terms of how to design effective and inclusive questionnaires (Bauer et al., 2017; Group, 2014). Further, even with self-report textboxes that give the respondent the primacy and autonomy to express gender, downstream research often ignores such data or combines information in ways beyond the control of the respondent.[20] Also, as is the case here, it is not easy to obtain self-reported historical information.
Social category detection can potentially lead to harm, for example, depriving people of opportunities simply because of their race or gender. However, one can also see the benefits of NLP techniques and social category detection in public health (e.g., developing targeted initiatives to improve health outcomes of vulnerable populations),
as well as in psychology and social science (e.g., to better understand the unique challenges of belonging to a social category).
A larger list of ethical considerations associated with the NLP Scholar project is available through the project webpage.[21] Mihaljevic et al. (2019) also discusses the ethical considerations in using author names to infer gender statistics in the Gender Gap in Science Project.[22]
We analyzed the ACL Anthology to show that only  $\sim 30\%$  have female authors,  $\sim 29\%$  have female first authors, and  $\sim 25\%$  have female last authors. Strikingly, even though some gains were made in the early years of NLP, overall FFA% has not improved since the mid 2000s. Even though there are some areas where FFA% is close to parity with male first authorship, most areas have a substantial gap in the numbers for male and female authorship. We found no correlation between popularity of research area and FFA%. We also showed how FFA% varied by paper type, venue, academic age, and area of research. We used citation counts extracted from Google Scholar to show that, on average, male first authors are cited markedly more than female first authors, even when controlling for experience and area of work. Thus, in NLP, gender gaps exist both in authorship and citations.
This paper did not explore the reasons behind the gender gaps. However, the inequities that impact the number of women pursuing scientific research (Roos, 2008; Foschi, 2004; Buchmann, 2009) and biases that impact citation patterns unfairly (Brouns, 2007; Feller, 2004; Gupta et al., 2005) are well-documented. These factors play a substantial role in creating the gender gap, as opposed to differences in innate ability or differences in quality of work produced by these two genders. If anything, past research has shown that self-selection in the face of inequities and adversity leads to more competitive, capable, and confident cohorts (Nekby et al., 2008; Hardies et al., 2013).
Many thanks to Rebecca Knowles, Ellen Riloff, Tara Small, Isar Nejadgholi, Dan Jurafsky, Rada Mihalcea, Isabelle Augenstein, Eric Joanis, Michael Strube, Shubhanshu Mishra, and Cyril
<sup>19</sup>Note that as per widely cited definitions, nonbinary people are considered transgender, but most transgender people are not non-binary. Also, trans people often use a name that is more associated with their gender identity.
$^{20}$ https://realifemag.com/counting-the-countless/
<sup>21</sup> http://saifmohammad.com/WebPages/nlpscholar.html
$^{22}$ https://gender-gap-in-science.org
Goutte for the tremendously helpful discussions. Many thanks to Cassidy Rae Henry, Luca Soldaini, Su Lin Blodgett, Graeme Hirst, Brendan T. O'Connor, Lucia Santamaría, Lyle Ungar, Emma Manning, and Peter Turney for discussions on the ethical considerations involved with this work.
Richard Alford. 1987. Naming and identity: A cross-cultural study of personal naming practices. Hraf Press.
Jens Peter Andersen and Mathias Wullum Nielsen. 2018. Google Scholar and Web of Science: Examining gender differences in citation coverage across five scientific disciplines. Journal of Informetrics, 12(3):950-959.
Ashton Anderson, Dan McFarland, and Dan Jurafsky. 2012. Towards a computational history of the ACL: 1980-2008. In Proceedings of the Workshop on Rediscovering 50 Years of Discoveries, pages 13-21.
Herbert Barry III and Aylene S Harper. 2014. Unisex names for babies born in pennsylvania 1990-2010. *Names*, 62(1):13-22.
Greta R Bauer, Jessica Braimoh, Ayden I Scheim, and Christoffer Dharma. 2017. Transgender-inclusive measures of sex/gender for population surveys: Mixed-methods evaluation and recommendations. *PloS one*, 12(5):e0178043.
Lutz Bornmann and Hans-Dieter Daniel. 2009. The state of h index research. EMBO reports, 10(1):2-6.
Margo Brouns. 2007. The making of excellence: Gender bias in academia. In *Exzellenz in Wissenschaft und Forschung - neue Wege in der Gleichstellungspolitik*, pages 23-42. Wissenschaftsrat.
Claudia Buchmann. 2009. Gender inequalities in the transition to college. Teachers College Record, 111(10):2320-2345.
Catherine Connell. 2010. Doing, undoing, or redoing gender? Learning from the workplace experiences of transpeople. Gender & Society, 24(1):31-55.
Helana Darwin. 2017. Doing gender beyond the binary: A virtual ethnography. *Symbolic Interaction*, 40(3):317-334.
Michelle L Dion, Jane Sumner, and Sara McLaughlin Mitchell. 2018. Gendered citation patterns across political science and social science methodology fields. Political Analysis, 26(3):312-327.
Jordi Duch, Xiao Han T Zeng, Marta Sales-Pardo, Filippo Radicchi, Shayna Otis, Teresa K Woodruff, and Luís A Nunes Amaral. 2012. The possible role of resource requirements and academic career-choice risk on gender differences in publication rate and impact. *PloS one*, 7(12):e51332.
Irwin Feller. 2004. Measurement of scientific performance and gender bias. In Gender and Excellence in the Making, pages 35-39. Luxembourg: Office for Official Publications of the European Communities.
Marta Foschi. 2004. Blocking the use of gender-based double standards for competence. In Gender and Excellence in the Making, pages 51-55. Luxembourg: Office for Official Publications of the European Communities.
Juan Miguel Gallego and Luis H Gutierrez. 2018. An integrated analysis of the impact of gender diversity on innovation and productivity in manufacturing firms. Technical report, Inter-American Development Bank.
Gita Ghiasi, Vincent Larivière, and Cassidy Sugimoto. 2016. Gender differences in synchronous and diachronous self-citations. In 21st International Conference on Science and Technology Indicators-STI 2016. Book of Proceedings.
GenIUSS Group. 2014. Best practices for asking questions to identify transgender and other gender minority respondents on population-based surveys. eScholarship, University of California.
Namrata Gupta, Carol Kemelgor, Stefan Fuchs, and Henry Etzkowitz. 2005. Triple burden on women in science: A cross-cultural analysis. Current science, pages 1382-1386.
Malin Håkanson. 2005. The impact of gender on citations: An analysis of college & research libraries, journal of academic librarianship, and library quarterly. College & Research Libraries, 66(4):312-323.
Dalia S Hakura, Mumtaz Hussain, Monique Newiak, Vimal Thakoor, and Fan Yang. 2016. Inequality, gender gaps and economic growth: Comparative evidence for sub-Saharan Africa. International Monetary Fund.
Foad Hamidi, Morgan Klaus Scheuerman, and Stacy M Branham. 2018. Gender recognition or gender reductionism? The social implications of embedded gender recognition systems. In Proceedings of the 2018 CHI conference on human factors in computing systems, pages 1-13.
Kris Hardies, Diane Breesch, and Joel Branson. 2013. Gender differences in overconfidence and risk taking: Do self-selection and socialization matter? Economics Letters, 118(3):442-444.
Janet Shibley Hyde, Rebecca S Bigler, Daphna Joel, Charlotte Chucky Tate, and Sari M van Anders. 2019. The future of sex and gender in psychology: Five challenges to the gender binary. American Psychologist, 74(2):171.
Suzanne J Kessler and Wendy McKenna. 1978. Gender: An ethnomethodological approach. IL: The University of Chicago Press.
Molly M King, Carl T Bergstrom, Shelley J Correll, Jennifer Jacquet, and Jevin D West. 2017. Men set their own cites high: Gender and self-citation across fields and over time. *Socius*, 3:2378023117738903.
Vincent Lariviere, Chaoqun Ni, Yves Gingras, Blaise Cronin, and Cassidy R Sugimoto. 2013. Bibliometrics: Global gender disparities in science. Nature News, 504(7479):211.
Stanley Lieberson, Susan Dumais, and Shyon Baumann. 2000. The instability of androgynous names: The symbolic maintenance of gender boundaries. American Journal of Sociology, 105(5):1249-1287.
Linda L Lindsey. 2015. The sociology of gender theoretical perspectives and feminist frameworks. In Gender roles, pages 23-48. Routledge.
The Linguistic Society of America LSA. 2017. The state of linguistics in higher education annual report 2017. Technical report, The Linguistic Society of America.
Sangeeta Mehta, Karen EA Burns, Flavia R Machado, Alison E Fox-Robichaud, Deborah J Cook, Carolyn S Calfee, Lorraine B Ware, Ellen L Burnham, Niranjan Kissoon, John C Marshall, et al. 2017. Gender parity in critical care medicine. American journal of respiratory and critical care medicine, 196(4):425-429.
Helena Mihaljevic, Marco Tullney, Lucia Santamaría, and Christian Steinfeldt. 2019. Reflections on gender analyses of bibliographic corpora. Frontiers in Big Data, 2:29.
Helena Mihaljevic-Brandt, Lucia Santamaría, and Marco Tullney. 2016. The effect of gender in the publication patterns in mathematics. PLoS One, 11(10):e0165367.
Shubhanshu Mishra, Brent D Fegley, Jana Diesner, and Vetle I Torvik. 2018. Self-citation is the hallmark of productive authors, of any gender. *PloS one*, 13(9):e0195773.
Saif M. Mohammad. 2019. The state of NLP literature: A diachronic analysis of the ACL Anthology. arXiv preprint arXiv:1911.03562.
Saif M. Mohammad. 2020a. Examining citations of natural language processing literature. In Proceedings of the 2020 Annual Conference of the Association for Computational Linguistics, Seattle, USA.
Saif M. Mohammad. 2020b. NLP Scholar: A dataset for examining the state of NLP research. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC-2020), Marseille, France.
Saif M. Mohammad. 2020c. NLP Scholar: An interactive visual explorer for Natural Language Processing literature. In Proceedings of the 2020 Annual Conference of the Association for Computational Linguistics, Seattle, USA.
Lena Nekby, Peter Thoursie, and Lars Vahtriak. 2008. Gender and self-selection into a competitive environment: Are women more overconfident than men? Economics Letters, 100(3):405-407.
Caroline Criado Perez. 2019. Invisible women: Exposing data bias in a world designed for men. Random House.
Jane Pilcher. 2017. Names and "doing gender": How forenames and surnames contribute to gender identities, difference, and inequalities. *Sex roles*, 77(11-12):812-822.
Cristina Richards, Walter Pierre Bouman, and M Barker. 2017. Non-binary genders. London: Pal grave Macmillan.
Patricia A Roos. 2008. Together but unequal: Combating gender inequity in the academy. Journal of Workplace Rights, 13(2):185-199.
Natalie Schluter. 2018. The glass ceiling in NLP. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2793-2798, Brussels, Belgium.
Inger Skjelsboek and Dan Smith. 2001. Gender, peace and conflict. Sage.
Brittany N Smith, Mamta Singh, and Vetle I Torvik. 2013. A search engine approach to estimating temporal changes in gender orientation of first names. In Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries, pages 199-208. ACM.
Vetle I Torvik and Neil R Smalheiser. 2009. Author name disambiguation in medline. ACM Transactions on Knowledge Discovery from Data (TKDD), 3(3):1-29.
Adam Vogel and Dan Jurafsky. 2012. He said, she said: Gender in the ACL Anthology. In Proceedings of the Special Workshop on Rediscovering 50 Years of Discoveries, pages 33-41, Jeju Island, Korea.
World Economic Forum WEF. 2018. The global gender gap report 2018. Technical report, World Economic Forum, Geneva, Switzerland.
Cassandra Willyard. 2011. Men: A growing minority. GradPSYCH Magazine, 9(1):40.
Jonathan Woetzel et al. 2015. The power of parity: How advancing women's equality can add $12 trillion to global growth. Technical report, McKinsey Global Institute.
Matthew Zook, Solon Barocas, Danah Boyd, Kate Crawford, Emily Keller, Seeta Pena Gangadharan, Alyssa Goodman, Rachelle Hollander, Barbara A. Koenig, Jacob Metcalf, Arvind Narayanan, Alondra Nelson, and Frank Pasquale. 2017. Ten simple rules for responsible big data research. PLOS Computational Biology, 13(3):1-10.