Abstract: The ACL Anthology (AA) is a digital repository of tens of thousands of articles on Natural Language Processing (NLP). This paper examines the literature as a whole to identify broad trends in productivity, focus, and impact. It presents the analyses in a sequence of questions and answers. The goal is to record the state of the AA literature: who and how many of us are publishing? what are we publishing on? where and in what form are we publishing? and what is the impact of our publications? The answers are usually in the form of numbers, graphs, and inter-connected visualizations. Special emphasis is laid on the demographics and inclusiveness of NLP publishing. Notably, we find that only about 30% of first authors are female, and that this percentage has not improved since the year 2000. We also show that, on average, female first authors are cited less than male first authors, even when controlling for experience. We hope that recording citation and participation gaps across demographic groups will encourage more inclusiveness and fairness in research.

Introduction: The ACL Anthology (AA) is a digital repository of tens of thousands of articles on Natural Language Processing (NLP) / Computational Linguistics (CL). It includes papers published in the family of ACL conferences as well as in other NLP conferences such as LREC and RANLP. AA is the largest single source of scientific literature on NLP. This project, which we call NLP Scholar, examines the literature as a whole to identify broad trends in productivity, focus, and impact. We will present the analyses in a sequence of questions and answers. The questions range from fairly mundane to oh-that-will-be-good-to-know. Our broader goal here is simply to record the state of the AA literature: who and how many of us are publishing? what are we publishing on? where and in what form are we publishing? and what is the impact of our publications? The answers are usually in the form of numbers, graphs, and inter-connected visualizations. We focus on the following aspects of NLP research: size, demographics, areas of research, impact, and correlation of citations with demographic attributes (age and gender).  Target Audience: The analyses presented here are likely to be of interest to any NLP researcher. This might be particularly the case for those that are new to the field and wish to get a broad overview of the NLP publishing landscape. On the other hand, even seasoned NLP'ers have likely wondered about the questions raised here and might be interested in the empirical evidence.  Data: The analyses presented below are based on information about the papers taken directly from AA (as of June 2019) and citation information extracted from Google Scholar (as of June 2019). Thus, all subsequent papers and citations are not included in the analysis. A fresh data collection is planned for January 2020.  Interactive Visualizations: The visualizations we are developing for this work (using Tableau) are interactive—so one can hover, click to select and filter, move sliders, etc. Since this work is high in the number of visualizations, the main visualizations are presented as figures in the paper and some sets of visualizations are pointed to online. The interactive visualizations and data will be made available through the first author's website after peer review.  Related Work: This work builds on past research, including that on Google Scholar BIBREF0, BIBREF1, BIBREF2, BIBREF3, on the analysis of NLP papers BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, on citation intent BIBREF10, BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15, and on measuring scholarly impact BIBREF16, BIBREF17, BIBREF18, BIBREF19, BIBREF20, BIBREF21.  Caveats and Ethical Considerations: We list several caveats and limitations throughout the paper. A compilation of these is also available online in the About NLP Scholar page. The analyses presented here are also available as a series of blog posts.

Question: Which 5 languages appear most frequently in AA paper titles?
Answer: English, Chinese, French, Japanese and Arabic