SlowGuess/ACL_2020_OCR huggingface dataset:
12Other approaches such as clustering are also reasonable; however, results with those might not be easily reproducible. We chose the title bigrams approach for its simplicity.
Discussion: The graph shows, for example, that the bigram machine translation occurred in 1,659 AA' papers that have a median citation count of 14, while the average is 68.8. The average is one of the highest among the bigrams, despite the median being more middle of the pack. This suggests the presence of heavily cited, outlier, papers. Indeed, the most cited paper in all of AA' is an MT paper with more than 9000 citations (Papineni et al., 2002). Note that not all MT papers have machine translation in the title. Although non-random, this sample of 1,659 papers is arguably a reasonably representative sample of MT papers.
Third in the list are papers with statistical machine in the title—most commonly from the phrase statistical machine translation. One expects considerable overlap across these sets of papers. However, machine translation likely covers a broader range of research including work done before statistical MT was introduced, as well as work on neural MT and MT evaluation.
The bigrams with the highest median include: sentiment classification (31), anaphora resolution (30), and entity recognition (25). The bigrams with the lowest median include: language resources (5), textual entailment (8), translation system (9), and cross language (9). The bigrams with the highest average include: sentiment classification (181.6), speech tagging (107.9), sentiment analysis (104.0), and statistical machine (90.1).<sup>13</sup> One can access the lists of highly cited papers, pertaining to each of the bigrams, through the interactive visualization.
We list below some ideas of future work that we did not explore in this paper:
- Analyze NLP papers that are published outside of the ACL Anthology.
- Measure involvement of the industry in NLP publications over time.
- Measure the impact of research publications in other ways beyond citations. Identify papers that have made substantial contributions in non-standard ways.
A list of limitations and ethical considerations associated with this work is available online. $^{14}$
We extracted citation information for  $\sim 1.1\mathrm{M}$  papers from Google Scholar profiles of researchers who published at least three papers in the ACL Anthology. We used the citation counts of a subset ( $\sim 27\mathrm{K}$  papers) to examine patterns of citation across paper types, venues, over time, and across areas of research within NLP.
We showed that only about  $56\%$  of the papers are cited ten or more times. CL Journal has the most cited papers, but the citation gap between CL journal and top-tier conferences has reduced in recent years. On average, long papers get almost three times as many citations as short papers. In case of popular shared tasks, the task-description papers and competition-winning system-description papers often receive a considerable number of citations. So much so that the average number of citations for the shared task papers is higher than the average for non-top-tier conferences. The papers on sentiment classification, anaphora resolution, and entity recognition have the highest median citations. Workshop papers and the shared task papers have higher median and average citations than the non-top-tier conferences.
The analyses presented here, and the associated dataset of papers mapped to citations, have a number of uses including, understanding how the field is growing and quantifying the impact of different types of papers. In separate work, we explored the use of the dataset to detect gender disparities in authorship and citations (Mohammad, 2020a). The dataset can potentially also be used to compare patterns of citations in NLP with those in other fields. Finally, we note again that citations are not an accurate reflection of the quality or importance of individual pieces of work. A crucial direction of future work is to develop richer ways of capturing scholarly impact.
This work was possible due to the helpful discussion and encouragement from a number of awesome people including: Dan Jurafsky, Tara Small, Michael Strube, Cyril Goutte, Eric Joanis, Matt Post, Patrick Littell, Torsten Zesch, Ellen Riloff, Iryna Gurevych, Rebecca Knowles, Isar Nejadgholi, and Peter Turney. Also, a big thanks to the ACL Anthology and Google Scholar Teams for creating and maintaining wonderful resources.
<sup>13</sup>Note that simply composing titles with these high-citation bigrams is not expected to attract a large number of citations. <sup>14</sup>https://medium.com/@nlpscholar/about-nlp-scholar-62cb3b0f4488
Ashton Anderson, Dan McFarland, and Dan Jurafsky. 2012. Towards a computational history of the acl: 1980-2008. In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, pages 13-21.
Selcuk Aya, Carl Lagoze, and Thorsten Joachims. 2005. Citation classification and its applications. In Knowledge Management: Nurturing Culture, Innovation, and Technology, pages 287-298. World Scientific.
Steven Bird, Robert Dale, Bonnie J Dorr, Bryan Gibson, Mark Thomas Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir R Radev, and Yee Fan Tan. 2008. The acl anthology reference corpus: A reference dataset for bibliographic research in computational linguistics.
Lutz Bornmann and Hans-Dieter Daniel. 2009. The state of h index research. EMBO reports, 10(1):2-6.
Arthur R Bos and Sandrine Nitza. 2019. Interdisciplinary comparison of scientific impact of publications using the citation-ratio. Data Science Journal, 18(1).
Zoe Bulaitis. 2017. Measuring impact in the humanities: Learning from accountability and economics in a contemporary history of cultural value. Palgrave Communications, 3(1):7.
Daniel Gildea, Min-Yen Kan, Nitin Madnani, Christoph Teichmann, and Martin Villalba. 2018. The ACL anthology: Current state and future directions. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 23-28, Melbourne, Australia. Association for Computational Linguistics.
Michael Gusenbauer. 2019. Google scholar to overshadow them all? comparing the sizes of 12 academic search engines and bibliographic databases. Scientometrics, 118(1):177-214.
Jared L Howland. 2010. How scholarly is google scholar? a comparison to library databases.
John PA Ioannidis, Jeroen Baas, Richard Klavans, and Kevin W Boyack. 2019. A standardized citation metrics author database annotated for scientific field. PLoS biology, 17(8):e3000384.
Madian Khabsa and C Lee Giles. 2014. The number of scholarly documents on the public web. *PloS one*, 9(5):e93949.
Joseph Mariani, Gil Francopoulos, and Patrick Paroubek. 2018. The nlp4nlp corpus (i): 50 years of publication, collaboration and citation in speech and language processing. Frontiers in Research Metrics and Analytics, 3:36.
Alberto Martin-Martin, Enrique Orduna-Malea, Mike Thelwall, and Emilio Delgado López-Cózar. 2018. Google scholar, web of science, and scopus: A systematic comparison of citations in 252 subject categories. Journal of Informetrics, 12(4):1160-1177.
John Mingers and Loet Leydesdorff. 2015. A review of theory and practice in scientometrics. European journal of operational research, 246(1):1-19.
Saif Mohammad, Bonnie Dorr, Melissa Egan, Ahmed Hassan, Pradeep Muthukrishan, Vahed Qazvinian, Dragomir Radev, and David Zajic. 2009. Using citations to generate surveys of scientific paradigms. In Proceedings of human language technologies: The 2009 annual conference of the North American chapter of the association for computational linguistics, pages 584-592.
Saif M. Mohammad. 2019. The state of nlp literature: A diachronic analysis of the acl anthology. arXiv preprint arXiv:1911.03562.
Saif M. Mohammad. 2020a. Gender gap in natural language processing research: Disparities in authorship and citations. In Proceedings of the 2020 Annual Conference of the Association for Computational Linguistics, Seattle, USA.
Saif M. Mohammad. 2020b. Nlp scholar: A dataset for examining the state of nlp research. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC-2020), Marseille, France.
Saif M. Mohammad. 2020c. Nlp scholar: An interactive visual explorer for natural language processing literature. In Proceedings of the 2020 Annual Conference of the Association for Computational Linguistics, Seattle, USA.
Hidetsugu Nanba, Noriko Kando, and Manabu Okumura. 2011. Classification of research papers using citation links and citation types: Towards automatic review article generation. Advances in Classification Research Online, 11(1):117-134.
Enrique Orduña-Malea, Juan Manuel Ayllón, Alberto Martín-Martín, and Emilio Delgado López-Cózar. 2014. About the size of google scholar: playing the numbers. arXiv preprint arXiv:1407.6239.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318.
Son Bao Pham and Achim Hoffmann. 2003. A new approach for scientific citation classification using cue phrases. In Australasian Joint Conference on Artificial Intelligence, pages 759-771. Springer.
Jason Priem and Bradely H Hemminger. 2010. Scientometrics 2.0: New metrics of scholarly impact on the social web. First monday, 15(7).
Vahed Qazvinian, Dragomir R Radev, Saif M Mohammad, Bonnie Dorr, David Zajic, Michael Whidby, and Taesun Moon. 2013. Generating extractive summaries of scientific paradigms. Journal of Artificial Intelligence Research, 46:165-201.
Dragomir R Radev, Mark Thomas Joseph, Bryan Gibson, and Pradeep Muthukrishnan. 2016. A bibliometric and network analysis of the field of computational linguistics. Journal of the Association for Information Science and Technology, 67(3):683-706.
James Ravenscroft, Maria Liakata, Amanda Clare, and Daniel Duma. 2017. Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements. *PloS one*, 12(3):e0173152.
Simone Teufel, Advaith Siddharthan, and Dan Tidhar. 2006. Automatic classification of citation function. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 103-110.
Alan M Turing. 1950. Computing machinery and intelligence-am Turing. Mind, 59(236):433.
Alan M Turing. 2009. Computing machinery and intelligence. In *Parsing the Turing Test*, pages 23-65. Springer.
Dani Yogatama, Michael Heilman, Brendan O'Connor, Chris Dyer, Bryan R Routledge, and Noah A Smith. 2011. Predicting a scientific community's response to an article. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 594-604.
Xiaodan Zhu, Peter Turney, Daniel Lemire, and André Vellino. 2015. Measuring academic influence: Not all citations are equal. Journal of the Association for Information Science and Technology, 66(2):408-427.