We are Who We Cite: Bridges of Influence Between Natural Language Processing and Other Academic Fields
Authors: Jan Philip Wahle
Terry Ruas
Mohamed Abdalla
Bela Gipp
Saif M. Mohammad
Venue: EMNLP 2023 Main (2023)
Keywords: responsible nlp
scientific influence
interdisciplinarity
bibliometrics

Abstract: Natural Language Processing (NLP) is poised to substantially influence the world. However, significant progress comes hand-in-hand with substantial risks. Addressing them requires broad engagement with various fields of study. Yet, little empirical work examines the state of such engagement (past or current). In this paper, we quantify the degree of influence between 23 fields of study and NLP (on each other). We analyzed ~77k NLP papers, ~3.1m citations from NLP papers to other papers, and ~1.8m citations from other papers to NLP papers. We show that, unlike most fields, the cross-field engagement of NLP, measured by our proposed Citation Field Diversity Index (CFDI), has declined from 0.58 in 1980 to 0.31 in 2022 (an all-time low). In addition, we find that NLP has grown more insular---citing increasingly more NLP papers and having fewer papers that act as bridges between fields. NLP citations are dominated by computer science; Less than 8% of NLP citations are to linguistics, and less than 3% are to math and psychology. These findings underscore NLP's urgent need to reflect on its engagement with various fields.

We are Who We Cite: Bridges of Influence Between Natural Language Processing and Other Academic Fields
Jan Philip Wahle
Terry Ruas
Mohamed Abdalla
National Research Council Canada
University of Göuinngen
Institute for Better Health Toronto, Canada
University of Göuingen Germany
Göutingen, Germany meaiiiigggtttngnnnnn de
wahle www.euiningetttigennco
msaêcs. toronto. edu
Bela Gipp University of Göttingen Göttingen, Germany -ooeuun--geettineeeee de
Saif M. Mohammad
National Research Council Canada
Ottawa, Canada
saif. mohammadenrc--nne
Abstract
thus, making an empirical determination of the de- gree of influence difficult.
Natural Language Processing (NIP) poised substantially influence the world. However, significant progress comes hand-in-hand with substantial risks. Addressing them requires broad engagement with various fields of study. Yet, little empirical work examines the state of such engagement (past or current). this paper, we quantify the degree of influence be- tween fields study and NI.P (on each other). We analyzed ~-77k NLP papers, ~m citations from NLP papers to other papers, and 1..8m citations from other papers NLP pa- pers. We show that, unlike most fields, the cross-field engagement of NLP, measured by our proposed Citation Field Diversity Index (CFDI), has declined from 0.58 in 1980 to 0.31 2022 (an all-time low). In addition, we find that NLP has grown more insular-cine citing in- creasingly more NLP papers and having fewer papers that act as bridges between fields. NLP citations dominated
In this work, we focus on a specific subset of influence: the scientific influence of one field of study another. Stated another way, we explore the degree to which the ideas and scientific artifacts of various fields of study impact target field of interest and the degree which the ideas of target field impact other fields.
Who is inftuencing us (our field)? Who influencing?
Mechanisms of field-to feld-to-field influence are also complex, but one notable marker of scientific influ- ence is citations. Thus, we propose that the extent which source field cites a target field is rough indicator of the degree of influence of the target on the source. We note here, though, that not all citations are equal-sone some cited work may have been much more influential to the citing work than others (Zhu et al., 2015; Valenzuela et al., 2015). Further, citation patterns are subject to various bi- ases (Mohammad, 2020a; loannidis et al., 2019). Nonetheless, meaningful inferences at an aggregate level; for example, if the propor- tion of citations from field markedly increased as compared to the proportion of citations from other fields to the target, then it is likely that the influence of x on has grown.
computer science; Less than 8% of NL.P citations are to linguistics, and less than 3% are math and psychology. These findings underscore NLP''s urgent need to reflect on its engagement with various fields.
be drawn
1 Introduction
target field y has
The degree to which the ideas and artifacts field study are useful to the world (for improving our understanding of the world or for practical applica- tions) is measure of its influence. Developing better sense of the influence of a field has several benefits, such understanding what fosters greater innovation and what stifles it; what a field has suc- cess at understanding and what remains elusive; who are the most prominent stakeholders benefit- ing and who are being left behind, etc. However, the modes of influence are numerous and complex;
Agency of the Researcher: A key point about scientific influence is that as researchers, we are not just passive objects of influence. Whereas some inffuences hard to ignore (say, because of substantial buy-in from one's peers and the review process), others are direct result of active engagement by the researcher with relevant literature (possibly from a different field). We

---

(can) choose whose literature engage with, and thus benefit from. Similarly, while times we may happen to influence other fields, we (can) choose to engage with other fields so that they are more aware of our work. Such an engagement can be through cross-field exchanges at conferences, blog posts about one's work for target audience in another field, etc. Thus, examining which fields of study are the prominent sources of ideas for field is a window into what literature the field as a whole values, engages with, and benefits from.
papers, ~3.1m citations from NLP papers to other papers, and ~1.8m citations from other pupers NLP papers. Notably, the metadata includes the field of study and year of publication of the papers cited by the NLP papers, the field of study and year of publication of papers citing NLP papers, and the NLP subfields relevant to each NLP paper. We trace hundreds of thousands of citations in the dataset systematically and quantitatively examine broad trends in the influence of various fields of study on NLP and NLP's influence on them. Specifically, we explore
Whose literature are we (our field) choosing to engage with?
1. Which fields of study are influencing NLP? And what degree?
Who is engaging with our literature?
Further, since cross-field engagement leads to
2. Which fields of study are influenced by NLP?
citations, we argue:
And to what degree?
Who we cite says a lot about us. And,
3. To what extent is NLP
on its own ideas as opposed to being hotbed of cross-pollination by actively bringing in ideas and innovations from other fields of study?
Who cites us also says a lot about us.
Why NLP: While studying influence is useful for any field of study, and our general approach and ex- perimental setup are broadly applicable, we focus on Natural language Processing (NLP) research for
4. How have the above evolved over time?
This study enables broader reflection on the ex- tent to which we are actively engaging with the communities and literature from other fields-an an important facet of responsible NLP.
one critical reason.
NLP is at an inflection point. Recent develop- ments in large language models have captured the imagination of the scientific world, industry, and the general public. Thus, NLP is poised to exert substantial influence, despite significant risks (Buo- lamwini and Gebru, 2018; Mitchell et al., 2019; Bender al., 2023; Mohammad, 2022; Wahle et al.. 2023; Abdalla et al., 2023). Further, language is social, and NLP applications have complex social implications. Therefore, responsible rescarch and development need engagement with a wide swathe of literature (arguably, more so for NLP than other fields). Yet, there is no empirical study on the bridges of influence between NLP and other fields. Additionally, even though NLP is interdisci- plinary and draws on many academic disciplines, including linguistics, computer science (CS), and artificial intelligence (AI), CS methods and ideas dominate the field. Relatedly, according to the 'NL Community Metasurvey', most respondents believed that NLP should do more to incorporate insights from relevant fields, despite believing that the community did not care about increasing in- terdisciplinarity (Michael et al., 2022). Thus, we believe it is important measure the extent to which CS and non-CS fields influence NLP.
Related Work
Previous studies responsible research in NLP have examined aspects such as author gender diver- sity (Vogel and Jurafsky, 2012; Schluter, 2018: Mo- hammad, 2020b), author location diversity (Rungta et al., 2022), temporal citation diversity (Singh et al., 2023), software package diversity (Gururaja et al., 2023), and institutional diversity (Abdalla et al., 2023). However, core component of respon- sible research is interdisciplinary engagement with other research fields (Porter et al., 2008; Leydes- dorff et al., 2019). Such engagement can steer the direction of research. Cross-field research engage- ment can provide many benefits in the short-term, by integrating expertise from various disciplines to address a specific issue, and in the long-term to generate new insights or innovations by blending ideas from different disciplines (Hansson, 1999).
Many significant advances have emerged from the synergistic interaction of multiple disciplines, e.g., the conception of quantum mechanics, the- ory that coalesced Planck's 's idea of quantized en- ergy levels (Planck, 1901), Einstein's photoclectric effect (Einstein, 1905), and Bohr's model of the
In this paper, we describe how we created a new dataset of metadata associated with 777k NLP

---

atom (Bohr, 1913). In addition, entirely new fields have emerged such as ecological economics. A key concept in ecological economics is the notion of "ecosystem services', which originated in biol- ogy (Postel et al., 2012) and ecology (Pearce and Turner, 1989). The field of medicine has integrated neuroscience (Bear et al., 2020) with engineering principles (Saltzman, 2009). Within NLP, in the 1980s, there was marked growth in the use of sta- tistical probability tools to process speech, driven by researchers with electrical engineering back- grounds engaging with NLP (Jurafsky, 2016).
Time range
1965-2022
#papers
209 016 314
#citations
2,521 776 124
#pupers NLP
76 745
#out-citations from NI.P
3 115 126
#in-citations XLP
1 847 873
Table I: Overall dataset statistics.
3 Data
Central to work on field-to-field influence is a dataset where the field of each of the papers is known. We would like to note here that a paper may belong to many fields, each to varying extents, and it can be difficult for humans and automatic systems to determine these labels and scores per- feetly. Further, it is difficult obtain complete set of papers pertaining to each field; even deter- mining what constitutes field and what does not is complicated. Nonetheless, meaningful inferences can be drawn from large samples of papers labeled as per some schema of field labels.
To quantify the interdisciplinarity of body of re- search, researchers have proposed various metrics (on Science et al., 2004; Wang et al., 2015). These metrics can range from simple counting of refer- enced subjects (Wang et al., 2015) to more complex metrics such the Gini-index (Leydesdorff and Rafols, 2011) I). Our work seeks understand the level of interdisciplinarity in NLP by tracking how often and how much it references other fields and how this pattern evolves over time.
As in past work (Mohammad, 2020a; Abdalla et al., 2023), we chose the ACL Anthology (AA) as suitable sample of NLP papers.' We extracted metadata associated with the NLP (AA) papers and with papers from various fields using Semantic Scholar (S2) dump. 2 It includes papers published between 1965-2022, totaling 209m papers and 2.5b citations. S2's field of study annotation labels cach paper as belonging to one or more of 23 broad fields of study. It uses a field-of-study classifier based on abstracts and titles (S2FOS²) with an accuracy of 86%. The S2 dump also includes information about which paper cites another paper. Table provides an overview of key dataset statistics.
Several studies have reported an increasing trend of interdisciplinarity across fields (Porter and Rafols, 2009; Larivière et al., 2009; Van No- orden et al., 2015; Truc et al., 2020) and within subfields (Pan al., 2012). For those observing increasing interdisciplinarity, the growth is predom- inantly seen in neighboring fields, with minimal increases in connections between traditionally dis- tant research areas, such as materials sciences and clinical medicine (Porter and Rafols, 2009). The findings are contested, with other researchers find- ing that the "scope of science" has become more limited (Evans, 2008), that the findings are field- dependent, exemplified by contrasting citation pat- terns in sub-disciplines within anthropology and physies (Choi, 1988; Pan al., 2012).
Since we expect NLP papers to have cited CS substantially, we also examine the influence be- tween specific subfields of CS and NLP. Thus, we make use of the subfield annotations of the Com- puter Science Ontology (CSO) version 3.3 (Salatino et al., 2020) that is available through the D3 dataset (Wahle et al., 2022). The CSO classifier uses pa- per's titles, abstracts, and keywords and has an fll score of 74% (Salatino et al., 2019).
Within computer science (CS), Chakraborty (2018) used citational analysis to show increas- ing interdisciplinarity in CS sub-fields, positing a three-stage life cycle for citation patterns of fields: a growing, maturing, and interdisciplinary phase.
Within NLP, Anderson et al. (2012) examined papers from 1980 to 2008 to track the popularity of research topics, the influence of subfields on each other, and the influence of government interven- tions on the influx of new researchers into NLP. By examining the trends of NLP's engagement with other fields, our work seeks to provide a discipline- specific perspective.
Within the CSO ontology, we consider only the 15 most populous subfields of CS (e.g., AI, com- puter vision (CV)). We also further split AI into Machine Learning (ML) and the rest of AI (Al') simply because ML is known influencer of NLP.
hhups::/www.aclwwaet http://ww.wwgggw.oog.......
2 https://pppi sewanticscholar..oggg/oooo smmamisshhblarroo/pppddeedsssss
https://tinyurl. com.///mmm//////

---

The source code to process our data and repro- duce the experiments is available on GitHub:
CS (7.55, 81.8%
NLP (3.1m 1.8m)
C$ (1 Am, 79.44)
https://ggthub..omm/pwaaaa (imtp22-ciiatiooo-fiell------fee
Non-CS (361 99, 20.616X
Non- C5 (566.3k 18.2%
cucgoing ctations citx norvts
Incoming diations cl
4 Experiments
(a) CS, Non-CS
We use the dataset described above to answer a series of questions about the degree of influence between various fields and NLP.
Linguistics (240.7%, 42.4%)
QL. Which fields do NLP papers cite? In other words, how diverse are the outgoing citations by NLP papers-do we cite papers mostly just from computer science, or do we have a marked number of citations to other fields of study, as well?
Linguistics 73.4k, 45,1
Marhematics (86. Sk. 15.3%)
NLP (567.2k, 354 4k)
Separately, which fields cite NLP papers? How diverse are the incoming citations to NLP papers?
Psychology (81.2k, 14.39
Paychology (60.5k, 15.7%)
Sociology 53019)
Mathe mar 4.k 8.2%;
Ans. For each NLP paper, we look at the citations to other papers and count the number of outgoing citations going to each field (or, outcites). Similarly, we track papers citing NLP work and count the number of incoming citations from each field (or, incites). If paper is in multiple fields, we assign one citation to each field.
Biology 3.7kk
Surcolgge mig.0k S
@Phitosophy 3.092
Medichee 022.3 3.226a
UMedicine [16.5K: 2.5%0
Engineer 1..3 3/10
Philosophy
UBuusness 2.556)
Biolotx
méhysics (12:0%.2:199
Business 1. 2:7M
mEngineennn 19:5 1.6%)
mEduration (3.9k 1.2%)
Education 2:0%i
méconorics
Econommics
0.8%)
Art
Marerisi 2.0
Political
Eny Scionce 13 0.5%
0.6%)
Egu. Geography 122
History
Material Eeo 020 0-30
We calculate the percentage of outgoing cita- tions: the percent of citations that go to a specific field from NLP over all citations from NLP to any field. Similarly, we calculate the percentage for incoming citations: the percent of citations go- ing from specific field to NLP over all citations from any field to NLP. Because we expect CS to be a marked driver of NLP citations, we split cita- tions into CS and non-CS and then explore citations within CS and non-CS for additional insights.
Siannes (148, 0.0%)
Food
(b) Non-CS Fields
All (1 5m,, 27.008)
(637.44, 11.5%)
Al (1. (1.1m, 29.154)
ML (618.1k, 11.1%)
148899 12,054
Dota Mining (561.9K 10.1/))
NLP (2.6m. 1.5m)
(198.1) 10.89:
Interntt (396.84, 7.194]
Data Mining (364.34
Results. Figure shows three Sankey plots. The right side of each figure shows citations from other fields to NLP (Hincoming citations, Wincomcing ci- tations), and the left side shows citations from NLP to other fields (Houtgoing citations, loutgoing ci- tations). The middle part shows the overall number of outgoing and incoming citations from/to NLP (#outgoing citations, #incoming citations). The width of the grey flowpath is proportional to the number of citations.
cV (378.5k 5.8%)
Computer Networks 138..4, 6, 09)
Computer Neterrs ts 1847 7.546.664 6.39%
Computer Programming 1237.34, 4.3%)
incemnet teas 54, 6.19)
Computer 1210.0k 3.00k.33.8 3.8%)
Compater Programt wornen & 1922 4
Hllumnnoompuee uler inter interaction (190,3k,
3
Computee uter Systom (146 de Bum9)
Software Engineering (17144 3.1%)
Software Engineer
mélonnoomsh (F1.88 2,0%)
Human-Compurer
Dorfformttio
Eomppttrr Ceempptsr-eettoor (63 1.2%))
Techriding 4k. (.48)
Computer Neowors
wiobrntc (61 1k. (.19)
Computer Communication Networks
(c) CS Subfields
Figure I: Citations from other fields to NLP (right) and from NLP to other fields (left).
Figure (a) shows the flow of citations from CS and non-CS papers to NLP (right) and from NLP to CS and non-CS papers (left). Overall, the number of incoming citations to NLP is lower (1.9m) than the number of outgoing citations mm). The dis- crepancy between incoming and outgoing citations shows that NLP is overall cited less than it cites
other fields. This is not uncommon: we found that #incites is lower compared to #outcites for all 23 fields of study (figures not shown here).
79.4% of citations received by NL.P are from CS papers. Similarly, a large majority (81.8% of cita- tions from NLP papers go to CS papers. Citations

---

(a) Outgoing Citation Percentage, 1990-2020
(b) Incoming Citation Percentage, 1990-2020
C5
Non-CS
CS
Non-CS
100%
100%
75%
75%
50%
50%
25%
25%
0%
0%
1990
2000
2010
2020
1990
2000
2010
2020
Figure 2:
The pereentage of citations (a) from NLP to CS and non-CS and (b) from CS and non-CS to NLP over all citations from and to NLP with moving average of three years.
(a) Outgoing Citation Percentage, 1990-2020 Socio.
(b) Incoming Citation Percentage, 1990-2020 -2020
Psych.
Math.
Ling.
Socio.
Psych.
Math.
Ling.
80%
75%
60%
50%
25%
20%
0%
0%
1990
2000
2010
1990
2000
2010
Figure 3: The percentage citations (a) from NLP to non-CS fields and (b) non-CS CS fields to NLP in relation to all non-CS citations from and to NLP.
to individual non-CS fields are markedly lower than CS: linguistics (7.6%), math (2.8%), psychology (2.6%), and sociology (1.0%) (Figure 13 in the Appendix shows the percentages for all 23 fields).
However, these results are for all citations and do not tell us whether the citational influences of indi- vidual fields are rising or dropping.
Q2. How have NLP's outgoing (and incoming) ci- tations to fields changed over time?
To get a better sense the variation across non- CS fields, Figure (b) shows a Sankey plot when we only consider the non-CS fields. Observe that linguistics receives 42.4% of all citations from NLP to non-CS papers, and 45.1 % of citations from non- CS papers to NLP are from linguistics. NLP cites mathematics more than psychology, but more of NLP's incites are from psychology than math.
Ans. Figure 2 shows NLP's outgoing (a) and in- coming (b) citation percentages CS and non-CS papers from 1990 to 2020. Observe that while in 1990 only about 54% the outgoing citations were to CS, that number has steadily increased and reached 83% by 2020. This is a dramatic transfor- mation and shows how CS-centri NLP has become over the years. The plot for incoming citations (Fig- ure (b)) shows that NLP receives most citations from CS, and that has also increased steadily from about 64% to about 81% 2020.
Figure (c) shows the flow of citations from CS subfields to NLP and from NLP to CS sub- fields. Observe that NLP cites the following sub- fields most: AI' (27.0%), ML (11. (111%) and infor- mation retrieval (IR) (1 1.5%).. ML receives roughly many citations from NLP as all non-CS fields combined (2.5x #citations as linguistics).
Figure 3 shows the citation percentages for four non-CS fields with the most citations time. Observe that linguistics has experienced marked (relative) decline in relevance for NL.P from 2000 to 2020 (60.3% to 26.9% for outcites; 62.7% to 39.6 99.6% for incites). Details on the diachronic trends for CS subfields can be found in the Appendix A.3, Discussion. Over time, both the in- and out- citations of CS have had an increasing trend. These
Discussion. Even though expected high CS- NLP citational influence, this work shows for the first time that ~80 of the citations and of NLP are from and to CS. Linguisties, mathematics, psychology, and social science are involved a major portion of the remaining in- and out-citations.

---

results also show waning influence of linguistics and sociology and an exponential rise in the influ- ence of mathematics (probably due to the increas- ing dominance of sathhemmtics-hharyy deep learning and large language models) and psychology (prob- ably due to increased use of psychological models of behavior, emotion, and well-being in NLP ap- plications). The large increase in the influence of mathematics seems have largely eaten into what used be the influence of linguistics.
average
overage
Computer cience
Linguistics
M Mathematics
hilosophy
History
Agr. Food 5 science
Geography
Geology
Q3. Which fields cite NLP more than average and which do not? Which fields are cited by NLP more than average, and which are not?
Education
Physks
Sociology
Materials Science
Engineering
Ans. As we know from QI, 15.3% of NLP's non- CS citations go to math, but how does that compare to other fields citing math? Are we citing math more prominently than the average paper? That is the motivation behind exploring this question. To answer it, calculate the difference between NLP's outgoing citation percentage a field / and the macro average of the outgoing citations from various fields to f. We name this metric Outgoing Relative Citational Prominence (ORCP). If NLP has an ORCP greater than for f, then NLP cites ff more often than the average of all fields to f.
Economics
Politicll Science
Chemistry
Business
Psychology
Env. Science
Blology
Medicine
-15%
0%
15%
30%
45%
60%
Figure 4: NLP's Outgoing Relative Citational Promi- nence (ORCP) scores for 23 fields of study.
chology, linguistics, and math. Here, the gray hori-
ORCP NLP(F) x(f) Evieen CNLP-IS
((f--(1)
(1) zontal bands indicate the range from min. max.
CNIR-S
(2) of all 23 fields of study. Among the 23 fields, the highest ORCP to a field is reached by the field it- self, showing that, as expected, citations to papers within the same field are higher than cross-field citations. However, none of the 23 fields cite CS much as NLP: NLP cites CS 73.9 percentage points more than average, whereas CS cites itself only 32.6 percentage points more than average. Linguis- tics cites psychology substantially above average with 15% ORCP and CS markedly too with 6.8% ORCP. Psychology has a strong medicine focus citing it more than 16 percentage points above av- erage. Math cites physies and engineering above average but not nearly as much as CS.
Y()= 1 EE
Ch-s
Exirer Clry
where is the set of all fields, N is the number all fields, and Chi+ss is the number of citations from field fi to field fz-
Similar to ORCP, we calculate the Incoming Rel- ative Citational Prominence (IRCP) from each field to NLP. IRCP indicates whether field cites NLP more often than the average of all fields NLP. Results. Figure 4 shows plot of NLP's ORCPs with various fields. Observe that NLP cites CS markedly more than average (ORCP = 73.9%). Even though linguistics is the primary source for theories languages, NLP papers cite only 6.3 per- centage points more than average (markedly lower than CS). It is interesting to note that even though psychology is the third most cited non-CS field by NLP (see Figure (b)), it has an ORCP of 5.0. indicating that NLP cites psychology markedly less than how much the other fields cite psychology.
Figures 15 and 16 (Appendix) show plots for incoming RCP. Similar trends are observed as seen for outgoing RCP.
Discussion. Overall, the analyses show dramatic degree of CS-centric NLP citations.
Q4. Is there a bottom-line metric that captures the degree of diversity of outgoing citations? How did the diversity of outgoing citations NLP papers (as captured by this metric) change over time? And the same questions for incoming citations?
Figure 14 in the Appendix is a similar plot as Figure& 4, but also shows ORCP scores for CS, psy-

---

(a) O Outgoing CFDI, 1980-2020
(b) Incoming CFDI, 1980-2020
Avg.
Ling. Matn.
Psych.
Avg.
Ling.
Math.
Psych.
0.7
0.6
0.6
0.4
1980
2000
Figure 5: CFDI of NLP and the three largest fields cited by NIPP for (a) outgoing citations and (b) incoming citations that field. The macro-ave avero-everage shows CFDI for the average over all 23 fields.
Ans. To measure citational field diversity in sin- gle score, propose the Citation Field Diversity Index (CFDI). Outgoing CFDI is defined as:
dicating that incoming citations are coming largely from one (or few fields) as opposed many. Fig- ure 5 (b) shows the plot. While the incoming CFDIs for other fields, as well as the average, have been platcauing from 1980 to 2010, NLP's incoming CFDI has decreased from 0.59 to 0.42. In the 2010s, all fields declined in CFDI, but NLP had a particularly strong fall (0.42 to 0.31).
CFDI = - -IEfffields Dÿ where pf / and X E 24
(4)
(5)
(6)
where zy is the number of papers in field f. and N is the total number of citations. CFDI has range of [0, IJ. Scores close to indicate that the num- ber of citations from the target field (in our case. NLP) each of the 23 fields is roughly uniform. A score of O indicates that all the citations go just one field. Incoming CFDI calculated similarly, except by considering citations from other fields to the target field (NLP).
Discussion. The decline in both incoming and out- going field diversity within the NLP domain indi- cates significantly less cross-field engagement and reliance existing NLP / CS research. In contrast, other large fields like math and psychology have maintained or increased CFDI scores. NLP was a much smaller field in the 80s and 90s compared to more established fields like psychology or math; and NLP has had marked trajectory of growth that may have been one of the driving forces of the observed trends. Nonetheless, this stark contrast between NLP and other fields should give the NLP community pause and ask whether we engaging enough with relevant literature from various fields.
Results. The overall outgoing CFDI for NLP pa- pers 0.57, and the overall incoming CFDI 0.48. Given the CS dominance in citations observed car- lier in the paper, is not surprising that these scores are closer to 0.5 than to 1.
Figure 5 (a) shows outgoing CFDI over time for NLP papers. It also shows CFDIs for linguistics, math, and psychology (the top 3 non-CS fields citing and cited by NLP), as well as the macro average of the CFDIs of all fields. Observe that while the average outgoing CFDI has been slowly increasing with time, NLP has experienced a swift decline over the last four decades. The average outgoing CFDI has grown by 0.08 while NLP's outgoing CFDI declined by about 30% (0.18) from 1980-2022 2020. CFDI for linguistics and psychology follow similar trend to the average while math has experienced large increase in outgoing CFDI of 0.16 over four decades.
Q5. Given that a paper can belong to one or more fields, on average, what the number of fields that an NLP paper belongs (how interdisciplinary is NLP)? How does that compare to other academic fields? How has that changed over time?
Ans. We determine the number of field labels each NLP paper has and compare it to the average for papers from other fields of study.
Results. Figure 6 shows that the average number of fields per NLP paper of other fields in 1980. However, the trends for NLP and other fields from 1980 to 2020 diverge sharply. While papers in other fields have become slightly more interdisciplinary, a finding that is consistent with related work (Porter and Rafols,
comparable to that
Similar to outgoing CFDI, NLP pupers also have a marked decline in incoming CFDI over time; in-

---

Avg. Number Fields per Paper, 1960-2022
of Intra Feldd Citations per Field, 1980-2022
Other Fielss
Avg. Ling. Math. NLP Psych.
W
60%
1.6
1.4
1.2
20%
NLP
1.0
0%
1980
1990
2000
2010
2020
1960
1990
2000
2010
2020
Figure Avg. number fields per paper.
Figure 7: Intra-field citation percentage of fields.
2009: Van Noorden et al., 2015; Truc et al., 2020), NLP papers have become, on average, less concerned with multiple fields.
Discussion. The rise of within-field citations in NLP signifies increasing insularity, potentially due to its methodological specialization. It is still un- clear why NLP citing itself proportionally more than other fields. One can argue that as the body of work in NLP grows, there is plenty to cite in NLP itself; perhaps the cited work itself cites out- side work, but authors of current work do not feel the need to cite original work published in outside fields, as much as before. Yet, not clear why this occurs only in NLP and not in other fields (which are also growing).
Discussion. NLP papers demonstrate a decreas- ing trend in interdisciplinarity in contrast to other fields that show increased heterogeneity in their publications. This decline in NLP possibly reflects the field's increased specialization and focus on computational approaches, at the expense of work that holistically situates NLP systems in society. Q6. To what extent do NLP papers cite other NLP papers as opposed to outside-NLP papers?
Q7. Do well-cited NLP papers have higher cita- tional field diversity?
Ans. For NLP and each of the 23 fields of study, we measure the percentage of intra-field citations i.c., the number of citations from field to itself over citations to all fields (including itself).
Ans. We measure outgoing CFDI for each paper in nine different citation bins to distinguish highly cited papers from low cited papers: 0, 1-9, 10-49, 50-99, 100-499, 500-999, 1000-1999, 2000-4999, and 5000+ citations. To track trends of outgoing citational field diversity over time, we group papers in four time ranges: 1965-1990, 1990-2000 2000- 2010, and 2010-2020.
Results. Figure 7 shows the percentage of intra- field citations for NLP, linguistics, math, psychol- ogy, and the macro average of all 23 fields. In 1980, 5%, or every 20th citation from an NLP paper, was to another NLP paper. Since then, the proportion has increased substantially, to 20% in 2000 and 40% in 2020, trend of 10% per decade. In 2022, NLP reached the same intra-field citation percent- age the average intra-field citation percentage over all fields. 40% is still lower bound we use the AA to assign a paper to NLP. This percentage might be higher when relaxing this condition to pa- pers outside the AA. Compared to other fields, such as linguistics, NLP experienced particularly strong growth in intra-field citations after a lower score start. This is probably because NLP, field, is younger than others and much smaller during the 1980s and 90s. Linguistics's ss intra-field citation percentage has increased slowly from 21% in 1980 to 33% 2010 but decreased ever since. Math and psychology had platcauing percentage over time but recently, experienced a swift increase again.
Results. Figure 8 shows the results. Observe that for both 1965- 1990 and 1990 1990-2000, higher eited papers have higher outgoing CFDI (with few ex- ceptions). From 2000-2010, the overall outgoing CFDI is lower, as seen before in Figure (a), but the difference in CFDI between citation groups has plateaued. The trend then reverses between 2010- 2020, and higher cited papers have less outgoing citational field diversity.
Discussion. Papers may garner citations for various reasons, and those that get large amounts of cita- tions are not necessarily model papers (or perfect in every way). However, by virtue of their visibility, highly-cited papers markedly impact research and how early researchers perceive papers should be written. There have been concerns the academic

---

Average Field Diversity
Mean Average Field Diversity
1965-1990
1990-2000
2000-2010
2010-2020
0.8
Outgengg CFDI
0.7
0.8
0.5
0.4
1-9
10-40
50-99
100-480
500-090
1098-1999
2000-1980
1-9
10440
60:90
100-499
600-990
1020
1999
2000-4090
5080+
1-8
10-40
60-09
100-490
600-999
1600- 1988
2000-4998 1900-4999
5000
1-0
10-40
80-80
100-100
600-800 8000000
1000-1999 1989
-000-0000
50001
Figure 8: Outgoing citational field diversity for NLP papers in different citation groups over time.
community that early-stage researchers often cat- egorize papers not from their immediate field of study as irrelevant. This coincides with a marked increase in papers posted on arXiv, and researchers feel that it is difficcult to keep track of core NLP papers itself. Thus, engaging with literature from outside of NLP and CS may be getting sidelined even further. This recent trend highly cited pa- pers not being as diverse in their engagement with literature as those from earlier decades is alarming.
tics and sociology have diminished over time. The diversity of NLP citing papers from other fields and how NLP is cited by other fields has decreased since 1980 and is particularly low compared to other fields such as linguistics and psychology. NLP presents increasing insularity reflected by the growth of intra-field citations and a decline in multi- disciplinary works. Although historically, well- cited papers have higher citational field diversity, concerningly, this trend has reversed in the 2010s.
Q8. Is there an online tool that allows one to easily determine the diversity of fields being cited by a paper (or set of papers)?
Over the last 5 years or so, we see NLP technolo- gies being widely deployed, impacting billions of people directly and indirectly. Numerous instances have also surfaced where it is clear that adequate thought was not given to the development of those systems, leading to various adverse outcomes. It has also been well-established that a crucial ingre- dient in developing better systems is to engage with a wide ensemble of literature and ideas, especially bringing ideas from outside of CS (say, psy- chology, social science, linguistics, etc.) Against this backdrop, the picture of NLP's striking lack of engagement with the wider research literature especially when marginalized communities con- tinue to face substantial risks from its technologies is an indictment of the field of NLP. Not only are we not engaging with outside literature more than the average field or even just the same as other fields, our engagement is, in fact, markedly less. A trend that is only getting worse.
Ans. We have developed a freely accessible web- based tool to promote cognizance of disciplinary diversity in academic citations. Users can upload a paper's PDF, input an ACL Anthology or Se- mantic Scholar link (including author profiles or proceedings), and the system produces salient data and visualizations concerning the diversity of fields of the cited literature. Details in Appendix A.I.
5 Concluding Remarks
In this work, we examined the citational influence between NLP and various fields of study. We cre- ated a distinct dataset of metadata that includes 77k NLP papers, citations to these papers (and the fields the citing papers belong to), and citations by the NLP papers (and the fields the cited papers belong to). We analyzed this data using various metrics such as Citational Field Diversity Index and Relative Citational Prominence to show that, unlike most other fields, the diversity of both incoming and outgoing citations to and from NLP papers has steadily declined over the past few decades.
The good news is that this can change. It is a myth to believe that citation patterns are "meant to happen",, or that individual researchers and teams do not have a choice in what they cite. We can ac- tively choose what literature we engage with. We can work harder to tie work with ideas in lin- guistics, psychology, social science, and beyond. We can work more on problems that matter to other fields, using language and computation. By keep- ing an eye on work in other fields, we can bring their ideas to new compelling forms in NLP.
Our experiments show that NLP citations are dominated by CS, accounting for over 80% of ci- tations, with particular emphasis on AI, ML, and IR. Contributions from non-CS fields like linguis-

---

Limitations
Much of this work on the papers from the ACL Anthology (AA). It includes papers published in the family of ACL conferences as well as in other NLP conferences such as LREC and RANLP. However, there exist NL.P papers outside of AA as well, e.g., in AI journals and regional conferences.
As stated earlier, mechanisms of field-to-field in- fluence are complex and subject to a variety of influences such as social factors (Cheng et al., 2023), centralization of the field on benchmarks and software (Gururaja et al., 2023), or publishing structures (Kim et al., 2020). This paper primarily makes use of citations to quantify influence which comes with many limitations. One such limitation is the lack of nuance that arises from counting cita- tions; not all citations are equal-some cited work may have been much more influential to the cit- ing work than others (Zhu et al., 2015). Further, citation patterns are subject to various biases (Mo- hammad, 2020a; loannidis et al., 2019; Nielsen and Andersen, 2021).
Ethics Statement
As most of experiments use the number of ci- tations as a proxy to characterize research fields, some concerns should be discussed to avoid mis- interpretation or misuse of our findings. The low number of ingoing or outgoing citations should not be used to justify diminishing a particular field or removing their investments. Because of our choice of citations as primary metric, the other dimen- sions of our study, i.e., CFDI, influence, diversity, and insularity, are also subject to the same concerns. Decisions involving science should be evaluated through more than one aspect. Although still im- perfect, a multi-faceted evaluation incorporating characteristics such as popularity, relevance, re- sources available, impact, location, and time could help to mitigate the problems of shallow analysis.
Although this current work relies on the assign- ment of hard field labels to papers, in reality, a pa- per may have a fuzzy membership to various fields. Work on automatically determining these fuzzy memberships will be interesting, but we leave that for future work. Similarly, there is no universally agreed ontology of fields of study, and different such ontologies exist that differ from each other in many ways. We have not done experiments to determine if our conclusions hold true across other ontologies. To counter these limitations, there exist works to study interdisciplinary and impact with- out bibliometrics (Saari, 2019; Carnot et al., 2020; Rao et al., 2023), and future work should explore upplying these approaches to our questions.
A Acknowledgements
This work was partially supported by the DAAD (German Academic Exchange Service) under grant no. 9187215, the Lower Saxony Ministry of Sci- ence and Culture, and the VW Foundation. Many thanks to Roland Kuhn, Andreas Stephan, Amnika Schulte- Hürmann and Tara Small for their thought- ful discussions.
For this work, rely on the S2FOS classifier categorize our 209m papers into research fields. Although S2FOS accuracy at 86% for its 23 fields (c.g., medicine, biology), other alternatives such as S2AG (Kinney et al., 2023) can also be incorporated to obtain more accurate results. The first-level subfields attributed to the papers in our CS corpus use the CSO (Salatino et al., 2020) as a proxy, which also applied in other domains (Ruas et al., 2022; Wahle al., 2022). However, other alternatives such as the Web of Science, and SCOPUS can also be considered.
References
Mohamed Abdalla, Jan Philip Wahle, Terry Lima Ruas, Aurélie Névéol, Fanny Ducel, Saif Mohammad, and Karen Fort. 2023. The elephant in the room: Ana- lyzing the presence of big tech in natural language processing research. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 13141- 13160, Toronto, Canada. Association for Computa- tional Linguisties.
Additionally, recent work suggests adjusting for the general increase in publications (Kim et al., 2020) and the citational rate in each field (Althouse et al., 2009), finding that it affected their findings of citational diversity within fields and impact factors respectively. We do not expect this to change the conclusions we draw in this work, but it might be worth verifying.
Benjamin M Althouse, Jevin West, Carl Bergstrom, and Theodore Bergstrom. 2009. Differences in im- pact factor across fields and over time. Journal of the American Society for Information Science and technology. 60(1):27-34
Ashton Anderson, Dan Jurafsky, and Daniel A. Mc- Farland. 2012. Towards a computational history of

---

the ACL: 1980-2008. Proceedings of the ACL- 2012 Special Workshop on Rediscovering 50 Years of Discoveries, pages 13-21, Jeju Island, Korea, Asso- ciation for Computational Linguistics.
Bengt Hansson. 1999. Interdisciplinarit: For what purpose? Policy Sciences, 22(4):339-343
John PA loannidis, Jeroen Baas, Richard Klavans, and Kevin W Boyack. 2019. A standardized citation metrics author database annotated for scientific field. PLoS biology, 17(8)c3000384
Mark Bear, Barry Connors, and Michael A Paradiso. 2020. Neuroscience: exploring the brain, enhanced edition: exploring the brain. Jones & Bartlett Learn- ing.
Dan Jurafsky. 2016. Ketchup, interdisciplinarity, and the spread of innovation in speech and language pro- cessing. In Interspeech 2016, 17th Annual Confer- ence of the International Speech Communication As- sociation, San Francisco, CA, USA, September 8-12, 2016, page 3111. ISCA.
Emily M. Bender, Timnit Gebru, Angelina McMillan-
Major, and Shmargaret Shmitchell. 2023. On the dangers of stochastic parrots: Can language models be too big? volume abs/2305.188 188554
Niels Bohr. 1913. I. on the constitution of atoms and molecules. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 26(151):1-25.
Lanu Kim, Christopher Adolph, Jevin D West, and Katherine Stovel. 2020. The influence of changing marginals on measures of inequality in scholarly cita- tions: Evidence of bias and resampling correction. Sociological Science, 7:314-341
Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. 2007. Large language mod- els in machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Nat- ural Language Processing and Computational Nat- ural Language Learning (EMNLP-CONE) pages 858-867, Prague, Czech Republic. Association for Computational Linguistics.
Rodney Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy. Jonathan Bragg, Alexandra Buraczyn- ski, Isabel Cachola, Stefan Candra, Yoganand Chan- drasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph Gorne, David Gra- ham, Fangzhou Hu, Regan Huff, Daniel King. Se- bastian Kohlmeier, Bailey Kuchl, Michael Langan, Daniel Lin, Haokun Liu, Kyle Lo. Jaron Lochner, Kelsey MacMillan, Tyler Murray, Chris Newell, Smita Rao, Shaurya Rohatgi, Paul Sayre, Zejiang Shen, Amanpreet Singh, Luca Soldaini, Shivashankar Subramanian. Amber Tanaka, Alex D. Wade, Linda Wagner, Lucy Lu Wang. Chris Wilhelm, Caroline Wu, Jiangjiang Yang. Angele Zamarron, Madeleine Van Zuylen, and Daniel S. Weld. 2023. The semantic scholar open data platform.
Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in com- mercial gender classification. In Conference on fair- ness, accountability and transparency, pages 77-91. PMLR.
Miriam Louise Carnot, Jorge Bernardino, Nuno Laran- jeiro, and Hugo Gonçalo Oliveira. 2020. Applying text analyties for studying research trends in depend- ability. Entropy, 22(11):1305
Tanmoy Chakraborty. 2018. Role of interdisciplinarity in computer sciences: quantification. impact and life trajectory. Scientometrics. 114:1011-103
Vincent Larivière. Yves Gingras, and Éric Archambault. 2009. The decline in the concentration of citations, 1900-2009. Journal of the American Society for Information Science and Technology, 60(4):858-66..
Mengjie Cheng, Daniel Scott Smith, Xiang Ren, Hancheng Cao, Sanne Smith, and Daniel A McFar- land. 2023. How new ideas diffuse in science. Amer- ican Sociological Review, 88(3):522-561.
Loet Leydesdorfi and Ismael Rafols. 2011. Indicators of the interdisciplinarity of jouraals: Diversity, central- ity, and citations. Journal of Informetrics, 5(1):87- 100.
Jin M Choi. 1988. Citation analysis of intra-and inter- disciplinary communication patterns of anthropology the usa. Behavioral Social Sciences Librarian, 6(3-4):65-84.
Loet Leydesdorff, Caroline S Wagner, and Lutz Born- mann. 2019. Interdisciplinarity as diversity in cita- tion patterns among journals: Rao-Stirling diversity. relative variety, and the gini coefficient. Journal of Informetrics. 13(1):255-269
Albert Einstein. 1905. Über einen die erzeugung und verwandlung des lichtes betreffenden heuristischen gesichtspunkt.
Julian Michael, Ari Holtzman, Alicia Parrish, Aaron Mueller, Alex Wang. Angelica Chen, Divyam Madaan, Nikita Nangia, Richard Yuanzhe Pang. Ja- son Phang, al. 2022. What do nlp researchers believe? results of the nlp community metasurvey. ArXiv preprint, 122852
James A Evans. 2008. Electronic publication and the narrowing of science and scholarship. science, 321(5887)395-399
Sireesh Gururaja. Amanda Bertsch, Clara Na. David Gray Widder, and Emma Strubell. 2023. To build our future, must know our past: Contextual- izing paradigm shifts in natural language processing. ArXiv preprint, abs/2310.0774
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson. Elena Spitzer, Inioluwa Deborah Raji, and Timnit

---

Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, account- ability, and transparency, pages 220-229.
Terry Ruas, Jan Philip Wahle, Lennart Küll, Saif M Mohammad, and Bela Gipp. 2022. CS Insights: A System for Analyzing Computer Science Research. ArXiv preprint, abs/22 10.06688.
Saif Mohammad. 2022. Ethics shects for Al tasks. In
Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 8368-8379, Dublin, Ireland. As- sociation for Computational Linguistics.
Mukund Rungta, Janvijay Singh, Saif M. Mohammad, and Diyi Yang. 2022. Geographic citation gaps in NLP research. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 1371-1383 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Saif M. Mohammad. 2020a. Examining citations of nat- ural language processing literature. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5199-5209, On-
Eemeli Saari. 2019. Trend analysis in ai research over time using nip techniques. B.S. thesis.
line. Association for Computational Linguistics.
Angelo A Salatino, Francesco Osborne, Thiviyan Thana- palasingam, and Enrico Motta. 2019. The CSO clas- sifier: Ontology- driven detection of research topics in scholarly articles. In Digital Libraries for Open Knowledge: 23rd International Conference on The- ory and Practice of Digital Libraries, TPDL 2019, Oslo, Norway, September 9-12, 2019, Proceedings 23, pages 296-311. Springer.
Saif M. Mohammad. 2020h. Gender gap in natural lan-
guage processing research: Disparities in authorship and citations. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7860-7870, Online. Association for Computational Linguistics.
Mathias Wullum Nielsen and Jens Peter Andersen. 2021. Global citation inequality is on the rise. Proceedings of the National Academy of Sciences, (1807)::0112201111 18.
Angelo A. Salatino, Thiviyan Thanapalasingam, Andrea Mannocci, Aliaksandr Birukou, Francesco Osborne, and Enrico Motta. 2020, The Computer Science On- tology: A Comprehensive Taxonomy of Research Areas. Data Intelligence, 2(3):399-416.
Engineering Committee on Science, Public Policy, Institute of Medicine (US), National Academies (US). Committee on Facilitating Interdisciplinary Re- search, National Academy of Engineering (US), and National Academy of Sciences (US). 2004. Facilitat- ing interdisciplinary research. National Academies Press.
cemmatically-Genrratee
W Mark Saltzman. 2009. Biomedical engineering: bridging medicine and technology. Cambridge Uni- versity Press.
Raj Kumar Pan, Sitahhra Sinha, Kimmo Kaski, and Jari Saramäki. 2012. The evolution of interdisciplinarity in physics research. Scieniffic reports, 2(1):551.
Natalie Schluter. 2018. The glass ceiling in NLP. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2793-2798, Brussels, Belgium. Association for Com- putational Linguistics.
David Pearce and Kerry Turner. 1989. Economics of natural resources and the environment. Johns Hopkins University Press.
Janvijay Singh, Mukund Rungta, Diyi Yang. and Saif Mohammad. 2023. Forgotten knowledge: Examin- ing the citational amnesia in NLP. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6192-6686666 6208, Toronto, Canada. Association for Computational Linguistics.
Max Planck. 1901. On the law of distribution of en- ergy in the normal spectrum. Annalen der physik, 4(553):1.
Alan Porter and Ismael Rafols. 2009. Is science be- coming more interdisciplinary? measuring and map- ping six research fields over time. Scientometrics, 81(3):719-745.
A Alexandre Truc, Olivier Santerre, Yves Gingras, and François Claveau. 2020. The interdisciplinarity of economics. Available at SSRN 3669335.
Alan Porter, David Roessner, and Anne E Heberger. 2008. How interdisciplinary is a given body of re- scarch? Research evaluation, (744)273-
Marco Valenzuela, Vu Ha, and Oren Etzioni. 2015. Iden- tifying meaningful citations. In Proc. of AAAL
Sandra Postel, Kamaljit Bawa, Les Kaufman, Charles H Peterson, Stephen Carpenter, David Tillman, Paul Dayton, Susan Alexander, Kalen Lagerquist, Larry Goulder, et al. 2012. Nature's 's services: Societal dependence on natural ecusystems. Island Press.
Richard Van Noorden et al. 2015. Intendisciplinary research by the numbers. Nature, 525(7569)3306 307.
Adam Vogel and Dan Jurafsky. 2012. He said, she said: Gender in the ACI Anthology. In Proceedings of the ACL-2012 Special Workshop on Rediscovering Years of Discoveries, pages 33 43, Jeju Island, Korea, Association for Computational Linguistics.
Susie Xi Rao, Peter H Egger, and Ce Zhang. 2023. Hierarchical classification of research fields in the" web science" using deep learning. ArXiv preprint, abs/2302.00390.

---

Jan Philip Wahle, Terry Ruas, Saif Mohammad, and Bela Gipp. 2022. D3: massive dataset of schol- arly metadata for analyzing the state of computer sci- ence research. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 2642-2651, Marseille, France. European Language Resources Association.
Category
Count (J)
Medicine
45 693 661
Biology
22 899 911
Computer Science
15 102 871
Chemistry
014 729
Engineering
991 616
Physies
809 343
Materials Science
12211 647
Jan Philip Wahle, Terry Ruas, Saif M Mohammad, Nor- man Meuschke, and Bela Gipp. 2023. Al Usage Cards: Responsibly Reporting Al-generated Content. ArXiv preprint, abs/2303.03886.
Psychology
594 542
Environmental Science
7115 511
Business
738
Education
197 121
Mathematics
6142 561
Economics
6335 779
Jian Wang. Bart Thijs, and Wolfgang Glänzel. 2015. Interdisciplinarity and impact: Distinct effects of variety, balance, and disparity. PloS one, 10(5):c0127298.
Political Science
5.578 326
Agricultural And Food Sciences
5 230 208
Sociology
4 4994 792
History
3 669 722
Art
3 397 837
Xiaodan Zhu, Peter Turney, Daniel Lemire, and André Vellino. 2015. Measuring academic influence: Not all citations are equal. Journal of the Association for Information Science and Technology. 66(2))408-42
Geology
3 3372 109
Geography
3 173 289
Philosophy
1 680 010
Law
9-14 678
Linguistics
880 043
Appendix
Table 2: Number of papers per field for all 23 fields.
A.1 Demo for Citation Field Diversity
A.3.1 Results for NLP Subfields
We have developed freely accessible web-based demonstration promote cognizance of disci- plinary diversity in academic citations. Users can input any paper's Semantic Scholar ID, ACL An- thology link, and PDF file, and our system yiclds salient data concerning the interdisciplinary scope of the cited literature. One can also input author profiles or proceeding links from Semantic Scholar and ACL Anthology. The interface visualizes the distribution of the CFDI for all NLP papers pub- lished until 2022, juxtaposing this with the CFDI the paper inputted by the user. Figure in the Appendix shows an overview of the demo, which is available at
We extended our analysis also for NLP subfields, with three additional questions answered in the fol- lowing.
SQ1. Which subfields within NLP have been the most frequently cited by works outside NLP? Which subfields within NLP have cited research from other fields the most?
Ans. We categorize papers into subfields of NLP by computing the 200 most frequent bigrams of paper titles and manually assigning them to one of the 24 ACL Rolling Review (ARR) categories (e.g., ethics, generation). For example, the paper titled "Large Language Models in Machine Trans- lation" (Brants al., 2007) was assigned the NLP subfield "machine translation". We add an additional category for shared tasks (e.g., SemEval tasks). We then measure the number of citations between subfields and other research fields.
https://huggingfee http:/aacccooopcce/e//aaaea co/sspacss/jpwahlle fiyldddiversity.
A.2 Supplementary Dataset Details
Table shows the number of papers per field, show- ing that medicine, with 45.7m papers, is the largest field, followed by biology with 22.9m papers. CS is the third largest field, with 15m publications. Linguistics is the smallest one, with only 880k pub- lications overall.
Results. Figure 9 shows the percentage of citations from an NLP subfield to another CS subfield (a) or non-CS research field (b). The denominator for (a) is all CS citations and for (b) non-CS citations. 9 (b) Linguistics plays significant role in non-CS, influencing lexical semantics (32%) and machine translation (30%). Often surpassing other non-CS fields, math is cited most by MI. for NLPP (35%) and machine translation (32%) over all non-CS fields. Psychology also stands out, impacting NLP
A.3 Supplemental Experimental Results
addition to the primary results presented in this paper, we describe supplementary results in the form of additional statistics and plots.

---

applications (19%), lexical semanties (16%), and sentiment analysis (16%).
were only recently recorded without subfield as- signments since the 2000s.
Discussion. The consistently high diversity in mul- tilinguality and lexical semantics suggests a broad interdisciplinary engagement, reflecting these ar- eas' inherent complexity and wide-ranging appli- cations. The declining diversity in machine trans- lation, dialogue, and interactive systems could in- dicate consolidation of research within specific methodologies or theories existing techniques rather than integrating new per- spectives. The recent resurgence in field diver- sity within machine learning for NLP is encourag- ing, as it may signal a renewed openness to cross- disciplinary influences and innovative approaches. SQ3. To what extent do papers in NLP subfields cite themselves? How does that score compare be- tween subfields?
9 (a) Shifting the focus to CS disciplines, Al' and ML have a substantial influence on the subfield of lexical semantics, with Al' contributing 32% and ML 16% of the influence. For sentiment analy- sis, a broader mix of disciplines comes into play, with AI' (24%) and data mining (16%) leading the charge, followed by information retrieval (IR) and ML, each contributing 12%.
focus on refining
Discussion. The results indicate that machine translation, ML for NLP, and lexical semantics exhibit substantial influences from various disci- plines, such as AI, mathematics, and linguistics. NLP applications have broad influence, particu- larly in psychology and medicine, spanning both CS and non-CS domains.
When connecting these subfield findings to the general trends in NLP, we can see the broader pic- ture of the field's interdisciplinary nature and evolu- tion. The influence of linguistics and sociology has decreased over time, aligning with the overall shift in NLP towards more quantitative and empirical methodologies. On the other hand, the exponen- tial increase in the influence of mathematics and the growing significance of psychology is reflected even more strongly in specific subfields.
Ans. We measure the percentage of intra-field ci- tations, i.e., the number of citations from an NLP subfield to itself over the total number of citations from that subfield. A high intra-field citation per- centage means a field is relying predominantly on works from within, while low-self citations indi- cate a high amount of reliance on works outside that field
Results. Figure 11 shows the intra-field citation percentage of NLP subfields. Machine translation has the largest amount with 65.7% while summa- rization and sentiment analysis are at 49.6% and 47 17,4%, respectively. Shared tasks, applications, or ethics, have much lower ratios of 22.7%, 16.5%. and 1,1,, respectively.
SQ2. How diversely specific subfields of NLP cite other research fields?
Ans. We measure the average CFDI for NLP.
Results. The horizontal black line at zero in Fig- ure 10 shows the average outgoing CFDI for NLP per year. While the overall citational field diversity NLP has been declining over time, as discussed in Q3, specific subfields deviate from that downward trend (i.e., they even decline more or they do not decline as much). Machine translation and ML for NLP started at average CFDI (considering all doc- umented fields) in 1990 but rapidly decreased to 0.14 (machine translation) and 0.06 (ML for NLP) in 2010. MI. for NLP came back close to average NLP diversity in 2022 (-3.4%), but machine trans- lation has remained at 10 diversity compared to the average. Dialogue and interactive systems started with much higher-thae waaier-tth---arragge diversity in 1996 ++0.11) but dropped below average in 2018 and 2022 (-0.05). Multilinguality and language di- versity are clear winners, with consistently more than a +0.10 average diversity index. It is worth noting that multilinguality and language diversity
Discussion. Some fields, such as machine trans- lation, have grown strongly over time. Therefore, more relevant works are published in a growing community, and more researchers are keen to cite work from their community. However, fields of comparable sizes, such ML for NLP, have much lower intra-field citation percentage. There- fore. fields like machine translation should be conscious about whether they are engaging with enough relevant work from fields outside of their own niche.
A.3.2 Limitations of NLP Subfield Analysis
We limited our categorization to subfields to bi- grams of NLP paper titles (80k). While in many cases, such assignments are precise, there are some caveats. We also did not account for any other n- grams in the paper's 's text, as the coverage of all

---

35
-35
Artifciaa Intellignce
27
37
23
-3
-30
Machine Leaming
14
16
12
13
Linguistics
17
30
17
12
25
-25
20
Information Retrieval
10
10
6.4
11
12
20
Psychology
13
16
11
19
16
15
Data Mining
8.6
9.5
4.7
10
15
Medicine
14
4.9
15
12
10
-10
Computer
5.7
7.3
Biology
1.5
5
Machine Laaming NLP
Semantia: Lexioal
Machine Transiaionn
NL Applcations
Santiment Analyoie
Machine Lourning NLP
Semantica Locial
Machina Trandation
AL Applicatone
Sentiment Anobysis
(a) CS fields.
(b) Non-CS Fields.
Figure 9: The percentage of citations subfield cites (a) CS subfields and non-CS fields for the five most cited fields and subfields, respectively. Percentages are using (a) citations to CS and (b) overall citations to non-CS fields as denominators.
sagg-age
% of Cit. per
PLP NLP Subfield
Dulogue and Inturacive Systers Mediin Transiaion
OthenUrinown
Machine Translation
Discourse and Progmates Summarizatin Information Retrieval and Text Mining Sentiment Amalysis
10%
Relative Outgoing
Speech and Multmoda ity Applications Resourzes and Evol NLP
0%
-0%%
rescerce metrods NLP
-20%
1980
1990
2000
2010
2020
Figure II: The percentage of intra-field citations the five most self-citing and five least self-citing NLP subfields.
Figure 10: The outgoing CFDI of the six largest sub- fields relative average annual NLP diversity.
citations to ML (3.9% to 10.1% 1% outgoing CS cita- tions). Citations to IR relevance decreased during the 1980s, but in the 1990s, returned to previous all-time highs.
possible combinations would be restrictive. Addi- tionally, the mapping of these bi-grams to one of the categories in the ACL Rolling Review (ARR) only considers the instructions of the 2023 ACL reviewer form which can change over time. We assume the instructions provided for 2023 result from an extensive discussion and curation process between the conference organizers over the years.
A.3.4 Comparison of Field Citations Overall
Figure 13 (a) shows the percentage of outgoing citations from NLP to non-CS fields and CS sub- fields with the same denominator, i.e., all outgoing citations. Using this plot, we can visualize how many citations linguistics receives from NLP in comparison to ML. Similarly, Figure 13 (b) shows the percentage of incoming citations from NLP non-CS fields and CS subfields in the same scale.
A.3.3 Diachronic Trends of CS Subfields
In the following, we provide additional diachronic analysis of Q1 for CS subfields. Figure 12 shows the citation percentage of the three most promi- nent, and two additionally selected CS subfields with steep increases declines in citation percent- age. AT' has consistently the most influence on NLP (range: 21.9%-21 27.5% outgoing CS citations). the 1980s, there has been a substantial rise in
A.3.5 Extended Results Relative Citation Prominence
Figure 14 shows the ORCP from Q3 for NLP, CS, linguistics, math, and psychology as well as the

---

(a) Outgoing Citation Percentage, 1990-2020
(b) Incoming Citation Percentage, 1990-2020
CV
DM
IR
ML
AI'
CV
DM
IR
ML
Al'
40%
40%
30%
30%
20%
20%
10%
10%
0%
1990
2000
2010
1990
2000
2010
Figure 12: The percentage of citations from (a) NLP to CS subfields and (b) subfields to NLP in relation all CS citations from and NLP. CV Computer Vision; DM Data Mining: IR Information Retrieval; ML- Machine Learning; AT Artificial Intelligence.
(a) % of Outgoing Citations vOvralll
(b) of Incoming Citations verall
15
15
10
10
5
Arúficial intelignec
Information Retrieval
Machine Learning
Data Mining
Internet
Computer Imaging And Vision
Computer Networks
Linguistics
Computer Programming Computer S System
Human Computer Interaction Software Engineering
Bioinformalics
Matherattics
Psychologyy
Information Technology
Computer Network
Robotics
Computer Communication Networks
Genetic Algorithms
Software
Sociology
Artificial Intelligence
Machine Learning
Informaioon Retrieval
Data Mining
Computer Imaging And Vision
Computter Networks
Internet
Linguistiis
Computer Programmming
Computer System
Software Engineering
Computer Interaction
blainformatcc
Psychology
Computer Network
Software
Networks
Robolics
Information Technology
Mathemattcss
Genetic Algorithis
Sociology
Computer Communication
Human Human-Comep cooon-Compllsss
Figure 13: The overall percentages of (a) outgoing citations from each field over all citations from NLP and (b) of
incoming citations from each field over all citations to NL.P.
range of minimum and maximum ORCP over all 23 general fields of study. Figure 16 and Figure 15 show the respective TRCP plots.

---

CS
lin..
Math.
NLP
Psych.
Below everage
Above pverage
Computer Science
Linguistics
Low Philosaphy
Art Agr. Food Science
Geography
Education
Mathematics
History
Geology
Engineering
Business
Political science
Materials Science
Sociology
Below everage
Above everge
Physics
Psychology
Economics
Computer Science
Env. Science
Linguistics
Chemistry
Law hilosophy
Blology
c00
Medicine
100
Art
Agr: Food S sieecc
-15%
0%
15%
30%
45%
60%
Geography E Education M Mathematics
Figure 14: Outgoing Relative Citational Prominence (ORCP) scores between NIP, CS, linguistics, math, and psychology and 23 fields of study.
H History Geology Engineering
CS
Ling. Math.
NLP
Psych.
Political Sriece
Materials Science
Socialogy
average
everage
Physics
Psychology
Computer Lingulstics
Economics
Env. Science
Chemisty
Philosophy
Biology
Medicine
Agr. Food
Geography
Education
15%
0%
15%
30%
45%
60%
Mathematics
History
Figure 16: NLP's Incoming Relative Citational Promi- nence (IRCP) scores for 23 fields of study.
Geology
Engineering
Business
Poliical
Materials
Soclology
P Physks
Psychology
Economics
Env. Science
Chemistry
Blology
Medicine
-1%
0%
15%
30%
45%
60%
Figure 15: Incoming Relative Citational Prominence (IRCP) scores between NLP, CS, linguistics, math, and psychology and all 23 fields of study.

---

Semantic Scholar
ACL Anthologyy Link
PDF File
Upload your paper
EMNLPZS-Inf Infl-innnlneeeeeeeee pdf
MB
Title Author Name Venue Name:
Who We Cite: Bridges Influence Between Natural Processing and Other Academic Fields
Numbee references
Top fields cited:
Computer Science: 14
Economics 7
Education:
CFDI
Diversity
Avg.
2.0
1.5
Density
1.0
paper
0.5
0.0
Citation Index
Figure 17: A web demo to compute field diversity metrics for paper, author, or proceeding.


Reviews:
[{"review_id": "ekcMg4dN0n", "reviewer": "Reviewer_k77Z", "summary": "", "questions": "Thank you for an interesting read -- lots to engage with! Some questions:                                                                                               \n                                                                                                                                                                        \n* There is past work indicating that several measures used for assessing citation inequality, including the Gini coefficient, may need correction to account for paper/citation increases over time (_The influence of changing marginals on measures of inequality in scholarly citations: Evidence of bias and a resampling correction_, Kim et al., Sociological Science 2020). That scenario differs slightly as the measurement is of citation inequality across papers, not fields; is a similar correction necessary for your Gini-based measure at the field level (CFDI)?                                                                                                              \n                                                                                                                                                                        \n* The x-axis ranges in charts are a bit inconsistent -- Fig. 2 & 3 are 1990-2020, Fig. 5 is 1980-2020, and Fig. 6 & 7 are 1980-2022 -- is there a particular reason for doing so? For other charts aggregating time away (e.g., Fig. 1 and 4), is the data used from the full 1965-2022 range or a similar subset as the other figures?         \n                                                                                                                                                                        \n* For the field information in Semantic Scholar & CSO, do you have a sense of whether accuracy may degrade for papers that are interdisciplinary / highly unusual to a field?                                                                                                                                                                   \n                                                                                                                                                                        \n* Not a question, but please feel free to engage with other points above and/or note if I've misunderstood/misrepresented anything from your paper in this review.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their detailed and insightful review. \n\nRe it feels odd to \"penalize\" areas of CS that actually align tightly with those goals:\n\nWe are not sure whether we got the main point here. We are not penalizing any particular field; instead, the goal was to provide information on who NLP cites and who cites NLP through multiple ways of aggregation (from and to various fields, from and to AI, etc.).\n\nRe NLP citing certain fields less frequently compared to the average across all fields is itself not necessarily a concern:\n\nYes, we agree. It is not a concern on its own that NLP cites specific fields more/less than others. This work provides the information, and it is up to the community or individual researchers to reflect on it for themselves. We will make this clearer in the text.\n\nRe rhetoric in the introduction and discussion/conclusion:\n\nThis is indeed tricky, and so we appreciate the opportunity to discuss this with you; especially given your thoughtful review and substantive engagement with our submission. Our line of thought is as follows. \nNLP technologies are being widely deployed, and numerous cases have surfaced where it is clear that adequate thought was not given before developing those systems, leading to harm, especially to marginalized communities. It has also been well-established that a crucial ingredient in developing better systems is to engage with literature and bring in team members from outside of CS (say, psychology, social science, linguistics, etc.) \nGiven this context, we think it is particularly damning for us as a community that not only are we not engaging with outside literature more than the average field or even just the same as other fields, we are in fact, doing so markedly less! And this trend is only getting worse. We will have an extra page if the paper is accepted, and we can make this point more clearly with an explicit “Discussion” section. It is better for us as a community to look critically at ourselves rather than outsiders raise critical issues.\n\nRegarding the engagement discussion in introduction:\n\nWe do not mean to exert moral judgment. However, our goal is to convey that it is a myth to think that citation patterns that have occurred were meant to happen, and we did not have a choice in that. As a scientific community, we should be willing to take responsibility for what literature we choose to engage with and what we choose to ignore. And if we believe that, then it is important to know about the broad trends in what literature we engage with and who we cite. \n\nQuestion A: Re corrections to Gini for scale. \n\nAnswer A: Thank you for bringing this to our attention. Our study measures inequality as a “per-paper” metric. We measure how unequal a single paper cites different fields in their references. Our aggregator over all papers then becomes the mean, which, in its limits, is not affected by the amount of papers. Indeed, for incoming citations to a paper, our metric could be mildly affected, and we are exploring suitable methods to correct for this impact. However, we do not expect it to change any of the broad conclusions of our work, especially since we are interested in comparisons across fields.\n\nQuestion B: Re x-axis ranges. \n\nAnswer B: Aggregated plots use the entire time range. Citation-based plots generally use 2020 as the limit because papers published within the last three years didn’t have enough time to accumulate citations—the number of citations a paper receives in AA peaks after 3 years. Plots not affected by the absolute citation count of papers (e.g., Fig 6: Avg. number of fields per paper) also use the most recent papers until 2022. We will add a comment on the difference in dates prominently in the main body of the paper.\n\nQuestion C: For the field information in Semantic Scholar & CSO, do you have a sense of whether accuracy may degrade for papers that are interdisciplinary / highly unusual to a field? \n\nAnswer C: Semantic Scholar and CSO provide results on their classification as aggregated results, not individually for multi-field papers. The variation in detection results could be higher because the domain shift is less likely to fall in the same distribution as the original field. From our non-systematic manual exploration of results, we found only very few cases in which the multi-field assignment was incorrect.\n\nOnce again, thank you for your time and for engaging with the material in this paper and our response. We see the time and thought you have put into this and very much appreciate it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper offers valuable meta-analysis of NLP in- and out-citation trends: NLP is a fundamentally interdisciplinary field that has broad applications due to the ubiquity of language, and having a better understanding of research influence in both directions can help increase awareness of potential insularity.                       \n                                                                                                                                                                        \n* No lack of substance in the paper -- the analysis goes in depth examining in- and out-bound citation percentages (absolute and in relation to norms across fields), diversity in cited fields based on the Gini index, whether well-cited papers tend to have high citational field diversity, and in the appendix additional results on subfields of NLP. (The 8 page limit is a major constraint here.)                                                                                                                                    \n                                                                                                                                                                        \n* The writing is generally clear and the paper is well-organized.", "reasons_to_reject": "I think most of my concerns with this paper stem from the \"position paper\" framing, mostly in the introduction and conclusion, that asserts interdisciplinary influence from other fields to NLP and vice versa (as a function of citation diversity) as ethical imperative. I don't personally disagree with that assertion, but the analysis presented in section 3 feels somewhat disjoint from that:                                                                                                                  \n\n* I didn't fully understand why the emphasis between CS and non-CS in considering interdisciplinarity -- my instinct would be toward focusing on AI/ML vs. not (if NLP has simply been subsumed into ML to the detriment of everything else) or something like quantitative/statistical vs. not (if NLP has become overwhelmingly quantitative, with loss of linguistics/psychology/other grounding). Given the motivation to better account for social risks/implications, it feels odd to \"penalize\" areas of CS that actually align tightly with those goals (e.g., HCI and security). I would love to see a bit more nuance in that aspect even if the results don't significantly change.  \n\n* Similarly, NLP citing certain fields less frequently compared to the average across all fields is itself not necessarily a concern -- e.g., for more distant fields (e.g., agriculture & food science, materials science, and geography), it is not unreasonable that NLP cites them less frequently than other fields do (Figure 4) -- a bit more nuance in why fields may be cited (methods, substantive grounding, ethical concerns, etc.) may help clarify when lower-than-average / decreased citation proportion is indeed concerning.\n                                                                                                                                                                        \n* I felt that the rhetoric in the introduction and discussion/conclusion verged towards assumption of moral judgement and/or popular opinion that detracted from the substance of the paper. (Tone is of course subjective; though the argumentation could be refined further, I don't think this itself is a strong reason for rejection.) \n                                                                                                                                                                        \n  - Arguments are sometimes attributed to community opinion in ways that should warrant either specific citation or reframing to come from the authors (all italics mine): \"While _some have argued_ that CS dominance has become extreme (and harmful)...\" (L124); \"_There have been concerns in the academic community_ that early-stage researchers often categorize papers not from their immediate field of study as irrelevant... _researchers feel_ that it is difficult to keep track of core NLP papers itself\" (L536).                                                                                                                                                                \n                                                                                                                                                                        \n  - The paper ends with: \"this striking lack of engagement with the wider research literature, especially when marginalized communities face substantial risks, is an indictment of the field of NLP\" (L590). The results certainly support trends towards increased citation insularity of NLP (alone and in comparison to other fields), but it is a tall jump from that to moral indictment. (A devil's advocate question: in this measurement setup, what might we deem to be sufficient interdisciplinary citation to minimize such hazards? e.g., constant ratio over time, similar ratio to other fields, etc.) \n\n_edit/clarification: please interpret the length of this section more as \"this paper was engaging and I had some contrary thoughts\", not \"these are major reasons to reject.\"_", "questions_to_authors": "Thank you for an interesting read -- lots to engage with! Some questions:                                                                                               \n                                                                                                                                                                        \n* There is past work indicating that several measures used for assessing citation inequality, including the Gini coefficient, may need correction to account for paper/citation increases over time (_The influence of changing marginals on measures of inequality in scholarly citations: Evidence of bias and a resampling correction_, Kim et al., Sociological Science 2020). That scenario differs slightly as the measurement is of citation inequality across papers, not fields; is a similar correction necessary for your Gini-based measure at the field level (CFDI)?                                                                                                              \n                                                                                                                                                                        \n* The x-axis ranges in charts are a bit inconsistent -- Fig. 2 & 3 are 1990-2020, Fig. 5 is 1980-2020, and Fig. 6 & 7 are 1980-2022 -- is there a particular reason for doing so? For other charts aggregating time away (e.g., Fig. 1 and 4), is the data used from the full 1965-2022 range or a similar subset as the other figures?         \n                                                                                                                                                                        \n* For the field information in Semantic Scholar & CSO, do you have a sense of whether accuracy may degrade for papers that are interdisciplinary / highly unusual to a field?                                                                                                                                                                   \n                                                                                                                                                                        \n* Not a question, but please feel free to engage with other points above and/or note if I've misunderstood/misrepresented anything from your paper in this review.", "paper_topic_and_main_contributions": "This meta paper examines influence between NLP and other fields through the lens of citation frequency, and in doing so calls for increased engagement with other fields as an important aspect of responsible NLP research. Contributions include:                                                                                             \n                                                                                                                                                                        \n* Publication, citation, and field of study data compiled from the ACL Anthology and Semantic Scholar. (The paper also indicates that a tool for exploring this data will be made available upon acceptance.)                                                                                                                                   \n* Suggested metrics for measuring in- and out- direction citations for a field relative to others (incoming/outgoing relative citational prominence) and citation field diversity.                                                                                                                                                              \n* Analysis using said metrics to address a number of research questions: e.g., diversity of NLP citations to itself vs. other fields (CS or not), citation frequency by papers in other fields to NLP, changes in citation patterns over time, and other related effects.                                                                       \n* Results demonstrate increased insularity of NLP relative to other fields.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "* A recent (contemporaneous) paper that may be of interest for citing community opinion: _What do NLP researchers believe? Results of the NLP community metasurvey_ (Michael et al., ACL 2023).                                                                                                                                                 \n                                                                                                                                                                        \n* Assorted work on citation networks from Jevin West's group may be relevant, e.g. _The influence of changing marginals on measures of inequality in scholarly citations: Evidence of bias and a resampling correction_ (Kim et al., Sociological Science 2020), mentioned above; _Bursting scientific filter bubbles: Boosting innovation via novel author discovery_ (Portenoy et al., CHI 2022), which introduces a system to enable discovery of other researchers outside one's \"bubble\".", "Typos_Grammar_Style_and_Presentation_Improvements": "Minor suggestions/errors:                                                                                                                                               \n                                                                                                                                                                        \n* Some words are unnecessarily capitalized (e.g., \"language processing\", \"less\", \"linguistics\" in the abstract).                                                        \n* Table 1 hyphenates in-citations and out-citations; perhaps consider doing the same for in-cites and out-cites more broadly (incites is a very different word! got a bit tripped up every time I saw it).                                                                                                                                      \n* Fig. 14 & 15: the colors for math & CS read as basically identical to me."}}, {"review_id": "aPMtHFmSv0", "reviewer": "Reviewer_GqaD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for finding the research questions raised to be interesting and the paper to be well-written.\n\nRegarding the use of citations to measure influence:\n\nThis study aimed to measure **citational influence** between fields. This is because even though one field can influence another field in various ways, the amount of citations to a field is a direct and substantive indicator of influence. In fact, it is hard even to imagine what else one can use that will be as effective in determining influence other than citations. Engagement with literature, researchers, or scientific events from other fields eventually leads to citations from one work to another. Of course, we welcome ideas for other measurable markers of influence that we can add (in addition to looking at citations).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and the research questions interesting, even though the results somehow expected, mainly for some aspects (e.g., the domination of computer science citations).", "reasons_to_reject": "As stated also by the authors in the Limitations Section, the paper primarily makes use of citations to quantify influence.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper analyses citations of/in NLP papers to assess the influence of NLP research on/from other fields, including a diachronic analysis.\nTo conduct such an investigation, authors propose a dataset of metadata associated with ∼77k NLP papers, ∼3.1m citations from NLP papers to other papers, and ∼1.8m citations from other papers to NLP papers. Notably, the metadata includes the field of study and year of publication of the papers cited by the NLP papers, the field of study and year of publication of papers citing NLP papers, and the NLP subfields relevant to each NLP paper.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fxc8iWnor6", "reviewer": "Reviewer_H61G", "summary": "", "questions": "Question A: The method for collecting citations, especially from other fields, is not completely clear. Are they accessible from Semantic Scholar? The actual process for getting them should be provided.\n\nQuestion B: Why are some plots ending in 2020 and not 2022?\n\nQuestion C: Would it be possible to correlate some findings with actual works or trends? For instance, any idea for the cause of the 2006(?) peak in the plot of Figure 2a? Or for the start of the decrease of outgoing linguistics citations in the plot of Figure 3a?\n\nQuestion D: As mentioned, arXiv and related services seem to have an impact on Q7 (Do well-cited NLP papers have higher citational field diversity?), possibly hiding the real picture. I would add the fact that NLP has become more applicational / it is very easy to use state-fo-the-art tools out-of-the-box, so researchers that use these tools tend not to cite them or end up just including their URL.\nWould it be possible to speculate on the trend after 2022, when virtually every researcher heard of NLP and the usage of Large Language Models exploded?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and for finding the work to bring out interesting insights on the influence between NLP and other academic fields. We also thank you for highlighting its \"potential of raising very interesting discussions\" within the academic community.\n\nQuestion A: Re: method for collecting citations\n\nAnswer A: The citation information (which paper cites which paper, year of publication of a paper, etc.) is accessible through SemanticScholar’s data dump, which is regularly updated and openly available. We will provide the source code on GitHub. We will also update the paper with additional details to clarify the data processing so that the work can be easily reproduced.\n\nQuestion B: Why are some plots ending in 2020 and not 2022?\n\nAnswer B: Citation-based plots generally use 2020 as the limit because papers published within the last three years didn’t have enough time to accumulate citations—the number of citations a paper receives in AA peaks after 3 years. Plots not affected by the absolute citation count of papers (e.g., Fig 6: Avg. number of fields per paper) also use the most recent papers until 2022. We will make this clearer in the main body of the paper.\n\nQuestion C: Would it be possible to correlate some findings with actual works or trends? For instance, any idea for the cause of the 2006(?) peak in the plot of Figure 2a? Or for the start of the decrease of outgoing linguistics citations in the plot of Figure 3a?\n\nAnswer C: We enjoyed the idea of pinpointing specific works to trends in our study. For example, one could measure which highly cited papers have led to much more/less linguistics citations over time. We will include that thought for our future work. \n\nQuestion D: NLP has become more applicational / it is very easy to use state-fo-the-art tools out-of-the-box, so researchers that use these tools tend not to cite them or end up just including their URL. Would it be possible to speculate on the trend after 2022, when virtually every researcher heard of NLP and the usage of Large Language Models exploded?\n\nAnswer D: We think tracking citations of other fields to NLP is especially interesting going forward. Given the tremendous and widespread impact of LLMs, we expect a much higher amount of citations and a much wider set of fields using and citing NLP technologies. As you point out, perhaps *some* of these works will use NLP technologies but not cite NLP papers. However, we cannot predict the extent to which this will happen. Our guess though is that we will still see a marked rise in citations to NLP papers from other fields. In that regard, the results in this paper act as a great comparison point – and we hope future work will revisit the questions raised here for papers published between 2022 and a later point in time.\n\nOnce again, thank you for your time and for engaging with the material in this paper and our response. We see the time and thought you have put into this and very much appreciate it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A paper with the potential of raising very interesting discussions.\n\nUseful insights on NLP and its connection to other academic fields, illustrated by very nice graphics.\n\nResults are accessible through a web application, which further provides visualisations for given papers.", "reasons_to_reject": "Discussions are very limited.\n\nRelated Work is not focused on previous studies on research trends for specific areas, their methodologies and adopted metrics.", "questions_to_authors": "Question A: The method for collecting citations, especially from other fields, is not completely clear. Are they accessible from Semantic Scholar? The actual process for getting them should be provided.\n\nQuestion B: Why are some plots ending in 2020 and not 2022?\n\nQuestion C: Would it be possible to correlate some findings with actual works or trends? For instance, any idea for the cause of the 2006(?) peak in the plot of Figure 2a? Or for the start of the decrease of outgoing linguistics citations in the plot of Figure 3a?\n\nQuestion D: As mentioned, arXiv and related services seem to have an impact on Q7 (Do well-cited NLP papers have higher citational field diversity?), possibly hiding the real picture. I would add the fact that NLP has become more applicational / it is very easy to use state-fo-the-art tools out-of-the-box, so researchers that use these tools tend not to cite them or end up just including their URL.\nWould it be possible to speculate on the trend after 2022, when virtually every researcher heard of NLP and the usage of Large Language Models exploded?", "paper_topic_and_main_contributions": "This is a study on Natural Language Processing (NLP) research trends with a special focus on academic fields typically cited by or citing NLP papers.\nIt is based on metadata from the ACL Anthology and Semantic Scholar, and provides very interesting insights on the trends, overall and its evolution since 1980, which really makes us think.\n\nThe paper is structured on several questions and their answers, illustrated with nice plots. However, in most cases, the answers are limited to describing the contents of plot, with very limited discussion (Q6 and Q7 are an exception) and extrapolated conclusions. It would be interesting to go deeper on the possible causes of some observed trends, even if speculating, to some extent.\n\nConsidering the previous limitation as well as the plots left in the Appendix, I wonder whether this paper is better suited for a journal than for a conference.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Not that it is imperative to cite them, but the following are examples that analyse (or try to predict) research trends in different areas, generally focusing on used text, which might be worth looking at:\n\nDietz, L.; Bickel, S.; Scheffer, T. (2007). Unsupervised prediction of citation influences. ACM Int. Conf. Proc. Ser. 2007, 227, 233–240.\n\nFrancopoulo, G.; Mariani, J.; Paroubek, P. (2016). Predictive modeling: Guessing the NLP terms of tomorrow. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), Portorož, Slovenia, 23–28 May 2016; pp. 336–343.\n\nSaari, E. (2019). Trend Analysis in AI Research. Ph.D. Thesis, Tampere University, Tampere, Finland\n\nCarnot, M. L., Bernardino, J., Laranjeiro, N., & Gonçalo Oliveira, H. (2020). Applying text analytics for studying research trends in dependability. Entropy, 22(11), 1303.", "Typos_Grammar_Style_and_Presentation_Improvements": "Why NLP -> Why Natural Language Processing (NLP)\nfocus on Natural Language Processing (NLP) -> focus on NLP\n\nThe Introduction should end with a description of the paper structure.\n\nPart of the Related Work is not really \"related work\". For instance, given that the study does not focus specific publications, are the citations to examples of \"synergistic interaction\" of multiple disciplines really necessary?\n\nSome acronyms like IR and CV are not introduced in the body of the paper (only in the Appendix)."}}]

Author rebuttal:

Meta-review: This manuscript examines how deeply NLP research engages with other fields as approximated by citations outside of NLP venues. The manuscript provides in-depth analysis of some of their findings. 

In general, reviewers find many reasons to accept this manuscript and only very few reasons not to, some of which are an excellent basis for further work investigating this. Indeed, the limitation section covers many of the limitations identified by the reviewers. 
The most compelling reason for rejecting the manuscript is that certain analyses sections are primarily engaged with describing the figures rather than discussing the results themselves. However, this seems very easily addressable by the CR deadline.

Moreover, it's unclear to me why the authors rely on classification, when publication venues (e.g., ACM Transactions on Asian and Low-Resource Language Information Processing or GeoJournal) often explicitly name the field in the name. This information should be available from both citations and the publication venue of different papers. Using a classifier seems like an unnecessary way to introduce error into the analysis.
Decision: Accept-Main