To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing
Authors: Sireesh Gururaja
Amanda Bertsch
Clara Na
David Gray Widder
Emma Strubell
Venue: EMNLP 2023 Main (2023)
Keywords: paradigm shift
future of nlp research
incentives
benchmarking
software
science of science
TL;DR: We study how recent paradigm shifts in NLP are similar and different from historical shifts in the field.

Abstract: NLP is in a period of disruptive change that is impacting our methodologies, funding sources, and public perception.
In this work, we seek to understand how to shape our future by better understanding our past. 
We study factors that shape NLP as a field, including culture, incentives, and infrastructure by conducting long-form interviews with 26 NLP researchers of varying seniority, research area, institution, and social identity.
Our interviewees identify cyclical patterns in the field, as well as new shifts without historical parallel, including changes in benchmark culture and software infrastructure.
We complement this discussion with quantitative analysis of citation, authorship, and language use in the ACL Anthology over time.
We conclude by discussing shared visions, concerns, and hopes for the future of NLP.
We hope that this study of our field's past and present can prompt informed discussion of our community's implicit norms and more deliberate action to consciously shape the future.

To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing
Sireesh Gururaja Amanda Bertsch Clara Na
David Gray Widder Emma Strubell
I Language Technologies Institute, Cameeie Mellon University, Pittsburgh, PA, USA
Digital Life Initiative, Cornell Tech, Cornell University, New York City, NY, USA
3 *Allen Institute for Artificial Intelligence, Seattle, WA, USA
(sgururaj, abertsch, csna, estrubel]ee j@cs. cmu cmu.edu, edu, david. g. wi wwwderr@gaiil.ccmmmm com
Abstract
Unique authors publishing in *CL venues
1+ papers
NLP is in period of disruptive change that is impacting our methodologies, funding sources, and public perception. In this work, we seek to understand how to shape our future by better understanding our past. We study factors that shape NLP as field, including culture, incen- tives, and infrastructure by conducting long- form interviews with 26 NLP researchers of varying seniority, research area, institution, and social identity. Our interviewees identify cycli- cal patterns in the field, as well new shifts without historical parallel, including changes in benchmark culture and software infrastructure. We complement this discussion with quantita- tive analysis of citation, authorship, and lan- guage use in the ACI. Anthology over time. We conclude by discussing shared visions, con- cerns, and hopes for the future NLP. We hope that this study of field's past and present can prompt informed discussion of our com- munity's implicit norms and more deliberate action to consciously shape the future.
21 papers
papers
13000
S+ papers
12500
10000
3000
2500
Figure 1: The number of unique researchers publishing in ACL venues has inereased dramatically, from 715 unique authors in 1980 to 17,829 in 2022.
of dubious scientific value," and that Al technol- ogy could lead a catastrophic event this century (Michael et al., 2022). More recently, there has been discussion of the increasing prevalence closed-source models NLP and how that will shape the field and its innovations (Rogers, 2023; Solaiman, 2023; Liao and Vaughan, 2023). In order to tackle these big challenges, we must understand the factors norms, incentives, technology and culture that led to our current crossroads.
1 Introduction
Natural language processing (NLP) is in a period of flux. The successes of deep neural networks and large language models (LLMs) in NLP coin- cides with shift not only in the nature of our research questions and methodology, but also in the size and visibility of our field. Since the mid- 2010s, the number of first-time authors publishing the ACL Anthology has been increasing expo- nentially (Figure 1). Recent publicity around NLP technology, most notably ChatGPT, has brought our field into the public spotlight, with correspond- ing (exitee) and scrutiny.
We present study of the community in its cur- rent state, informed by series of long-form ret- rospective interviews with NLP researchers. Our interviewees identify patterns throughout the his- tory of NLP, describing periods of research pro- ductivity and stagnation that recur over decades and appear at smaller scale around prominent meth- ods ($3). Interviewees also point out unparalleled shifts in the community's norms and incentives. Aggregating trends across interviews, we identify key factors shaping these shifts, including the risc and persistence of benchmarking culture ($4) and the maturation and centralization of software in- frastructure ($5). Our quantitative analysis of cita- tion patterns, authorship. and language use the ACL Anthology over time provides a complemen-
In the 2022 NLP Community Metasurvey, many NLP practicioners expressed fears that private firms exert excessive influence on the field, that "a ma- jority of the research being published in NLP is
"Denotes equal contribution.

---

tary view of the shifts described by interviewees, grounding their narratives and our interpretation in measurable trends. Through our characterization the current state of the NLP research community and the factors that have led us here, we aim to offer foundation for informed reflection on the future that we community might wish to see.
was not present in the interview applied the closed coding frame to the data. In weekly analysis meetings, new codes arose to capture new themes or provide greater specificity, in which case the closed coding scheme was revised, categories refined, and data re-coded in an iterative process. Analysis reported here emerged first from this coded data, was refined by subsequent review of raw transeripts check context, and developed in discussion between all authors.
2 Methods
2.1 Qualitative methods
We recruited 26 researchers to participate in inter- views using purposive (Campbell et al., 2020) and snowball sampling (Parker et al.. 2019): asking participants to recommend other candidates, and purposively selecting for diversity in affiliation, ca- reer stage, geographic position, and research area (see participant demographics in Appendix A.I). Our sample had 69-31% % academia-industry split, was 19% women, and 27% of participants identi- fied as part of a minoritized group in the field. Of our academic participants, we had a near-even split of carly-, mid-, and late-career researchers; indus- try researchers were 75% individual contributors and 25% managers.
2.2 Quantitative Methods
We use quantitative methods primarily as coher- ence check on our qualitative results. While our work is largely concerned with the causes and com- munily reception of changes in the community, our quantitative analyses provide evidence that these changes have occurred. This includes analyzing authorship shifts (Figures I and 4), citation pat- terns (Figure 2), terminology use (Figure 2,3) in the ACL anthology over time; for more details on reproducing this analysis, see Appendix B.
Exploit-explore cycles of work
Our participants described cyclical behavior in NLP research following methodological shifts ev- ery few years. Many participants referred to these methodological shifts "paradigm shifts"", with similar structure, which we characterize as explore and exploit phases.
Interviews were semi-structured (Weiss, 1995), including a dedicated notetaker, and recorded with participant consent, lasting between 45-73 minutes (mean: 58 minutes). Interviews followed an in- terview guide (see Appendix A.3) and began by contrasting the participant's experience of the NLP community the start of their career and the cur- rent moment, then moved to discussion of shifts in the community during their career. Interviews were conducted between November 2022 and May 2023; these interviews were coincidentally contempora- neous with the release of ChatGPT in November 2022 and GPT-4 in March 2023, which frequently provided points of reflection for our participants. Following procedures of grounded the- ory (Strauss and Corbin, 1990), the authors present for the interview produced analytical memos for early interviews (Glaser et al., 2004). As interviews proceeded, authors began a process of independently open coding the data, an interpre- rive (Lincoln et al., 2011) 1) analytical process where researchers assign conceptual labels to segments of data (Strauss and Corbin, 1990). After this, authors convened discuss their open codes, systematizing recurring themes and contrasts to construct a preliminary closed coding frame (Miles and Huberman, 1994). After this, an author who
First wave: exploit. Participants suggested that after a key paper, wave of work is published that demonstrates the utility of that method across varying tasks benchmarks. Interviewees vari- ously describe this stage "following the band- wagon" (17), "land grah stuff" (9), or "picking the low-hanging fruit" (8). A researcher with prior experience computer vision drew parallels be- tween the rise of BERT and the computer vision community after AlexNet was released, where "it felt like every other paper was, 'I have fine luned ImageNet trained CNN some dataset' (17). However, participants identified benefits of this "initial wave of showing things work" (17) in demonstrating the value of techniques across tasks or domains; in finding the seemingly obvi- ous ideas which do not work, thus exposing new areas to investigate; and in developing downstream applications. When one researcher was asked to
'Inspired by the verbiage of reinforcement learning, egg. Sutton and Barto (2018).

---

Fro suaning
Frampting
Maskram Enerayy
Attertion
FTR
Rand/eec
Jenry Leariing to Alge Theileee
3955
3536
Pirt Partle
Seqp
REXT. CAT
Sessions, ENILPP
Whrninn and Schuttee
Symbels Mathods Daminake
Matrics become
DANIS
bunzra revelution NuP
Mechine Trandiaarr Models
Puring MT dominate *DL conferences
Pramaratic basett maurral nomarth
Nonay neural mothods the us
Statittical NLF (insluding tapic models PCMls deminatte
Word ambeddiran fusl SD1A chasing
Diecussiona data scale everything
Tre-rain Ims-tune
work meuraa NLP
Cumer alion over cheallfattion
Prampting
Figure Quantitative and qualitative timeline. The lower half of this diagram captures historical information that participants felt was relevant, along with their reported date ranges. The upper half captures quantitative information that parallels that timeline. Bar charts indicate fraction of papers that cite a given paper, while line charts indicate the fraction of papers that use particular term.
identify their exploit work, they joked could simply "sort [their] Google Scholar from most cited least cited" (9), describing another incentive to publish the first paper applying new methodology to a particular task. Participants ad- ditionally described doing exploit work early in their careers as a hedge against riskier, longer-term projects. However, most participants ascribed low status to exploit work?, with participants calling these projects "low difficulty" (4) and "obvious idea[s]" (9) with "high probability of success" (4). Participants also discussed increasing risk of being "scooped" (7,9,4) 4) on exploit work as the community grows larger.
work may focus interpretability, bias, data compute efficiency. While some participants see this as time of "stalled" (8) progress, others de- scribed this as "the more interesting research after the initial wave of showing things work" (18). A mid-career participant identified this as the work they choose to focus on: "I'm at the stage of my career where don't 't want to just push numbers, you know. I'll let the grad students do that. want do interesting stuff" (22). Participants of- ten saw "pushing numbers" as lower-status work, appropriate for graduate students and important for advancing the field and one's career, but ultimately not what researchers hope to explore.
Participants felt that the exploit phase of work is not sustainable. Eventually, "the low hanging fruit has been picked" (8); this style of work be- comes unsurprising and less publishable. As one researcher put it: "If fine tune (BERT) for some new task, it's probably going to get some high ac- curacy. And that's 's fine. That's the end." (17).
Some work to improve benchmark performance was also perceived as explore work, particularly when it involved developing new architectures. One participant described distinction between "entering a race" and "forging in new direction" (4) with a project, which focuses the exploit/explore divide more on the perceived surprisingness an idea rather than the type of contribution made. Ex- ploration often leads to a new breakthrough, causing the cycle to begin anew.
Second wave: explore After some time in the exploit phase, participants described a state where obvious extensions to the dominant methodology have already been proposed, and fewer papers demonstrate dramatic improvements over the state of the art popular benchmarks.
3.1 Where are we now?
Placing the current state of the field along the exploit-explore cycle requires defining the current methodological "paradigm". Participants identi- fied similar patterns varying scales, with some disagreement on the timing of recent trends.
In this phase, work identifying the ways that the new method is flawed gains prominence. This
'Data work is also commonly perceived as low-status (Sambasivan et al., 2021); participants agreed data work was previously undervalued in NI.P but described trend of in- creasing respect, one calling dataset curation "valorized." (7)
Prompting as a methodological shift Several participants described prompting as a paradigm

---

shift or a direction that the community found promising, but most participants viewed current work on prompt engineering or "ChaIGPT for X" (9) as something that people are working on "in- stead [..]] of something that might make a funda- mental difference" (14). One participant described both prompt engineering and previous work on fea- ture engineering as "psnedoscience [...] just poking at the model" (6). The current flurry of prompting work was viewed by several participants as lower- status work exploiting a known method.
featuring greater centralization on fewer methods (see $5 for more discussion). Some expressed con- cern: "a technique shows some promise, and then more people investigate it. That's perfectly appro- priate and reasonable, but I think it happens a little too much. [...] Everybody collapses on this one approach [.-.] everything else gets abandoned." (19). Another participant described peers from lin- guistics departments who left NLP because they felt alienated by the focus on machine learning.
Issues with peer review Some felt that peer re- view was inherently biased toward incremental work because peer reviewers are invested in the suc- cess of the current methodological trends, with one participant arguing that you you want to break the paradigm and do something different, you 're gonna get bad reviews, and that's fatal these days" (21). Far more commonly, participants did not express inherent opposition to peer review but raised con- cerns because of the recent expansion of the field, with one senior industry researcher remarking that peer reviewers are now primarily junior researchers who "have not seen the effort that went into ʃear- lier] papers" (12). Another participant asserted that "my peers never review my papers" " (22). Par- ticipants additionally suggested that the pressure on junior researchers to publish more causes an acceleration in the pace of research and reinforce- ment of current norms, as research that is farther from current norms/methodologies requires higher upfront time investment.
"Era of scale" For participants who discussed larger-scale cycles, pre-trained models were fre- quently identified as the most recent methodologi- cal shift. Participants disagreed on whether scaling up pre-trained models (in terms of parameter count, training time, and/or pre-training data) was form of exploiting or exploring this method. Some par- ticipants found current approaches to scale to be "a reliable recipe where we, when we put more re- sources in, we get [...] more useful behavior and capabilities out" (4) and relatively easy to perform: " "Once you have that GPU [..]] it's like, super sim- ple" (5). This perception of scaling up as both high likelihood of success and low difficulty places it as exploit work, and researchers who described scale in this way tended to view it as exploiting "obvious" trends. One researcher described scale way of establishing what is possible but "ac- tually bad way to achieve our goals." " (4), with further (explore-wave -ave) work necessary to find effi- cient ways to achieve the same performance.
This competitiveness can manifest in harsher reviews, and one participant described "deadly combination " (19) of higher standards for papers and lower quality of reviews. Some participants described this as a reason they were choosing to engage less with NLP conferences; one industry re- searcher stated that "I just find it difficult to publish papers in *CL [venues] that have ideas in them." (22).
A minority of participants argued that, while historical efforts to scale models or extract large noisy corpora from the internet were exploit work, current efforts to scale are different, displaying "emergence in addition to scale, whereas previously we just saw [...] diminishing returns" " (22). Some participants also emphasized the engineering work required to scale models, saying that some were "underestimating the amount of work that goes into training large model" (8) and identifying people engineering teams as a major resource neces- sary to perform scaling work. The participants who described scaling work as producing surprising re- sults and being higher difficulty also described seal- ing as higher status, more exploratory work.
4
Benchmarking culture
4.1
The rise of benchmarks
Senior and emeritus faculty shared consistent recollcction of the ACL community before the prominence of benchmarks centralized around few US institutions and characterized by "pa- tient money" (21): funding from DARPA that did not require any deliverables or statements of work. Capabilities in language technologies were show-
"Deep learning monoculture" There was a sense from several participants that the current cy- cle has changed the field more than previous ones,

---

cased with technical "toy" (26, 19) demonstrations that were evaluated qualitatively: "the performance metrics were, 'Oh my God it does that? No ma- chine ever did that before. " (21). Participants repeatedly mentioned how small the community was; at conferences, "'veeybodyy knew each other. Everybody was conversing, in all the issues" (26). The field was described as "higher trust" (22). with social mediation of research quality able to function in the absence of standardization because of the strong interconnectednnss of the community.
ple together to exchange ideas. [...] I think this really helped the field to move forward." (2). Other participants similarly argued that a culture of quan- titative measurement was key for moving on from techniques that were appealing for their "elegance" (14) but empirically underperforming.
4.2 The current state of benchmarks
Roughly twenty years on from the establishment of benchmarks as a field-wide priority, our partici- pants' attitudes towards benchmarks had become significantly more complex. Many of our partici- pants still found benchmarks necessary, but nearly all of them found them increasingly insufficient.
Many participants recalled the rise of bench- marks in the late 1990s and early 2000s, coinciding with major expansion in the NLP community in the wake of the "statistical revolution," where participants described statistical models displacing more traditional rules-based work (see Figure 2). the words of one participant, the field became of "such a big snowballing size that nobody owned the first evers anymore." " (26). Instead, after the release of the Penn Treebank in 1993 and the reporting of initial results on the dataset, "the climb started" (25) to increase performance. Some participants attributed these changes an influx of methods from machine learning and statistics, while others described them as methods understand and orga- nize progress when doing this coordination through one's social network was no longer feasible.
Misaligned incentives Many participants, par- ticular early- and late-career faculty, argued that the field incentivizes the production of benchmark results to the exclusion of all clse: "the typical re- search paper...heir immediate goal has to be to get another 2% and get the boldface black eniry on the table." " (21). For our participants, improvements on benchmarks in NLP are the only results that are self-jjutiff to reviewers. Some participants felt this encourages researchers exploit model- ing tricks to get state-of-the-art results on bench- marks, rather than explore the deeper mechanisms by which models function (see $3).
Over time, this focus on metries seems to have overtaken the rest of the field, in part through the operationalization of metrics as a key condition of DARPA funding. One participant credited this to Anthony Tether, who became director of DARPA in 2001: they described earlier DARPA grants as funding "the crazy [...] stuff that just might be a breakthrough" (21) and DARPA under Tether "show me the metrics. We'r re going to run these metrics every year." (21).3
"We're 'rere solving NLP" Some participants per- ceive a degradation in the value of benchmarks be- cause of the strength of newer models. Participants appreciated both the increased diversity and fre- quency of new benchmark introduction, but noted that the time for new approaches to reach "superhu- man" (6,22) levels of performance on any specific benchmark is shortening. One common compar- ison was between part of speech tagging ("a hill that was climbed for [...] about 20 years" (15)) and most modern benchmarks ("solved" within a few years, even months). Some went further, describing "solving NLP" (8) or naming 2020 as the time when "classification was solved" (15).
Some participants mourned the risk appetite of a culture that prioritized "first-evers, criticizing the lack of funding for ideas that did not immediately perform well at evaluations (notably leading to the recession of neural networks until 2011). However, there was sharp disagreement here; many other participants celebrated the introduction of bench- marks, with one stating that comparing results on benchmarks between methods "really brought peo-
However, when participants were asked for clar- ification on what it meant to "solve" problem, most participants hedged in similar ways; that datasets and benchmarks could be solved, with the correct scoping. but problems could rarely or never be solved. Many participants argued that the standard for solving task should be human equiv- alency, and that this was not possible without new benchmarks, metrics, or task definition.
*Other participants named DARPA program managers Charles Wayne and J. Allen Sears as additional key players. A recent tribute to Wayne (Church, 2018) provides additional context reflecting on DARPA's shift in priorities in the mid-
1980s.

---

NLP in the wild Some participants argued that many benchmarks reflect tasks that "aren't that useful in the world" (13), and that this has led to a situation where "[NLP], a field that, like funda- mentally, is about something about people, knows remarkably little about people" " (3). Industry par- ticipants often viewed this distinction between their work and the academic community, with one stating that "most of the academic benchmarks there are not real tasks" " (12). Many academics articulated desire for more numan-centered NLP, and most participants described feeling pressure over the unprecedented level of outside interest in the field. One participant contrasted the interna- tional attention on ChatGPT with the visibility of earlier NLP work: "It's not like anyone ever went to like parseryahoo.comm com to parser some- thing" " (3). Participants argued that, given this out- side attention, the benchmark focus of NLP is too narrow, and that benchmarks fail to capture no- tions of language understanding that translate to wider audiences, and that we should move on from benchmarks not when they are saturated but when "it wouldn't 't really improve the world to im- prove this performance anymore" (9). This echoed common refrain: many participants, especially early- and mid-career researchers, saw positive so- cial change as a goal of progress in NLP.
Mentions of software libraries by year
0.30
HuagingRace
PyTorch
Fraction of papere thak merrion
0.25
Theano
DyNet
0.20
Moses
Tentorflaw
0.15
0.10
0.05
0.00
2010
2012
2014
2016
2018 2020
2022
Year
Figure 3: Mentions of libraries over time in the ACL Anthology. Note the cyclic pattern and increasing con- centration the dominant framework over time. While some libraries are built on others, the shift in mentions over time captures the primary level of abstraction that researchers consider important. See appendix B for de- tails on how we handle ambiguity in mentions.
simpler to manage, simpler to train" " (17). Previ- ously, participants described spending "like 90% of our time re- re-implementing papers" (12): as more papers began releasing code implemented in pop- ular frameworks, the cost of using those methods as baselines decreased. One participant stated that "things that software makes easy, people are going to do " (18); this further compounds centralization onto the most popular libraries, with little incen- tive to stray from the mainstream: "everybody uses PyTorch, so now use PyTorch too" (8); "we just use Hugging for pretty much everything" " (18). Figure 3 visualizes mentions of frameworks across papers in the ACL Anthology, showing both the rise and fall in their popularity. The rising peaks of popularity reflect the centralization over time. While some communities within NLP had previously seen some centralization on toolkits (e.g. machine translation's 's use of Moses (Koehn et al., 2007)), the current centralization transcends sub- fields.
5 Software lotteries
Hooker (2021) argues that machine learning re- search has been shaped by hardware lottery: an idea's success is partially tied to its suitability for available hardware. Several participants spoke about software in ways that indicate an analogous software lottery in NLP research: as the community centralizes in its software use, it also centralizes in methodology, with researchers' choices of methods influenced by relative ease of implementation. This appeared to be a relatively new phenomenon; partic- ipants described previously using custom-designed software from their own research group, or writing code from scratch for each new project.
Centralization on specific models Participants identified another shift after the release of BERT and subsequent development of Hugging Face. Be- cause of pre-training, participants moved from merely using the same libraries to "everyone us[ing] the same base models" (9). Participants expressed concern that this led to further centraliza-
Centralization on frameworks As deep learn- ing became more popular in NLP, participants de- scribed the landscape shifting. As TensorFlow (Abadi et al., 2015) increased support for NLP modeling, PyTorch (Paszke et al., 2019) was re- leased, along with NLP-specific toolkits such as DyNet (Neubig et al., 2017) and AllenNLP (Gard- ner et al., 2018), and "everything started being [...]
"In this section, we primarily discuss open source frame- works commonly used by academic and industry researchers. However, many of our participants working in industry also describe affordances and constraints of internal toolkits that are used at their respective companies.

---

tion in the community, with one participant identi- fying& trend of some people "not usfing] anything else than BERT [...] that's a problem" " (5). This concentration around particular modeling choices has reached greater heights than any previous concentration on method; in 2021, 46.7% of papers in the ACL anthology cited BERT (Devlin et al. (2019); Figure 2)².
citing the lack of well-supported NLP toolkits or community use in other languages. This an in- stance of a software lottery at higher level, where the dominance of a single programming language has snowballed with the continued development of research artifacts in that language.
5.1 Consequences of centralization
Other large models are only available to re- scarchers via an API. One participant who works on LLMs in industry argued that black-box LLMs, while "non-scientific" (15) in many ways, were like large-scale technical tools constructed in other disciplines, drawing a parallel to particle physies: "computer science is getting to have its CERN mo- ment now [..]] there's 's only one Large Hadron Col- lider, right?" (15). This participant argued that NLP has become a field whose frontiers require tools that are beyond most organizations' resources and capabilities to construct, but nonetheless are widely adopted and set bounding parameters for future research as they see wide adoption. In this vision, black-box LLMs take on the same role as the LHC or the Hubble Space Telescope (both no- tably public endeavors, unlike most LLMs), as tools whose specifications are decided on by a few well-rexoureed organizations who shape significant parts of the field. But most participants expressed skepticism about the scientific validity of experi- ments on black-box LLMs, with one participant referencing critiques of early-2000s IR research on Google (Kilgarriff, 2007).
This increasing centralization of the modern NLP stack has several consequences. One of the primary ones, however, is the loss of control of design deci- sions for the majority of researchers in the commu- nity. Practically, researchers can now choose from handful of well-established implementations, but only have access to software and models once the decisions on how to build them have already been reified in ways that are difficult to change.
Lower barriers Beyond time saved (re-) imple- menting methods, many participants identified a lower barrier to entry into the field as a notable ben- efit of centralization on specific software infrastruc- ture. Participants described students getting state of the art results within an hour of tackling a problem; seeing the average startup time for new students decrcasing from six months to a few weeks; and teaching students with no computer science back- ground to build NLP applications with prompting.
Obscuring what's "under the hood" One par- ticipant recalled trying to convince their carlier students to implement things from scratch in or- der to understand all the details of the method, but no longer doing so because "I don't 't think it's pos- sible [...] it's just too complicated" (11); others attributed this to speed more than complexity, stat- ing that "the pace is so fast that there is no time to properly document, there is no time to properly en- gage with this code, you't re just using them directly" (5). However, this can cause issues on opera- tional level: several participants recalled instances where a bug or poor documentation of shared soft- ware tools resulted in invalid research results. One participant described using a widely shared piece of evaluation code that made an unstated assumption about the input data format, leading to "massively inflated evaluation numbers" (3) on well-cited dataset. Another participant described working paper where they realized, an hour before the pa- per deadline, that the student authors had used two different tokenizers in the pipeline by mistake: "we decided that well, the results were still valid, and the results would only get better if was fixe]]..
Centralization on Python While most early- career and late-career participants did not express strong opinions about programming languages, many mid-career participants expressed strong dis- like for Python, describing it as "a horrible lan- guage" (22) with efficiency issues that are "an impediment to geiling things done" (20) and data structure implementations that are "a complete disaster in terms of memory" (9). One participant described their ideal next paradigm shift in the field as shift away from using Python for NLP.
Yet even the participants who most vehemently op- posed Python used it for most of their research,
'While not every paper that cites BERT uses a BERT model, this indicates how central BERT is both as a model and as a frame for the discussion of other work. For compari- son, only two other papers have been cited by more than 20% of anthology papers in a single year: "Attention is All You Need" (Vaswani et al., 2017) with 27% in 2021 and GloVe (Pennington et al., 2014) with 21% 5 in 2019.

---

the paper went out. It was published that way." (26) Software bugs in research code are not a new prob-
Disparities in compute access Many felt that building large-scale systems was increasingly out of reach for academics, echoing concerns previ- library is correct most the time" " (8), even as they spoke of finding "many, many, many" (8) bugs in toolkits including Hugginglace and PyTorch.
lem, o but participants described bugs in toolkits as difficult to diagnose because they "trust that the
ously described by Ahmed and Wahed (2020). Participants worried that "we are building an up- per class of AI" (6) where most researchers must "build on top of [large models]" (15) that they can- not replicate, though others expressed optimism that "clever people who are motivated to solve these problems" (22) will develop new efficient methods (Bartoldson et al., 2023). Industry partici- pants from large tech companics also felt resource- constrained: "modern machine learning expands to fo.fit the available compute." " (4).
Software is implicit funding Participants sug- gested that tools that win the software lottery act as a sort of implicit funding: they enable research groups to conduct work that would not be possible in the tools' absence. and many of our participants asserted that the scope of their projects expanded as result. However, they also significantly raise the relative cost of doing research that does not fall neatly into existing tools' purview. As one par- ticipant stated, "You't re not gonna just build your own system that's 's gonna compete on these major benchmarks yourself. You have to start [with] the infrastructure that is there" " (19). This is true even of putatively "open" large language models, which do not necessarily decentralize power, and can of- ten entrench it instead (Widder et al., 2023). This set of incentives pushes researchers to follow cur- rent methodological practice, and some participants feared this led toward more incremental work.
6 Related Work
The shifts we explore in this paper have not hap- pened in vacuum, with adjacent research commu- nities such as computer vision (CV) and machine learning (ML) experiencing similar phenomena, in- spiring a number of recent papers discussing norms in Al more broadly. Birhane et al. (2022) analyze a set of the most highly cited papers at recent ML conferences, finding that they eschew discussion of societal need and negative potential, instead empha- sizing limited set of values benefiting relatively few entities. Others have noticed that corporate interests have played an increasing role in shaping research, and quantified this with studies of author affiliations over time in machine learning (Ahmed and Wahed, 2020) and NLP (Abdalla et al., 2023). Su and Crandall (2021) study the tangible emo- tional impact of recent dramatic growth in the CV community by asking community members to write stories about emotional events they experienced members of their research community.
5.2 Impact on Reproducibility
A common sentiment among participants was that centralization has had an overall positive impact on reproducibility, because using shared tools makes it easier to evaluate and use others' research code. However, participants also expressed concerns that the increasing secrecy of industry research compli- cates that overall narrative: "things are more open, reproducible... except for those tech companies who share nothing" " (14).
While we focus on summarizing and synthesiz- ing the views of our participants, some of the over- arching themes identified in this work have been discussed more critically. Fishman and Hancox- Li (2022) critique the unification of ML research around transformer models on both epistemic and ethical grounds. Position papers have critiqued the notion of general purpose benchmarks for Al (Raji et al., 2021), and emphasized the importance of more careful and deliberate data curation in NLP (Rogers, 2021; Bowman and Dahl, 2021).
Shifts in expectations One participant described a general shift in focus to "making sure that you make claims that are supported rather than repro- ducing prior work exactly" (12) in order to match reviewers' shifting expectations. However, partic- ipants also felt that the expectations for baselines had increased: "fin the past,] everybody knew that the Google system was better because they were rumning on the entire Internet. But like that was not a requirement [to] match Google's 's accuracy. But now it is, right?" (8).
The NLP Community Metasurvey (Michael et al., 2022) provides a complementary view to this work, with their survey eliciting opinions from a broad swath of the *CL community on a set 32
"Tambon et al. (2023) describe silent bugs in popular deep learning frameworks that escape notice due to undetected crror propagation.

---

controversial statements related the field. The survey also asked respondents to guess at what the most popular beliefs would be, eliciting sociologi- cal beliefs about the community. While there is no direct overlap between our questions and Metasur- vey questions, participants raised the topics of scal- ing up, benchmarking culture, anonymous peer re- view, and the role of industry research, which were the subject of Metasurvey questions. Where can map between our thematic analysis and Meta- survey questions, we see agreement- e.g. many of our participants discussed others valuing scale, but few placed high value themselves on scaling up as a research contribution.
community, A shift like this can feel threatening to the fundamental nature of NLP research, but this is not the first period of flux in the field, nor are the fundamental forces enabling LLMs' dominance and other changes entirely new.
Our participants described cycles of change in the NLP community from mid-80s to the present, with common themes of first exploiting and then exploring promising methooologies. Each methodological shift brought corresponding cul- tural change: the shift from symbolic to statistical methods brought about the rise of benchmark cul- ture and the end of the socially mediated, small- network ACL community. Neural methods be- gan the centralization on software toolkits and the methodologies they support. Pre-training intensi- fied this software lottery, causing unprecedented levels of centralization on individual methods and models. Current models have called into question the value of benchmarks and catapulted NLP into the public eye. Our participants largely agree on the resulting incentives- to beat benchmark results, to do the easiest thing rather than the most fulfilling, to produce work faster and faster- while largely expressing frustration with the consequences.
The availability of the ACL Anthology has en- abled quantitative studies of our community via pat- terns of citation, authorship, and language use over time. Anderson et al. (2012) perform a topic model analysis over the Anthology to identify different cras of research and better understand how they develop over time, and analyze factors leading au- thors to join and leave the community. Mohammad (2020) analyze citation patterns in FL conferences across research topics and paper types, and Singh al. (2023) specifically inspect the phenomenon wherein more recent papers are less likely to cite older work. Pramanick et al. (2023) provide a view of paradigm shifts in the NLP community comple- mentary to ours based on a diachronic analysis of the ACL Anthology, inferring causal links between datasets, methods, tasks and metrics.
We hope that this contextualization of the cur- rent state of NLP will both serve to inform newer members of the community and stir informed dis- cussion on the condition of the field. While we do not prescribe specific solutions, some topics of discussion emerge from the themes of this work: Shifts in norms and methods in science more broadly has been studied outside computing-related fields. Most notably, Kuhn (1970) coined the term paradigm shift in The Structure of Scientific Revo- lutions. His theory of the cyclic process of science over decades or centuries has some parallels with the (shorter timescale) exploit-explore cycles dis- cussed in this work. Note that in this work, we did not prime participants with an a priori defini- tion of paradigm shift, allowing each participant to engage with the term according to their own inter- pretation, which often differed from Kuhn's notion of paradigm shift.
Who holds the power to shape the field? How can a broad range of voices be heard?
Do the incentives in place encourage the be- havior we would like to see? How can we improve reviewing to align with our values? What affects the ability to do longer-term work that may deviate from current norms? How can the community arrive at an actively mediated consensus, rather than passively be- ing shaped by forces like the ones we discuss?
We personally take great hope for our commu- nity from this project. The care with which all participants reflected on the shape of the field sug- gests to us that many people are concerned about these issues, invested in the community, and hope- ful for the future. By sharing publicly what peo- ple so thoughtfully articulate privately, we hope to prompt further discussion of what the community can do to build our future.
7 The Future
The rise of large language models has coincided with disruptive change in NLP: accelerating central- ization of software and methodologies, questioning of the value of benchmarks, unprecedented public scrutiny of the field, and dramatic growth of the

---

Limitations
early career researchers and those who have not yet published but are interested in NLP research.
Western bias The most notably irrepresentative sampling bias in our participant pool is the lack of non-Western institutional affiliation (and the strong skew toward North American affiliations). This bias has arisen likely in part due to our own institu- tional affiliation and conceptions of the community. That being said, given the Association for Compu- tational Linguisties' historically US- and English- centric skews, this allows us to gather historical perspectives. Additionally, considering that West- emn institutions constitute a citation network largely distinct from Asian networks (Rungta et al., 2022). we believe that our sample allows us to tell rich and thorough story of factors which have shaped the Western NLP research community, which both informs and is informed by other communities of NLP researchers.
Ethics Statement
Following Institutional Review Board recommen- dations, we take steps to preserve the anonymity of our participants. including aggregating or gen- eralizing across demographic information, avoid- ing the typical practice of providing a table of per-interviewee demographics, using discretion redact or not report quotes that may be identifying. and randomizing participant numbers. Participants consented to the interview and to being quoted anonymously in this work. This work underwent additional IRB screening for interviewing partici- pants in GDPR-protected zones.
We view our work as having potential for posi- tive impact the ACL community, its members to engage in active reflection. We be- lieve that, given recent developments the field and the co-occuring external scrutiny, the current moment is a particularly appropriate time for such reflection. Additionally, we hope that our work can serve those currently external to the community an accessible, human-centered survey of the field and factors that have shaped it over the decades, pri- oritizing sharing of anecdotes and other in-group knowledge that may be difficult for outsiders learn about otherwise.
Our inclusion crite- ria for our participants- three or more publications in *CL., IR, or speech venues?- necessarily means that we have limited perspectives on and from more jumior NLP researchers (such as jumior graduate students), those hoping to conduct NLP research in the future, those who have engaged with NLP research in the past and decided not to continue before developing a publication record, and those who have consciously decided not to engage with NLP research in the first place. In general, although we gathered perspectives from participants across variety of demographic backgrounds, our partic- ipants represent those who have been successful and persisted in the field. This is especially true for our participants in academia; of our participants' current academic affiliations, only RI institutions (if in the US) and institutions of comparable re- search output (if outside the US) are represented. We therefore may be missing perspectives from certain groups of researchers, including those who primarily engage with undergraduate students or face more limited resource constraints than most of the academic faculty we interviewed.
Lack of early career voices
prompt
Acknowledgements
This work would not have been possible without our participants, who were generous with their time and engaged deeply with this material. We would additionally like thank Jill Fain Lehman, An- jalie Field, Dawn Nafus, Momin M. Malik, Gra- ham Neubig and the anonymous reviewers for their helpful comments.
This work was supported in part by a grant from the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE214077 147799. Widder gratefully acknowledges the support of the Digital Life Initiative at Comell Tech. Any opinions, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.
Future research could further examine differ- ences between geographic subcommunities in NLP and more closely examine influences on people's participation in and disengagement from the com- munity. Additionally, we leave to future work a more intentional exploration of perspectives from
"In order to capture perspectives of the community chang- ing over time, and to select for people who are part of these communities.

---

References
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- tre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistiex: Humun Lunguage Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp. Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefow- icz, Lukasz Kaiser, Manjunath Kudlur, Josh Leven- berg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ily Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Mar- tin Wattenberg, Martin Wicke, Yuan Yu, and Xiao- qiang Zheng. 2015. TensorFlow: Large-scale ma- chine learning on heterogeneous systems. Software available from tensorflow.org.
Nic Fishman and Leif Hancox-Li. 2022. Should at- tention be all need? the epistemic and ethical implications of unification in machine learning. In 2022 ACM Conference on Faimess, Accountability, and Transparency, pages 1516-1527.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi. Nelson F, Liu, Matthew Pe- ters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language pro- cessing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1-6, Melbourne, Australia. Association for Computational Linguistics.
Mohamed Abdalla, Jan Philip Wahle, Terry Ruas, Au- rélie Névéol, Fanny Ducel, Saif M. Mohammad, and Karën Fort. 2023. The elephant in the room: Ana- lyzing the presence of big tech in natural language processing research.
Nur Ahmed and Muntasir Wahed. 2020. The de- democratization of ai: Deep learning and the compute divide in artificial intelligence research.
Barney Glaser, Judith Holton, et al. 2004. Remod- cling grounded theory. In Forum qualitative sozial- forschung/forum: qualitative social research, vol- ume 5.
Ashton Anderson, Dan Jurafsky, and Daniel A. Me- Farland. 2012. Towards computational history of the ACL: 1980-2008. In Proceedings of the ACI- 2012 Special Workshop Rediscovering 50 Years of Discoveries, pages 13-21, Jeju Island, Korea, Asso- ciation for Computational Linguistics.
Sara Hooker. 2021. The hardware lottery. Communica- tions of the ACM, 64( 64(11))58-65.
Adam Kilgarriff. 2007. Last words: Googleology is bad science. Computational Linguistics, 33(1)1147-151
Brian R. Bartoldson, Bhavya Kailkhura, and Davis Blalock, 2023. Compute-efficient deep learning: Al- gorithmic trends and opportunities. Journal of Ma- chine Learning Research, 24(122):1-77.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the As- sociation for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2022. The values encoded in machine learning research. In Pro- ceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAceT '22, page 173-184, New York, NY, USA. Association for Com- puting Machinery.
Samuel R. Bowman and George Dahl. 2021. What will it take to fix benchmarking in natural language under- standing? In Proceedings of the 2027 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 4843 4843-4855, Online. Association for Computational Linguistics.
Thomas S. Kuhn. 1970. The structure of scientific revo- lutions. University of Chicago Press, Chicago.
Q. Vera Liao and Jennifer Wortman Vaughan. 2023, Ai transparency in the age of llms: A human-centered research roadmap.
Yvonna S Lincoln, Susan A Lynham, and Egon G Guba. 2011. Paradigmatic controversies, contradictions, and emerging confluences, revisited. The Sage hand- book of qualitative research, 4:97-128.
Steve Campbell, Melanie Greenwood, Sarah Prior, Toniele Shearer, Kerrie Walkem, Sarah Young, Danielle Bywaters, and Kim Walker. 2020. Purpo- sive sampling: complex or simple? research case examples. Journal of research in Nursing, 25(8):652- 661.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kin- ney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4969-4983, Online. Asso- ciation for Computational Linguistics.
Kenneth Ward Church. 2018 Emerging trends: A trib- ute to charles wayne. Natural Language Engineering, 24( I): 155-160.

---

Julian Michael, Ari Holtzman, Alicia Parrish, Aaron Mueller, Alex Wang. Angelica Chen, Divyam Madaan, Nikita Nangia, Richard Yuanzhe Pang. Ja- son Phang, and Samuel R. Bowman. 2022. What do nip researchers believe? results of the nlp community metasurvey.
pages 2182-2194, Online, Association for Computa- tional Linguistics.
Anna Rogers. 2023, Closed ai models make bad base- lines.
Mukund Rungta, Janvijay Singh, Saif M. Mohammad, and Diyi Yang. 2022. Geographic citation gaps in NLP research. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 1371-1389 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Matthew B Miles and A Michael Huberman. 1994. Qualitative data analysis: An expanded sourcebook. sage,
Saif M. Mohammad. 2020. Examining citations of nat-
ural language processing literature. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5199-5209, On-
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. 2021. "everyone wants to do the model work, not the data work": Data cascades in high stakes ai. In Proceedings of the 202] CHI Conference on Human Factors in Computing Systems, CHI '21, New York, NY, USA. Association for Computing Machinery.
line. Association for Computational Linguistics.
high-stakes
Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopou- los, Miguel Ballesteros, David Chiang, Daniel Cloth- jaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chai- tanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra. Swabha Swayamdipta, and Pengeheng Yin. 2017. Dynet: The dy- namic neural network toolkit. arXiv preprint arXiv:1701.03980
Janvijay Singh, Mukund Rungta, Diyi Yang. and Saif M. Mohammad, 2023. Forgotten knowledge: Examin- ing the citational amnesia in nlp.
Irene Solaiman. 2023. release: Methods and considerations.
The gradient of generative ai
Anselm Strauss and Juliet Corbin. 1990. Basics of qual- itative research. Sage publications.
Charlie Parker, Sam Scott, and Alistair Geddes. 2019. Snowball sampling. SAGE research methods founda- tions.
Norman Makoto Su and David J. Crandall. 2021. The affective growth of computer vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9291-9300.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Im- perative Style, Perforreffo Deep Learning Li- brary. Curran Associates Inc., Red Hook, NY, USA.
Richard S Sutton and Andrew G Barto. 2018. Reinforce- ment learning: An introduction. MIT press.
Florian Tambon, Amin Nikanjam, Le An, Foutse Khomh, and Giuliano Antoniol. 2023. Silent bugs in deep learning frameworks: An empirical study of keras and tensorflow.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguisties.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Alex D Wade. 2022. The semantic scholar academic graph (s2ag). Companion Proceedings of the Web Conference 2022.
Aniket Pramanick, Yufang Hou, and Iryna Gurevych. 2023. A diachronic analysis of the nip research paradigm shift: When, how, and why?
Samuel Way, Daniel Larremore, and Aaron Clauset. 2016. Gender, productivity, and prestige in computer science faculty hiring networks.
Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. 2021. Ai and the everything in the whole wide world benchmark. In Proceedings of the Neural Information Process- ing Systems Track on Datasets and Benchmarks, vol- ume 1. Curran.
Robert S Weiss. 1995. Learning from strangers: The art and method of qualitative interview studies. Simon and Schuster.
David Gray Widder, Sarah West, and Meredith Whit- taker. 2023. Open (for business): Big tech, concen- trated power, and the political economy of open ai. Concentrated Power, and the Political Economy of Open Al (August 17, 2023).
Anna Rogers. 2021. Changing the world by changing the data. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers),

---

A Details of Qualitative Methodologies
A.1 Participant demographies
Of the academics interviewed, there was an even split between early, mid, and late career (6/33% each). Of those in industry, 6 (75%) were individual contributors and 2 (25%) were research managers. Our sample is only 19% women, which is likely representative, women comprise approximately 15% of tenure-track computer science faculty in the US (Way et al., 2016). For more discussion of the sample characteristics, see 7. Our positive response rate was 82%.
Demographic trait
sample
In academia
69%
Women Minoritized group
19%
27%
Born outside US
38%
Currently works outside US
4%
Table Self reporred demographic makeup subjects.
A.2 Consent protocol
Interviewees were asked for verbal consent unless they were in a GDPR-protected region at the time of the interview, in which case they provided written consent via DocuSign instead. This consent script aapappodo
Hi, thanks for taking the time to talk with me! My collaborators on this project and work for CMU. We can be reached at [emails] should you have any questions for us during the study or after.
This interview will take between 45 minutes and an hour. There will be no compensation for participation.
Participation is always voluntary, and you may refuse to participate in the research study or stop participation at any time.
You will not be identified in any reports we release from this research. This data will be deidentified, which means you will not be identified by name or any other specific characteristic. We may quote you anonymously. Just in case, though, please do not reveal any private or personally-identffaabl information about yourself or others your answer to our questions.
I'd like to record the audio of this interview as a memory aid. You can ask me to stop the recording at any time. Only the members of our research team will have access to these recordings.
We really appreciate your participation, and we hope publish this research to advance our understanding of the factors shaping the NLP research community.
Do you have any questions? If not, do have your consent to participate in this study?
[answer any questions; if consented, begin recording]
Alright, I've started the recording. Just to confirm, do have your consent to participate in this study, and to record this interview?

---

A.3 Interview Guide
These questions are intentionally open-ended, and the interviewers asked non-scripted followups or additional questions where appropriate. Over time, as early themes emerged, additional questions were added, particularly on funding and pace of work.
1. First, I have few questions about your relationship to the NLP community. You can be specific vague as you'd d like with your responses.
(a) What do you consider be your main/home/primary research community?
(b) More specifically, what venues do you follow and/or publish in?
(c) What subarea(s) or subfields are you most active in?
(d) this different from what you have considered your main community other points in your career? (Prompt: if so, what changed?)
(e) What would you define as the start of your NLP research career? (e.g. start of PhD, research an undergrad, etc). (Prompt: When was this?)
2. I'd like hear your thoughts on what the field was like near the beginning of your career.
(a) When you started in your field, what did people generally think were the most promising directions? (Prompt: do you agree?)
(b) How do you interpret the term "promising"?
(c) What do you think the research community prioritized when you started?
(d) What was the scope of the work that your research group did?
(c) What was your relationship to computing resources the start of your carcer?
(f) What did your software workflow look like when you first started doing rescarch? (Prompt: What tools, frameworks, libraries did you use?)
Where did funding for your work come from? (Prompt: what were the major costs involved with your research?)
Now, I'd like to compare this with the current state of the field.
(a) What do you think others in your field would say the most promising directions? (Prompt:
do you agree?)
(b) What do you think the research community prioritizes now?
(c) What is the scope of the work that your research group does?
(d) How do computing resources affect your group's work now?
(e) How does the software workflow look like for you or your students now?
(f) What tools, frameworks, libraries do you your students use?
Have these tools, frameworks, and libraries made an impact on your (or your students') research?
(h) What impacts do you think these tools, frameworks, and libraries have made on your commu- nity's s rescarch?
Where does the funding for your work come from? (Prompt: what are the major costs involved your research?)
i When you your students start new project, how long average do you expect it to take, from the start of work paper submission?
(k) Has this changed over your career? (Prompt: what has led to this change?)
Now I'd like hear your thoughts on the changes you'l ve observed in your career.
(a) Are there paradigm shifts that you would identify in the field over the course of your career?
(b) Did your community change at all, as result of these paradigm shifts?
(c) Prompt: what years would you ascribe to each shift?

---

(d) How frequently do you feel the community undergoes paradigm shift? (Prompt: is this
frequency changing?)
(e) Are there concerns you have with the direction of the field?
(I) If there were to be paradigm shift in the next few years, in what direction should it go?
(g) Do you think changes in the research community have changed your teaching? (Prompts: how? How do you feel about this shift?)
Now have some demographic questions, which will help us understand the range of people that we talk to. If you'd d prefer not to answer any of them, just let me know.
(a) With which gender do you identify?
i. Man
ii. Woman
iii. Or, feel free to specify as you wish
(b) I going read some age brackets. Can you indicate when read bracket that your age falls into?
i. 18-24
25-34
iii. 35-44
iv. 45-54
v. 55-64
vi. 65+
(c) Which country were born in?
(d) (if not already known) Which country are you currently based in?
(c) What stage of your academic career would you consider yourself in?
(f) Do you consider yourself part of minoritized group in your field?
Anything else in your background that feels relevant or that you want to add?
3. Finally, we'd d like to hear from more people about these issues. Is there anyone you could introduce us to who you think would have interesting answers to these questions?

---

Active authors in *CL venues over time
8000
All active authors
Newcomers
Outgoers
4000
0
Year
Figure 4: The number of "active" researchers publishing ACL. venues has increased dramatically, with more newcomers to the field year over year
B Detailed quantitative methodology
We begin with the ACL Anthology and focus on papers between 1980 and 2022. Using the Semantic- Scholar (S2) API (Wade, 2022), we obtain author and citation information of papers indexed by S2 (some venues, such some workshops, and some Findings papers, are not indexed, leaving 77, 235 papers), with focus citations of papers identified by our participants having been influential the NLP community. We select from this set of influential papers generate the bar plots for Figure Figure uses author publication information linked the individual papers considered.
We rely on S2ORC (Lo et al., 2020) for full-text PDF parses of a subset of these papers, which we use match for mentions of software toolkits identified by our participants as having been influential to the community, for Figure 3, as well as for mentions of influential techniques in the line plot of Figure 2. Note that quantifying mentions is noisy: framework/libray names can be spelled in a variety of ways, and names like "Moses" are also used for authors in the ACL Anthology. For figure 3, we normalize by lowercasing all text, and using the most common normalized spelling, e.g. tensorflow We estimate that this will overestimate the presence of Moses, due its other usages. and underestimate the presence of Hugging Face, which is officially spelled "Hugging Face", but much more often used as "HuggingFace" Despite this, the incredible growth and popularity of Hugging Face relative to other Frameworks is still prominently visible.
huggingf ace.
We present an alternative view of data, similar to that seen Figure 1, in Figure 4. Here, we define members of the community as authors with at least three papers total in *CL venues. Authors are counted "leaving" the community the year after their last *CL publication. We only consider trends in authorship until 2020, it is difficult to determine if authors who did not publish in the last few years have left the community indefinitely.
*As of October 2023, while some Findings papers, such as from ACL 2021 and EMNLP 2020, are automatically indexed by $2, Findings papers from some conferences such as EMNLPP 2022 are not. While some of these papers (or versions of them) may still be indexed by $2 due to also being on ArXiv or similar service, we do not include them in our set of papers.


Reviews:
[{"review_id": "5JG0BgekPO", "reviewer": "Reviewer_bhU3", "summary": "", "questions": "Question 1: Reflecting on the theme, what roles do large language models play in shaping the current state of NLP?\n\nQuestion 2: Among the challenges highlighted in this paper, which ones are unique to the field of NLP, and which ones extend to the broader AI community?\n\nQuestion 3: Can you provide some specific actions that various stakeholders could undertake based on the findings of this study?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for your kind words! We are glad to hear that you find our work well-written, thought-provoking, and timely, and that you anticipate it to be impactful. Please see our response below.\n\n> **Neglect of practitioners' viewpoints**\n\n> **lack of mention regarding the growing challenge of reproducing research papers and establishing fair comparisons between baselines**\n\nOur least senior academic interviewees were post-docs and first-year faculty, and only one of our industry interviewees was in a management position, as opposed to a hands-on, IC role. That is to say, a significant number of our interviewees are practitioners. \nParticipants generally leaned toward discussing factors like funding and resource availability; this may be because we framed our questions to focus on the factors that shape research more broadly, which may have led our respondents to take a more high-level view.\n\nWe touch on reproducibility in Section 5.2, although there we focus on reproducibility concerns arising from disparities in compute access. Due to the page limit, we cut a previous section on reproducibility more broadly. A common sentiment was that converging on similar tools and frameworks for research has benefited reproducibility overall, although participants expressed concerns that large models and increasing secrecy by tech companies were harming reproducibility. Some participants also tied this to a shift in peer review, feeling that  reviewers’ expectations overall have shifted away from valuing reproducibility.   \nTo address the reviewer’s concern, we will add back discussion of these concerns in the next version of the paper. \n\n\n> **Absence of novice voices**\n\nThis is a great point, and one that we discussed in the final stages of this project. Our framing was focused on the factors shaping the field, both now and historically, and as such, we wanted to ask people that tend to have higher leverage than graduate students - professors who advise labs and industry researchers with experience. We would like to do follow up work here about the experience of more junior researchers – and there is some excellent prior work (e.g. [1]) – but we did not feel that interviewing junior researchers naturally fit into the scope of this paper. \n\n[1] Ignat, Oana, Zhijing Jin, Artem Abzaliev, Laura Biester, Santiago Castro, Naihao Deng, Xinyi Gao, Aylin Gunal, Jacky He, Ashkan Kazemi, Muhammad Khalifa, Nam Ho Koh, Andrew Lee, Siyang Liu, Do June Min, Shinka Mori, Joan Nwatu, Verónica Pérez-Rosas, Siqi Shen, Zekun Wang, Winston Wu and Rada Mihalcea. “A PhD Student's Perspective on Research in NLP in the Era of Very Large Language Models.” ArXiv abs/2305.12544 (2023).\n\n\n> **Absence of tangible action items**\n\n> **Question 3: Can you provide some specific actions that various stakeholders could undertake based on the findings of this study?**\n\nThis is another great point – and one that we debated ourselves. We share the reviewer’s sentiment in being hopeful for positive change, though we intentionally decided not to prescribe any particular solutions to our potential audience, especially given the absence of junior researchers in our sample. Instead, we came up with specific *questions* surfaced by our research that we believe the NLP community would benefit from considering. These questions direct action and reflection to points of controversy without prescribing any specific path forward. Though we shortened these significantly for length in Section 7, we have decided to include an expanded list in the next version.\n\nAdditional questions we will raise in the expanded discussion:\n> 1. What is the role of benchmarks when the community has exploded in size and “NLP kinda works now”? What alternatives might we embrace in addition to existing prevalent methods of evaluation? How can we embrace more holistic evaluations of models, despite the difficulty of scaling them? How can conference organizers set guidelines for reviewing to help align evaluation of work with the things we value?\n> 2. Do the incentives that our field has in place encourage the behavior we'd like to see? What kinds of people and research are valued by those incentives? What kind of work has higher and lower status?\n> 3. How do we feel about the pace and pressure of NLP research today? How can we encourage people to do longer-term work that may deviate from current norms?\n> 4. Even among our sample of 26, there were substantial disagreements. How do we move forward as a community? How do we wish to define the community and the relationships between its subcommunities?\n\n\n> **Question 1: Reflecting on the theme, what roles do large language models play in shaping the current state of NLP?**\n\nThis was one of the questions that we started with in this work, but in the process of conducting the interviews, we realized that while language models appear to be the driving force of change in NLP today, several of the cultural changes that we attribute solely to language models arise from the way NLP developed through the 2010s more broadly. The increasing pace of work and burgeoning community size, for instance, especially seen in relation to language model research, is fed by the centralization of methods, software and design decisions that we highlight.\n\nHowever, LLMs have intensified these trends, and several of the trends that we see now. In particular, the increasing public perception of the field now that NLP “kinda works” and the centralization around a few particular models because of their size and the difficulty in reproducing them are linked directly to language models.\n\nEach major methodological shift (e.g. statistical NLP, neural NLP, pretrained models) that our interviewees discussed also heralded cultural shifts. The ways that we react as a community to current concerns– e.g. peer review, centralization, and compute disparities–  will define the future cultural impact of LLMs. We believe that this makes this the perfect time to reflect on previous shifts in the community, and to learn from how we responded to those shifts. \n\nWe will update the next version of the paper to clarify and highlight these connections. \n\n\n> **Question 2: Among the challenges highlighted in this paper, which ones are unique to the field of NLP, and which ones extend to the broader AI community?**\n\nThis is a great question– while we don’t have evidence to make strong claims here, some interviewees did discuss their experiences in NLP in comparison to CV, speech, and ML.\nSome commonalities identified by participants include the public visibility of the umbrella category of “AI” (e.g. the common gaze on things like self-driving cars and LLMs), the cycles of work after breakthroughs, and concerns over peer review– although participants disagreed on the scale of NLP’s challenges relative to those of related communities. Another point of reflection was *where* NLP placed in a broader community– some participants referred to NLP as a “subfield of ML/AI,” while others drew strong distinctions between ML as a method for NLP and as a central point of the community. We will add some discussion of this in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strength 1: This community study is solid and beautifully written. The qualitative methodologies adhere to ethical standards and appropriate protocols for interactions with human subjects. The presentation of information is clear and thought-provoking.\n\nStrength 2: This paper is remarkably timely, arriving at a juncture where NLP is undergoing substantial transformation. The crescendo of voices and sentiments surrounding the success and hype of large language models is undeniable. While this study refrains from offering definitive predictions about the future, it adeptly highlights numerous deeply entrenched issues. The paper's impact is readily foreseen in its role of raising public awareness about several concerns within the NLP community. \n", "reasons_to_reject": "While I resonate with the sentiments expressed in this paper, which reflect some of the concerns I share, I'd like to draw the authors' attention to a few limitations that I've identified.\n\nWeakness 1: Neglect of practitioners' viewpoints: I was surprised to find that the topics covered in this paper largely revolve around research management matters, such as funding and resources. I noticed a lack of mention regarding the growing challenge of reproducing research papers and establishing fair comparisons between baselines. This raises concerns that the study participants might primarily consist of research managers in industry or faculty members in academia, rather than NLP practitioners or students deeply involved in research implementation. \n\nWeakness 2: Absence of novice voices: To enhance its comprehensiveness, this work could benefit from including more perspectives from beginners and first-time researchers. Many students within this community face even more challenging conditions, with restricted access to computational resources and limited opportunities to secure research assistant roles within established research groups (which are crucial if they need recommendation letters for higher education applications).\n\nWeakness 3: Absence of tangible action items. While this work effectively raises public awareness regarding numerous concerns within the NLP community, it appears to lack a concrete list of actionable steps. What are some venues to trigger these discussions? How can current conference and reviewing systems improve to at least take initiative? The absence of such discourse left me somewhat disappointed.\n", "questions_to_authors": "Question 1: Reflecting on the theme, what roles do large language models play in shaping the current state of NLP?\n\nQuestion 2: Among the challenges highlighted in this paper, which ones are unique to the field of NLP, and which ones extend to the broader AI community?\n\nQuestion 3: Can you provide some specific actions that various stakeholders could undertake based on the findings of this study?\n", "paper_topic_and_main_contributions": "This work is a community study that provides a concrete review of the history in the landscape of NLP, from the perspectives of norms, incentives, technology, and culture that have contributed to its present state. The methodology involves in-depth interviews with 26 NLP researchers of varied backgrounds, and their insights are thoughtfully categorized into various pertinent themes. Additionally, the study includes a quantitative assessment of citations, authorship, and terminology usage within ACL conferences, designed to enhance the ongoing discourse. It appears that many of the discussions within the paper are predominantly centered around Western institutions and funding structures, potentially introducing a significant bias. To the authors' credit, they do acknowledge this issue in their limitations section, and the value of their work remains intact. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 181: an extra \".\""}, "extra_text": {}}, {"review_id": "6JiPAFt60F", "reviewer": "Reviewer_nTbt", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper! Please see our response below.\n\n> **Most of the paper seems anecdotal relying on the interviews from 26 anonymized NLP researchers.**\n\nOur primary methodology is qualitative. While qualitative methods originated and have robust grounding in fields as diverse as sociology, psychology and anthropology, they have long been used in NLP and related fields such as HCI and linguistics. Some recent examples of qualitative research conducted primarily via long-form interviews include work from NAACL 2022 [1], FAccT 2023 [2], and one of the best papers from CHI 2023 [3].\n\nWe followed norms from qualitative methods and grounded theory to ensure that we carried out a rigorous study, including a deliberate and justified sample, carefully constructed and administered interview guide, and established data coding and analysis methods. 26 participants is well within the norm for this kind of work. We took care to collect a representative sample across multiple dimensions of identity, and we discuss the ways in which we interpret our results in light of our sample characteristics in our Limitations section. \n\nLike all methods, interview methods have limitations, some of which are remedied by our complementary quantitative methods (section 2.2), and the remaining of which we carefully document in the limitations section. We welcome pointers to any more specific limitations we may have omitted. We are confident that our methods yield insights that would be difficult to uncover otherwise, including understandings of researchers’ thought processes that cannot be explored through quantitative methods or short-form surveys.\n\n[1] Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, and Alexandra Olteanu. 2022. Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 314–324, Seattle, United States. Association for Computational Linguistics.\n\n[2] Wesley Hanwen Deng, Nur Yildirim, Monica Chang, Motahhare Eslami, Kenneth Holstein, and Michael Madaio. 2023. Investigating Practices and Opportunities for Cross-functional Collaboration around AI Fairness in Industry Practice. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT '23). Association for Computing Machinery, New York, NY, USA, 705–716. https://doi.org/10.1145/3593013.3594037\n\n[3] Lauren Wilcox, Renee Shelby, Rajesh Veeraraghavan, Oliver L. Haimson, Gabriela Cruz Erickson, Michael Turken, and Rebecca Gulotta. 2023. Infrastructuring Care: How Trans and Non-Binary People Meet Health and Well-Being Needs through Technology. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 489, 1–17. https://doi.org/10.1145/3544548.3581040\n\n\n> **The paper doesn't focus on LLMs but a general thought process of how the 26 NLP researchers have viewed NLP since the time they entered into the field v/s the current state. For ex., issues with peer reviews.**\n\nWhile current discussion in the field is centered around LLMs and the disruptive changes they have brought, our work contextualizes our present and future by analyzing the historical precedent of this shift. LLMs are only one of many other historical “paradigm shifts”, which we uncover, analyze and learn from in this work. Our analysis shows that cultural shifts, such as growing discontent with peer review, emerge alongside methodological shifts– e.g. with statistical methods and benchmarking, and with neural methods and increased software centralization. Thus, we believe discussing current perceptions of the field both culturally and methodologically is appropriate for the theme track. For instance, participants attribute issues with peer review partially to the burgeoning size of the community; the community’s growth is a consequence of the growing public perception of the field, which in turn was catalyzed by LLMs leading to more public-facing NLP applications. One of the contributions of our work is uncovering and analyzing these connections, which we will update the next version of the paper to clarify and highlight. \n\nPlease see our response to Reviewer bhU3’s Question 1 for additional context.\n\n> **The paper isolates only prompting as a methodology whereas there are many other strategies employed currently. For ex., zero shot/ few shot, fine-tuning only the classification layer, adapters etc.**\n\nTo avoid biasing interview respondents, we do not *a priori* identify methodologies of note– we discuss methodologies that interviewees themselves identify as major shifts or otherwise bring up organically when discussing changes in the field. While one or two interviewees mentioned these other methods, nearly all of our interviewees discussed prompting, suggesting that prompting is dominating the conversation around recent methodologies.\n\n> **Comparing HuggingFace and Pytorch/Tensorflow usage isn’t the right comparison because HuggingFace is built using Pytorch/Tensorflow**  \n\nWhile it’s true that using HuggingFace requires also using Pytorch/Tensorflow, Figure 3 compares explicit references to each toolkit/library in the text of papers. The intuition here is that people generally reference the toolkit/library they are using most directly, rather than all software they use (in the same way that people who use PyTorch directly do not generally also write that they’re using Python). This is not to say that people who discuss using HuggingFace are not using a PyTorch/Tensorflow backend. \n\nThis comparison is useful because the abstractions that HuggingFace provides are different from using PyTorch directly. We discuss the benefits and drawbacks of increasing software abstraction in Section 5. We will clarify this in the caption of Figure 3 and in discussion of this figure.\n\n> **Figure 3 isn't referenced in the main text.**\n\nThanks! We will correct this in the next version.\n\n\n> **I feel generalizing voices of 26 NLP researchers over a field of thousands can create a bias in the minds of junior researchers or new researchers trying to enter the field.**\n\nThis is a thoughtful concern. We hope that the measures we took can alleviate some of these potential risks. We took care to describe the limitations and sampling biases of our study explicitly (see our Limitations section), as well as highlight both agreements and disagreements between our participants. We welcome feedback calling attention to any specific additional limitations, blind spots, or potential negative consequences of our study. \n\nIt is also (perhaps increasingly) true that new and junior researchers are entering the field with their own preconceptions about it. We are sympathetic to the many stresses and pressures the community’s junior researchers may be experiencing, due to factors such as recent developments in research, media coverage of AI/NLP in general, and competitive admissions to graduate programs. We believe that our work has greater potential to help new and junior researchers *expand* their own perspectives by increasing awareness of perspectives from other “eras,” subfields, and institutions, than to bias them in any specific way. From our ethics statement:\n\n> We view our work as having potential for positive impact on the ACL community, as we prompt its members to engage in active reflection. We believe that, given recent developments in the field and the co-occuring external scrutiny, the current moment is a particularly appropriate time for such reflection. Additionally, we hope that our work can serve those currently *external* to the community as an accessible, human-centered survey of the field and factors that have shaped it over the decades, prioritizing sharing of anecdotes and other in-group knowledge that may be difficult for outsiders to learn about otherwise.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I like:\n1) (Briefly) discussing some of the issues of the current state of NLP. For ex., issues with scale and the rat race of publishing.\n", "reasons_to_reject": "The paper would better fit as a technical/interviews report.\n1) Most of the paper seems anecdotal relying on the interviews from 26 anonymized NLP researchers. \n2) The paper doesn't focus on LLMs but a general thought process of how the 26 NLP researchers have viewed NLP since the time they entered into the field v/s the current state. For ex., issues with peer reviews.\n3) The paper isolates only prompting as a methodology whereas there are many other strategies employed currently. For ex., zero shot/ few shot, fine-tuning only the classification layer, adapters etc. \n\n\nSide: The paper compares the trends of softwares but includes toolkits and libraries in the same graph (which is not the right comparison). For ex., HuggingFace is a toolkit that is built using Pytorch/Tensorflow, and the authors compare Pytorch, Tensorflow usage to HuggingFace.\nAlso, figure 3 that makes this comparison is not referenced in the main text.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper aims to document a qualitative survey of 26 NLP researchers over the history of NLP in terms of Explore-Exploit nature of studies, the current state, the benchmarking culture, and the software availability.\nThe authors try to depict certain historical timelines with the quantitative measures of papers, citations etc. in ACL anthology.\nThe author's state their motivation as driving the future of NLP by knowing the past.", "excitement": 2, "reproducibility": null, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 3 isn't referenced in the main text."}, "extra_text": {"Justification_for_Ethical_Concerns": "I feel generalizing voices of 26 NLP researchers over a field of thousands can create a bias in the minds of junior researchers or new researchers trying to enter the field.\n\nDisclaimer: This is a personal opinion."}}, {"review_id": "zoiQrWFLtI", "reviewer": "Reviewer_tGxC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for your kind words! We are glad to hear that you appreciate the rigor and timeliness of our work and share in our excitement about its potential impact. We will be happy to address any questions in the discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I greatly enjoyed reading this article. I am not a sociologist, but having collaborated with them I have some awareness of the methodology. The sampling strategy, the way the interviews were conducted and the coding protocol appear sound and rigorous to me. I also appreciate the effort to validate conclusions as much as possible with quantitative methods on the corpus of publications.\n\nThe gold nugget in my view is the qualitative analysis of the interviews. It is clear, lucid and well-written. It touches on all the fundamental topics that I perceive to be crucial to reflect about at this point: the history and cycles of the field, its research and publication practices, the need to reward more novelty and risk, the direct and also complex possible consequences of a model and software mono-culture, benchmarking culture (its pros and cons), the growing influence of private companies, the close models, the rise of \"prompt engineering\", the \"CERN\" era of NLP. I can't think of one topic that preoccupied me in the last year in relation to NLP that is not covered.\n\nIn short: this work is great and the timing is more than appropriate. I believe that its publication will be a good service to the community.", "reasons_to_reject": "I see no reason to reject this article.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work is a sociological study of the field of NLP as currently perceived by its practitioners, primarily based on the long-form interviews of a sample of 26 researchers and supplemented by qualitative longitudinal analysis of citation, terminology and authorship patterns in ACL anthology. A formal protocol was established to sample the participants, conduct and code the interviews.\n\nThe author(s) then provide a qualitative analysis of the various topics of the interviews, describing NLP researcher's self reflections on the history of the field (introducing the idea of recurrent cycles of exploration and exploitation), and also of the particularly strong pull that the latest breakthroughs produced, potentially leading to a \"deep learning monoculture\". A perception of the fundamental shift from symbolic to statistical methods is described, along with deep methodological and cultural changes, such as the rise of \"benchmarking culture\" and the dispersion of a community that used to be small and highly connected. Another topic is the centralization not only of methods but also of tools, where not long ago most code was written from scratch or maintained at the lab level, now a few libraries such as PyTorch came to dominate. This trend is qualitatively confirmed with ACL Anthology data. Researcher reflect on the \"software lottery\" effect, where methods are more likely to be studied if they happen to match the capabilities of the dominant software, another phenomenon likely to hurt diversity. These now ubiquitous libraries are also described as \"implicit funding\", leveraging what can be achieved by a lab. The current level of competition in the field, dominance in numbers of junior researchers (also at the peer review level) and heavy exploitation phase around transformers are perceived by some researchers to make it very hard for original ideas to actually be published, and create a career risk in pursuing them.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents interviews with 26 NLP practitioners about the current state of the field. While the sample size is small, the study was conducted following proper protocols and included in-depth qualitative analyses of the various topics in the interview, supplemented with qualitative longitudinal analyses of citations, terminology, and authorship patterns in the ACL Anthology.
Decision: Accept-Main