A Diachronic Analysis of Paradigm Shifts in NLP Research: When, How, and Why?
Authors: Aniket Pramanick
Yufang Hou
Saif M. Mohammad
Iryna Gurevych
Venue: EMNLP 2023 Main (2023)
Keywords: Scholarly Document Processing
NLP Scientometrics
TL;DR: In this study, we propose a systematic framework for analyzing the evolution of research topics in a scientific field using causal discovery and inference techniques.

Abstract: Understanding the fundamental concepts and trends in a scientific field is crucial for keeping abreast of its continuous advancement. 
In this study, we propose a systematic framework for analyzing the evolution of research topics in a scientific field using causal discovery and inference techniques. We define three variables to encompass diverse facets of the evolution of research topics within NLP and utilize a causal discovery algorithm to unveil the causal connections among these variables using observational data. Subsequently, we leverage this structure to measure the intensity of these relationships. By conducting extensive experiments on the ACL Anthology corpus, we demonstrate that our framework effectively uncovers evolutionary trends and the underlying causes for a wide range of NLP research topics. Specifically, we show that tasks and methods are primary drivers of research in NLP, with datasets following, while metrics have minimal impact.

A Diachronic Analysis of Paradigm Shifts in NLP Research: When, How, and Why?
Aniket Pramanick', Yufang Hou?, Saif M. Mohammad', Iryna Gurevych' 'Ubiquitous Knowledge Processing Lab (UKP Lab)
Department of Computer Science and Hessian Center for Al (hessian.Al)
IBM Research Europe, Ireland
National Research Council Canada
www. wkp. tu- darmstadt..o de, yhou@ie. chou@ie.ibm.comm com, saif. mohamadenre-cnrc.ccoo
Abstract
method track the temporal evolution of research topics can be beneficial in offering an overview of the field and assisting researchers in staying abreast of advancements more efficiently.
Understanding the fundamental concepts and trends in scientific field is crucial for keeping abreast of its continuous advancement. In this study, we propose systematic framework for analyzing the evolution of research topics in scientific field using causal discovery and in- ference techniques. We define three variables to encompass diverse facets of the evolution of research topics within NLP and utilize causal discovery algorithm unveil the causal connec- tions among these variables using observational data. Subsequently, we leverage this structure measure the intensity of these relationships. By conducting extensive experiments on the ACL Anthology corpus, we demonstrate that our framework effectively uncovers evolution- ary trends and the underlying causes for a wide range of NL.P research topics. Specifically, we show that tasks and methods are primary drivers of research in NLP, with datasets fol- lowing, while metrics have minimal impact.'
In this work, we propose a systematic framework to examine the evolutionary journey of research topics within the realm of Natural Language Pro- cessing (NLP), harnessing causal discovery and inference techniques. Prior research on historical analysis of NLP has predominantly concentrated scrutinizing metadata associated with research papers (Hall et al., 2008; Mohammad, 2019; Uban et al.. 2021; Singh et al.. 2023; Wahle al., 2023) such as number of citations, title, author profile, af- filiation, and publication venue. These studies have examined the research trends through unigram or bigram frequency analysis, but they do not provide insights into the underlying causes propelling these research topics.
Our study centers on four distinct fundamental types of entities in NLP research: tasks represent- ing well defined problems; methods, signifying the solutions approaches employed to tackle the tasks; datasets, indicating the relevant textual sources such corpora and lexicons; and metries, encompassing the evaluation techniques tailored to specific tasks. We abbreviate these types as TDMM for short. Specifically, we examine the interplay between an NLP task that is commonly viewed as a focused research topic (c.g., Machine Translation) and the key entities that exert pivotal influence on the target task (such as "BLEU" (Pa- pineni al., 2002) "Transformers" (Vaswani et al., 2017)).
1 Introduction
Experts in field sometimes conduct historical stud- ies to synthesize and document the key research ideas, topics of interest, methods, and datasets that shaped a field of study. They document how new research topies celipsed older ones and contributed to shaping the trajectory of the research area (Kuhn, 1970). Aspiring scientists learn the craft of their discipline by delving into the examination of past scientific accomplishments documented in research papers. However, conducting such historical study is challenging: Experts in field rely on years of experience and peruse large amounts of past published articles to determine the chronolog- ical progression of a research field. Further, the exponential growth of scientific publications in re- cent years has rendered it arduous even for domain experts to stay current. Therefore, automated
Our goal is to identify the TDMM entities (E) associated with a specific task (t) and assess their causal influence on the task's research trends (TDMM-Tase causal analysis). Specifically, we address the following key research questions asso- ciated with a task entity t: (a) Which entities E effectively indicate the research trends for this task
WWo publish the code and dataset for our experiments at https://github com/ wwt.Laaaaaaaaaaaaeeaaaaaaa

---

Research trends and the top causal entities machine translation
Symbolic NLP Period
Statistical NLP Period
Neural NLP Period
Pretrained Period
Type
Entiyy
Causal
Larguuggo Gencraaioo
1550-2002
Type
Endity
Caunal Effect
2010-2002
Metlood
Statisiiall Matels
0,2505
Type
Entiy
Caural Effect
Metric
Perplesity
0.2914
Method
Trandteners
02515
Dotosct
Wordrct
0,2497
Metric
Metsor
Lerguage
0.202
Language
Figure I: Evolution of Machine Translation (MT) research. Blue line: Number MT papers (1979-2022). Tables
show the top causal entities/types for different periods (excluding 1979-1989 due to limited MT papers).
t? (b) Are there discernible causal relationships between t and E? (c) What is the extent of the causal impact exerted by E on t?
ture surveys by identifying related research areas and enable young researchers to delve into new re- search focuses by establishing connections among different research areas.
Unlike Uhan et al. (2021) and Koch et al. (2021) that heavily rely on manual annotations and have limited coverage, our analysis based on TDMM entities automatically extracted from 55K papers in the ACL Anthology?. Our framework not only recognizes the key entities driving the research di- rection of a research topic but also measures the causal effects of these entities the target topic in an end-to-end fashion. Figure shows the most influential entities for Machine Translation (MT) different time periods. For instance, "statistical models" used to be the popular method for MT in 1990-2002, and the evaluation metric "BLEU" is one of the top causal entities driving the MT research in 2003-2017. In the era of pre-trained large language models (LLMs) starting from 2018, "transformer" has become the popular method for MT. For another research topic of "Speech recogni- tion", our framework uncovers the influential role of "language modeling" between 1979 2022, where speech recognition models utilize probability scores from language models to recognize coherent text from speech (Negri et al., 2014).
In summary, we make three-fold contributions in this study: Firstly, we propose a framework quantify research activities, including (1) trends and stability of an NLP research task, and (2) re- lation intensity between TDMM entities and NLP research tasks. Secondly, we employ causal anal- ysis algorithms to uncover causal structures and measure effects between tasks and related TDMM entities (TDMM-Task causal analysis). To the best of our knowledge, this represents the first histori- cal study of scientific research anthology from a causal perspective. Finally, through extensive experiments on the ACL Anthology, we offer an empirical overview of the NLP research landscape. In the following sections, we will refer to TDMM- Task causal analysis causal analysis.
2 Related Work
Scientific Trends Analysis The analysis of sci- entific trends has been a research focus since Hall et al. (2008). the field of "scientometrics" exten- sive literature explores citation patterns and utilizes topological measures in citation networks for trend analysis (Small, 2006; Shibata et al., 2008; Boyack and Klavans, 2022).
In this work, we analyze 16 tasks from diverse set of research areas identified by ACL 2018 orga- nizers. Our framework is versatile and applicable to other tasks and domains, benefiting both young and experienced researchers. It can aid litera-
Another line of research focuses on metadata and content analysis. For instance, Prabhakaran et al. (2016) employed rhetorical framing to exam-
"https://accannthooogy.ou /rgg/

---

Step 1: Pre-processing
Step 2: Multiple linear regression
What type of entitics (X) influence the
Grobid Parser
Step Causal Analysis
PDF Table Parser
TDMM Entity Taggers
3.1 calculate variable values for each task
frepl(s
Astabliit(()
(x())
Texk
Morieee Fondlation
3.2 causal structure discovery
3.3 causal inference analysis
Papers from the ACL Anthology
(DirectLiNGAM)
Talk
Aesc triti
Eht
0,2515
Tumolation
0.997
Leaase
Causal Graph
Figure 2: System architecture.
ine trend patterns. Grudin (2009), Liu al. (2015), and Mohammad (2019) investigated the interac- tion between the topics in publications, research grants, author profiles, highly impactful papers, and dataset usage patterns. Additionally, Koch et al. (2021) studied dataset usage patterns among differ- ent research communities, while Uban et al. (2021) analyzed relationships between NLP research top- ies based their co-occurrence in text and the degree of correlation between their popularity over time. our work, we develop entity recognition models to extract TDMM entities from NLP re- search papers and focus on analyzing the causal re- lations between a task entity and its related TDMM entities.
3 Data Collection
ACL Anthology Corpus Following prior work by Mohammad (2020), we utilize ACL Anthology as the source of NLP Research papers. For this work, we collect 55,366 NLP papers that belong to the "ACL Events" category² from the ACL anthol- ogy published between 1979 and 2022. For each paper, we use GROBID (GRO, 2008-2022) and the PDF table parser from Hou al. (2019) to extract sentences from each the individual sections as well as from the table and figure captions. In a post-pro post-processing step, we remove all the URLs from the extracted sentences. On average, we have 1,258 papers per year and 1,117 sentences per paper.
It is worth noting that certain NLP paper preprints may become accessible servers before they are officially published in the ACL Anthology. However, we argue that the peer review process in ACL Anthology serves as a ro- bust quality assurance mechanism. Hence, we con- sider ACL Anthology a more reliable source com- pared to preprint servers.
Causality in NLP Existing works on NLP ap- plying causal analysis algorithms mainly focus on two directions. The first line of work discovers causal relations among textual features or expres- sions of events in texts and uses them in various downstream tasks, such as question answering (Oh al., 2016), commonsense reasoning (Bosselut al., 2019; Sap et al., 2019), and relation extrac- tion (Do ct al., 2011; Mirza and Tonelli, 2014; Duni- etz et al., 2017).
preprint
TDMM Entity Extraction To identify tasks, datasets, metrics, and methods entities from NLP papers, we developed two entity taggers based on Flair (Akbik et al., 2018). The first tagger based on the TDMSci annotations (Hou et al., 2021) for recognizing task, dataset, and metric entities. The second tagger is trained using the SciERC dataset
In another avenue of this field, researchers rep- resent causal elements using textual features (Jin et al., 2021; Fong and Grimmer, 2016; Veitch et al., 2020; Keith et al., 2020) and define the causal graph structure based on domain knowledge. Our work falls within this line research, where we employ causal algorithms to analyze the trends in NLP re- search topics and the underlying causes.
"This category covers major NLP conferences. workshops. and journals including ACL, NAACL, EMNLP, EACL, AACL, CL, and TACL. Additionally. we also include papers published at COLING from the "Non-ACL. events category".

---

Period
Years
Key Research Themes
Early Years
1979-1999
Foundational work in syntactic parsing, machine translation, and information retrieval.
Formative Years
1990-2002
Advances in language modeling, named entity recognition, and discourse analy- sis (research focus shifted towards data-driven approaches).
Statistical Revolution & Neural Networks
2003-2017 Focus on statistical techniques (text classification, statistical machine translation, etc.) and resurgence neural networks (word emheddings, neural machine translation, etc.)
Deep Learning Era
2018-2022 Dominance of transformer-buse architectures (BERT and their variants).
Table 1: Chronological Periods of NLP Research.
(Luan et al., 2018) to extract method entities. On the testing datasets of TDMSci and SciERC, the of 0.77 and 0.78 for the type partial match (Segura-Bedmar et al., 2013), respectively. In type partial match, a predicted entity is considered correct if it par- tially overlaps with gold entity and has the same type. For example, "Penn Treebank" is counted as correct prediction even if the corresponding gold annotation is "Penn Treebank dataset".
Variables
R-Squared (t)
unique tasks
0.87
two taggers achieve a micro-average
+ unique datasets
0.91
+ unique methods
0.93
+ unique metrics
0.97
Table 2: Variable Selection for Regression.
influence on the research direction of NLP. To achieve this understanding, we employ Multiple Linear Regression (see Appendix D for details), widely utilized tool in economies rescarch (Bar- rios and Hochberg, 2020). Figure 2 (step1/step2) illustrates the framework.
To further improve the precision of the TDMM taggers, we include only entities that appear in more than five papers in the dataset. For each pa- per, we collect the most frequent task mentions appearing in the title, abstract, experiment section, table, and figure captions approximate the tasks that the paper has done research
Our analysis assumes that if the TDMM entities have played role in the emergence or disappear- ance of task entities, this influence will be reflected in the number of unique task entities in subsequent years, which can be captured through regression analysis. While the study does not provide spe- cific information on the precise influence each TDMM entity on individual task entities, the partial regression coefficients shed light on the types of entities responsible for influencing the overall task entity landscape.
Taxonomy for Periods of Reference In order to facilitate in-depth analysis. in this paper, we adopt a taxonomy that partitions our reference time frame (1979-2022) into four distinct intervals. Table 1 illustrates the defined intervals. These intervals have been designed approximate the overarching trends observed in NLP research throughout the years, aligning with our perspective on the field's evolution. is important to acknowledge that the exact boundaries and thematic emphases may differ based on varying perspectives and specific research areas within NLP. However, we highlight that our framework and methodologies are highly adaptable, allowing end users to effortlessly apply them to any desired time interval specific analysis.
Method. Mathematically, predict the number of task entities Y in given year t as a function of the cumulative counts of all types of entities [X(-1) (TDMM entities) until that year, f- given by Y - ro + Err {ri} quantifies the re- lationship strength between the predicted variable (number of task entities) and the independent vari- ables (number of TDMM entities).
4 Entity Influence in NLP Research: Regression Analysis
Evaluation. We evaluate the regression model using the R² measure (coefficient of determina- tion) to assess the goodness of fit. Additionally, we perform a null hypothesis test to determine the statistical significance of the partial regression co-
Before conducting the causal analysis, we aim to identify the key variables that significantly impact the evolution of NLP Research. Specifically, we investigate which types of entities exert the most

---

efficients.
Partial Regression Coefficient
Years
Tasks
Datasets
Methods
Metries
Results and Discussion.
1) Optimized Number of Variables. In our initial experiment, we determine the optimal number of variables and summarize the corresponding R² values in Table 2. Additionally, all regression coefficients statistically significant at 5% level, indicating their strong relationship with the predicted variable. Discussion: The overall results indicate that the model achieves good to the data when all four variables (number of tasks, datasets, metrics, and method entities) are used to predict the number of task entities in subsequent years. We also explore the possibility of reducing the number of variables while maintaining similar performance. Interestingly, using only one variable results in a significant drop of 0.1 in the R² value (R2 value 0.87), indicating poor fit to the model. Conversely, increasing the number of variables improves the model fit, suggesting the significance of all four variables in analyzing research trends (R² value 0.97). It is worth noting that we exhaustively explored various combinations of variables, including those presented in the table, and consistently obtained similar results.
1979-1989
0.35
2.24
0.21
0.02
1990-2002
0.82
0.89
2.86
0.81
2003-20117
5.37
6.26
7.00
0.69
2028-2022
1.47
3.38
1.79
0.41
1979
2022
3.50
1.07
2.92
0.54
Table 3: Variables Influencing NLP task entities.
newer datasets in relatively short span of time, driven by the research needs and the data require- ments of deep learning models. These highlight key factors inftuencing research trajectory over time.
Causal Methodology for NLP Research Analysis
Drawing on the insights gained from the Regres- sion Analysis (Section 4), we now establish the cornerstone of our study by defining three causal variables that drive the causal analysis in the sub- sequent sections. Using causal discovery and in- ference techniques, we analyze the causal relation- ships among the variables and measure the impact of TDMM entities target task entities based these relationships. Figure 2 illustrates the archi- tecture that underpins our framework.
2) Influence of the Variables. In the second ex- periment, we assess the association between the target variable and cach independent variable. In Table 3, present the regression coefficients cor- responding each entity type. Larger values of regression coefficients indicate stronger relation- ship between the target variable and the respective independent variable. Discussion: Overall, we note that the gradual emergence of newer tasks has been driving force behind research progress. However, when we analyze the trends within each year interval, we uncover more nuanced patterns. During the Early Years (1979-199))) 1989), when NLP was in its nascent stage as an independent research field, the focus was on creating new datasets to fuel research advancements. In the Formative Years (1990-2002), we witnessed the introduc- tion of new methods, particularly data-driven ap- proaches, which played a crucial role in shaping the field. Subsequently. from 2003 to 2017, statis- tical methods underwent revolution, and later in the same period, neural network methods experi- enced resurgence, indicating significant shifts in research trends. Now, in the present Deep Learning Era (2018-2022), we observe a rapid creation of
5.1 Causal Variables
Task Frequency Shift Value: Distinguishing from the previous approaches (Tan al., 2017; Prabhakaran al., 2016), that rely on word frequen- cies, we define task frequency f(y) as the number of published papers focusing on a specific task y in a given year f, normalized by the total number of papers published on the same year. The task frequency shift value Afreq'; (v) captures the av- erage change in the number of published papers between two years t t2. This value serves measure of the research trends associated with the task during that time interval, indicating whether it experienced growth or decline. The frequency shift value is given by: Afreq, (v)
f(y),, f(y), 22-h
Task Stability Value: We introduce the concept of task stability value to measure the change in the research context of a given task, y, between two years, 4 l2. This value quantifies the overlap neighboring TDMM entities that appear in the same publication as y within the specified time interval. To calculate task stability, we adapt

---

the semantic stability approach of Wendlandt et al. (2018) to our setting and define specifically for task entities. Initially, we represent each paper in our dataset as a sequence of TDMM entity men- tions, removing non-entity tokens. We then em- play "Skip-gr gram with negative sampling" (Mikolov et al., 2013) to obtain embeddings from this rep- resentation. Formally, let e1, e2..... en be this en- tity representation of a paper, and the objective of skip-gram to maximize the mean log proba- bility # Ei= E-esisa logp(e)+ sle)), where is called the context window size. Finally, the task stability value Astability!, (v) of y between i and 22 is computed as the percentage overlap between the nearest I neighboring entities of the given task in two representation spaces. The stability value My, (w)n,, N, (v) is given by: Astability (y) wi (u)uNi, (v)
5.2 Causal Algorithms
Causal Structure Discovery To uncover the causal structure among variables from observa- tional data, we employ DirectLinGAM (Shimizu et al., 2011), which assumes a non-Gaussian data- generating process. Since the variables in Sec- tion 5.1 come from non-Gaussian frequency dis- tributions, DirectL.nGAAM is suitable. It uses an entropy-based measure to subtract the effect of each independent variable successively. Unlike PC- Stable (Colombo and Maathuis, 2014), it does not require iterative search algorithmic parameters. We apply DirectLiNGAM with a 5% significance level for causal discovery (see Appendix for de- tails).
Causal Inference Once the causal structure between the variables has been established, we leverage this structure to assess the causal effects. Specifically, we measure the causal effects by the entity change value of entity x on the frequency shift and subsequently on the stability values associated with a given task y. For this purpose, we use the probability density function instead of probability mass, as all our causal variables are continuous in nature. We measure the causal effects in two steps: first, we estimate the probability density of the entity change variable using a linear regression model. In the next step, regress the frequency shift and stability against the entity change value, weighted by the inverse probability densities obtained in the previous step. We model the functional form of this regression using a spline to avoid bias due to misspecification. Finally, we calculate the causal effect as Veitch and Zaveri (2020); pf A fred' and similarly, p(Astabilityy) (staability((())) E Astability] (v) (})
where (1) the set of neighbours of y in the representation space of year L. In this study, we consider the context window e to encompass the entire document, and we set the value of / to 5.
Entity Change Value: We use entity change value to track emerging and disappearing of spe- cific TDMM entities associated with task, quan- tifying these changes and capturing related entity occurrences within a specific time period. Put sim- ply, we measure the difference in the co-occurrence frequency of a TDMM entity x and a task y be- tween two years l and 22. When we identify a significant change in the co-occurrence frequency of x and y over this period, it likely signals a shift in the relation between and y and, in turn, a shift in NLP Research trends. We define entity change value 8, ( of an entity x of type 7(x) E [task, dataset, metric, method] with respect to task y as the absolute difference in frequencies of x co- occurring with y in the same sentence, between years t and f2 normalized by the total number of entities of the same type as x that co-occur with in both years. The entity change value is given by: City (G.a))
E[Afreg (1)l
Results and Analysis
Correlation-based measures provide a simple way to quantify the association between variables. How- ever, they fall short of explaining complex cause- effect relationships and can yield misleading re- sults. Causality is essential for gaining a deeper understanding of variable relationships, enhancing the robustness and reliability of our findings be- yond the limitations of correlation. We discuss more about the importance of causal methods over correlation- measures in Section 7. In this
Everreeenr(n) (Ce, (eg)) (ra(e))) where the frequency of x co-occurring with in year t is given by Cz(z,)) y).
(2)) =
In summary, quantify task trends and re- search context changes using task frequency change and task stability values. Below we explore the re- lationship between entity change values and these two variables and estimate the causal impact of TDMM entities on task research landscapes.

---

Auiidated Tall
Ditaet
Nasdica Monioe
Naactand Merhod
Q1. What role do the methodologies play in causally driving the shift in NLP tasks?
New methodologies have a significant influence on research in various areas of Natural Language Pro- cessing (NLP). In the field of Language Modeling, observe shift in influence between different methodologies over time.
Figure 3: Causal Graph of TDMM Entities (entity change values) and Task Entity Frequency Shift.
section, our focus is on uncovering relationships among causal variables (Section 6.1) and measur- ing the impact of TDMM entities on target task entities (Section 6.2).
Between 2003 and 2017, Recurrent Neural Net- works (RNNs) had the most decisive impact on Language Modeling research. However, this trend shifted with the emergence of Transformers, which have since become the dominant influence in search on this task.
6.1 Causal Relation between the Variables
Figure 3 shows the discovered causal graph for the frequency shift of task entities. Overall, we observe that the entity change values of associated tasks, datasets, metrics, and methods have direct causal effect on the frequency shift values of the target tasks. Since frequency shift value quantifies the trend in NLP research, we infer from the causal graph that the trend of a task is governed primarily by the life cycles of its associated TDMM enti- ties. We see similar causal relation task stability value (see Figure 4, Appendix A). Evaluation: We perform a sensitivity analysis of the causal graph by adding Gaussian noise with zero mean and unit variance to the entity change values in the data (Cinelli et al., 2019). This gives an estimate of the robustness of the graph in the presence of unob- served confounders. We observe that the graph is stable to unobserved confounding, giving all edge probabilities greater than 0.5.
Dialogue Systems, which involve automatic re- sponse generation, are closely related to Language Modeling, Therefore, research in this area is highly influenced by Generative Models. From 1990 2002, Probabilistic Models played a crucial role in shaping Dialogue Systems research, while RNNs took the lead between 2003 and 2017.
Machine Translation, another task related to Language Modeling, requires the generation of the translated text. Naturally, we observe the influence of similar entities in Machine Translation research. Probabilistic Models had the most decisive impact between 1990 and 2002. In recent years (2018- 2022), Transformers have emerged as the dominant influence in this research area. In the field of Speech Recognition, Hidden Markov Models (HMMs) have shown a signifi- cant influence. HMMs have played a crucial role in shaping Speech Recognition research between 1979 to 2002.
6.2 Causal Impact of the Variables
Named Entity Recognition (NER) has also been influenced by Hidden Markov Models, par- ticularly in its early days (1990-2002), as NER is often formulated as a sequence tagging problem. Various parser algorithms were employed to solve the problem in the period between 2003 and 2017. For Semantic Parsing, parser algorithms have been instrumental and have had significant impact on research in this area. Between 1979 and 1989, Grammar Induction techniques were used to elicit the underlying semantic parse trees.
The organizers of ACL 2018 categorize NLP research into 21 areas, and provide a set of popular tasks for each area. Out of those, we curate 16 areas and select one task from each based on its frequency of occurrence in our corpus. We estimate the effect of TDMM entities (entity change value) behind the development of these tasks (frequency shift value) (see Section 5.1) and summarize the results in Table 4. Since we do not have confounders (Section 1.,,, evaluating the causal effect reduces to estimating the conditional expectation of the frequency shift values given the entity change values. We present detailed results in Appendix A.2. We examine the results by addressing the following set of inquiries.
From 1990 to 2002, researchers employed vari- ous statistical models in Morphological Analysis, which evident from our results.
In Semantic Role Labeling, Support Vector Machines and Neural Network Models have been widely used solve this task.
"https://acl20" wtt.///oro/llloo//. coo.///ollooorppprrsss-.
In Co-reference Resolution, Neural Network

---

Primary
Tisk
2018-3022
1999-2022
Language Modeling
Recument Networiczll
Transfomerr"
Diálogue System
Probabiisikk Gesecative Models
Recuncat Netwodaa
MullWoe"
Muli
Machite Transiaionn
Probabilssii Generative Midele
WMT Doza
Transbemens
Transfomery"
Spooch Recogrition
Fidken Yarkew Midicaef
Fidden Markes Midela
Machine Translation'
Machine Trandaaiue"
Hidlon Markee Madris
Namall Friiyy Resegattion
Hiklen Markes Modds
PO6 Targing"
Ralation Estracion"
POS Targing
POS Tegaing
Text Classilorioon
Puser Alpocillams
Word Seanectation
Segisemationn
Semantic Parsigg
Induction o
Paner Algonthms
Algetttmmn M
Dependency Paning?
Alponthuss
Morphological Analysic
Striatical Midele
Dependency Parving"
(In Truehani"
Somaitcal Modele
Screntic Rol Labeling
Suppori Vecka Machincs"
Nearal Network Morkcs
Suppont Vistor Machises"
Co-cefacace Reseluion
MEC.VI Text Collection"
litege: Litex Programaing" of
Nestal Netwur Model.
Nound Netwock Models
Word Serse Dismbiguutoo
Wondeer
Maximum Models
Neatal Newwork Models"
Wardner"
Sestinest Analysis
Twitter Dutast"
Claschotion'
Teur Classilaation
Asguneat Mining
Clasification
Saniinere Analsse"
Sevtieere Analsii"
Quedhnn Amowning
Paronn Algonthens!
Informaion Instrationn
Infermsion Eximzztion
mo-Taieedl M.MM
Information Farraction"
Testual Eatsitnent
Stadical Mudes. M
hh-Tninne LLMr
Pop-Traised LLMs"
Summoristioon
Wimber
Semence Compresiion
Pn-Tiineed M.
Pee-Trained E1 Me
Table 4: Causal analysis identifies the main drivers (Methods, Tasks, Datasets) of frequency shifts in NLP tasks across four periods, with "-" indicating insufficient data for analysis.
models have gained prominence starting in 201 8. However, from 2003 to 2017, Integer Linear Pro- gramming also utilized to address this prob- lem.
pendency Treebank dataset is frequently used as benchmark, indicating its importance in driving research in this area.
During the period of 1990-2002, the creation of the MUC-VI dataset played a crucial role in advancing research in Co-reference resolution. In the field of Sentiment Analysis, the Twit- ter dataset holds significant importance in driving research in this domain.
Pre-trained Language Models (LLMs) have demonstrated superior performance in several NLP tasks, including Question Answering. Re- searchers have also explored parsing algorithms to parse questions and align them with potential answers.
Overall, our analysis underscores the vital role of datasets in shaping and driving research across various NLP tasks.
Furthermore, Textual Entailment and Sum- marization have been heavily influenced by pre-trained LLMs between 2018 and 2022, as evident from our results.
Q3. Do evaluation metrics drive paradigm shifts in NLP research?
Q2. How have changes in data availability con- tributed to the NLP Research Tasks?
Most NLP tasks rely on a standard set of metrics borrowed from other domains, such as machine learning and computer vision, to evaluate system performance. However, there is limited research dedicated to improving these metrics within the field of NLP, it often requires theoretical knowledge beyond the scope of NLP itself. Despite this, our analysis Table 5 reveals some noteworthy exceptions. Metrics explicitly designed for evaluating NLP tasks, such as BLEU and METEOR, have demonstrated significant impact in advancing Machine Translation research. Similarly, the metric ROUGE has influenced research in the field of Summarization. While perplexity scores are commonly used measure the generalization capabilities of probability distributions, they are predominantly utilized for evaluating language models NLP tasks.
High-quality datasets play a crucial role in advanc- ing NLP research. While new methodologies are important, they cannot fully propel the field for- ward without the support of high-quality datasets. Researchers understand the significance of dataset quality and actively curate datasets to drive ad- vancements in the field. Our findings further con- firm the prevalence of this trend, highlighting the strong emphasis on dataset quality in NLP research.
In the early stages of deep neural models, such as Recurrent Neural Networks (RNNs), the creation of large datasets became essential for efficient model training. Between 2018 and 2022, several datasets were curated, with MultiWoz being the most widely used dataset for research in Dialogue Systems.
In the domain of Machine Translation, the sig- nificance of datasets in shaping research direction cannot be overlooked. The influence of WMT datasets Machine Translation research evi- dent from our findings.
Q4. What is the causal impact of cross-pollina- tion of ideas between related NLP tasks? We consistently observe a pattern of related NLP tasks evolving in tandem, borrowing ideas and tech-
For Morphological Analysis, the Universal De-

---

niques from one another. This trend is clearly re- flected in our findings. For instance, Speech Recog- nition and Machine Translation are linked as re- searchers explore end-to-end systems that translate speech, and our results show that Machine Trans- lation has had the greatest influence on Speech Recognition research between 2003 and 2022.
get task entity due its prominent presence in our corpus and the NLP research landscape. We calcu- late the PMI scores of Machine Translation with all other TDMM entities. The PMI score represents the probabilities of co-occurrence between two en- tities in sentences from rescarch papers, normalized by their individual occurrence probabilities.
Named Entity Recognition (NER) is commonly approached sequence tagging problem, and it is influenced by related tasks such as POS Tag- ging (2003-2017) and Relation Extraction (2018- 2022), as these problems are often jointly solved. Similarly, POS Tagging initially posed as text classification problem (1990-2002), is significantly impacted by the word segmentation task, as evi- dent from our results in the period of 2018-2022.
Interestingly, we find that accuracy, an entity of type metric, has the highest PMI score with Ma- chine Translation among all other entities. How- ever, is important note that accuracy is widely used metric across various NLP tasks, and it is not specifically developed for machine translation, nor has machine translation influenced the concept of accuracy. This observation emphasizes the insuffi- ciency of relying solely on correlation-based met- rics to understand and analyze research influence on an entity.
In recent years (2018-2022) dependency and se- mantic parsing have been jointly solved using the same neural model, highlighting the influence of dependency parsing on research in semantic pars- ing. Sentiment Analysis has garnered consider- able research interest and is commonly framed as a text classification problem. Additionally, Argu- ment Mining, which involves understanding the sentiments behind arguments, is influenced by sen- timent analysis. Furthermore, the classification of various argument components, such as claims and evidence, often approached as text classification problems, as evidenced by our results.
We observe that relying solely on correlations can lead to misleading results and interpretations. Therefore, in order to understand the influence of associated TDMM entities NLP Task entities, we utilize causal algorithms that enable to gain insights into the ande-ean--fffet dynamics among the variables we study.
8 Concluding Remarks
In this paper, we retrospectively study NLP re- search from a causal perspective, quantifying re- search trends of task entities and proposing sys- tematic framework using causal algorithms to iden- tify key reasons behind the emergence or disappear- ance of NLP tasks. Our analysis reveals that tasks and methods are the primary drivers of research in NLP, with datasets following their influence, while metrics have minimal impact. is important note that in our analysis, we have structured the ref- erence time into four distinct intervals (see Table 1); however, it can be applied to diverse timeframes, ranging from longer periods to brief intervals, in- cluding single years. This adaptability, in the con- text of rapid recent advancements in NLP, allows to zoom in on local trends and developments that might otherwise go unnoticed (such as the influ- ence of in-context learning on NLP tasks).
7 Discussion: Correlation and Causation Causation and correlation, although related, are distinct concepts. While they can coexist, corre- lation does not simply imply causation. Causa- tion signifies direct cause-and-effect relationship. where one action leads to specific outcome. In contrast, correlation simply indicates that two ac- tions are related in some way, without one neces- sarily causing the other.
"correlation does not imply causation" Pearson (1892)
In our work, we focus on causal inference from data. While correlation-based measures provide a straightforward method for quantifying associa- tions between variables, they often fall short when it comes to explaining complex cause-and-effect relationships.
We believe our causal analysis enhances under- standing of the interplay of research entities in NLP. contributing to the growing body of work on causal- ity and NLP (Feder et al., 2021). We provide with additional analysis and insights in Appendix B.
To demonstrate the effectiveness of our frame- work, we establish simple baseline using a PMI- based correlation measure (Bouma, 2009). For this analysis, we select Machine Translation as our tar-

---

Limitations
Acknowledgements
This work is centered on NLP research papers from ACL Anthology, with a focus on papers from the "ACL Events" category. The "ACL Events" cate- gory encompasses major conferences, workshops. and journals, including ACL,, NAACL, EMNLP, EACL, AACL, CL., and TACL.. We also include papers published COLING from the "non-ACL Events" category. Nevertheless, it is important to acknowledge the presence NLP papers beyond ACL Anthology in Al journals, regional confer- ences, and preprint servers. Furthermore, we recog- nize that certain NLP papers may become available preprint servers before their official publication in peer-reviewed venues. In this study, we focus ACL Anthology, which can introduce time lag when assessing the early impact of influential pa- pers released as preprints (e.g., BERT) or only on preprint servers (e.g.. RoBERTa). To address such challenges, we leave the curation and inclusion of NLP research papers from these alternative sources for future works.
We thank Ilia Kuznetsov for his feedback on the initial version of this work. We appreciate all the anonymous reviewers for their helpful comments and suggestions for further analysis. This work has been funded by the German Research Founda- tion (DFG) part of the Research Training Group KRITIS No. GRK 2222.
References
2008-2022 Grobid. https://gttubb.co///emmiit grobid.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence label- ing. In Proceedings of the 27th International Con- ference on Computational Linguistics, pages 1638- 1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
John M Barrios and Yael Hochberg. 2020. Risk per- ception through the lens of politics the time of the covid-19 pandemic. Working Paper 27008, National Bureau of Economic Research.
Our framework requires research papers tagged with entities as input. Hence, the quality of the tags plays a crucial role in the causal inference of our proposed method. The taggers generate noisy out- puts and, thus, might require human intervention to denoise the tags. Moreover, causal algorithms require large amount of data to produce statis- tically significant results. Hence, research areas that are less explored or newly emerging may not always be suitable for this framework to be applied. Additionally, we highlight that in this work, we do not consider extra-linguistic factors like author affiliations, funding, gender, etc. We leave them for future research work.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762-4779, Flo- rence, Italy. Association for Computational Linguis- tics.
Gerlof Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. Proceedings of GSCL, 30:31-40
Kevin W. Boyack and Richard Klavans. 2022. An im- proved practical approach to forecasting exceptional growth in research. Quantitative Science Studies, 3(3):672-6933
Ethics Statement
Carlos Cinelli, Daniel Kumor, Bryant Chen, Judea Pearl, and Elias Bareinboim. 2019. Sensitivity analysis of linear structural causal models. In Proceedings of the 36th International Conference on Machine Learn- ing, volume 97 of Proceedings of Machine Learning Research, pages 1252-1261. PMLR.
In this work, we use publicly available data from ACL Anthology and do not involve any personal data. It is important to recognize that, while our framework is data-driven, individual perspectives toward research are inherently subjective. Deci- sions involving science should consider data as well as ethical, social, and other qualitative factors. Furthermore, underscore that the low influence of TDMM entities in our analysis should not be the sole reason for devaluing research papers or reducing their investments. Ethical and academic considerations should guide decisions on research evaluation and resource allocation.
Diego Colombo and Marloes H. Maathuis. 2014. inder-independent constraint- causal structure learning. Journal of Machine Learning Research, 15(1 15(116):3921-33
Quang Do, Yee Seng Chan, and Dan Roth. 2011. Min- imally supervised event causality identification. Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 294- 303, Edinburgh, Scotland, UK. Association for Com- putational Linguisties.

---

Jesse Dunietz, Lori Levin, and Jaime Carbonell. 2017. The BECauSE corpus 2.0: Annotating causality and overlapping relations. In Proceedings of the 11th Linguistic Anmotation Workshop, pages 95-104, Va- lencia, Spain. Association for Computational Lin- guistics.
Bernard Koch, Emily Denton, Alex Hanna, and Ja- cob Gates Foster. 2021. Reduced, reused and re- cycled: The life of a dataset in machine learning search. In Thirty-fitth Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track (Round 2).
Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brandon M. Stewart, Victor Veitch, and Diyi Yang. 2021. Causal inference in natural language processing: Estimation, prediction, interpretation and beyond. CoRR, 0ass/209.00772
Thomas S Kuhn. 1970. The structure of scientific revo- lutions, volume 111, Chicago University of Chicago Press.
Shixia Liu, Yang Chen, Hao Wei, J. Yang, Kun Zhou, and Steven Mark Drucker. 2015. Exploring topi- cal lead-lag across corpora. IEEE Transactions on Knowledge and Data Engineering, 27:1
Christian Fong and Justin Grimmer. 2016. Discovery of treatments from text corpora. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1600-1609, Berlin, Germany, Association for Com- putational Linguistics.
Yi Luan, Luheng He, Mari Ostendorf, and Hannanch Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 3219-3232, Brussels, Belgium. Association for Computational Linguisties.
Jonathan Grudin. 2009. Al and HCI: Two fields divided by a common focus. Al Magazine, 30(4):48-48.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. In Advances in Neural Information Processing Sys- tems, volume 26. Curran Associates, Inc.
David Hall, Daniel Jurafsky, and Christopher D. Man- ning. 2008. Studying the history ideas using topic models. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 363-371. Honolulu, Hawaii. Association for Computational Linguistics.
Paramita Mirza and Sara Tonelli. 2014, An analysis of causality between events and its relation to temporal information. In Proceedings of COLING 2014, the 25th International Conference on Computational Lin- guistics: Technical Papers. pages 2097-2106, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2019. Identification of tasks, datasets, evaluation metrics, and numeric scores for scientific leaderboards construction. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5203- 5213, Florence, Italy. Association for Computational Linguistics.
Saif M. Mohammad. 2019. The state of NLP literature: A diachronic analysis of the acl anthology. ArXiv, 1////9911.3366
Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2021. TDMSci: A spe- cialized corpus for scientific literature entity tagging of tasks datasets and metrics. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 707-714, Online. Association for Computa- tional Linguistics.
Saif M. Mohammad. 2020. Examining citations of nat- ural language processing literature. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5199-5209, On- line. Association for Computational Linguistics.
Mattco Negri, Marco Turchi, José G. C. de Souza, and Danicle Falavigna. 2014. Quality estimation for automatic speech recognition. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1823, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
Zhijing Jin Zeyu Peng. Tejas Vaidhya, Bernhard Schoelkopf, and Rada Mihalcea. 2021. Mining the cause of political decision decision-making from social media: A case study of COVID-19 policies across the US states. In Findings of the Association for Compu- tational Linguistics: EMNLP 2021, pages 288-301, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jong-Hoon Hoon Oh, Kentaro Torisawa, Chikara Hashimoto, Ryu lida, Masahiro Tanaka, and Julien Kloetzer. 2016. A semi-supervised learning approach to why- question answering. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'6 16, page 3022-3002000 3029. AAAI Press.
Katherine Keith, David Jensen, and Brendan O' O'Connor. 2020. Text and causal inference: A review of using text to remove confounding from causal estimates. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 5332- 5344, Online. Association for Computational Lin- guistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: a method for automatic eval- uation of machine translation. In Proceedings of the

---

40th annual meeting of the Association for Computa- tional Linguistics, pages 311-318.
773-783, Vancouver, Canada. Association for Com- putational Linguistics.
Judea Pearl, Madelyn Glymour, and Nicholas P Jewell. 2016. Causal inference in statistics: A primer. John Wiley & Sons.
Ana Sabina Uban, Cornelia Caragea, and Liviu P. Dinu. 2021. Studying the evolution of scientific topics and their relationships. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1908-1922 Online. Association for Computa- tional Linguistics.
Karl Pearson. 1892. The grammar of science. Nature, 46(1185):247-27.
Vinodkumar Prabhakaran, William L.. Hamilton, Dan McFarland, and Dan Jurafsky, 2016. Predicting the rise and fall of scientific topies from trends in their Proceedings of the 54th An- nual Meeting of the Association for Compitational Linguistics (Volume 1: Long Papers), pages 1170- 1180, Berlin, Germany, Association for Computa- tional Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. & ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30, Curran Associates, Inc.
thetorical framing.
Victor Veitch, Dhanya Sridhar, and David Blei. 2020. Adapting text embeddings for causal inference. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), volume 124 of Proceed- ings of Machine Learning Research, pages 919-928. PMLR.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chan- dra Bhagavatula, Nicholas Lourie. Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if- then reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):307-30033
Victor Veitch and Anisha Zaveri. 2020. Sense and sensi- tivity analysis: Simple post-hoc analysis of bias due to unobserved confounding. In Advances Neural Information Processing Systems, volume 33, pages 10999-11009, Curran Associates, Inc.
Isabel Regura-Bedmar, Paloma Martínez, and María Herrero-Zazo. 2013. SemEval-20113 task 9: Extrac- tion of drug-drug interactions from biomedical texts (DDIExtraction 2013). In Second Joint Conference on Lexical and Computational Semantics ( (*SEM). Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013). pages 341-350, Atlanta, Georgia, USA. Association for Computational Linguistics.
Jan Philip Wahle, Terry Ruas, Mohamed Abdalla, Bela Gipp, and Saif M. Mohammad. 2023. We are who we cite: Bridges of influence between natural language processing and other academic fields. In Proceedings of the 2023 Conference on Empirical Methods Natural Longuage Processing. Singapore, Singapore. Association for Computational Linguisties.
Naoki Shibata, Yuya Kajikawa, Yoshiyuki Takeda, and Katsumori Matsushima. 2008. Detecting emerging research fronts based on topological measures in ci- tation networks of scientific publications. Technova- tion, 28( 28(11)11158 (1)77558777
Laura Wendlandt, Jonathan K. Kummerfeld, and Rada Mihalcea. 2018. Factors influencing the surprising instability of word embeddings. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 7 [Long Pa- pers), pages 2092-2102, New Orleans, Louisiana. Association for Computational Linguisties.
Shohei Shimizu, Takanori Inazumi, Yasuhiro Sogawa, Aapo Hyvarinen, Yoshinobu Kawahara, Takashi Washio, Patrik O Hoyer, Kenneth Bollen, and Pa- trik Hoyer. 2011. Directlingam: A direct method for learning a linear non-gaussian structural equa- tion model. Journal of Machine Learning Research- JMIR, 1225-11222-255
Christopher KI Williams and Carl Edward Rasmussen. 2006. Gaussian processes for machine learning, vol- ume 2. MIT press Cambridge, MA.
Janvijay Singh, Mukund Rungta, Diyi Yang. and Saif Mohammad, 2023. Forgotten knowledges Examin- ing the citational amnesia in NLP. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages
Dongxiang Zhang, Lei Wang. Nuo Xu, Bing Tian Dai, and Heng Tao Shen. 2018. The gap of semantic parsing: A survey on automatic math word problem solvers. CoRR, abs/1808.07290.
Toronto, Canada. Association for Computational Linguistics.
Henry Small. 2006. Tracking and predicting growth areas in science. Sciemtometrics, 68(3):995-610
A Appendix: Additional Results
Chenhao Tan, Dallas Card, and Noah A. Smith. 2017. Friendships, rivalries, and trysts: Characterizing re- lations between ideas in texts. In Proceedings of the 55th Annual Meeting of the Association for Compu- lational Linguistics (Volume I: Long Papers), pages
A.1 Causal Relation
In Figure 4, we observe that entity change values of tasks, datasets, metrics and methods have direct causal influence on task stability value.

---

Associated Tasks
Associated Datasets
Associated Metrics
Associated Methods
Target Task Stability
Figure 4: Causal Graph: The graph shows that the emergence and disappearance of TDMM entities (entity change values) have direct causal effect on the stability of task entities.
A.2 Causal Effects
bank" dataset emerged pivotal influence, sig- nificantly impacting tasks such as "Language Mod- cling," " "POS Tagging," and "Semantic Parsing." Using our framework, a similar analysis could also be done between any chosen periods.
In Table 5, we observe the entities (Tasks, Datasets, Methods, and Metrics) that influence research given NLP Task.
B Appendix: Supplementary Analysis
B.2 Entitiy Influence on Task Frequency and Stability
addition to the primary results presented in the paper (Section 6), in this section, we describe the supplementary analysis.
Influence of Research Entities on Task Stability. We measure the causal effect of research entities on Task Stability Value (see Section 5.1). From the resulting causal graph (Figure 4), we observe that the entity change values of associated tasks, datasets, metries, and methods directly impact the stability value of the target task, similar the task frequency shift value.
B.1 NLP Tasks and Their Dataset Evolution
Frequently Pursued NLP Tasks. From Table in our paper, we observe that overall (from 1979- 2022), among all the tasks, "Text Classification" (column 6) holds remarkable position. This prominence stems from the frequent usage of var- ious NLP tasks being framed or aligned "Text Classification" or borrowing concepts from it to ad- dress other tasks such as "Sentiment Analysis" or "Word Sense Disambiguation." Additionally, our framework offers the flexibility to perform a similar analysis between any chosen periods.
Correlations Between Task Frequency Change and Stability. We observe a slightly positive cor- relation between frequency change and stability of research tasks with Pearson coefficient of 0.08. This is because when a new task emerges, initially, a few researchers start working on it, which grad- ually increases its frequency of appearance. At the same time, researchers experiment with various methods and datasets to solve these newly emerged tasks, causing high instability (e.g., Math Problem Solving (Zhang al., 2018)). On the contrary. the opposite not always true: well-defined tasks are often the most researched, and yet researchers always explore new ideas on these tasks, which harms stability.
Evolution of Datasets in NLP Tasks. Refer- ring to Table 5 in our paper, in the context of "Speech Recognition," we observe a shift in in- fluential datasets over different periods. Between 1990-2002, the "WSJ Corpus" took the lead, while in the subsequent period of 2003-2017, the "ATIS Dataset" had more influence. Interestingly, be- tween 2018-2022, trend shifted once again to the "Switchboard Dataset".
Overview and Insights. Our analysis shows that research in NLP is primarily driven by tasks and methods; the influence of datasets follows them, and metrics have minimum impact. Our analysis of frequency shift values reveals the gradual paradigm shift in NLP research. Initially, the focus was practical problems such Speech Recognition and Machine Translation. However, over time, researchers ventured into more complex areas like textual entailment and argument mining, neces- sitating domain knowledge and extensive data rea-
A similar trend is reflected in the "Summariza- tion" task as well: in the years 1990-2002, "Word- net" played significant role, while the "Gigaword Dataset" took over in 2003-2017. However, in the most recent period of 2018-2022, "Pubmed" emerged as the notable dataset for the "Summariza- tion" task.
Common Datasets Across NLP Tasks. We ob- serve from Table (column 6) that across the entire span from 1979 to 2022, the "Penn Tree-

---

Primury Cause
Tisk
1979-1989
1900-2002
2005-2017
2018-30122
1979-3022
Recunctt Neural Networks
Transformess"
Transfomees"
Language Modelling
Machine Trassmiia" Porn Treehonk
Text Generation
Perplesity"
Text Generason" Perplesity m
Perpleuy"
Super
Treebunk P
Recuneet Neura Networks 2f
MuilWoe" Transtormens" Mf
MulliWos" "
MultWwr P
"Tansfoomers" A
Dialogue Systes
Language Geoention"
Resporse Genzasson" Ronge"
Resporse Generasea" Rouge"
Perplesnt"
Probubiliticc Genzrazive Models
WMI Dita"
Transformees"
Transformess " METEOR Language Generasio" WMT Dura
Machina Trindatioon
Spesch Recagnition
Perplosit"
Aftention Mochanism M
BLEU*
Language Modeling"
METBOR"
Pern Trrebank
Langunge Generation 7
WMT Dan!"
Hiltenn Markos Modkle Madiinne Tranilationn Perplesity
Hidden Mirkos Miskcle
Machine Transatioon
Michine Tranchtionn
Hiälen Markew Medeleff
Sperch Recogrition
WSI Ciepus
Perplesnt
Hildon Madkey Moddle
ATIS Dutuser
Swachbourd Dutare; D
Accudic Modcl
Language Modeling"
Perplesity ATIS Danser
Language Modelling"
Wonl Rase"
Wonl limor
Hidden Markon Modele
POS Tagging"
Relution Ustractionn
POS Taggiing
Information Extraction"
Conditional Random Hielss
Wisipedaa Cocpuç
Kandom Elelds
Namted Emily Recognition
Genia Corpuc
Pohmad
Pro My
Cunditional Choonctes O Soove"
Soore"
Sccee"
Soor"
Teat Classittation"
Parser Algocittums"
Word Segincttaioo"
Word Seamncitaiioa
POS Tigging
Disonminative Mo
Pern Trrehank Seorc
Wond Segmentzion"
Porn Trachank
Neural Network Middets Pern Trrahank Soore"
Neural Network Middll Penn Treehink D Secre"
Sccee"
Wooinar"
Maximum Ettropy Modes
Neural Network Modals
Wordner Neural Network Models Test Clasiftzaiion Fl Score"
Semamic Tagging"
Text Chastheatsnn
Test Ulaseticrionn
Word Sense Disuntiguation
Discimivation Medrishf
Winhe"
Wordhar
Accuracy"
Fl Seore"
FI Soore"
Sattzcal Midels
Dependeacy
"rrebank"
Statitical Middlls Af
Worl Segmenttton
Statistical Models
Pre-Tiieed M
Dependency Phesing
Morphotogical Anilyis
Tirehank
Thashenk" P
Lammutintioon
LTD Treehank P Accussy"
Accuracy
Accussy"
F1 Soare"
In M
Ap 3f Information Estraction"
Algoriihmeli
Depredency Parsin"
Algerttme Pf Pes Treehask"" 0 Degendeuzy Parking" Fl Score"
Infoemation Ratiivall Accuacy
Dependescy Preving"
Purser Algorihume 3
Semantic Parsing
Pein Treebank" Score"
Pana Treekauk"
Pesa Treebnkk
F1 Sccce"
Score"
Support Vactor Machineek
Neuaa Network Modhelef
Suppurt Victor Muchinee
Semantic Labeling
Relation Estration"
Nanee Eaity Recogration
Namial Emity Recognilson
Propbask"
Propbaak" F1 Score"
Progbask"
P1 Sccre"
Pll Score"
MUC--VI Teat Colliation Discriminator Modcl if Word Sense Disaniiuation Scare"
Integer incer Programing"
Neusal Nataork Modielef
Neural Network Modele
Cu-refeencce Recolation
Maniia Detection"
Oxtonotes
Manion Desecion?
Oatonmees"
Ontonotes" ! Sccrem Mentina Detection
Soare"
Soarc"
Twite:: Datsset
Teat Classiteation
Tout Clasiffestion
Sentiment Analysis
Text Classtication
Models
Amazon Reviess
Inined L
Newral Netwok Models A
Twaner Daisse P Fl Score"
Sccre"
F1 Score"
Texa Clissifictiou"
Seatiness Analysis
Saninteat Analysis
Neural Nework Motess ot
Neural Network Modaish
Neural Network Moddlss o
Argument Miniag
Wilipedia Compus" /
Wikipedia Corpuc"
Wilipedis Corpur"
Parsing Algonthme 1 Informotion Ratioval" Accuracy
Information Estraction" Wortinr?
Information Ixtraction' F
Pre Trannd LLMs
Intormation Extroction Pre-Traine L.M Squad
Sqndt
Question Answeiing
Aucuracy Statidiial Midellef
Algerilun
Summtrizatios
Secre"
Suome"
Statistical Modea Informtion Fairation"
Pre Trinnee "
Pte 'Trained LLM," SNT.1 D
SNI INIITTane Tarrus
Textual Entaliment
Score
Text Clacttiaaion
Text Claseiicatonn Seree" Pre "
Worting
Sentence Compression
LLM??
Probabilistic Genzrative Models I'l Suore" Information Retrieval"
Recureet Neural Nttworss
Konge"
Rouge" Quesion Answinin" Pohmaol
Summurization
Rouge" Gipaoced
Puhmedd
Queetion Asvwering"
Table 5: The primary reason behind the frequency shift of the tasks. We analyze the trends in four different periods of reference. Most influential TTsk(T), Dataset(D), Method(M) and Metric(m) are given in the decreasing order of their influence. "." means there is not enough data instances for the causal analysis.
soning. Examining stability values, we note that pre-trained language models have emerged ver- satile solutions, reducing the need for task specifce approaches.
C
Appendix: Algorithms
C.1 DirectLinGAM
In Algorithm 1, we describe the DirectLinGAM algorithm (oracle version) in high level as described by Shimizu et al. (2011).

---

D Appendix: Multiple Linear Regression
We use multiple linear regression to regress a variable on several variables (Pearl et al., 2016). For instance, if we want to predict the value of a variable Y using the values of variables X2, .... Xk-1,-B; we perform multiple linear regression of Y on (X1, X2, .... Xx 1, X&), and estimate regression relationship (Eqn. 1), which represents an inclined plane through the (k + 1)- dimensional coordinate system.
Y T0 +
rx
Algorithm 1: Causal Graph Discovery:
DicccLinGAA--lgorrimm
The Gauss-Markov theorem (Williams and Ras- mussen, 2006) simplifies the computation of par- tial regression coefficients (r1, (r,,-"7 P'k in Eqn 1). It states that if write Y as a linear combination of X1, X2, X 1, Xg and noise term 6,
Given p-dimensional random vector set of its variable subscripts U and data matrix of the random vector as X, initialize an ordered list of variables
K := and m :=
2 Repeat until p- subscripts are appended to K: Perform least square regression of x; and zy. Vi U K(i + j) and compute the residual vectors أ(5 and the residual data matrix R(5) from the matrix X, Vj U K. Final variable Tm independent its residuals and append to the end of K:
Y +
+i +e
(2)
2=1
then, regardless of the distributions of the vari- ables Y, X1, X2, Xs. the best least-square coef- ficients obtained when is uncorrelated with each regressors, i.e.,
Cou(e, X;) -,, 1,,,
(3)
3 Append the remaining variable to the end of K:
Construct a strictly lower triangular matrix B by following the order in K, and estimate the connection strengths by by using some conventional covariance-based regression such as least squares and maximum likelihood approaches on the original random vector x and the original data matrix X:


Reviews:
[{"review_id": "F6b0fmvKYS", "reviewer": "Reviewer_s1DD", "summary": "", "questions": "Suggestions\n\n* Provide more findings and insights\n\nI would appreciate it if more findings and insights are provided. If this work is task-oriented, then could you show which task scientists are biased? or is there any task which scientists over-relying? over-used because it is simply easy to evaluate? Which task has multiple datasets while which task only has a common standard dataset? These interesting questions will make this work more valuable, and it will improve the value of this work.\n\n* More specific language\n\n\"Causality\" is used a lot in this paper, but it feels too strong to describe this work. Rather, what would you think of revising by putting more specific terms, such as \"task-oriented\"?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful comments. In the following, we address your questions with clarifications. \n\n**About providing more findings and insights**\n\nThank you for your excellent suggestions. Indeed, our framework can be used to answer a wide range of specific questions. Below, we provide analysis answers to your questions, and we will add these additional analysis results in the final version of our paper.\n\n1. **Are there any tasks scientists over-rely on?**\n\n   *From Table 5 in our paper, we observe that overall (from 1979-2022), among all the tasks, “Text Classification” (column 6) holds a remarkable position. This prominence stems from the frequent usage of various NLP tasks being framed or aligned as “Text Classification” or borrowing concepts from it to address other tasks such as “Sentiment Analysis” or “Word Sense Disambiguation.” Additionally, our framework offers the flexibility to perform a similar analysis between any chosen periods.*\n\n2. **Which task has multiple datasets?**\n\n   *Referring to Table 5 in our paper, in the context of “Speech Recognition,” we observe a shift in influential datasets over different periods. Between 1990-2002, the “WSJ Corpus” took the lead, while in the subsequent period of 2003-2017, the “ATIS Dataset” had more influence. Interestingly, between 2018-2022, the trend shifted once again to the “Switchboard Dataset.”* \n\n   *A similar trend is reflected in the “Summarization” task as well: in the years 1990-2002, “Wordnet” played a significant role, while the “Gigaword Dataset” took over in 2003-2017. However, in the most recent period of 2018-2022, “Pubmed” emerged as the notable dataset for the “Summarization” task.* \n\n3. **Which tasks have a common dataset?** \n\n   *We observe from Table 5 (column 6) that across the entire span from 1979 to 2022, the “Penn Treebank” dataset emerged as a pivotal influence, significantly impacting tasks such as “Language Modeling”, “POS Tagging”, and “Semantic Parsing”. Using our framework, a similar analysis could also be done between any chosen periods.* \n\n\n\n**About using more specific language to describe causality**\n\nThank you for your suggestion. We acknowledge the importance of explicitly distinguishing between correlation and causation and avoiding overstatements. Inspired by your suggestion, we think **“TDMM-Task” causal analysis** would be a more accurate term to describe our work since we leverage causal analysis algorithms to model the relationships between \"TDMM entity change values\" and two task-oriented variables (\"task frequency\" and \"task stability\"). Furthermore, apart from Lines 411- 424, we put more details on the merits of causal analysis over correlation analysis in Appendix C. In the final version, we will integrate the specific terms mentioned above to enhance the clarity of our work’s scope. In addition, we will move the key points from Appendix C into the main paper to clarify the advantage of causal analysis over correlation analysis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is important to provide insights into a holistic research view. It helps scientists not to be biased toward one very trendy topic but also it can suggest scientists pay attention to other topics", "reasons_to_reject": "* Lack of deeper findings and insights\n\nThe value of this work relies on how deeply they investigate findings and sights since this work deploys the proposed model. However, the provided findings are rather obvious and simple, such as \"neural networks\" are commonly studied. I was expecting more specific questions from this automatic framework. Sections 2-4 can be shortened or moved to the appendix, then I believe more space can be used for this.\n\n\n* Ambiguous term for causality\n\nAs far as I understand, this work models the relationships between \"entity change values\" and two variables of task-oriented, \"task frequency\" and \"task stability\". It seems too strong to use \"causal\" for this.", "questions_to_authors": "Suggestions\n\n* Provide more findings and insights\n\nI would appreciate it if more findings and insights are provided. If this work is task-oriented, then could you show which task scientists are biased? or is there any task which scientists over-relying? over-used because it is simply easy to evaluate? Which task has multiple datasets while which task only has a common standard dataset? These interesting questions will make this work more valuable, and it will improve the value of this work.\n\n* More specific language\n\n\"Causality\" is used a lot in this paper, but it feels too strong to describe this work. Rather, what would you think of revising by putting more specific terms, such as \"task-oriented\"?", "paper_topic_and_main_contributions": "This work proposes a framework for analyzing research topics from the scientific article corpus.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5HergwUKyr", "reviewer": "Reviewer_2oor", "summary": "", "questions": "Quesiton A: the advantage of causal analysis over correlation analysis is important for understanding the motivation of causal analysis. Can you include these discussions in the main paper instead of the appendix?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your recognition of the strengths of our work. Thank you for your comments and feedback to improve our paper. We will fix the typo and incorporate your suggestions into the final version of the paper. \n\n**Why not ArXiv papers?**\n\nWe acknowledge the point about some papers being posted on ArXiv before appearing on ACL Anthology. However, it's important to note that ArXiv lacks the peer-review process that guarantees the quality of papers, resulting in a mix of reliable and less reliable content. Therefore, for our paper's analysis, we prioritize the use of ACL Anthology data due to its rigorous peer-review process.\n\nIn response to your example involving the BERT paper, we recognize that while the BERT paper was available on ArXiv from October 2018, it was included in ACL Anthology in 2019. Our framework, which is based on automatically identified Task/Dataset/Method/Metric (TDMM) entities in the whole paper, can effectively measure BERT's influence on ACL Anthology papers mentioning “BERT” prior to 2019 or published at the same venue as the BERT paper (NAACL 2019). Nevertheless, we acknowledge that there is indeed a slight time lag when analyzing the starting point of impact for an influencing paper that is first published at ArXiv (e.g., BERT) or only published at ArXiv (RoBERTa). Currently, our analysis results can only accurately reflect the impact of TDMM entities among papers published at the ACL Anthology. In the future, such problems can be addressed by including the high-quality ArXiv papers into our analysis dataset.\n\n**About Identifying Local Research Trends**\n\nWe argue that our framework is flexible in identifying local research trends by adjusting the time window as needed. We motivate this flexibility with the example of “Question Answering” (Table 4, row 14). We recognize that while \"Information Extraction\" generally holds a significant role in \"Question Answering\" research (column 6), the shorter-term analysis (2018-2022) highlights the predominant influence of Pre-trained LLMs on \"Question Answering\" (column 5).\n\nWe believe that our framework could be applied to even shorter time periods to analyze local trends. In light of your suggestion, if one applies our framework to recent data within an appropriate time frame, we will be able to measure the impacts of recent developments like instruction-following LLMs and in-context learning on NLP Tasks. \n\nWhile we acknowledge that this point was not explicitly highlighted in our paper, we would like to point out that, using our framework, one will be able to measure the year-wise causal influence of multi-task learning, reinforcement learning, and instruction learning on NLP Tasks, and comparing these values will enable us to mark the year when the research paradigm shifted from multi-task learning and reinforcement learning to instruction learning. \n\nWe will further clarify these in the final version of the paper. \n\n\n\n**About including Appendix C on merits of causal analysis over correlation analysis in the main paper**\n\nThank you for the suggestion; in the final version, we will include this in the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses a meaningful research question: automatic historical analysis is helpful for both senior and junior researchers\n2. The paper conducts a neat and smart causal analysis, designing novel methods to quantitatively represent task stability and shifting.\n3. The analyses cover a long period of NLP research and a wide range of tasks.", "reasons_to_reject": "1. The paper split the papers according to their publication years on the ACL anthology. However, many papers are posted on arxiv much earlier than ACL anthology. For example, the BERT paper is available on arxiv from Oct. 2018 (when its influence started), but is on ACL Anthology from 2019. This may more or less influence the soundness of the causal analysis.\n2. The paper claims that the framework aids the literature survey, which is true. However, the listed overviews are mostly \"Global Research Trends\" that are well-known by the community. For example, it is known that BLEU and Transformers fuel machine translation. The paper didn't illustrate how to use the framework to identify \"Local Research Trends\" which are more useful for daily research: for example, what is the impact of instruction-following LLMs and in-context learning on NLP tasks? How do multi-task learning and reinforcement learning transfer to instruction learning?", "questions_to_authors": "Quesiton A: the advantage of causal analysis over correlation analysis is important for understanding the motivation of causal analysis. Can you include these discussions in the main paper instead of the appendix?", "paper_topic_and_main_contributions": "Given the sheer volume of NLP publications, it is hard for researchers to conduct historical studies. The paper proposes a casual analysis-based framework to quantitatively and automatically analyze the trend of NLP research and the relations between task, data, method, and metric entities.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 575 should be BLEU instead of BLUE"}, "extra_text": {}}, {"review_id": "QhBj6moy8q", "reviewer": "Reviewer_Lzm4", "summary": "", "questions": "A) The paper says: \"Our goal is to address key research questions associated with a task entity t: (a) Which entities E effectively indicate the research trends for this task t? (b) Are there discernible causal relationships between t and E? (c) What is the extent of the causal impact exerted by E on t?\"\n\nI find this somehow unclear. \"Task\" is one of the four entities in TDMM, and the objective is to analyze how these entities relate to a specific task and shape the research trends for it. So how can the variable \"task\" be on both sides at the same time? How do you analyze how tasks influence /relate to a task? \n\nB) Are research topic and task interchangeable concepts in the paper?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback aimed at improving the clarity of our paper's explanation. We contend that these adjustments are straightforward to implement, and we will certainly include them in the final version of the paper. Below, we provide clarifications for the raised queries.\n\n\n\n**How can the variable \"task\" be on both sides at the same time? How do you analyze how tasks influence/relate to a task?**\n\nYes, “task” is indeed one of the Task/Dataset/Method/Metric (TDMM) entities. We clarify that we measure the influence on a target task entity by other related task entities. We illustrate this in lines 076-090 and Figure 1, where we showcase the causal effect on the target task “Machine Translation” by the related task “Language Generation” between the years 1980 and 2022. Further details are elucidated in Table 4. For example, we observe that overall “Named Entity Recognition” is most influenced by “POS Tagging” (Table 4, row 5, column 6); similarly, the “Text Classification” task influences “Sentiment Analysis” the most (Table 4, row 12, column 6), and “Information Extraction” influences “Question Answering” the most (Table 4, row 14, column 6). We will further clarify these in the final version of the paper. \n\n\n\n**Are research topic and task interchangeable concepts in the paper?**\n\nNo, tasks and topics are not interchangeable, but we utilize tasks to approximate a research topic. We have acknowledged this distinction in our paper, specifically in line 048, where we state that “... an NLP Task is commonly viewed as a focused research topic … ”. We will clarify this in the final version of our paper. \n\n\n\n**The flow of experiments and analysis**\n\nThe logical flow of the steps is illustrated in Figure 2, where each subsequent step builds upon the previous ones; we have adhered to the logical flow of these steps throughout the paper and further refer to them in Section 4 (line 219) and Section 5 (line 305). We will further clarify it in the final version of the paper.\n\n\n\n**Suggestions on stylistic improvements**\n\nThank you for your suggestions; we will incorporate them in the final version of our paper.\n\n1. **the formula in 235 is not explained:** We use the standard formula of multiple-linear regression in line 235, citing the relevant paper (line 218). However, we will certainly add an explanation of the variables in the final version of the paper.\n2. **formula in 363 needs more explanation:** In addition to lines 355-362, we will add a detailed explanation of the formula in line 363 in the final version of the paper. \n3. **is machine learning a different domain?** Yes, for the analysis of this paper, we distinguish machine learning as a separate domain (following the ArXiv Category Taxonomy) where the primary focus is on general learning algorithms and techniques. The same applies to computer vision. We will clarify this in the final version. \n4. **Explanation of Table 3:** While we explain Table 3 in lines 273-294, we will add a detailed explanation in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a bibliometric application of NLP that can provide a novel researcher with a good picture of the evolution of their research topic over the years, and which methods, datasets and metrics have played a role in each task at each moment. For this paper, the authors have applied their framework, based on NLP techniques, to NLP literature, but it can be applied to any other research field.\n\nThe discussion of the results, in the form of question-answer is very refreshing and helps understand the outcomes of the approach. Particularly, I enjoyed the results of Q4, which I found the most interesting, as they unveil more hidden relations, which can actually be very useful not only for novel researchers but also for experienced researchers who wish to try new approaches.", "reasons_to_reject": "I feel the paper still needs some work, as I found it quite confusing. I think some information needs to be said before so the reader understands the work developed in this paper from the beginning. For instance, the authors talk about how TDMM influence tasks, but tasks are one of the elements of TDMM. So do they analyze how tasks influence /relate to tasks? In line 254, they mentioned NUMBER of tasks, as one of the entities, but I think they don't refer to this again in the rest of the paper, and they go back to call task to both, the main variable and one of the entities. I find that very confusing and I am still not sure that I understand it well. \n\nThe sentence\n\n\"all four variables (number of tasks,\ndatasets, metrics, and method entities) are used to\npredict the number of task entities in subsequent\nyears\".\n\nfrom line 254, clarifies something, but it might be more useful if introduced sooner.\n\nAlso, I found it difficult to follow the flow of the experiments, I think the structure of the paper could be reviewed. There is more information needed in the introduction. Some formulas need (more) explanation.", "questions_to_authors": "A) The paper says: \"Our goal is to address key research questions associated with a task entity t: (a) Which entities E effectively indicate the research trends for this task t? (b) Are there discernible causal relationships between t and E? (c) What is the extent of the causal impact exerted by E on t?\"\n\nI find this somehow unclear. \"Task\" is one of the four entities in TDMM, and the objective is to analyze how these entities relate to a specific task and shape the research trends for it. So how can the variable \"task\" be on both sides at the same time? How do you analyze how tasks influence /relate to a task? \n\nB) Are research topic and task interchangeable concepts in the paper?", "paper_topic_and_main_contributions": "The paper explores The ACL Anthology of NLP academic publications from 1979 to 2022, in order to unveil the connection between NLP tasks and what the authors call tasks, methods, datasets and metrics (TDMM), over the years, with the objective of offering a picture of the evolution of NLP research, both from a general perspective and looking at the most popular NLP tasks individually.\n\nThe authors use statistical, correlation-based methods to quantify the relation between variables and causal algorithms to explain to what extent that connection shapes the evolution of tasks of focus in NLP research.\n\nThis work presents a bibliometric application of NLP that can be useful, especially for novel researchers in a specific area. I feel that the amount and quality of new information that the framework can provide to more experimented researchers is limited. However, I understand it can be beneficial for new researchers.\n\nI found the first part of the paper somehow confusing, as I started to understand this work in Section 4.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Some smaller things that I have found and you might want to consider:\n\n- Consider mentioning in the introduction the time period considered in the dataset. (it is in the data section, but it might be useful to know it before).\n\nin the Method section: \n -t is used to denote time (or year), but previously has been used to denote task.\n\n- formula in 235 not explained\n- formula in 363 needs more explanation\n\n- consider including the paragraph in line 446 at the beginning\n\n- is machine learning a different domain?? line 567-568\n\n- Table 3 could benefit from more explanation.\n\n- Subsection in section 4: results and discussion. Section 6: results and analysis. Consider naming the sections differently for more clarity."}}]

Author rebuttal:

Meta-review: Focusing on the ACL Anthology from 1979 to 2022, this paper analyzes the correlational and causal effects of tasks, methods, datasets and metrics (TDMM) on other tasks. For example, the authors measure the causal effect of a change in the “BLEU” score appearing in papers on frequency shifts in the “machine translation” task. 

Overall, reviewers found the paper sound and exciting. A few reviewers noted that the findings were pretty well-known by the NLP community. However, the approach the authors took to get there is extremely thorough and sound: from TDMM Entity Extraction to regression analysis on the entity influence, to using state-of-the-art causal discovery and causal estimation approaches to estimate the final causal effects. Even if these relationships could have been highlighted by researchers in the NLP community a-priori, I’m guessing that those researchers would not be able to *quantify* the causal effects. 

This seems like an interesting, well-developed, and extensive empirical pipeline that the rest of the community could have as an example for other large-scale data driven analyses with text and causal inference.
Decision: Accept-Main