Venue: EMNLP 2023

# A Diachronic Analysis of Paradigm Shifts in NLP Research:

When, How, and Why?

Aniket Pramanick1, Yufang Hou2, Saif M. Mohammad3, Iryna Gurevych1

###### Abstract

Understanding the fundamental concepts and trends in a scientific field is crucial for keeping abreast of its continuous advancement. In this study, we propose a systematic framework for analyzing the evolution of research topics in a scientific field using causal discovery and inference techniques. We define three variables to encompass diverse facets of the evolution of research topics within NLP and utilize a causal discovery algorithm to unveil the causal connections among these variables using observational data. Subsequently, we leverage this structure to measure the intensity of these relationships. By conducting extensive experiments on the _ACL Anthology_ corpus, we demonstrate that our framework effectively uncovers evolutionary trends and the underlying causes for a wide range of NLP research topics. Specifically, we show that tasks and methods are primary drivers of research in NLP, with datasets following, while metrics have minimal impact.1

Footnote 1: We publish the code and dataset for our experiments at [https://github.com/UKPLab/CausalNLPTrends](https://github.com/UKPLab/CausalNLPTrends)

1Ubiquitous Knowledge Processing Lab (UKP Lab) Department of Computer Science and Hessian Center for AI (hessian.AI)

2IBM Research Europe, Ireland

3National Research Council Canada www.ukp.tu-darmstadt.de, yhou@ie.ibm.com, saif.mohamad@nrc-cnrc.gc.ca

## 1 Introduction

Experts in a field sometimes conduct historical studies to synthesize and document the key research ideas, topics of interest, methods, and datasets that shaped a field of study. They document how new research topics eclipsed older ones and contributed to shaping the trajectory of the research area Kuhn (1970). Aspiring scientists learn the craft of their discipline by delving into the examination of past scientific accomplishments documented in research papers. However, conducting such a historical study is challenging: Experts in a field rely on years of experience and peruse large amounts of past published articles to determine the chronological progression of a research field. Further, the exponential growth of scientific publications in recent years has rendered it arluous even for domain experts to stay current. Therefore, an automated method to track the temporal evolution of research topics can be beneficial in offering an overview of the field and assisting researchers in staying abreast of advancements more efficiently.

In this work, we propose a systematic framework to examine the evolutionary journey of research topics within the realm of Natural Language Processing (NLP), harnessing causal discovery and inference techniques. Prior research on historical analysis of NLP has predominantly concentrated on scrutinizing metadata associated with research papers Hall et al. (2008); Mohammad (2019); Uban et al. (2021); Singh et al. (2023); Wahle et al. (2023) such as number of citations, title, author profile, affiliation, and publication venue. These studies have examined the research trends through unigram or bigram frequency analysis, but they do not provide insights into the underlying causes propelling these research topics.

Our study centers on four distinct fundamental types of entities in NLP research: **tasks** representing well defined problems; **methods**, signifying the solutions or approaches employed to tackle the tasks; **datasets**, indicating the relevant textual resources such as corpora and lexicons; and **metrics**, encompassing the evaluation techniques tailored to specific tasks. We abbreviate these types as **TDMM** for short. Specifically, we examine the interplay between an NLP task that is commonly viewed as a focused research topic (e.g., _Machine Translation_) and the _key entities_ that exert pivotal influence on the target task (such as "_BLEU_" Papineni et al. (2002) or "_Transformers_" Vaswani et al. (2017)).

Our goal is to identify the TDMM entities (\(E\)) associated with a specific task (\(t\)) and assess their causal influence on the task's research trends (**TDMM-Task causal analysis**). Specifically, we address the following key research questions associated with a task entity \(t\): (a) Which entities \(E\) effectively indicate the research trends for this task

\(t\)? (b) Are there discernible causal relationships between \(t\) and \(E\)? (c) What is the extent of the causal impact exerted by \(E\) on \(t\)?

Unlike Uban et al. (2021) and Koch et al. (2021) that heavily rely on manual annotations and have limited coverage, our analysis is based on TDMM entities automatically extracted from 55K papers in the ACL Anthology2. Our framework not only recognizes the key entities driving the research direction of a research topic but also measures the causal effects of these entities on the target topic in an end-to-end fashion. Figure 1 shows the most influential entities for _Machine Translation_ (MT) in different time periods. For instance, "_statistical models_" used to be the popular method for MT in 1990-2002, and the evaluation metric "_BLEU_" is one of the top causal entities driving the MT research in 2003-2017. In the era of pre-trained large language models (LLMs) starting from 2018, "_transformer_" has become the popular method for MT. For another research topic of "_Speech recognition_", our framework uncovers the influential role of "_language modeling_" between 1979 to 2022, where speech recognition models utilize probability scores from language models to recognize coherent text from speech Negri et al. (2014).

Footnote 2: [https://aclanthology.org/](https://aclanthology.org/)

In this work, we analyze 16 tasks from a diverse set of research areas identified by ACL 2018 organizers. Our framework is versatile and applicable to other tasks and domains, benefiting both young and experienced researchers. It can aid literature surveys by identifying related research areas and enable young researchers to delve into new research focuses by establishing connections among different research areas.

In summary, we make _three-fold_ contributions in this study: **Firstly**, we propose a framework to quantify research activities, including (1) trends and stability of an NLP research task, and (2) relation intensity between TDMM entities and NLP research tasks. **Secondly**, we employ causal analysis algorithms to uncover causal structures and measure effects between tasks and related TDMM entities (_TDMM-Task causal analysis_). To the best of our knowledge, this represents the first historical study of a scientific research anthology from a causal perspective. **Finally**, through extensive experiments on the ACL Anthology, we offer an empirical overview of the NLP research landscape. In the following sections, we will refer to _TDMM-Task causal analysis_ as _causal analysis_.

## 2 Related Work

Scientific Trends AnalysisThe analysis of scientific trends has been a research focus since Hall et al. (2008). In the field of "scientometrics", extensive literature explores citation patterns and utilizes topological measures in citation networks for trend analysis Small (2006); Shibata et al. (2008); Boyack and Klavans (2022).

Another line of research focuses on metadata and content analysis. For instance, Prabhakaran et al. (2016) employed rhetorical framing to exam

Figure 1: Evolution of Machine Translation (MT) research. Blue line: Number of MT papers (1979-2022). Tables show the top causal entities/types for different periods (excluding 1979-1989 due to limited MT papers).

 

ine trend patterns. Grudin (2009), Liu et al. (2015), and Mohammad (2019) investigated the interaction between the topics in publications, research grants, author profiles, highly impactful papers, and dataset usage patterns. Additionally, Koch et al. (2021) studied dataset usage patterns among different research communities, while Uban et al. (2021) analyzed relationships between NLP research topics based on their co-occurrence in text and the degree of correlation between their popularity over time. In our work, we develop entity recognition models to extract TDMM entities from NLP research papers and focus on analyzing the causal relations between a task entity and its related TDMM entities.

Causality in NLPAExisting works on NLP applying causal analysis algorithms mainly focus on two directions. The first line of work discovers causal relations among textual features or expressions of events in texts and uses them in various downstream tasks, such as question answering Oh et al. (2016), commonsense reasoning Bosselut et al. (2019); Sap et al. (2019), and relation extraction Do et al. (2011); Mirza and Tonelli (2014); Dunietz et al. (2017).

In another avenue of this field, researchers represent causal elements using textual features Jin et al. (2021); Fong and Grimmer (2016); Veitch et al. (2020); Keith et al. (2020) and define the causal graph structure based on domain knowledge. Our work falls within this line of research, where we employ causal algorithms to analyze the trends in NLP research topics and the underlying causes.

## 3 Data Collection

ACL Anthology CorpusFollowing prior work by Mohammad (2020), we utilize ACL Anthology as the source of NLP Research papers. For this work, we collect 55,366 NLP papers that belong to the "ACL Events" category3 from the ACL anthology published between 1979 and 2022. For each paper, we use GROBID (GRO, 2008-2022) and the PDF table parser from Hou et al. (2019) to extract sentences from each of the individual sections as well as from the table and figure captions. In a post-processing step, we remove all the URLs from the extracted sentences. On average, we have 1,258 papers per year and 1,117 sentences per paper.

Footnote 3: This category covers major NLP conferences, workshops, and journals including ACL, NAACL, EMNLP, EACL, AACL, CL, and TACL. Additionally, we also include papers published at COLING from the “Non-ACL events category”.

It is worth noting that certain NLP paper preprints may become accessible on preprint servers before they are officially published in the ACL Anthology. However, we argue that the peer review process in ACL Anthology serves as a robust quality assurance mechanism. Hence, we consider ACL Anthology a more reliable source compared to preprint servers.

TDMM Entity ExtractionTo identify _tasks_, _datasets_, _metrics_, and _methods_ entities from NLP papers, we developed two entity taggers based on Flair Akbik et al. (2018). The first tagger is based on the TDMSci annotations Hou et al. (2021) for recognizing _task_, _dataset_, and _metric_ entities. The second tagger is trained using the SciERC dataset

Figure 2: System architecture.

 

(Luan et al., 2018) to extract _method_ entities. On the testing datasets of TDMSci and SciERC, the two taggers achieve a micro-average F1 of 0.77 and 0.78 for the type partial match (Segura-Bedmar et al., 2013), respectively. In type partial match, a predicted entity is considered correct if it partially overlaps with a gold entity and has the same type. For example, "_Penn Treebank_" is counted as a correct prediction even if the corresponding gold annotation is "_Penn Treebank dataset_".

To further improve the precision of the TDMM taggers, we include only entities that appear in more than five papers in the dataset. For each paper, we collect the most frequent task mentions appearing in the _title_, _abstract_, _experiment section_, _table_, and _figure captions_ to approximate the tasks that the paper has done research on.

Taxonomy for Periods of ReferenceIn order to facilitate in-depth analysis, in this paper, we adopt a taxonomy that partitions our reference time frame (1979-2022) into four distinct intervals. Table 1 illustrates the defined intervals. These intervals have been designed to approximate the overarching trends observed in NLP research throughout the years, aligning with our perspective on the field's evolution. It is important to acknowledge that the exact boundaries and thematic emphases may differ based on varying perspectives and specific research areas within NLP. However, we highlight that our framework and methodologies are highly adaptable, allowing end users to effortlessly apply them to any desired time interval or a specific analysis.

## 4 Entity Influence in NLP Research: A Regression Analysis

Before conducting the causal analysis, we aim to identify the key variables that significantly impact the evolution of NLP Research. Specifically, we investigate which types of entities exert the most influence on the research direction of NLP. To achieve this understanding, we employ Multiple Linear Regression (see Appendix D for details), a widely utilized tool in economics research (Barrios and Hochberg, 2020). Figure 2 (step1/step2) illustrates the framework.

Our analysis assumes that if the TDMM entities have played a role in the emergence or disappearance of task entities, this influence will be reflected in the number of unique task entities in subsequent years, which can be captured through regression analysis. While the study does not provide specific information on the precise influence of each TDMM entity on individual task entities, the partial regression coefficients shed light on the types of entities responsible for influencing the overall task entity landscape.

Method.Mathematically, we predict the number of task entities \(Y^{t}\) in a given year \(t\) as a function of the cumulative counts of all types of entities \(\{X_{i}^{t_{-1}}\}\) (TDMM entities) until that year, \(t_{-1}\), given by \(Y^{t}=r_{0}+\sum_{i}r_{i}X_{i}^{t_{-1}}\). \(\{r_{i}\}\) quantifies the relationship strength between the predicted variable (number of task entities) and the independent variables (number of TDMM entities).

Evaluation.We evaluate the regression model using the \(R^{2}\) measure (coefficient of determination) to assess the goodness of fit. Additionally, we perform a null hypothesis test to determine the statistical significance of the partial regression co

\begin{table}
\begin{tabular}{l c l} \hline \hline Period & Years & Key Research Themes \\ \hline Early Years & 1979–1989 & Foundational work in syntactic parsing, machine translation, and information retrieval. \\ \hline Formative Years & 1990–2002 & Advances in language modeling, named entity recognition, and discourse analysis (research focus shifted towards data-driven approaches). \\ \hline Statistical Revolution \& Neural Networks & 2003–2017 & Focus on statistical techniques (text classification, statistical machine translation, etc.) and resurgence of neural networks (word embeddings, neural machine translation, etc.) \\ \hline Deep Learning Era & 2018–2022 & Dominance of transformer-based architectures (BERT and their variants). \\ \hline \hline \end{tabular}
\end{table}
Table 1: Chronological Periods of NLP Research.

\begin{table}
\begin{tabular}{l c} \hline \hline Variables & R-Squared (\(\uparrow\)) \\ \hline unique tasks & 0.87 \\ + unique datasets & 0.91 \\ + unique methods & 0.93 \\ + unique metrics & **0.97** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Variable Selection for Regression.

 

efficients.

**Results and Discussion.****1) Optimized Number of Variables.** In our initial experiment, we determine the optimal number of variables and summarize the corresponding \(R^{2}\) values in Table 2. Additionally, all regression coefficients are statistically significant at \(5\%\) level, indicating their strong relationship with the predicted variable. **Discussion:** The overall results indicate that the model achieves a good fit to the data when all four variables (number of tasks, datasets, metrics, and method entities) are used to predict the number of task entities in subsequent years. We also explore the possibility of reducing the number of variables while maintaining similar performance. Interestingly, using only one variable results in a significant drop of 0.1 in the \(R^{2}\) value (\(R^{2}\) value \(0.87\)), indicating a poor fit to the model. Conversely, increasing the number of variables improves the model fit, suggesting the significance of all four variables in analyzing research trends (\(R^{2}\) value \(0.97\)). It is worth noting that we exhaustively explored various combinations of variables, including those presented in the table, and consistently obtained similar results.

**2) Influence of the Variables.** In the second experiment, we assess the association between the target variable and each independent variable. In Table 3, we present the regression coefficients corresponding to each entity type. Larger values of regression coefficients indicate a stronger relationship between the target variable and the respective independent variable. **Discussion:** Overall, we note that the gradual emergence of newer tasks has been a driving force behind research progress. However, when we analyze the trends within each year interval, we uncover more nuanced patterns. During the _Early Years (1979-1989)_, when NLP was in its nascent stage as an independent research field, the focus was on creating new _datasets_ to fuel research advancements. In the _Formative Years (1990-2002)_, we witnessed the introduction of new _methods_, particularly data-driven approaches, which played a crucial role in shaping the field. Subsequently, _from 2003 to 2017_, _statistical methods_ underwent a revolution, and later in the same period, _neural network methods_ experienced a resurgence, indicating significant shifts in research trends. Now, in the present _Deep Learning Era (2018-2022)_, we observe a rapid creation of newer _datasets_ in a relatively short span of time, driven by the research needs and the data requirements of deep learning models. These highlight key factors influencing research trajectory over time.

## 5 Causal Methodology for NLP Research Analysis

Drawing on the insights gained from the Regression Analysis (Section 4), we now establish the cornerstone of our study by defining three causal variables that drive the causal analysis in the subsequent sections. Using causal discovery and inference techniques, we analyze the causal relationships among the variables and measure the impact of TDMM entities on target task entities based on these relationships. Figure 2 illustrates the architecture that underpins our framework.

### Causal Variables

Task Frequency Shift Value:Distinguishing from the previous approaches Tan et al. (2017); Prabhakaran et al. (2016), that rely on word frequencies, we define task frequency \(f(y)_{t}\) as the number of published papers focusing on a specific task \(y\) in a given year \(t\), normalized by the total number of papers published on the same year. The task frequency shift value \(\Delta freq_{t_{1}}^{t_{2}}(y)\) captures the average change in the number of published papers on \(y\) between two years \(t_{1}<t_{2}\). This value serves as a measure of the research trends associated with the task during that time interval, indicating whether it experienced growth or decline. The frequency shift value is given by: \(\Delta freq_{t_{1}}^{t_{2}}(y)=\frac{f(y)t_{2}-f(y)t_{1}}{t_{2}-t_{1}}\).

Task Stability Value:We introduce the concept of task stability value to measure the change in the research context of a given task, \(y\), between two years, \(t_{1}<t_{2}\). This value quantifies the overlap in neighboring TDMM entities that appear in the same publication as \(y\) within the specified time interval. To calculate task stability, we adapt

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{Years} & \multicolumn{4}{c}{Partial Regression Coefficient} \\ \cline{2-5}  & Tasks & Datasets & Methods & Metrics \\ \hline
1979–1989 & 0.35 & **2.24** & 0.21 & 0.02 \\
1990–2002 & 0.82 & 0.89 & **2.86** & 0.81 \\
2003–2017 & 5.37 & 6.26 & **7.00** & 0.69 \\
2018–2022 & 1.47 & **3.38** & 1.79 & 0.41 \\ \hline
1979 - 2022 & **3.50** & 1.07 & 2.92 & 0.54 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Variables Influencing NLP task entities.

 

the semantic stability approach of Wendlandt et al. (2018) to our setting and define it specifically for task entities. Initially, we represent each paper in our dataset as a sequence of TDMM entity mentions, removing non-entity tokens. We then employ "Skip-gram with negative sampling" Mikolov et al. (2013) to obtain embeddings from this representation. Formally, let \(e_{1},e_{2},...,e_{n}\) be this entity representation of a paper, and the objective of skip-gram is to maximize the mean log probability \(\frac{1}{n}\sum_{i=1}^{n}\sum_{-c\leq j<c}logp(e_{i+j}|e_{i})\), where \(c\) is called the context window size. Finally, the task stability value \(\Deltastability_{t_{1}}^{l_{2}}(y)\) of \(y\) between \(t_{1}\) and \(t_{2}\) is computed as the percentage overlap between the nearest \(l\) neighboring entities of the given task in two representation spaces. The stability value is given by: \(\Deltastability_{t_{1}}^{t_{2}}(y)=\frac{|\mathcal{N}_{t_{1}}^{l}(y)\cap\mathcal{ N}_{t_{2}}^{l}(y)|}{|\mathcal{N}_{t_{1}}^{l}(y)\cup\mathcal{N}_{t_{2}}^{l}( y)|}\), where \(\mathcal{N}_{t}^{l}(y)\) is the set of \(l\) neighbours of \(y\) in the representation space of year \(t\). In this study, we consider the context window \(c\) to encompass the entire document, and we set the value of \(l\) to \(5\).

Entity Change Value:We use entity change value to track emerging and disappearing of specific TDMM entities associated with a task, quantifying these changes and capturing related entity occurrences within a specific time period. Put simply, we measure the difference in the co-occurrence frequency of a TDMM entity \(x\) and a task \(y\) between two years \(t_{1}\) and \(t_{2}\). When we identify a significant change in the co-occurrence frequency of \(x\) and \(y\) over this period, it likely signals a shift in the relation between \(x\) and \(y\) and, in turn, a shift in NLP Research trends. We define entity change value \(\delta_{y}(x)_{t_{1}}^{t_{2}}\) of an entity \(x\) of type \(\tau(x)\in\) {task, dataset, metric, method} with respect to a task \(y\) as the absolute difference in frequencies of \(x\) co-occurring with \(y\) in the same sentence, between years \(t_{1}\) and \(t_{2}\) normalized by the total number of entities of the same type as \(x\) that co-occur with \(y\) in both years. The entity change value is given by: \(\delta_{y}(x)_{t_{1}}^{t_{2}}=\frac{|C_{t_{1}}(x,y)-C_{t_{2}}(x,y)|}{\sum_{v: r(e)=r(x)}(C_{t_{1}}(e,y)+C_{t_{2}}(e,y))}\), where the frequency of \(x\) co-occurring with \(y\) in year \(t\) is given by \(C_{t}(x,y)\).

In summary, we quantify task trends and research context changes using _task frequency change_ and _task stability values_. Below we explore the relationship between _entity change values_ and these two variables and estimate the causal impact of TDMM entities on task research landscapes.

### Causal Algorithms

Causal Structure DiscoveryTo uncover the causal structure among variables from observational data, we employ DirectLinGAM Shimizu et al. (2011), which assumes a non-Gaussian data-generating process. Since the variables in Section 5.1 come from non-Gaussian frequency distributions, DirectLinGAM is suitable. It uses an entropy-based measure to subtract the effect of each independent variable successively. Unlike PC-Stable Colombo and Maathuis (2014), it does not require iterative search or algorithmic parameters. We apply DirectLiNGAM with a \(5\%\) significance level for causal discovery (see Appendix C for details).

Causal InferenceOnce the causal structure between the variables has been established, we leverage this structure to assess the causal effects. Specifically, we measure the causal effects by the _entity change value_ of entity \(x\) on the _frequency shift_ and subsequently on the _stability values_ associated with a given task \(y\). For this purpose, we use the probability density function instead of probability mass, as all our causal variables are continuous in nature. We measure the causal effects in two steps: first, we estimate the probability density of the _entity change variable_ using a linear regression model. In the next step, we regress the _frequency shift_ and _stability_ against the _entity change value_, weighted by the inverse probability densities obtained in the previous step. We model the functional form of this regression using a spline to avoid bias due to misspecification. Finally, we calculate the causal effect as Veitch and Zaveri (2020): \(\mu(\Delta freq_{t_{1}}^{t_{2}}(y))=\mathbb{E}[\Delta freq_{t_{1}}^{t_{2}}(y)| \delta_{y}(x)_{t_{1}}^{t_{2}}]\) and similarly, \(\mu(\Delta stability_{t_{1}}^{t_{2}}(y))=\mathbb{E}[\Delta stability_{t_{1}}^{ t_{2}}(y)|\delta_{y}(x)_{t_{1}}^{t_{2}}]\).

## 6 Results and Analysis

Correlation-based measures provide a simple way to quantify the association between variables. However, they fall short of explaining complex cause-effect relationships and can yield misleading results. Causality is essential for gaining a deeper understanding of variable relationships, enhancing the robustness and reliability of our findings beyond the limitations of correlation. We discuss more about the importance of causal methods over correlation-based measures in Section 7. In this 

section, our focus is on uncovering relationships among causal variables (Section 6.1) and measuring the impact of TDMM entities on target task entities (Section 6.2).

### Causal Relation between the Variables

Figure 3 shows the discovered causal graph for the frequency shift of task entities. Overall, we observe that the entity change values of associated tasks, datasets, metrics, and methods have a direct causal effect on the frequency shift values of the target tasks. Since frequency shift value quantifies the trend in NLP research, we infer from the causal graph that the trend of a task is governed primarily by the life cycles of its associated TDMM entities. We see similar causal relation on task stability value (see Figure 4, Appendix A). **Evaluation:** We perform a sensitivity analysis of the causal graph by adding Gaussian noise with zero mean and unit variance to the entity change values in the data [50]. This gives an estimate of the robustness of the graph in the presence of unobserved confounders. We observe that the graph is stable to unobserved confounding, giving all edge probabilities greater than 0.5.

### Causal Impact of the Variables

The organizers of ACL 20184 categorize NLP research into 21 areas, and provide a set of popular tasks for each area. Out of those, we curate 16 areas and select one task from each based on its frequency of occurrence in our corpus. We estimate the effect of TDMM entities (entity change value) behind the development of these tasks (frequency shift value) (see Section 5.1) and summarize the results in Table 4. Since we do not have confounders (Section 6.1), evaluating the causal effect reduces to estimating the conditional expectation of the frequency shift values given the entity change values. We present detailed results in Appendix A.2. We examine the results by addressing the following set of inquiries.

Footnote 4: [https://acl2018.org/call-for-papers/](https://acl2018.org/call-for-papers/)

Q1. **What role do the methodologies play in causally driving the shift in NLP tasks?**

New methodologies have a significant influence on research in various areas of Natural Language Processing (NLP). In the field of Language Modeling, we observe a shift in influence between different methodologies over time.

Between 2003 and 2017, Recurrent Neural Networks (RNNs) had the most decisive impact on **Language Modeling** research. However, this trend shifted with the emergence of Transformers, which have since become the dominant influence in research on this task.

**Dialogue Systems**, which involve automatic response generation, are closely related to Language Modeling. Therefore, research in this area is highly influenced by Generative Models. From 1990 to 2002, Probabilistic Models played a crucial role in shaping Dialogue Systems research, while RNNs took the lead between 2003 and 2017.

**Machine Translation**, another task related to Language Modeling, requires the generation of the translated text. Naturally, we observe the influence of similar entities in Machine Translation research. Probabilistic Models had the most decisive impact between 1990 and 2002. In recent years (2018-2022), Transformers have emerged as the dominant influence in this research area.

In the field of **Speech Recognition**, Hidden Markov Models (HMMs) have shown a significant influence. HMMs have played a crucial role in shaping Speech Recognition research between 1979 to 2002.

**Named Entity Recognition** (NER) has also been influenced by Hidden Markov Models, particularly in its early days (1990-2002), as NER is often formulated as a sequence tagging problem. Various parser algorithms were employed to solve the problem in the period between 2003 and 2017.

For **Semantic Parsing**, parser algorithms have been instrumental and have had a significant impact on research in this area. Between 1979 and 1989, Grammar Induction techniques were used to elicit the underlying semantic parse trees.

From 1990 to 2002, researchers employed various statistical models in **Morphological Analysis**, which is evident from our results.

In **Semantic Role Labeling**, Support Vector Machines and Neural Network Models have been widely used to solve this task.

In **Co-reference Resolution**, Neural Network

Figure 3: Causal Graph of TDMM Entities (entity change values) and Task Entity Frequency Shift.

 

models have gained prominence starting in 2018. However, from 2003 to 2017, Integer Linear Programming was also utilized to address this problem.

Pre-trained Language Models (LLMs) have demonstrated superior performance in several NLP tasks, including **Question Answering**. Researchers have also explored parsing algorithms to parse questions and align them with potential answers.

Furthermore, **Textual Entailment** and **Summarization** have been heavily influenced by pre-trained LLMs between 2018 and 2022, as evident from our results.

Q2. **How have changes in data availability contributed to the NLP Research Tasks?**

High-quality datasets play a crucial role in advancing NLP research. While new methodologies are important, they cannot fully propel the field forward without the support of high-quality datasets. Researchers understand the significance of dataset quality and actively curate datasets to drive advancements in the field. Our findings further confirm the prevalence of this trend, highlighting the strong emphasis on dataset quality in NLP research.

In the early stages of deep neural models, such as Recurrent Neural Networks (RNNs), the creation of large datasets became essential for efficient model training. Between 2018 and 2022, several datasets were curated, with MultiWoz being the most widely used dataset for research in **Dialogue Systems**.

In the domain of **Machine Translation**, the significance of datasets in shaping research direction cannot be overlooked. The influence of WMT datasets on Machine Translation research is evident from our findings.

For **Morphological Analysis**, the Universal Dependency Treebank dataset is frequently used as a benchmark, indicating its importance in driving research in this area.

During the period of 1990-2002, the creation of the MUC-VI dataset played a crucial role in advancing research in **Co-reference resolution**.

In the field of **Sentiment Analysis**, the Twitter dataset holds significant importance in driving research in this domain.

Overall, our analysis underscores the vital role of datasets in shaping and driving research across various NLP tasks.

Q3. **Do evaluation metrics drive paradigm shifts in NLP research?**

Most NLP tasks rely on a standard set of metrics borrowed from other domains, such as machine learning and computer vision, to evaluate system performance. However, there is limited research dedicated to improving these metrics within the field of NLP, as it often requires theoretical knowledge beyond the scope of NLP itself. Despite this, our analysis in Table 5 reveals some noteworthy exceptions. Metrics explicitly designed for evaluating NLP tasks, such as BLEU and METEOR, have demonstrated significant impact in advancing **Machine Translation** research. Similarly, the metric ROUGE has influenced research in the field of **Summarization**. While perplexity scores are commonly used to measure the generalization capabilities of probability distributions, they are predominantly utilized for evaluating language models in NLP tasks.

Q4. **What is the causal impact of cross-pollination of ideas between related NLP tasks?**

We consistently observe a pattern of related NLP tasks evolving in tandem, borrowing ideas and tech

\begin{table}
\begin{tabular}{l c l r r r}  & \multicolumn{6}{c}{**Primary Cause**} \\ \cline{2-6}

- & Recurrent Neural Networks* & Transforms* & Transformer* \\ Dialogue System &
- & Probabilistic Generative Models*/ & Recurrent Neural Networks*/ & MultiWoz* & MultiWoz* \\ Machine Translation &
- & Probabilistic Generative Models*/ & WMT* & Transform* & Transformer* \\ Speech Recognition & Hidden Markov Models*/ & Hidden Markov Models*/ & Machine Translation*/ & Machine Translation*/ & Hakeka Markov Models*/ \\ Named Identity Recognition &
- & Hidden Markov Models*/ & POS Mapping+ & PPS Categorie & Relation Extraction*/ & PostSpe Categories* \\ Cross Tagging &
- & Text Classification*/ & Pace Algorithms*/ & Word Segmentation*/ & Word Segmentation*/ \\ Semantic Parsing & Grammar Induction*/ & Parser Algorithms*/ & Paser Algorithms*/ & Dependency Penalty*/ & Parse Algorithm*/ \\ Metribological Analysis & Statistical Models*/ & Dependency Penalty*/ & Tunn Products/ & Supervised Models*/ \\ Semantic Rule Labeling &
- & Support Vector Machines*/ & Network Models*/ & Support Vector Machines*/ \\ Coreference Reconstruction &
- & MUC-VI Test Collection*/ & Intercept Processing*/ & Normal Network Models*/ & Neural Network Models*/ \\ World Sense Disambiguation &
- & Worknet*/ & Maximum Energy Models*/ & Neural Network Model*/ & Words*/ \\ Sentiment Analysis &
- & Twizer Dataset*/ & Text Classification*/ & Text Classification*/ \\ Argument Mining &
- & Text Classification*/ & Sentiment Analysis*/ & Sentence Analyses*/ \\ Question Answering & Persing Algorithms*/ & Information Extraction*/ & Informatics Function*/ & The Triangled LLS*/ & Inference Extraction*/ \\ Visual Entailment &
- & Student Models*/ & Pre-Trained LLS*/ & Pre-Trained LLR*/ \\ Summationation &
- & Worden*/ & Sentence Compression*/ & Pre-Trained LLS*/ & Pre-Trained LLS*/ \\ \end{tabular}
\end{table}
Table 4: Causal analysis identifies the main drivers (Methods, Tasks, Datasets) of frequency shifts in NLP tasks across four periods, with ”-” indicating insufficient data for analysis.
 

niques from one another. This trend is clearly reflected in our findings. For instance, **Speech Recognition** and **Machine Translation** are linked as researchers explore end-to-end systems that translate speech, and our results show that Machine Translation has had the greatest influence on Speech Recognition research between 2003 and 2022.

**Named Entity Recognition** (NER) is commonly approached as a sequence tagging problem, and it is influenced by related tasks such as **POS Tagging** (2003-2017) and **Relation Extraction** (2018-2022), as these problems are often jointly solved. Similarly, POS Tagging initially posed as a text classification problem (1990-2002), is significantly impacted by the **word segmentation task**, as evident from our results in the period of 2018-2022.

In recent years (2018-2022), **dependency** and **semantic parsing** have been jointly solved using the same neural model, highlighting the influence of dependency parsing on research in semantic parsing. **Sentiment Analysis** has garnered considerable research interest and is commonly framed as a text classification problem. Additionally, **Argument Mining**, which involves understanding the sentiments behind arguments, is influenced by sentiment analysis. Furthermore, the classification of various argument components, such as claims and evidence, is often approached as text classification problems, as evidenced by our results.

## 7 Discussion: Correlation and Causation

"correlation does not imply causation"

- Pearson (1892)

Causation and correlation, although related, are distinct concepts. While they can coexist, correlation does not simply imply causation. Causation signifies a direct cause-and-effect relationship, where one action leads to a specific outcome. In contrast, correlation simply indicates that two actions are related in some way, without one necessarily causing the other.

In our work, we focus on causal inference from data. While correlation-based measures provide a straightforward method for quantifying associations between variables, they often fall short when it comes to explaining complex cause-and-effect relationships.

To demonstrate the effectiveness of our framework, we establish a simple baseline using a PMI-based correlation measure Bouma (2009). For this analysis, we select **Machine Translation** as our target task entity due to its prominent presence in our corpus and the NLP research landscape. We calculate the PMI scores of **Machine Translation** with all other TDMM entities. The PMI score represents the probabilities of co-occurrence between two entities in sentences from research papers, normalized by their individual occurrence probabilities.

Interestingly, we find that **accuracy**, an entity of type **metric**, has the highest PMI score with Machine Translation among all other entities. However, it is important to note that accuracy is a widely used metric across various NLP tasks, and it is not specifically developed for machine translation, nor has machine translation influenced the concept of accuracy. This observation emphasizes the insufficiency of relying solely on correlation-based metrics to understand and analyze research influence on an entity.

We observe that relying solely on correlations can lead to misleading results and interpretations. Therefore, in order to understand the influence of associated TDMM entities on NLP Task entities, we utilize causal algorithms that enable us to gain insights into the cause-and-effect dynamics among the variables we study.

## 8 Concluding Remarks

In this paper, we retrospectively study NLP research from a causal perspective, quantifying research trends of task entities and proposing a systematic framework using causal algorithms to identify key reasons behind the emergence or disappearance of NLP tasks. Our analysis reveals that tasks and methods are the primary drivers of research in NLP, with datasets following their influence, while metrics have minimal impact. It is important to note that in our analysis, we have structured the reference time into four distinct intervals (see Table 1); however, it can be applied to diverse timeframes, ranging from longer periods to brief intervals, including single years. This adaptability, in the context of rapid recent advancements in NLP, allows to zoom in on local trends and developments that might otherwise go unnoticed (such as the influence of in-context learning on NLP tasks).

We believe our causal analysis enhances understanding of the interplay of research entities in NLP, contributing to the growing body of work on causality and NLP Feder et al. (2021). We provide with additional analysis and insights in Appendix B 



### Limitations

This work is centered on NLP research papers from ACL Anthology, with a focus on papers from the "ACL Events" category. The "ACL Events" category encompasses major conferences, workshops, and journals, including ACL, NAACL, EMNLP, EACL, AACL, CL, and TACL. We also include papers published at COLING from the "non-ACL Events" category. Nevertheless, it is important to acknowledge the presence of NLP papers beyond ACL Anthology in AI journals, regional conferences, and preprint servers. Furthermore, we recognize that certain NLP papers may become available on preprint servers before their official publication in peer-reviewed venues. In this study, we focus on ACL Anthology, which can introduce a time lag when assessing the early impact of influential papers released as preprints (e.g., BERT) or only on preprint servers (e.g., RoBERTa). To address such challenges, we leave the curation and inclusion of NLP research papers from these alternative sources for future works.

Our framework requires research papers tagged with entities as input. Hence, the quality of the tags plays a crucial role in the causal inference of our proposed method. The taggers generate noisy outputs and, thus, might require human intervention to denoise the tags. Moreover, causal algorithms require a large amount of data to produce statistically significant results. Hence, research areas that are less explored or newly emerging may not always be suitable for this framework to be applied. Additionally, we highlight that in this work, we do not consider extra-linguistic factors like author affiliations, funding, gender, etc. We leave them for future research work.

## Ethics Statement

In this work, we use publicly available data from ACL Anthology and do not involve any personal data. It is important to recognize that, while our framework is data-driven, individual perspectives toward research are inherently subjective. Decisions involving science should consider data as well as ethical, social, and other qualitative factors. Furthermore, we underscore that the low influence of TDMM entities in our analysis should not be the sole reason for devaluing research papers or reducing their investments. Ethical and academic considerations should guide decisions on research evaluation and resource allocation.

## Acknowledgements

We thank Ilia Kuznetsov for his feedback on the initial version of this work. We appreciate all the anonymous reviewers for their helpful comments and suggestions for further analysis. This work has been funded by the German Research Foundation (DFG) as part of the Research Training Group KRITIS No. GRK 2222.

## References

* Akbik et al. (2018) Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 1638-1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Barrios and Hochberg (2020) John M Barrios and Yael Hochberg. 2020. Risk perception through the lens of politics in the time of the covid-19 pandemic. Working Paper 27008, National Bureau of Economic Research.
* Bosselut et al. (2019) Antoine Bosselt, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4762-4779, Florence, Italy. Association for Computational Linguistics.
* Bouma (2009) Gerlof Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. _Proceedings of GSCL_, 30:31-40.
* Boyack and Klavans (2022) Kevin W. Boyack and Richard Klavans. 2022. An improved practical approach to forecasting exceptional growth in research. _Quantitative Science Studies_, 3(3):672-693.
* Cinelli et al. (2019) Carlos Cinelli, Daniel Kumor, Bryant Chen, Judea Pearl, and Elias Bareinboim. 2019. Sensitivity analysis of linear structural causal models. In _Proceedings of the 36th International Conference on Machine Learning_, volume 97 of _Proceedings of Machine Learning Research_, pages 1252-1261. PMLR.
* Colombo and Maathuis (2014) Diego Colombo and Marloes H. Maathuis. 2014. Order-independent constraint-based causal structure learning. _Journal of Machine Learning Research_, 15(116):3921-3962.
* Do et al. (2011) Quang Do, Yee Seng Chan, and Dan Roth. 2011. Minimally supervised event causality identification. In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_, pages 294-303, Edinburgh, Scotland, UK. Association for Computational Linguistics

Jesse Dunietz, Lori Levin, and Jaime Carbonell. 2017: The BECauSE corpus 2.0: Annotating causality and overlapping relations. In Proceedings of the 11th Linguistic Annotation Workshop, Vancouver, Canada, pp. 95-104. External Links: Link Cited by: SS3.
* A. Feder, Katherine A. Keith, Emaad Manzoor, Reid Przyant, Dhanya Sridhar, Zach Wood-Doughuty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, B. M. Stewart, V. Veitch, and Diyi Yang. (2021)Causal inference in natural language processing: estimation, prediction, interpretation and beyond. CoRRabs/2109.00725. External Links: Link, 2109.00725 Cited by: SS3.
* C. Fong and J. Grimmer (2016)Discovery of treatments from text corpora. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, pp. 1600-1609. External Links: ISBN 978-3-319-142-3, Document, Link Cited by: SS3.
* J. Grudin (2009)AI and HCI: two fields divided by a common focus. AI Magazine30 (4), pp. 48-48. External Links: ISSN 0002-921X, Document, Link Cited by: SS3.
* D. Hall, D. Jurafsky, and C. D. Manning (2008)Studying the history of ideas using topic models. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, Honolulu, Hawaii, pp. 363-371. External Links: Link, Document Cited by: SS3.
* Y. Hou, C. Jobim, M. Gleize, F. Bonin, and Debasis Ganguly (2021)Identification of tasks, datasets, evaluation metrics, and numeric scores for scientific leaderboards construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 5203-5213. External Links: Link, Document Cited by: SS3.
* Y. Hou, C. Jochim, M. Gleize, F. Bonin, and Debasis Ganguly (2021)TDMSci: a specialized corpus for scientific literature entity tagging of tasks datasets and metrics. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online, pp. 707-714. External Links: Link, Document Cited by: SS3.
* Z. Jin, Z. Peng, T. Vaidhya, B. Schoelkopf, and R. Mihalcea (2021)Mining the cause of political decision-making from social media: a case study of COVID-19 policies across the US states. In Findings of the Association for Computational Linguistics: EMNLP 2021, Punta Cana, Dominican Republic, pp. 288-301. External Links: Link, Document Cited by: SS3.
* K. Keith, D. Jensen, and B. O'Connor (2020)Text and causal inference: a review of using text to remove confounding from causal estimates. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 5332-5344. External Links: Link, Document Cited by: SS3.
* B. Koch, E. Denton, A. Hanna, and J. Foster (2021)Reduced, reused and recycled: the life of a dataset in machine learning research. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), External Links: Link Cited by: SS3.
* T. S. Kuhn (1970)The structure of scientific revolutions, volume 111. Chicago University of Chicago Press. External Links: Link Cited by: SS3.
* Shixia Liu, Y. Chen, H. Wei, J. Yang, K. Zhou, and S. Mark Drucker (2015)Exploring topical lead-lag across corpora. IEEE Transactions on Knowledge and Data Engineering27, pp. 115-129. External Links: Link, Document Cited by: SS3.
* Y. Luan, L. He, M. Ostendorf, and H. Hajishirzi (2018)Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 3219-3232. External Links: Link, Document Cited by: SS3.
* Tomas Mikolov, I. Sutskever, K. Chen, G. S Corrado, and J. Dean (2013)Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, Vol. 26, Curran Associates, Inc. Cited by: SS3.
* P. Mirza and S. Tonelli (2014)An analysis of causality between events and its relation to temporal information. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin, Ireland, Dublin City University and Association for Computational Linguistics, External Links: Link, Document Cited by: SS3.
* S. M. Mohammad (2019)The state of NLP literature: a diachronic analysis of the acl anthology. ArXivabs/1911.03562. External Links: Link, 1911.03562 Cited by: SS3.
* S. M. Mohammad (2020)Examining citations of natural language processing literature. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 5199-5209. External Links: Link, Document Cited by: SS3.
* Matteo Negri, M. Turchi, J. G. C. de Souza, and D. Falavigna (2014)Quality estimation for automatic speech recognition. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin, Ireland, pp. 1813-1823. External Links: Link, Document Cited by: SS3.
* J. Oh, K. Torisawa, C. Hashimoto, R. Iida, M. Tanaka, and J. Kloetzer (2016)A semi-supervised learning approach to why-question answering. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, pp. 3022-3029. External Links: Link, Document Cited by: SS3.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 

40th annual meeting of the Association for Computational Linguistics_, pages 311-318.
* Pearl et al. (2016) Judea Pearl, Madelyn Glymour, and Nicholas P Jewell. 2016. _Causal inference in statistics: A primer_. John Wiley & Sons.
* Pearson (1892) Karl Pearson. 1892. The grammar of science. _Nature_, 46(1185):247-247.
* Prabhakaran et al. (2016) Vinodkumar Prabhakaran, William L. Hamilton, Dan McFarland, and Dan Jurafsky. 2016. Predicting the rise and fall of scientific topics from trends in their rhetorical framing. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1170-1180, Berlin, Germany. Association for Computational Linguistics.
* Sap et al. (2019) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. _Proceedings of the AAAI Conference on Artificial Intelligence_, 33(01):3027-3035.
* Segura-Bedmar et al. (2013) Isabel Segura-Bedmar, Paloma Martinez, and Maria Herrero-Zazo. 2013. SemEval-2013 task 9 : Extraction of drug-drug interactions from biomedical texts (DDIExtraction 2013). In _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)_, pages 341-350, Atlanta, Georgia, USA. Association for Computational Linguistics.
* Shibata et al. (2008) Naoki Shibata, Yuya Kajikawa, Yoshiyuki Takeda, and Katsumori Matsushima. 2008. Detecting emerging research fronts based on topological measures in citation networks of scientific publications. _Technovation_, 28(11):758-775.
* Shimizu et al. (2011) Shohei Shimizu, Takanori Inzumi, Yasuhiro Sogawa, Aapo Hyvarinen, Yoshinobu Kawahara, Takashi Washio, Patrik O Hoyer, Kenneth Bollen, and Patrik Hoyer. 2011. Directlingam: A direct method for learning a linear non-gaussian structural equation model. _Journal of Machine Learning Research-JMLR_, 12(Apr):1225-1248.
* Singh et al. (2023) Janvijay Singh, Mukund Rungta, Diyi Yang, and Saif Mohammad. 2023. Forgotten knowledge: Examining the citational amnesia in NLP. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 6192-6208, Toronto, Canada. Association for Computational Linguistics.
* Small (2006) Henry Small. 2006. Tracking and predicting growth areas in science. _Scientometrics_, 68(3):595-610.
* Tan et al. (2017) Chenhao Tan, Dallas Card, and Noah A. Smith. 2017. Friendships, rivalries, and trysts: Characterizing relations between ideas in texts. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 773-783, Vancouver, Canada. Association for Computational Linguistics.
* Uban et al. (2021) Ana Sabina Uban, Cornelia Caragea, and Liviu P. Dinu. 2021. Studying the evolution of scientific topics and their relationships. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 1908-1922, Online. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Veitch et al. (2020) Victor Veitch, Dhanya Sridhar, and David Blei. 2020. Adapting text embeddings for causal inference. In _Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI)_, volume 124 of _Proceedings of Machine Learning Research_, pages 919-928. PMLR.
* Veitch and Zaveri (2020) Victor Veitch and Anisha Zaveri. 2020. Sense and sensitivity analysis: Simple post-hoc analysis of bias due to unobserved confounding. In _Advances in Neural Information Processing Systems_, volume 33, pages 10999-11009. Curran Associates, Inc.
* Wahle et al. (2023) Jan Philip Wahle, Terry Ruas, Mohamed Abdalla, Bela Gipp, and Saif M. Mohammad. 2023. We are who we cite: Bridges of influence between natural language processing and other academic fields. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, Singapore, Singapore. Association for Computational Linguistics.
* Wendlandt et al. (2018) Laura Wendlandt, Jonathan K. Kummerfeld, and Rada Mihalcea. 2018. Factors influencing the surprising instability of word embeddings. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2092-2102, New Orleans, Louisiana. Association for Computational Linguistics.
* Williams and Rasmussen (2006) Christopher KI Williams and Carl Edward Rasmussen. 2006. _Gaussian processes for machine learning_, volume 2. MIT press Cambridge, MA.
* Zhang et al. (2018) Dongxiang Zhang, Lei Wang, Nuo Xu, Bing Tian Dai, and Heng Tao Shen. 2018. The gap of semantic parsing: A survey on automatic math word problem solvers. _CoRR_, abs/1808.07290.

## Appendix A Appendix: Additional Results

### Causal Relation

In Figure 4, we observe that entity change values of tasks, datasets, metrics and methods have direct causal influence on task stability value 



### Causal Effects

In Table 5, we observe the entities (Tasks, Datasets, Methods, and Metrics) that influence research on a given NLP Task.

## Appendix B Appendix: Supplementary Analysis

In addition to the primary results presented in the paper (Section 6), in this section, we describe the supplementary analysis.

### NLP Tasks and Their Dataset Evolution

Frequently Pursued NLP Tasks.From Table 5 in our paper, we observe that overall (from 1979-2022), among all the tasks, "Text Classification" (column 6) holds a remarkable position. This prominence stems from the frequent usage of various NLP tasks being framed or aligned as "Text Classification" or borrowing concepts from it to address other tasks such as "Sentiment Analysis" or "Word Sense Disambiguation." Additionally, our framework offers the flexibility to perform a similar analysis between any chosen periods.

Evolution of Datasets in NLP Tasks.Referring to Table 5 in our paper, in the context of "Speech Recognition," we observe a shift in influential datasets over different periods. Between 1990-2002, the "WSJ Corpus" took the lead, while in the subsequent period of 2003-2017, the "ATIS Dataset" had more influence. Interestingly, between 2018-2022, the trend shifted once again to the "Switchboard Dataset".

A similar trend is reflected in the "Summarization" task as well: in the years 1990-2002, "Wordnet" played a significant role, while the "Gigaword Dataset" took over in 2003-2017. However, in the most recent period of 2018-2022, "Pubmed" emerged as the notable dataset for the "Summarization" task.

Common Datasets Across NLP Tasks.We observe from Table 5 (column 6) that across the entire span from 1979 to 2022, the "Penn Treebank" dataset emerged as a pivotal influence, significantly impacting tasks such as "Language Modeling," "POS Tagging," and "Semantic Parsing." Using our framework, a similar analysis could also be done between any chosen periods.

### Entitily Influence on Task Frequency and Stability

**Influence of Research Entities on Task Stability.** We measure the causal effect of research entities on Task Stability Value (see Section 5.1). From the resulting causal graph (Figure 4), we observe that the entity change values of associated tasks, datasets, metrics, and methods directly impact the stability value of the target task, similar to the task frequency shift value.

Correlations Between Task Frequency Change and Stability.We observe a slightly positive correlation between frequency change and stability of research tasks with a Pearson coefficient of 0.08. This is because when a new task emerges, initially, a few researchers start working on it, which gradually increases its frequency of appearance. At the same time, researchers experiment with various methods and datasets to solve these newly emerged tasks, causing high instability (e.g., Math Problem Solving [Zhang et al., 2018]). On the contrary, the opposite is not always true: well-defined tasks are often the most researched, and yet researchers always explore new ideas on these tasks, which harms stability.

Overview and Insights.Our analysis shows that research in NLP is primarily driven by tasks and methods; the influence of datasets follows them, and metrics have minimum impact. Our analysis of frequency shift values reveals the gradual paradigm shift in NLP research. Initially, the focus was on practical problems such as **Speech Recognition** and **Machine Translation**. However, over time, researchers ventured into more complex areas like **textual entailment** and **argument mining**, necessitating domain knowledge and extensive data rea

Figure 4: **Causal Graph: The graph shows that the emergence and disappearance of TDMM entities (entity change values) have a direct causal effect on the stability of task entities.**

soning. Examining stability values, we note that pre-trained language models have emerged as versatile solutions, reducing the need for task-specific approaches.

## Appendix C Appendix: Algorithms

### DirectLinGAM

In Algorithm 1, we describe the DirectLinGAM algorithm (oracle version) in high level as described by Shimizu et al. (2011).

\begin{table}
\begin{tabular}{l c c l l l}  & \multicolumn{4}{c}{**Primary Cause**} \\

- & Recurrent Neural Networks\({}^{M}\) & Transforms\({}^{M}\) & Transforms\({}^{M}\) \\ Language Modeling &
- & Machine Translation\({}^{S}\) & Text Generation\({}^{T}\) & Text Generation\({}^{T}\) \\  &
- & Penn Treebank\({}^{D}\) & Penpecty\({}^{D}\) & Perplexity\({}^{D}\) & Perplexity\({}^{D}\) \\  &
- & Preplexity\({}^{D}\) & SuperGULDE\({}^{D}\) & Penn Treebank\({}^{D}\) \\ \hline \multirow{5}{*}{Daibgene System} &
- & Recurrent Neural Networks\({}^{M}\) & MultiWay\({}^{D}\) & MultiWay\({}^{D}\) \\  &
- & MathWorks\({}^{M}\) & Transformer\({}^{M}\) & Transferring\({}^{M}\) \\  &
- & Langague Generation\({}^{T}\) & Repeeing Generation\({}^{T}\) & Reopene Generation\({}^{T}\) \\  &
- & Probabilistic Generative Models\({}^{M}\) & Word Data\({}^{D}\) & Twitter data\({}^{M}\) & Interaction\({}^{M}\) \\ Machine Translation &
- & Speech Recognition\({}^{T}\) & BLEU\({}^{m}\) & METEROR\({}^{m}\) & METEROR\({}^{m}\) \\  &
- & Vertexity\({}^{D}\) & Attention Mechanism\({}^{M}\) & Language Modelling\({}^{T}\) & Language Generation\({}^{T}\) \\  &
- & Penn Treebank\({}^{D}\) & Language Generation\({}^{T}\) & WWT Data\({}^{D}\) & WWT Data\({}^{D}\) \\ \hline \multirow{6}{*}{Speech Recognition} & Hidden Markov Model\({}^{TM}\) & Hidden Markov Models\({}^{TM}\) & Machine Transition\({}^{T}\) & Machine Transition\({}^{T}\) & Hidden Markov Models\({}^{M}\) \\  & Machine Translator\({}^{R}\) & WSJ Corpus\({}^{D}\) & Hidden Markov Models\({}^{TM}\) & Acoustic Models\({}^{M}\) & Language Modelling\({}^{T}\) \\  & Frequency\({}^{M}\) & Perplexity\({}^{D}\) & ArTS Dataset\({}^{D}\) & Switchboard Dataset\({}^{D}\) & Proplexity\({}^{D}\) \\  &
- & Language Modelling\({}^{T}\) & Word Error Rate\({}^{M}\) & Word Error Rate\({}^{M}\) & ATIS Dataset\({}^{D}\) \\ \hline \multirow{3}{*}{Nuned Entity Recognition} &
- & Hidden Markov Models\({}^{TM}\) & POS Engine\({}^{T}\) & Relation Extraction\({}^{TM}\) & Post Tagging\({}^{T}\) \\  &
- & Information Extraction\({}^{T}\) & Conditional Random Fields\({}^{M}\) & Wikipedia Copaya\({}^{D}\) & Random Fields\({}^{M}\) \\  &
- & Genia Corpus\({}^{D}\) & PhoneD\({}^{D}\) & Free-TimeLM/LM\({}^{M}\) & Confusion Connection\({}^{D}\) \\  &
- & F I Sever\({}^{D}\) & FI Sever\({}^{D}\) & FII Sever\({}^{D}\) & FII Sever\({}^{D}\) \\ \hline \multirow{6}{*}{POS Tugging} &
- & Test Classification\({}^{T}\) & Passer Algorithms\({}^{U}\) & Word Segmentation\({}^{T}\) & Wüd Segmentation\({}^{T}\) \\  &
- & Discriminative Module\({}^{M}\) & World Segmentation\({}^{T}\) & Neural Network Module\({}^{M}\) & Neural Network Module\({}^{M}\) \\  &
- & Penn Treebank\({}^{D}\) & Penn Treebank\({}^{D}\) & Penn Treebank\({}^{D}\) & Pnn Treebank\({}^{D}\) \\  &
- & F1 Secret\({}^{D}\) & FII Score\({}^{m}\) & FII Sever\({}^{D}\) & FII Sever\({}^{D}\) \\ \hline \multirow{3}{*}{Wud Seen Disambiguation} &
- & Worked & Maximum Entropy Models\({}^{M}\) & Neural Network Module\({}^{M}\) & Wüdden\({}^{U}\) \\  &
- & Semantic Tagging\({}^{T}\) & Text Unintention\({}^{T}\) & Text Unintention\({}^{T}\) & Text Unintention\({}^{T}\) \\  &
- & Discriminative Models\({}^{M}\) & Whether\({}^{D}\) & Worden \({}^{D}\) & Text Unintention\({}^{T}\) \\  &
- & Accuracy\({}^{M}\) & FII Score\({}^{m}\) & FII Sever\({}^{D}\) & FII Sever\({}^{D}\) \\ \hline \multirow{3}{*}{Morphological Analysis} &
- & Statistical Models\({}^{M}\) & Dependency Parsing\({}^{T}\) & UD Treebank\({}^{D}\) & Statistically Model\({}^{M}\) \\  &
- & Wood Segmentation\({}^{T}\) & Stanford Models\({}^{M}\) & Fine-Timed Lambda\({}^{M}\) & Dependency Testing\({}^{T}\) \\  &
- & UD Treebank\({}^{D}\) & Undemark\({}^{D}\) & Lemmentation\({}^{M}\) & UTreebank\({}^{D}\) \\  &
- & UC Inference\({}^{D}\) & Accursive\({}^{M}\) & FI Sever\({}^{D}\) & Accuracy\({}^{M}\) \\ \hline \multirow{3}{*}{Semantic Parsing Parsing} & Gramar Induction\({}^{M}\) & Faster Ageberings\({}^{M}\) & Pure Attribute\({}^{D}\) & Dependency Parsing\({}^{T}\) & Perser Algorithms\({}^{TM}\) \\  & Information Retrieval\({}^{T}\) & Informable Extraction\({}^{T}\) & Dependency Parsing\({}^{T}\) & Pinner Algebras\({}^{TM}\) & Pann Treebank\({}^{D}\) \\  & Accuracy\({}^{M}\) & Penn Treebank\({}^{D}\) & Penn Treebank\({}^{D}\) & Pern Treebank\({}^{D}\) \\  &
- & F1 Secure\({}^{D}\) & FII Sever\({}^{D}\) & FII Sever\({}^{D}\) & FII Sever\({}^{D}\) \\ \hline \multirow{3}{*}{Semantic Rule Labeling} &
- & Support Vector Machines\({}^{M}\) & Neural Network Module\({}^{M}\) & Support Vector Machines\({}^{M}\) \\  &
- & Relitue Extraction\({}^{T}\) & Named Entity Recognitized\({}^{T}\) & Named Entity Recognitized\({}^{T}\) \\  &
- & Propagation\({}^{D}\) & Prephan\({}^{D}\) & Prephan\({}^{D}\) \\  &
- & F1 Secure\({}^{D}\) & FII Sever\({}^{D}\) & F1 Secure\({}^{D}\) \\ \hline \multirow{3}{*}{Co-reference Resolution} &
- & MUC-VT Tel Collection\({}^{D}\) & Integral Linear Programming\({}^{M}\) & Neu Neu Neu Neu Module\({}^{M}\) \\  &
- & Discriminator Models\({}^{M}\) & Ontoctrees\({}^{D}\) & Omnutes\({}^{D}\) & Omnutes\({}^{D}\) \\  &
- & Word Sense Cunningham\({}^{T}\) & Menton Detection\({}^{TM}\) & Metaet Decoster\({}^{TM}\) & P1 Screen\({}^{TM}\) \\  &
- & F1 Secret\({}^{D}\) & FII Sever\({}^{D}\) & FII Sever\({}^{D}\) & Mentinen Distance\({}^{TM}\) \\ \hline \hline \multirow{3}{*}{Semanticment Analysis} &
- & Twitter Dataset\({}^{D}\) & Tet Classification\({}^{T}\) & Text Classification\({}^{T}\) \\  &
- & Text Unintentional & True-Timed LM\({}^{M}\) & Neu Neu Neu Neu Network Module\({}^{M}\) \\  &
- & Nearest Nowak Models\({}^{M}\) & Amazon Review\({}^{D}\) & Neu Neu Neu
 

```
1Given a p-dimensional random vector \(x\), a set of its variable subscripts \(U\) and a \(p\times n\) data matrix of the random vector as \(X\), initialize an ordered list of variables \(K:=\phi\) and \(m:=1\);
2Repeat until \(p\) - 1 subscripts are appended to \(K\): Perform least square regression of \(x_{i}\) and \(x_{j}\), \(\forall i\in U-K(i\neq j)\) and compute the residual vectors \(r^{(j)}\) and the residual data matrix \(R^{(j)}\) from the matrix \(X\), \(\forall j\in U-K\). Final a variable \(x_{m}\) independent of its residuals and append \(m\) to the end of \(K\);
3Append the remaining variable to the end of \(K\);
4Construct a strictly lower triangular matrix \(B\) by following the order in \(K\), and estimate the connection strengths \(b_{ij}\) by using some conventional covariance-based regression such as least squares and maximum likelihood approaches on the original random vector \(x\) and the original data matrix \(X\);
```

**Algorithm 1**Causal Graph Discovery: DirectLinGAM-Algorithm

## Appendix D Appendix: Multiple Linear Regression

We use multiple linear regression to regress a variable on several variables Pearl et al. (2016). For instance, if we want to predict the value of a variable \(Y\) using the values of variables \(X_{1},X_{2},...,X_{k-1},X_{k}\), we perform multiple linear regression of \(Y\) on \(\{X_{1},X_{2},...,X_{k-1},X_{k}\}\), and estimate a regression relationship (Eqn. 1), which represents an inclined plane through the \((k+1)\)-dimensional coordinate system.

\[Y=r_{0}+\sum_{i=1}^{k}r_{i}X_{i} \tag{1}\]

The Gauss-Markov theorem Williams and Rasmussen (2006) simplifies the computation of partial regression coefficients (\(r_{1},...,r_{k}\) in Eqn 1). It states that if we write Y as a linear combination of \(X_{1},X_{2},...,X_{k-1},X_{k}\) and noise term \(\epsilon\),

\[Y=r_{0}+\sum_{i=1}^{k}r_{i}X_{i}+\epsilon \tag{2}\]

then, regardless of the distributions of the variables \(Y,X_{1},X_{2},...,X_{k}\), the best least-square coefficients are obtained when \(\epsilon\) is uncorrelated with each regressors, i.e.,

\[Cov(\epsilon,X_{i})=0,\forall i=1,2,...,k\] (3)