# Improving Diachronic Word Sense Induction with a Nonparametric Bayesian method

**Ashjan Alsulaimani**

School of Computer Science
and Statistics
Trinity College of Dublin
alsulaia@tcd.ie

**Erwan Moreau**

School of Computer Science
and Statistics
Trinity College of Dublin
moreaue@tcd.ie

## Abstract

Diachronic Word Sense Induction (DWSI) is the task of inducing the temporal representations of a word meaning from the context, as a set of senses and their prevalence over time. We introduce two new models for DWSI, based on topic modelling techniques: one is based on Hierarchical Dirichlet Processes (HDP), a nonparametric model; the other is based on the Dynamic Embedded Topic Model (DETM), a recent dynamic neural model. We evaluate these models against two state of the art DWSI models, using a time-stamped labelled dataset from the biomedical domain. We demonstrate that the two proposed models perform better than the state of the art. In particular, the HDP-based model drastically outperforms all the other models, including the dynamic neural models.<fnref n="1" />

## 1 Introduction

Word meanings evolve over time. Recent research works have focused on how to model such dynamic behaviour. The unsupervised task of Diachronic Word Sense Induction (DWSI) aims to capture how the meaning of a word varies continuously over time, in particular when new senses appear or old senses disappear. DWSI takes the time dimension into account and assumes that the data spans over a long continuous period of time in order to model the progressive evolution of senses across time.

The dynamic behaviour of words contributes to semantic ambiguity, which is a challenge in many NLP tasks. DWSI can serve as an analytical tool to help building terminology resources and indexing documents more accurately and therefore can be beneficial for information retrieval tasks.

DWSI follows the probabilistic graphical modelling approach to approximate the true meanings from the observed data. Thus, in this paper, we explore the relation of DWSI with topic modelling in general and to the dynamic topic modelling techniques in particular: they both aim to discover a latent variable (sense or topic respectively) from a sequential collection of documents. Despite a close relation between the tasks, topic modelling techniques are not fully explored or compared against in the current state of the art of DWSI.

The state of the art of DWSI consists of only two models: (Emms and Kumar Jayapal, 2016) and (Frermann and Lapata, 2016). They are both designed specifically for DWSI; both are parametric; and both are dynamic, in the sense that they both introduce a time variable into the model in order to capture the evolution of the meaning over time. Emms and Kumar Jayapal (2016) propose a parametric generative model (NEO) where each sense is represented as a $|V|$-dimensional multinomial distribution over the vocabulary $V$, each document is represented as a mixture of senses, and the dependency of the sense proportions on time is represented as a $K$-dimensional multinomial distribution over the $K$ senses. The parameters of the model have finite Dirichlet priors. A more complex model called SCAN (Frermann and Lapata, 2016) allows each sense distribution over the vocabulary to evolve sequentially from adjacent time slices, as well as the senses proportion. The multinomial parameters of words and senses have logistic normal priors.

The two above-mentioned models are parametric, in the sense that the number of senses (which reflects the structure of the hidden meanings in the data) is a hyper-parameter which has to be known a priori. This is not ideal given the nature of the DWSI task, which is meant to infer senses from the

<footnote n="1">¹The code corresponding to this work is available at https://github.com/AshjanAlsulaimani/DWSI-advanced-models</footnote>
data. The same issue has been studied for the tasks
of topic modelling and WSI; Hierarchical Dirich-
let Processes (HDP), a nonparametric hierarchical
model introduced by Teh et al. (2006), offer an pow-
erful solution to this problem. HDP extends Latent
Dirichlet Allocation (LDA) (Blei et al., 2003) by
placing Dirichlet processes priors (DPs) (Ferguson,
1973) on the infinite-dimensional space of multino-
mial probability distributions. Thus the number of
mixture components is infinite a priori and to be
inferred from the data. In contrast, LDA posits a
predefined number *K* of topics, each of which is a
multinomial distribution over the vocabulary. Each
document has specific topic proportions from a
Dirichlet prior, and the topics are shared among the
documents. Additionally, the HDP model allows
sharing topics not only among documents but also
across hierarchical levels by the use of multiple
DPs.

The intuition behind our approach relies on the fact
that the hierarchical DPs allow “new” senses to ap-
pear as needed, thanks to the theoretically infinite
number of possible senses. Therefore, the hierar-
chical design of Dirichlet processes can capture the
dynamic behaviour of the words, while inferring
the optimal number of clusters directly from the
data across time.

Word embeddings are another natural direction of
potential improvement for DWSI. Introduced by
Rumelhart and Abrahamson (1973); Bengio et al.
(2003, 2006), they provide a distributed representa-
tion where words with similar meanings are close
in a lower-dimensional vector space. Recently, var-
ious models have been proposed which integrate
word embeddings for topic modelling, however
these models do not necessarily represent both
words and topics using embeddings. Dieng et al.
(2019) provide an elegant solution to this problem:
Dynamic Embedded Topic Model (DETM) is a
parametric generative model inspired by D-LDA
(Dynamic LDA) Blei and Lafferty (2006), in which
each word is represented with a word embedding,
and per-time topics are represented as embeddings
as well. Topics and topic proportions evolve se-
quentially from adjacent time slices. DETM also
directly models per-topic conditional probability
of a word as the exponentiated inner product be-
tween the word embeddings and per-time topic
embeddings. This results in a closer semantic cor-
respondence between words and topics, and thus

leads to better topics quality.

By contrast to previous contributions in DWSI
which were mostly theoretical, this paper is an
empirical contribution focusing on adapting differ-
ent existing topic modelling techniques to DWSI.
The aim is to set the state of the art DWSI mod-
els up against two serious competitors, in order to
check whether they actually fit the task of DWSI
optimally. In this perspective, we adapt HDP and
DETM to the task of DWSI, describing our ap-
proach in §3. We test the ability of these mod-
els to detect meaning change over time using the
evaluation framework proposed by (Alsulaimani
et al., 2020), described in §4: using a large corpus
of biomedical time-stamped data, including 188
ambiguous target words, we compare the proposed
models with the current state of the art models NEO
and SCAN. The results, presented in §5, show that
HDP-based models achieve the best results over
the dataset, establishing a new state of the art for
DWSI.

## 2 Related Work

Topic modelling techniques are hierarchical proba-
bilistic Bayesian models used originally for discov-
ering topics in a collection of documents (Blei et al.,
2010). Topic models have also been adopted for the
Word Sense Induction (WSI) task, as introduced
by (Brody and Lapata, 2009; Yao and Van Durme,
2011): word senses are treated as topics, and a short
window around the target word (context) is consid-
ered instead of a full document. Topic modelling
techniques have been extended further to similar
tasks, such as Novel Sense Detection.

Novel Sense Detection (NSD; also called Novel
Sense Identification), introduced by Lau et al.
(2012), consists of determining whether a target
word acquires a new sense over two independent
periods of time, separated by a large gap. Sev-
eral authors have used Hierarchical Dirichlet Pro-
cesses (HDP) for this task over a small set of target
words and/or small set of data (Lau et al., 2012,
2014; Cook et al., 2014). Yao and Van Durme
(2011); Lau et al. (2012) show in a preliminary
study that HDP is also superior to LDA for WSI,
due to its ability to adapt to varying degrees of
granularity. Lau et al. (2012) extend this study us-
ing an oracle-based method to identify new senses
from HDP predictions for the task of NSD, and
for only five target words. Sarsfield and Tay-
yar Madabushi (2020) used HDP for NSD on a
larger dataset (Schlechtweg et al., 2020), which
was proposed in a recent shared task about Lexi-
cal Semantic Change Detection (LSCD), a refined
version of NSD: LSCD intends to answer the ques-
tion of whether the meaning of a target word has
changed between two independent periods of time
(also separated by a large time gap). In the LSCD
task, methods based on static word embeddings
(where the meaning of the word is represented by
a single vector) achieved the highest performance.

In contrast to NSD/LSCD, DWSI takes the time di-
mension into account and thus the task of DWSI is
technically broader: it aims to discriminate senses
and also models the temporal dynamics of word
meaning across a long continuous period of time,
e.g. year by year. As a result, DWSI can track the
evolution of senses, the emergence of new senses
and detect the year where a new sense appears. The
DWSI task is introduced independently by Emms
and Kumar Jayapal (2016) and Frermann and Lap-
ata (2016); given a target word and a time-stamped
corpus, both models estimate two main parame-
ters: the senses as distributions over words, and the
senses proportions over time. Frermann and Lap-
ata (2016) extend this by also inferring the subtle
meaning changes within a single sense over time,
i.e. by allowing different word distributions over
time for the same sense.

However, these models are parametric and require
the number of senses to be chosen in advance. Pre-
vious approaches dealt with this issue by increas-
ing the number of senses. For example, Emms and
Kumar Jayapal (2016) vary the number of senses
manually for every target word, while Frermann
and Lapata (2016) choose an arbitrary fixed large
number of senses for all the target words.

Additionally, evaluating and comparing such mod-
els on the DWSI task is difficult: the lack of large
scale time-stamped and sense-annotated data hin-
ders direct quantitative evaluation. The state of
the art models, (Emms and Kumar Jayapal, 2016;
Frermann and Lapata, 2016), were originally evalu-
ated only qualitatively on a few hand-picked target
words, with a manual investigation of the quality of
the associated top words in each cluster; Frermann
and Lapata (2016) also evaluated their model on

several indirect tasks. Alsulaimani et al. (2020) demonstrate that these evaluation methods are insufficient, and consequently propose a quantitative evaluation of these DWSI models based on a large set of data. In particular, they show that the senses size distribution plays a significant role in capturing the senses representations and emergence of new senses. The number of senses is clearly a crucial hyperparameter for a DWSI model, the choice of which should in theory depend on the characteristics of the data.

3 Approach

3.1 Parameters Notation

DWSI aims to discover the senses *S* across time
*Y* for each target word in a sequential collection
of documents, where senses are latent variables
and the number of senses is unknown a priori. A
DWSI model estimates at least two multinomial
distributions:

* P(W|S), the word given sense distribution. The changes within senses across time can also be represented as P(W|S, Y), the word given sense and year distribution. These distributions represent the sense.

* P(S|Y), the sense given year distribution.
This distribution represents the relative prevalence of a sense over time.

3.2 HDP-DWSI

HDP allows senses (i.e. clusters) to appear when a
new context occurs, as the number of senses is de-
termined by the data. HDP-DWSI directly relies on
this property: in the first step, all the documents, in-
dependently from their year, are clustered by HDP.
Appendix A provides details about the description
of HDP. This means that in this step the documents
are assumed to be exchangeable, as opposed to
dynamic models in which documents are only ex-
changeable within a time period. In the second
step, the year of the document (observed variable)
is reintroduced and the time-related multinomial
parameters P(S = s|Y = y) are estimated by
marginalising across the documents of each year j
independently $\sum_{d \in y} \frac{freq(s_d)}{\sum_{s' \in y} freq(s'_d)}$, where freq(s_d)
the number of words predicted as sense *s* in the document *d*, and *d* ∈ *y* represents the condition that the document *d* belongs to year *y*.

HDP-DWSI is intended to be used as a nonparametric method, but a parametric mode is also proposed for the purpose of evaluation and comparison against parametric models. In the nonparametric mode, the model parameters are obtained directly as described above. In the parametric mode, an additional step is required to reduce the number of senses because HDP-DWSI tends to induce a higher number of clusters than the gold number of senses, i.e. to split senses into multiple clusters. Depending on the context of the application, it can also be relevant to reduce the number of senses even in the nonparametric mode. This can also be done with the method described below for the parametric mode, called HDP-DWSI<sub>m</sub>.

HDP-DWSI<sub>m</sub> consists in merging the predicted senses which are the most semantically similar. Agglomerative hierarchical clustering (Ward Jr, 1963) is used to merge senses, based on a sense cooccurrence matrix obtained from the HDP clustering output.

Pointwise Mutual Information (PMI) is used to represent how strongly two predicted senses are statistically associated, under the assumption of independence:

$$PMI(s_i, s_j) = \log_2 \frac{P(s_i, s_j)}{P(s_i)P(s_j)} \quad (1)$$

where *i* ≠ *j* and *P(s<sub>i</sub>, s<sub>j</sub>)* is the joint probability of observing both *s<sub>i</sub>* and *s<sub>j</sub>* in the same document. *P(s<sub>i</sub>)* (resp. *P(s<sub>j</sub>)*) is the probability of a predicted sense with respect to the entire corpus, i.e. an occurrence is counted for every document in which the predicted sense *s<sub>i</sub>* (resp. *s<sub>j</sub>*) independently occurs.

Moreover, since a pair of predicted senses with negative PMI is uninformative for the purpose of merging similar senses, Positive Pointwise Mutual Information (PPMI), as defined in Equation 2, is used for constructing the sense cooccurrence matrix.

$$PPMI = \begin{cases} PMI(s_i, s_j) & \text{if } PMI(s_i, s_j) > 0 \\ 0 & \text{else} \end{cases} \quad (2)$$

(P)PMI is sensitive to low frequency events, particularly in the event when one of the predicted

senses (or both of them) is/are less frequent with respect to the whole corpus; thus it is possible that two senses mostly cooccur together by chance, yet obtain a high (P)PMI value. In such a case, the two predicted senses are not semantically associated, so this is a potential bias in the merging process. To counter this bias, we use the linkage criterion defined in Equation 3 as the average of the PPMI values weighted by their corresponding joint probabilities. The linkage criterion for two clusters *C*<sub>1</sub>, *C*<sub>2</sub>:

$$\sum_{\substack{\forall s_1 \in C_1 \\ \forall s_2 \in C_2}} w(s_1, s_2) \times PPMI(s_1, s_2) \quad (3)$$

$$\text{where } w(s_1, s_2) = \frac{P(s_1, s_2)}{\sum_{\substack{\forall s_1 \in C_1 \\ \forall s_2 \in C_2}} P(s_1, s_2)}$$

The evaluation method proposed by Alsulaimani et al. (2020) (see §4) relies on the gold number of senses, as it is originally intended for parametric methods. In order to compare an HDP-based model against parametric models in an equivalent setting, the HDP-DWSI<sub>m</sub> merging method is used to reduce the predicted number of senses to the gold-standard number of senses.

## 3.3 DETM-DWSI

DETM represents not only the observed words but also latent topics/senses as embeddings, while preserving the traditional representation of a topic/sense as a probability distribution across words. The categorical distributions over the vocabulary is time dependent, i.e. *P(W|S, Y)* and is derived from the corresponding word embeddings and sense embedding at a given time. DETM also places time-dependent priors over senses proportions: the use of Markov chain over the sense proportions allows smoothness of the variations between the adjacent senses at neighboring times (see Appendix A for the description of DETM). We propose two modes for DETM-DWSI as follows:

* In the regular DETM-DWSI, both the word and sense embeddings are trained simultaneously. This mode does not require any additional resource but the corpus must be large enough for the embeddings to be accurate.

* In DETM-DWSI<sub>i</sub>, the model is trained with prefitted word embeddings. This mode leverages the external information contained in the
embeddings, potentially obtaining a more accurate representation of the senses as a consequence. It also allows the application of the model to text containing words not present in the corpus, as long as their embedding is available.

In the experiments described below, the DETM-DWSI<sub>i</sub> models are trained using the BioWord-Vec pretrained word embeddings<sup>2</sup> (Zhang et al., 2019). The fastText subword embedding model (Bojanowski et al., 2017) is a variant of the continuous skip-gram model (Mikolov et al., 2013). The fastText subword embedding can learn a distinct vector for each word while exploiting subword information in a unified n-gram embedding space. BioWordVec embeddings are trained with fastText on the PubMed text and MeSH terms, combined into a unified embedding space. In the biomedical domain, the advantage of a subword embedding model is that it can handle Out of Vocabulary (OOV) words (Zhang et al., 2019).<sup>3</sup> This leads to a more precise word representation, in theory better able to capture the semantics of specialised concepts. We use the intrinsic BioWordVec embeddings (as opposed to the extrinsic type), meant to represent the semantic similarity between words (Zhang et al., 2019).

# 4 Experimental Setup

## 4.1 Data

We use the DWSI evaluation framework proposed by Alsulaimani et al. (2020): the biomedical literature is used as a source of labelled and time-stamped data which covers the years 1946 to 2019.<sup>4</sup> The dataset is collected from resources provided by the US National Library of Medicine (NLM): PubMed (a platform which includes the major biomedical literature databases) and MeSH (a controlled vocabulary thesaurus, created manually to index NLM databases).<sup>5</sup> The data is preprocessed

as in (Alsulaimani et al., 2020). The data consists of 188 ambiguous target words and 379 gold-standard senses (Jimeno-Yepes et al., 2011): 75 ambiguous target words have 2 senses, 12 have 3 and one has 5 senses. The total data size is 15.36 × 10<sup>9</sup> words, and the average number of documents is 61,352 by sense. The input documents for every target word consist of the occurrences of the target word which are provided with a window of 5-word context on each side as well as the year of publication. The gold-standard sense label is also available for evaluation purposes.

## 4.2 Algorithms Settings

* The HDP-DWSI and HDP-DWSI<sub>m</sub> models are trained using the official C++ implementation of HDP.<sup>6</sup> No additional preprocessing is needed.

* The DETM-DWSI and DETM-DWSI<sub>i</sub> models are trained using the implementation provided by Dieng et al. (2019).<sup>7</sup> The preprocessing is adapted to the DWSI dataset: since the data is strongly imbalanced across time, stratified sampling is used in order to ensure a representative time distribution (with at least one instance by year) across the data partitions. The data is split into 85% of instances for training and 15% for validation. The document frequency thresholds are unused so as to include all the words. For efficiency reasons, during training the number of instances is capped at 2,000 instances per year.

## 4.3 Evaluation Methodology

Since DWSI is an unsupervised task (clustering) and our evaluation is based on the external sense labels, both the estimation of the model and the evaluation are performed on the full set of documents for each target word. The gold-standard number of senses of each ambiguous target word is provided for all the parametric models (excluding HDP-DWSI). The default parameters are used in all the systems,<sup>8</sup> except the number of itera-

<footnote n="2"><sup>2</sup>https://github.com/ncbi-nlp/BioSentVec.

<sup>3</sup>Note that the PubMed and MeSH terms are biomedical resources, collected from the US National Library of Medicine (NLM) and based on the database of 2019 and 2018 respectively. These are the same version for the DWSI evaluation data.

<sup>4</sup>https://github.com/AshjanAlsulaimani/DWSI-eval

<sup>5</sup>https://www.nlm.nih.gov/</footnote>

<footnote n="6"><sup>6</sup>https://github.com/blei-lab/hdp.

<sup>7</sup>https://github.com/adjidieng/DETM.

<sup>8</sup>This means that we do not tune any hyper-parameter for any of the systems. Since DWSI applications would usually not have access to any labelled data, the performance would be unrealistic if the parameters were tuned.</footnote>
tions/epochs (set to 500 for all the systems),<fnref n="9" /> and specifically for DETM-DWSI the batch size is set to 1000 and the dimension of the embeddings is set to 200.

After estimating each model for each ambiguous target word, the posterior probability is calculated for every document. The sense with the highest probability is assigned.

## 4.4 Evaluation Measures

We follow Alsulaimani et al. (2020) for the evaluation measures with some adjustments, detailed below.

The “Global Matching” method, presented by Alsulaimani et al. (2020), consists in determining a one-to-one assignment between predicted senses and gold senses based on their joint frequency: the pair with the highest frequency is matched first, and this process is iterated until all the senses are matched. In the case of HDP-DWSI, the number of predicted senses may be higher than the gold number of senses, and the instances of the predicted senses which remain unmatched are considered as false negative. This allows to compare HDP-DWSI with the parametric models, assuming that in theory the ideal nonparametric model would infer exactly the true number of senses. Of course, HDP-DWSI<sub>m</sub> is by definition more appropriate for a comparison in the parametric setting of HDP-based methods.

We also propose to use the V-measure as a different method of evaluation. The V-measure is introduced by Rosenberg and Hirschberg (2007), providing a different way to evaluate a clustering solution. In this case, it evaluates every cluster against every gold sense without relying on a matching method, thus providing an objective assessment even when the number of the clusters is higher than the true number of senses. The V-measure is based on entropy (entropy is a measure of the uncertainty associated with a random variable): it is defined as the harmonic mean of homogeneity and completeness, which are both based on the normalised conditional entropy.

Alsulaimani et al. (2020) also propose to evalu-

ate the emergence of a new sense by considering whether the system predicts the true emergence year of a sense. This requires a method to determine the year from the $P(S|Y)$ distribution, for which the original algorithm “EmergeTime” was proposed in Jayapal (2017). We introduce “LREmergeTime” (see Appendix B Algorithm 1), an improved version of “EmergeTime” using linear regression instead of multiple thresholds within a window. Indeed, the original algorithm is very sensitive to the noise which sometimes occurs in the emergence pattern. Linear regression handles this issue better, since it measures the global trend across the window.<fnref n="10" />

The emergence year is evaluated as in (Alsulaimani et al., 2020): (1) with standard classification measures, considering the sense as correctly predicted if the year is within 5 years of the true emergence year; (2) with (normalized) Mean Absolute Error, representing the average difference in number of years but also penalizing the wrongly predicted presence/absence of emergence.

Finally we also use the distance between the true and predicted evolution of the senses over time ($P(S|Y)$) as an evaluation method for DWSI, again following Alsulaimani et al. (2020).

# 5 Results

## 5.1 Qualitative exploration

We explore the temporal meanings of “SARS-associated coronavirus” over the years (2002-2018) as an example. The ambiguous word has two gold-standard senses described by UMLS concepts C1175175 and C1175743: *Severe Acute Respiratory Syndrome* (refers to the disease caused by the virus) and *SARS Virus* (refers to the virus related to the Coronavirus family causing the disease) respectively. The top words represented by the inferred parameter word given sense, identified by HDP-

<footnote n="9">⁹It has been verified that 500 epochs is sufficient for all models to become stable and therefore to achieve their optimal performance.</footnote>

<footnote n="10">¹⁰ The superiority of “LREmergeTime” was confirmed using a subset of manually annotated targets (the targets are chosen based on the visual clarity of the emergence pattern). The evaluation results on this subset show that “LREmergeTime” performs closer to the annotated senses. Following the evaluation measures by Alsulaimani et al. (2020), the results of “EmergeTime” and “LREmergeTime” are respectively 0.7 and 0.8 for Fscore, 12.06 and 6.74 for MAE, 0.21 and 0.11 for Normalised MAE. See Appendix C for details of algorithms outputs.</footnote>
DWSI<sub>m</sub> for the first sense are {patients, outbreak, sars, 2003, epidemic, health, case, transmission, hospital} and for the second sense are {cov, sars, coronavirus, patients, infection, protein, respiratory, acute, syndrome, cells}. Figure 1 shows the relative prevalence of the two inferred and gold senses over time, and Table 1 shows the top inferred words/usages associated with sense C1175175 at specific times.

In Figure 1, both senses data start in 2002, however the prevalence of sense C1175175 was decreasing progressively from 2002 to 2018 since SARS was successfully contained in 2004, while the prevalence of the sense C1175743 kept increasing since the research about the SARS virus became a priority for the public health around the world.

The temporal changes of the top words within C1175175 are highlighted in Table 1. Historically, the first known case of SARS appears in November 2002, causing the 2002-2004 SARS outbreaks in cities and hospitals. Global attention then started and in 2016, for instance, the top words shifted to *facemask*, *post*, *era*, *sars*. Finally, the year 2018 shows the concerns about a second wave of SARS.

<table><caption>Table 1: Temporal evolution of the top-7 words for the sense *Severe Acute Respiratory Syndrome* learned by HDP-DWSI<sub>m</sub>, at specific times.</caption><thead><tr><th>2002</th><th>2003</th><th>2004</th><th>→</th><th>2016</th><th>2017</th><th>2018</th></tr></thead><tbody><tr><td>case</td><td>patients</td><td>patients</td><td>outbreak</td><td>outbreak</td><td>outbreak</td><td>second</td></tr><tr><td>outbreak</td><td>outbreak</td><td>outbreak</td><td>facemask</td><td>2003</td><td>2003</td><td>impact</td></tr><tr><td>lessons</td><td>case</td><td>sars</td><td>post</td><td>patients</td><td>impact</td><td>impact</td></tr><tr><td>learned</td><td>health</td><td>transmission</td><td>2003</td><td>china</td><td>epidemic</td><td>epidemic</td></tr><tr><td>health</td><td>2003</td><td>hospital</td><td>era</td><td>data</td><td>wave</td><td>wave</td></tr><tr><td>chief</td><td>sars</td><td>case</td><td>sars</td><td>outbreaks</td><td>n't</td><td>n't</td></tr><tr><td>falls</td><td>hospital</td><td>patient</td><td>hong</td><td>health</td><td>link</td><td>link</td></tr></tbody></table>

<figure><img src="image_1.png" /><figcaption>Figure 1: Dynamic representations of “SARS-associated coronavirus”. On the Y-axis, $P(S|Y)$ shows the relative prevalence of the gold senses as well as the predicted senses across time estimated by HDP-DWSI<sub>m</sub>.</figcaption></figure>

## 5.2 Matching-based Evaluation

Table 2 shows the performance of the six models according to standard classification and regression measures using “Global Matching”. In general, DWSI models based on HDP perform well compared to NEO or SCAN. In the case of HDP-DWSI, “Global Matching” causes two observable effects: it increases precision, by allowing the system to choose the best predicted clusters matched with the gold senses; but it also decreases recall by introducing a large number of false negative cases due the discarded unmatched predicted clusters. Nevertheless the macro F1 score for HDP-DWSI is much higher than both NEO and SCAN, by 17.7% and 13.8% respectively. This shows that HDP-DWSI can distinguish minority senses significantly better. This can also be seen in Table 3 which shows the mean F1-score by sense size.

<table><caption>Table 2: Global performance results for all systems using “Global Matching”. P/R/F1 stand for Precision/Recall/F1-score (higher is better) MAE stands for Mean Absolute Error (lower is better). Best performance in bold.</caption><thead><tr><th rowspan="2">Systems</th><th colspan="3">Macro-average</th><th colspan="3">Micro-average</th><th rowspan="2">MAE</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>DETM-DWSI<sub>i</sub></td><td>0.553</td><td>0.561</td><td>0.557</td><td>0.704</td><td>0.704</td><td>0.704</td><td>0.401</td></tr><tr><td>DETM-DWSI</td><td>0.559</td><td>0.590</td><td>0.574</td><td>0.650</td><td>0.650</td><td>0.650</td><td>0.379</td></tr><tr><td>HDP-DWSI</td><td><strong>0.726</strong></td><td>0.599</td><td>0.657</td><td>0.739</td><td>0.424</td><td>0.539</td><td>-</td></tr><tr><td>HDP-DWSI<sub>m</sub></td><td>0.666</td><td><strong>0.681</strong></td><td><strong>0.674</strong></td><td><strong>0.744</strong></td><td><strong>0.744</strong></td><td><strong>0.744</strong></td><td><strong>0.26</strong></td></tr><tr><td>NEO</td><td>0.548</td><td>0.569</td><td>0.558</td><td>0.595</td><td>0.595</td><td>0.595</td><td>0.425</td></tr><tr><td>SCAN</td><td>0.562</td><td>0.591</td><td>0.577</td><td>0.558</td><td>0.558</td><td>0.558</td><td>0.444</td></tr></tbody></table>

The superiority of HDP-DWSI<sub>m</sub> is even clearer: the macro F1 score is 20.8% higher than NEO and 16.8% higher than SCAN; the performance difference in micro F1 score is even stronger: 21.0% above DETM-DWSI<sub>i</sub>, 17.4% higher than DETM-DWSI, 25.0% above NEO and 33.3% above SCAN. Contrary to the differences between NEO and SCAN, HDP-DWSI<sub>m</sub> improves performance significantly across the board: both precision and recall are drastically higher, according to both micro and macro scores. This means that HDP-based models are fundamentally much better at discriminating the different senses (with a very significant p-value < 0.05), as opposed to strategically favouring large senses for instance. This is confirmed in Table 3.<sup>11</sup>

The two DETM-based models perform very well, in particular achieving micro F1-score much higher than NEO and SCAN. However their macro-average performance is comparable to NEO and

<footnote n="11"><sup>11</sup>A Wilcoxon rank sum test is applied on the F1-scores of the senses for the results in Table 2 and 3.</footnote>
SCAN, a clear sign that they do not separate the senses better. Table 3 confirms that the DETM-based models perform closely to NEO and SCAN.

Finally the MAE scores confirm that DETM-DWSI<sub>i</sub> and DETM-DWSI perform better than NEO and SCAN, but also that these four models are drastically outperformed by HDP-DWSI<sub>m</sub>.

<table><caption>Table 3: Comparison of the performance by sense according to the “Global Matching” method, ranked by proportion within a target. The sense rank is ordered by the size of senses (in number of instances), from the smallest sense (rank first) to the largest (rank last). “-” means any number of senses (all the data). The systems are referred to by their initials.</caption><thead><tr><th rowspan="2">Number of Senses</th><th rowspan="2">Sense rank</th><th colspan="6">Mean F1 score</th></tr><tr><th>N</th><th>S</th><th>H</th><th>H<sub>m</sub></th><th>D</th><th>D<sub>i</sub></th></tr></thead><tbody><tr><td>-</td><td>first</td><td>0.299</td><td>0.321</td><td>0.532</td><td>0.438</td><td>0.314</td><td>0.283</td></tr><tr><td>-</td><td>last</td><td>0.732</td><td>0.692</td><td>0.658</td><td>0.857</td><td>0.739</td><td>0.772</td></tr><tr><td>2</td><td>first</td><td>0.315</td><td>0.335</td><td>0.557</td><td>0.462</td><td>0.330</td><td>0.294</td></tr><tr><td>2</td><td>second</td><td>0.740</td><td>0.6995</td><td>0.659</td><td>0.863</td><td>0.744</td><td>0.777</td></tr><tr><td>3</td><td>first</td><td>0.100</td><td>0.143</td><td>0.224</td><td>0.132</td><td>0.111</td><td>0.134</td></tr><tr><td>3</td><td>second</td><td>0.253</td><td>0.390</td><td>0.553</td><td>0.499</td><td>0.237</td><td>0.248</td></tr><tr><td>3</td><td>third</td><td>0.629</td><td>0.597</td><td>0.655</td><td>0.778</td><td>0.681</td><td>0.708</td></tr></tbody></table>

## 5.3 Entropy-based Evaluation

<table><caption>Table 4: V-measure, homogeneity and completeness for all the systems. Both the mean and median across targets are reported, because the strong differences between targets in terms of size and distribution of the senses may cause a bias with the mean.</caption><thead><tr><th rowspan="2">Systems</th><th colspan="2">V-measure</th><th colspan="2">homogeneity</th><th colspan="2">completeness</th></tr><tr><th>Mean</th><th>Median</th><th>Mean</th><th>Median</th><th>Mean</th><th>Median</th></tr></thead><tbody><tr><td>DETM-DWSI<sub>i</sub></td><td>0.093</td><td>0.021</td><td>0.106</td><td>0.059</td><td>0.089</td><td>0.016</td></tr><tr><td>DETM-DWSI</td><td>0.092</td><td>0.026</td><td>0.111</td><td>0.059</td><td>0.085</td><td>0.020</td></tr><tr><td>HDP-DWSI</td><td>0.213</td><td><strong>0.161</strong></td><td><strong>0.384</strong></td><td><strong>0.349</strong></td><td>0.157</td><td><strong>0.107</strong></td></tr><tr><td>HDP-DWSI<sub>m</sub></td><td><strong>0.272</strong></td><td>0.110</td><td>0.289</td><td>0.154</td><td><strong>0.268</strong></td><td>0.094</td></tr><tr><td>NEO</td><td>0.046</td><td>0.018</td><td>0.053</td><td>0.026</td><td>0.043</td><td>0.014</td></tr><tr><td>SCAN</td><td>0.080</td><td>0.021</td><td>0.098</td><td>0.041</td><td>0.074</td><td>0.015</td></tr></tbody></table>

Table 4 shows the results of the systems for V-measure, with details about homogeneity and completeness. HDP-DWSI and HDP-DWSI<sub>m</sub> perform the best at all three levels, with values far above the other systems. HDP-DWSI has the highest homogeneity mean, because this model produces a higher number of smaller predicted senses; these predicted senses are therefore more homogeneous in general, but also less complete since the gold senses are often split. HDP-DWSI<sub>m</sub> merges the senses predicted by HDP-DWSI, thus obtaining lower homogeneity but compensating with higher completeness, leading to higher mean V-measure.

Figure 2 offers a more precise picture of the differences between systems about their V-measure distribution. It confirms that DETM-DWSI, DETM-DWSI<sub>i</sub> and SCAN perform very similarly. It

<figure><img src="image_2.png" /><figcaption>Figure 2: Quantile plot of the V-measure scores by system, with the quantile rank shown on the X axis and the corresponding value on the Y axis. Example: for HDP-DWSI, the median (x=0.5) is y=0.16. The graph is obtained by sorting the values, then normalising their rank between 0 and 1.</figcaption></figure>

shows that the higher performance of DETM-DWSI, DETM-DWSI<sub>i</sub> and SCAN compared to NEO is due to a minority of targets, as their 75% lowest scores are almost identical. These targets cause most of the high difference in mean between NEO and SCAN, as the smaller difference in means shows.

By contrast, HDP-DWSI and HDP-DWSI<sub>m</sub> have a much smaller proportion of low scores. Interestingly, HDP-DWSI has higher low scores than HDP-DWSI<sub>m</sub>, i.e. HDP-DWSI performs better until both systems reach the median. However HDP-DWSI<sub>m</sub> skyrockets just after the median and surpasses HDP by having much higher high scores. This explains why the median is slightly lower for HDP-DWSI<sub>m</sub> than HDP while the mean is much higher for HDP-DWSI<sub>m</sub>.<fnref n="12" />

## 5.4 Comparison between Measures

<table><caption>Table 5: Pearson correlation coefficients: the relationship between the performance according to different measures. All the results are significantly correlated with p-value <= 5.6e-13. The systems are referred to by their initials.</caption><thead><tr><th rowspan="2">Measure</th><th colspan="6">V-measure</th></tr><tr><th>N</th><th>S</th><th>H</th><th>H<sub>m</sub></th><th>D</th><th>D<sub>i</sub></th></tr></thead><tbody><tr><td>Macro F-Score</td><td>0.730</td><td>0.799</td><td>0.856</td><td>0.901</td><td>0.795</td><td>0.794</td></tr><tr><td>Micro F-Score</td><td>0.583</td><td>0.850</td><td>0.806</td><td>0.714</td><td>0.713</td><td>0.494</td></tr></tbody></table>

V-measure can introduce a bias towards systems which predict a number of clusters larger than the number of gold senses. Such systems tend to have very high homogeneity scores and low completeness scores. However, this is not the case for HDP-DWSI. The HDP-DWSI performance is high not only according to the V-measure but also confirmed

<footnote n="12">¹²This can be verified visually on the quantile plot, because the area under the curve is equal to the mean.</footnote>
by the F1 scores. The number of senses predicted by HDP-DWSI in average is 8 senses, with the minimum 4 senses and the maximum 13 senses. The Pearson correlation between homogeneity and completeness is 0.853 and with very significant p-value, 2.2e-16. Also, it is found that there is virtually no correlation between the predicted number of senses and either the size of the data or V-measure by target: 0.065, 0.008 (non significant: p-value = 0.3746, 0.261). This indicates that HDP-DWSI is not biased towards generating more senses when the data is larger.

Table 5 shows that all the evaluation measures are significantly correlated. The macro-F1 scores are positively correlated in all four systems. However, the micro F-score favours systems that perform well on the majority sense, whereas the V-measure explicitly evaluates every cluster, taking into account not only the majority sense but also the minority one. Therefore systems which favour the majority sense, like NEO and DETM-DWSIᵢ, have a lower correlation.

## 5.5 Emergence-based Evaluation

<table><caption>Table 6: Sense emergence evaluation results for all the systems. The values in bold indicate the best score achieved among the systems.</caption><thead><tr><th>System</th><th>precision</th><th>recall</th><th>F1 score</th><th>global mean absolute error</th><th>normalised global mean absolute error</th></tr></thead><tbody><tr><td>DETM-DWSIᵢ</td><td><strong>0.500</strong></td><td>0.009</td><td>0.019</td><td>48.713</td><td>0.812</td></tr><tr><td>DETM-DWSI</td><td>0.385</td><td>0.050</td><td>0.088</td><td>45.685</td><td>0.761</td></tr><tr><td>HDP-DWSI<sub>m</sub></td><td>0.371</td><td>0.254</td><td>0.301</td><td><strong>23.148</strong></td><td>0.403</td></tr><tr><td>NEO</td><td>0.383</td><td><strong>0.397</strong></td><td><strong>0.390</strong></td><td>23.967</td><td><strong>0.399</strong></td></tr><tr><td>SCAN</td><td>0.374</td><td>0.162</td><td>0.226</td><td>39.634</td><td>0.666</td></tr></tbody></table>

DWSI systems can also be evaluated based on their ability to predict the year of emergence of a new sense. Table 6 shows the performance of the systems after applying “LREmergeTime” (see §4.4) on the predictions of the systems. HDP-DWSI<sub>m</sub> and NEO perfom closely to each other and much better than the other systems, according to both classification measures and MAE. NEO was designed and implemented with a focus on detecting sense emergence, this probably explains why it performs particularly well in this task (Jayapal, 2017).

## 5.6 Evaluation based on the predicted evolution over time

Table 7 shows for every system how well their prediction of $P(S|Y)$ matches the true evolution of sense. Among all the systems, HDP-DWSI<sub>m</sub> predicts the closest $P(S|Y)$ to the true evolution

<table><caption>Table 7: Mean distance between the true and predicted sense, measured by Dynamic Time Warping (DTW) and Euclidean distance (lower is better). The results in bold indicate the best system.</caption><thead><tr><th rowspan="2">System</th><th colspan="2">Distance Global mean</th></tr><tr><th>DTW</th><th>Euclidean</th></tr></thead><tbody><tr><td>DETM-DWSI</td><td>0.191</td><td>0.134</td></tr><tr><td>DETM-DWSI<sub>i</sub></td><td>0.165</td><td>0.106</td></tr><tr><td>HDP-DWSI<sub>m</sub></td><td><strong>0.115</strong></td><td><strong>0.067</strong></td></tr><tr><td>NEO</td><td>0.182</td><td>0.124</td></tr><tr><td>SCAN</td><td>0.222</td><td>0.142</td></tr></tbody></table>

according to both distance measures. This confirms that not only HDP-DWSI<sub>m</sub> produces accurate predictions of the emergence year of novel senses but also predicts accurately the $P(S|Y)$ trends in general, with significantly less errors than the other systems.

# 6 Conclusion and Discussion

In this paper we adapted two topic modelling methods to the task of DWSI and evaluated them against two state of art DWSI systems, NEO and SCAN, using the evaluation framework proposed by Alsu-laimani et al. (2020). We also compared using the V-measure, and proposed an improved version of the emergence algorithm.

The results show that HDP-based models are able to fit the data better than the parametric models. The results strongly show that merging HDP-DWSI clusters performs better than the DETM-DWSI models and LDA-like clustering, such as NEO and SCAN. The properties of HDP make it better at accurately fitting the topics/senses, in particular when there is a high imbalance between the senses proportions, i.e. with senses smaller in size (see Table 3). Furthermore, the fact that HDP-DWSI<sub>m</sub> outperforms all the other parametric models also demonstrates that these models do not find the optimal separation between the senses. It seems that the additional complexity of the time dimension together with the parametric constraints do not cope well with data imbalance across years.

One could naturally assume that models designed specifically for a task would perform better on it. Implicitly, the research community encourages the creation of new models and tends to reward theoretical contribution over empirical ones. Thus there might be a bias in favor of designing sophisticated ad-hoc models (like NEO and SCAN) rather than adapting existing robust models (like HDP).
# 7 Limitations

## 7.1 Biomedical Domain

The dataset used in these experiments belongs to the biomedical domain and it is in English language. There is no clear reason why the comparison between models would lead to different results on different domains, therefore we would expect the reported results (at least the major tendencies) to be also valid on the general domain.

Nevertheless this assumption would need to be tested experimentally. To our knowledge, there is no equivalent dataset available in the general domain which satisfies the two following conditions:

* Time-stamped documents spanning a relatively long period of time;

* Every document labelled with the sense of the target word.

## 7.2 Duration of the Training Stage

In the table below, we present the computational cost of training the different models presented in this paper. Most of the experiments were carried out on a computing cluster containing 20 to 30 machines with varying characteristics, thus the total duration is approximative.

Computing times are reported in hours of CPU/GPU activity required to train the total of 188 target datasets. It is important to note that the two DETM models are trained on GPUs, whereas all the other models are trained on regular CPUs. Thus in overall computing power, the DETM models are the most costly to train (more than HDP, despite the higher duration).

<table><thead><tr><td>System</td><td>Duration</td><td>Notes</td></tr></thead><tbody><tr><td>DETM-DWSI<sub>1</sub></td><td>523.4</td><td>Trained on GPU</td></tr><tr><td>DETM-DWSI</td><td>474.2</td><td>Trained on GPU</td></tr><tr><td>HDP-DWSI</td><td>2,471.4</td><td></td></tr><tr><td>HDP-DWSI<sub>m</sub></td><td>0.1</td><td>Only the merging process</td></tr><tr><td>NEO</td><td>25.1</td><td></td></tr><tr><td>SCAN</td><td>77.9</td><td></td></tr></tbody></table>

## Acknowledgements

We would like to thank the anonymous reviewers for their valuable comments. The first author is

grateful to the Custodian of the Two Holy Mosques Scholarship Program from the Saudi Arabian Government as well as to Mr. Saud Alsulaimani for supporting this work. This work was conducted using high-performance clusters facilitated by the ADAPT Centre, Trinity College of Dublin.

## References

Eneko Agirre and Aitor Soroa. 2007. Semeval-2007 task 02: Evaluating word sense induction and discrimination systems. In *Proceedings of the fourth international workshop on semantic evaluations (semeval-2007)*, pages 7–12.

Ashjan Alsulaimani, Erwan Moreau, and Carl Vogel. 2020. An evaluation method for diachronic word sense induction. In *Findings of the Association for Computational Linguistics: EMNLP 2020*, pages 3171–3180. Association for Computational Linguistics.

Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. journal of machine learning research, 3 (feb): 1137-1155, 2003. *Google Scholar Google Scholar Digital Library Digital Library*.

Yoshua Bengio, Holger Schwenk, Jean-Sébastien Senécal, Frédéric Morin, and Jean-Luc Gauvain. 2006. Innovations in machine learning. *Neural Probabilistic Language Models*, 194:137–186.

David Blei, Lawrence Carin, and David Dunson. 2010. Probabilistic topic models: A focus on graphical model design and applications to document and image analysis. *IEEE signal processing magazine*, 27(6):55.

David M Blei and John D Lafferty. 2006. Dynamic topic models. In *Proceedings of the 23rd international conference on Machine learning*, pages 113–120. ACM.

David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. *Journal of machine Learning research*, 3(Jan):993–1022.

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. *Transactions of the Association for Computational Linguistics*, 5:135–146.

Samuel Brody and Mirella Lapata. 2009. Bayesian word sense induction. In *Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics*, pages 103–111. Association for Computational Linguistics.

Paul Cook, Jey Han Lau, Diana McCarthy, and Timothy Baldwin. 2014. Novel word-sense identification. In *Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers*, pages 1624–1635.
Adji B Dieng, Francisco JR Ruiz, and David M Blei. 2019. The dynamic embedded topic model. *arXiv preprint arXiv:1907.05545*.

Martin Emmas and Arun Kumar Jayapal. 2016. Dynamic generative model for diachronic sense emergence detection. In *Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers*, pages 1362–1373.

Thomas S. Ferguson. 1973. A bayesian analysis of some nonparametric problems. *Annals of Statistics*, 1:209–230.

Lea Frermann and Mirella Lapata. 2016. A bayesian model of diachronic meaning change. *Transactions of the Association for Computational Linguistics*, 4:31–45.

Arun Jayapal. 2017. *Finding Sense Changes by Unsupervised Methods*. PhD thesis, Trinity College Dublin.

Antonio J Jimeno-Yepes, Bridget T McInnes, and Alan R Aronson. 2011. Exploiting mesh indexing in medline to generate a data set for word sense disambiguation. *BMC bioinformatics*, 12(1):223.

Jey Han Lau, Paul Cook, Diana McCarthy, Spandana Gella, and Timothy Baldwin. 2014. Learning word sense distributions, detecting unattested senses and identifying novel senses using topic models. In *Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 259–270.

Jey Han Lau, Paul Cook, Diana McCarthy, David Newman, and Timothy Baldwin. 2012. Word sense induction for novel sense detection. In *Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics*, pages 591–601. Association for Computational Linguistics.

Suresh Manandhar, Ioannis Klapaftis, Dmitriy Dligach, and Sameer Pradhan. 2010. SemEval-2010 task 14: Word sense induction &disambiguation. In *Proceedings of the 5th International Workshop on Semantic Evaluation*, pages 63–68, Uppsala, Sweden. Association for Computational Linguistics.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In *Advances in neural information processing systems*, pages 3111–3119.

Andrew Rosenberg and Julia Hirschberg. 2007. V-measure: A conditional entropy-based external cluster evaluation measure. In *Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL)*, pages 410–420.

David E Rumelhart and Adele A Abrahamson. 1973. A model for analogical reasoning. *Cognitive Psychology*, 5(1):1–28.

Eleri Sarsfield and Harish Tayyar Madabushi. 2020. UoB at SemEval-2020 task 1: Automatic identification of novel word senses. In *Proceedings of the Fourteenth Workshop on Semantic Evaluation*, pages 239–245, Barcelona (online). International Committee for Computational Linguistics.

Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020. Semeval-2020 task 1: Unsupervised lexical semantic change detection. *arXiv preprint arXiv:2007.11464*.

Yee Whye Teh, Michael I Jordan, Matthew J Beal, and David M Blei. 2006. Hierarchical dirichlet processes. *Journal of the american statistical association*, 101(476):1566–1581.

Joe H Ward Jr. 1963. Hierarchical grouping to optimize an objective function. *Journal of the American statistical association*, 58(301):236–244.

Xuchen Yao and Benjamin Van Durme. 2011. Non-parametric bayesian word sense induction. In *Proceedings of TextGraphs-6: Graph-based Methods for Natural Language Processing*, pages 10–14. Association for Computational Linguistics.

Yijia Zhang, Qingyu Chen, Zhihao Yang, Hongfei Lin, and Zhiyong Lu. 2019. Biowordvec, improving biomedical word embeddings with subword information and mesh. *Scientific data*, 6(1):1–9.
# A Hierarchical Bayesian Models Background

## A.1 Hierarchical Dirichlet Processes

Hierarchical Dirichlet Processes (HDP), introduced by Teh et al. (2006), uses Dirichlet processes priors (DPs), on the infinite-dimensional space of multinomial probability distributions and thus the number of mixture components (senses) is infinite a priori. The Hierarchical DPs allow new senses to emerge naturally at any point in time and guarantee the senses are shared within and across the documents. The DP provides a distribution on distributions over an arbitrary space. *H* is a symmetric Dirichlet on the word simplex and *γ* is a concentration parameter that controls the amount of variability of senses on the base distribution *G*₀, a distribution over senses drawn from a DP. *α* is also a concentration parameter that controls the amount of variability of per-document senses on *G*<sub>d</sub>, a multinomial probability distribution over senses drawn from a DP. Then, for each word *w* we draw a sense *β*<sub>d,n</sub> from *G*<sub>d</sub> and finally draw the word *w* from that sense *β*<sub>d,n</sub>. The graphical model and the generative story of HDP are described in Figure 3.

## A.2 Dynamic Embedded Topic Model

Dynamic Embedded Topic Model (DETM), introduced by Dieng et al. (2019), uses embedding representations of words and topics. For each term *v*, it considers an *L*-dimensional embedding representation *p*<sub>*v*</sub>. It also considers an embedding *α*<sub>*k*</sub><sup>*t*</sup> ∈ ℝ<sup>*L*</sup> for each topic *k* at a given time step *t* = 1, ..., *T*. The topics (i.e. distributions over the vocabulary) are represented by the normalised exponentiated dot product between the embedding representation of the word and the assigned topic's embedding at every time *t* for each word in a document *d*: *p*(*w*<sub>*d*,*n*</sub> = *v*|*z*<sub>*d*,*n*</sub> = *k*, *α*<sub>*k*</sub><sup>*t*</sup>) ∝ exp{*p*<sub>*v*</sub><sup>*T*</sup>*α*<sub>*k*</sub><sup>*t*</sup>}}. The DETM uses a Markov chain over the topic embeddings *α*<sub>*k*</sub><sup>*t*</sup> and thus they evolve under Gaussian noise with variance *δ*<sup>2</sup>. Moreover, DETM posits time-varying prior, the logistic-normal distribution 𝒩 over the topic proportions *θ*<sub>*d*</sub>, which depends on a latent variable *η*<sub>*t*</sub><sub>*d*</sub>.

# B Emergence Algorithm

"LREmergeTime" algorithm in 1 is linear regression based algorithm, an improved version of "EmergeTime" proposed by (Jayapal, 2017).

## ALGORITHM 1

Emergence Detection algorithm based on linear regression

Input $\pi$: $\pi[i]$ is the probability at time $i$, with $1 \le i \le N$

Input $r$: window size $\triangleright$ Value used for window size: 5

Input $s$: slope threshold. $\triangleright$ Value used for slope threshold: 0.04

function LREMERGTIME($\pi$, $r$, $s$)

Surges = $\phi$

for $n$:=1 to (N-r+1) do

if SurgeStart($n$, $\pi$, $s$) then

Surges = Surges $\cup$ $\{n\}$

end if

end for

if Surges $\neq$ $\phi$ then

return min(Surges)

else

return $\phi$

end if

end function

function SURGESTART($n$, $\pi$, $s$)

(slope, intercept) = fit linear regression model on $X$ = [$n$, ..., $n + r - 1$] and $Y$ = [$\pi[n]$, ..., $\pi[n + r - 1]$]

if slope < $s$ * max($\pi$) then

return false

end if

PrevYears = {$n'$ : $1 \le n' < n$}

if $|\{n' : n' \in \text{PrevYear and } \pi[n'] \le 0.1 * \max(\pi)\}| / |\text{PrevYear}| \ge 0.8$ then

return true

else

return false

end if

end function

# C Data: Gold Standard Dataset

The table C below shows the gold standard output (senses and year of emergence), as obtained by the "LREmergeTime" emergence detection algorithm based on the original gold data in (Alsulaimani et al., 2020).

The total number of targets which has emergence is 146 and which has no emergence is 42. This consists of 233 senses with emergence and 158 senses with no emergence. The table includes three types of emergence:

* N: Number of senses

* LRET: "LREmergeTime" emergence year,
<figure><img src="image_3.png" /><figcaption>Figure 3: Left: graphical representation of HDP for DWSI. Observed variables represented by shaded nodes and latent variables by clear nodes. Right: the corresponding generative process. Note that in DWSI the sense related variables replace the topic related variables.</figcaption></figure>

* Draw the base distribution over senses $G_0 \sim DP(\gamma, H)$,

* For $d \in \{1, ..., D\}$, draw the per-document distribution over senses $G_d \sim DP(\alpha, G_0)$,

* For each word $w \in \{1, ..., N_d\}$ in each document $d$,

- Draw the sense for the word $\beta_{d,n} \sim G_d$

- Draw the word $w_{d,n} \sim \text{Mult}(\beta_{d,n})$

<figure><img src="image_4.png" /></figure>

* Draw initial sense embedding $\alpha_k^{(0)} \sim \mathcal{N}(0, I)$

* Draw initial sense proportion mean $\eta_0 \sim \mathcal{N}(0, I)$

* For time step $t = 1, ..., T$:

- Draw sense embeddings $\alpha_k^{(t)} \sim \mathcal{N}(\alpha_k^{(t-1)}, \delta^2 I)$ for $k = 1, ..., K$

- Draw sense proportion means $\eta_t \sim \mathcal{N}(\eta_{t-1}, \delta^2 I)$

* For each document $d$:

- Draw sense proportions $\theta_d \sim \mathcal{LN}(\eta_{d}, a^2 I)$

- For each word $n$ in the document $d$:

* Draw sense assignment $z_{d,n} \sim \text{Cat}(\theta_d)$

* Draw word $w_{d,n} \sim \text{Cat}(\text{softmax}(p^T \alpha_{z_{d,n}}^{t_d}))$

<caption>Figure 4: Left: graphical representation of DETM for DWSI. Observed variables represented by shaded nodes and latent variables by clear nodes. Right: the corresponding generative process. Note that in DWSI the sense related variables replace the topic related variables.</caption>

* ET: "EmergeTime" emergence year,

* FYO: indicates the "First Year Occurrence" of a sense, determined by the start date of each sense in the data,

* MS: indicates the "Manual Surge", i.e. the visual manual annotations by the authors. The value "NA" indicates cases when no emergence found and "?" indicates visually ambiguous cases found during the manual annotation by the authors.

<table><thead><tr><th>Target</th><th>N</th><th>CUI</th><th>ET</th><th>LRET</th><th>FYO</th><th>MS</th></tr></thead><tbody><tr><td>AA</td><td>2</td><td>C0001972</td><td></td><td>1945</td><td>1947</td><td>1947</td></tr><tr><td>AA</td><td>2</td><td>C0002520</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>ADA</td><td>2</td><td>C0001457</td><td></td><td></td><td>1955</td><td>?</td></tr><tr><td>ADA</td><td>2</td><td>C0002456</td><td></td><td></td><td>1955</td><td>?</td></tr><tr><td>ADH</td><td>2</td><td>C0001942</td><td>1978</td><td>1977</td><td>1976</td><td>1979</td></tr><tr><td>ADH</td><td>2</td><td>C0003779</td><td></td><td></td><td>1975</td><td></td></tr><tr><td>ADP</td><td>2</td><td>C0001459</td><td></td><td></td><td>1956</td><td></td></tr><tr><td>ADP</td><td>2</td><td>C0004374</td><td>1958</td><td>1956</td><td>1959</td><td>1959</td></tr><tr><td>Adrenal</td><td>2</td><td>C0001625</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Adrenal</td><td>2</td><td>C0014563</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Ala</td><td>3</td><td>C0001898</td><td></td><td></td><td>1947</td><td></td></tr><tr><td>Ala</td><td>3</td><td>C0002563</td><td>1954</td><td>1949</td><td>1953</td><td>1973</td></tr><tr><td>Ala</td><td>3</td><td>C0051405</td><td></td><td>1982</td><td>1947</td><td>1979</td></tr><tr><td>ALS</td><td>2</td><td>C0002736</td><td></td><td></td><td>1948</td><td></td></tr><tr><td>ALS</td><td>2</td><td>C0003372</td><td></td><td>1964</td><td>1968</td><td>1968</td></tr><tr><td>ANA</td><td>2</td><td>C0002463</td><td>1962</td><td>1962</td><td>1963</td><td>1963</td></tr></tbody></table>

continued on next column or page
<table><thead><tr><th>Target</th><th>N</th><th>CUI</th><th>ET</th><th>LRET</th><th>FYO</th><th>MS</th></tr></thead><tbody><tr><td>ANA</td><td>2</td><td>C0003243</td><td></td><td></td><td>1962</td><td></td></tr><tr><td>Astragalus</td><td>2</td><td>C0039277</td><td></td><td></td><td>1947</td><td>?</td></tr><tr><td>Astragalus</td><td>2</td><td>C0330845</td><td></td><td></td><td>1947</td><td>?</td></tr><tr><td>B-Cell Leukemia</td><td>2</td><td>C0023434</td><td></td><td></td><td>1986</td><td></td></tr><tr><td>B-Cell Leukemia</td><td>2</td><td>C2004493</td><td>1986</td><td></td><td>1988</td><td>1988</td></tr><tr><td>BAT</td><td>2</td><td>C0006298</td><td></td><td></td><td>1949</td><td>1949</td></tr><tr><td>BAT</td><td>2</td><td>C0008139</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>BLM</td><td>2</td><td>C0005740</td><td></td><td></td><td>1971</td><td></td></tr><tr><td>BLM</td><td>2</td><td>C0005859</td><td>1978</td><td></td><td>1981</td><td>1981</td></tr><tr><td>Borrelia</td><td>2</td><td>C0006033</td><td></td><td></td><td>1979</td><td></td></tr><tr><td>Borrelia</td><td>2</td><td>C0024198</td><td>1980</td><td></td><td>1983</td><td>1983</td></tr><tr><td>BPD</td><td>2</td><td>C0006012</td><td></td><td></td><td>1980</td><td>?</td></tr><tr><td>BPD</td><td>2</td><td>C0006287</td><td>1980</td><td></td><td>1981</td><td>?</td></tr><tr><td>BR</td><td>2</td><td>C0006137</td><td>1946</td><td></td><td>1946</td><td>?</td></tr><tr><td>BR</td><td>2</td><td>C0006222</td><td></td><td></td><td>1946</td><td>?</td></tr><tr><td>Brucella abortus</td><td>2</td><td>C0006304</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>Brucella abortus</td><td>2</td><td>C0302363</td><td></td><td></td><td>1946</td><td>1958</td></tr><tr><td>BSA</td><td>2</td><td>C0005902</td><td></td><td></td><td>1952</td><td>?</td></tr><tr><td>BSA</td><td>2</td><td>C0036774</td><td>1952</td><td></td><td>1952</td><td>?</td></tr><tr><td>BSE</td><td>2</td><td>C0085105</td><td></td><td></td><td>1991</td><td>?</td></tr><tr><td>BSE</td><td>2</td><td>C0085209</td><td>1991</td><td></td><td>1991</td><td>?</td></tr><tr><td>Ca</td><td>3</td><td>C0006675</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Ca</td><td>3</td><td>C0006754</td><td>1945</td><td></td><td>1945</td><td>?</td></tr><tr><td>Ca</td><td>3</td><td>C0006823</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>CAD</td><td>2</td><td>C0011905</td><td></td><td></td><td>1983</td><td></td></tr><tr><td>CAD</td><td>2</td><td>C1956346</td><td>1983</td><td></td><td>1985</td><td>1985</td></tr><tr><td>Callus</td><td>2</td><td>C0006767</td><td></td><td></td><td>1972</td><td>?</td></tr><tr><td>Callus</td><td>2</td><td>C0376154</td><td></td><td></td><td>1972</td><td>?</td></tr><tr><td>CAM</td><td>2</td><td>C0007578</td><td></td><td></td><td>1981</td><td></td></tr><tr><td>CAM</td><td>2</td><td>C0178551</td><td>2002</td><td></td><td>2003</td><td>2003</td></tr><tr><td>CCD</td><td>2</td><td>C0008928</td><td></td><td></td><td>1965</td><td></td></tr><tr><td>CCD</td><td>2</td><td>C0751951</td><td>1997</td><td></td><td>1965</td><td>1998</td></tr><tr><td>CCI4</td><td>2</td><td>C0007022</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>CCI4</td><td>2</td><td>C0209338</td><td>1994</td><td></td><td>1991</td><td>1994</td></tr><tr><td>CDA</td><td>2</td><td>C0002876</td><td></td><td></td><td>1979</td><td></td></tr><tr><td>CDA</td><td>2</td><td>C0092801</td><td>1982</td><td></td><td>1983</td><td>1988</td></tr><tr><td>CDR</td><td>2</td><td>C0011485</td><td></td><td></td><td>1973</td><td></td></tr><tr><td>CDR</td><td>2</td><td>C0021024</td><td></td><td></td><td>1998</td><td>1998</td></tr><tr><td>Cell</td><td>2</td><td>C0007634</td><td></td><td></td><td>1969</td><td></td></tr><tr><td>Cell</td><td>2</td><td>C1136359</td><td>2010</td><td></td><td>1998</td><td>1999</td></tr><tr><td>Cement</td><td>2</td><td>C0011343</td><td></td><td></td><td>1957</td><td>?</td></tr><tr><td>Cement</td><td>2</td><td>C1706094</td><td></td><td></td><td>1957</td><td>?</td></tr><tr><td>CH</td><td>2</td><td>C0008115</td><td></td><td></td><td>1946</td><td>?</td></tr><tr><td>CH</td><td>2</td><td>C0039021</td><td>1946</td><td></td><td>1946</td><td>?</td></tr><tr><td>Cholera</td><td>2</td><td>C0008354</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Cholera</td><td>2</td><td>C0008359</td><td>1945</td><td></td><td>1946</td><td>1961</td></tr><tr><td>CI</td><td>2</td><td>C0008107</td><td></td><td></td><td>1949</td><td></td></tr><tr><td>CI</td><td>2</td><td>C0022326</td><td></td><td></td><td>1951</td><td>1955</td></tr><tr><td>Cilia</td><td>2</td><td>C0008778</td><td>1950</td><td></td><td>1950</td><td>?</td></tr><tr><td>Cilia</td><td>2</td><td>C0015422</td><td></td><td></td><td>1950</td><td>?</td></tr><tr><td>CIS</td><td>2</td><td>C0007099</td><td></td><td></td><td>1972</td><td></td></tr><tr><td>CIS</td><td>2</td><td>C0162854</td><td>1991</td><td></td><td>1992</td><td>1992</td></tr><tr><td>CLS</td><td>2</td><td>C0265252</td><td></td><td></td><td>2002</td><td>2002</td></tr><tr><td>CLS</td><td>2</td><td>C0343084</td><td></td><td></td><td>1996</td><td></td></tr><tr><td>Coffee</td><td>2</td><td>C0009237</td><td></td><td></td><td>1960</td><td></td></tr><tr><td>Coffee</td><td>2</td><td>C0085952</td><td>2001</td><td></td><td>1962</td><td>2002</td></tr><tr><td>Cold</td><td>3</td><td>C0009264</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Cold</td><td>3</td><td>C0009443</td><td>1945</td><td></td><td>1945</td><td></td></tr><tr><td>Cold</td><td>3</td><td>C0024117</td><td>1998</td><td></td><td>1959</td><td>1998</td></tr><tr><td>Compliance</td><td>2</td><td>C0009563</td><td></td><td></td><td>1974</td><td>?</td></tr><tr><td>Compliance</td><td>2</td><td>C1321605</td><td>1974</td><td></td><td>1974</td><td>?</td></tr><tr><td>Cortex</td><td>2</td><td>C0001614</td><td>1948</td><td></td><td>1950</td><td>?</td></tr><tr><td>Cortex</td><td>2</td><td>C0007776</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Cortical</td><td>3</td><td>C0001613</td><td>1945</td><td></td><td>1945</td><td></td></tr><tr><td>Cortical</td><td>3</td><td>C0007776</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Cortical</td><td>3</td><td>C0022655</td><td></td><td></td><td>1947</td><td>1971</td></tr><tr><td>CP</td><td>3</td><td>C0007789</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>CP</td><td>3</td><td>C0008925</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>CP</td><td>3</td><td>C0033477</td><td>1971</td><td></td><td>1969</td><td>1971</td></tr><tr><td>CPDD</td><td>2</td><td>C0008838</td><td>1971</td><td></td><td>1972</td><td>1972</td></tr><tr><td>CPDD</td><td>2</td><td>C0553730</td><td></td><td></td><td>1971</td><td></td></tr><tr><td>Crack</td><td>2</td><td>C0040441</td><td></td><td></td><td>1986</td><td></td></tr><tr><td>Crack</td><td>2</td><td>C0085163</td><td>1987</td><td></td><td>1990</td><td>1990</td></tr><tr><td>CRF</td><td>2</td><td>C0010132</td><td>1954</td><td></td><td>1956</td><td>1967</td></tr><tr><td>CRF</td><td>2</td><td>C0022661</td><td></td><td></td><td>1954</td><td></td></tr><tr><td>cRNA</td><td>2</td><td>C0056208</td><td>1981</td><td></td><td>1978</td><td>1982</td></tr></tbody></table>

continued on next column or page

<table><thead><tr><th>Target</th><th>N</th><th>CUI</th><th>ET</th><th>LRET</th><th>FYO</th><th>MS</th></tr></thead><tbody><tr><td>cRNA</td><td>2</td><td>C1321571</td><td></td><td></td><td>1975</td><td></td></tr><tr><td>CTX</td><td>2</td><td>C0010583</td><td></td><td></td><td>1960</td><td></td></tr><tr><td>CTX</td><td>2</td><td>C0238052</td><td>1997</td><td>1992</td><td>1974</td><td>1996</td></tr><tr><td>DAT</td><td>2</td><td>C0002395</td><td></td><td></td><td>1974</td><td></td></tr><tr><td>DAT</td><td>2</td><td>C0114838</td><td>1989</td><td>1988</td><td>1989</td><td>1991</td></tr><tr><td>DBA</td><td>2</td><td>C0025923</td><td></td><td></td><td>1972</td><td></td></tr><tr><td>DBA</td><td>2</td><td>C1260899</td><td>1999</td><td>1998</td><td>2001</td><td>2001</td></tr><tr><td>dC</td><td>2</td><td>C0011485</td><td>1971</td><td>1969</td><td>1973</td><td>1973</td></tr><tr><td>dC</td><td>2</td><td>C0012764</td><td></td><td></td><td>1966</td><td></td></tr><tr><td>DDD</td><td>2</td><td>C0011037</td><td></td><td></td><td>1962</td><td></td></tr><tr><td>DDD</td><td>2</td><td>C0026256</td><td></td><td></td><td>1963</td><td>1973</td></tr><tr><td>DDS</td><td>3</td><td>C0010980</td><td></td><td></td><td>1960</td><td></td></tr><tr><td>DDS</td><td>3</td><td>C0085104</td><td>1988</td><td>1987</td><td>1990</td><td>1990</td></tr><tr><td>DDS</td><td>3</td><td>C0950121</td><td>1999</td><td>1998</td><td>2001</td><td>2001</td></tr><tr><td>DE</td><td>2</td><td>C0011198</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>DE</td><td>2</td><td>C0017480</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>DI</td><td>2</td><td>C0011848</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>DI</td><td>2</td><td>C0032246</td><td></td><td></td><td>1946</td><td>1976</td></tr><tr><td>Digestive</td><td>2</td><td>C0012238</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Digestive</td><td>2</td><td>C0012240</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>DON</td><td>2</td><td>C0012020</td><td></td><td></td><td>1975</td><td></td></tr><tr><td>DON</td><td>2</td><td>C0028652</td><td>1979</td><td>1978</td><td>1981</td><td>1981</td></tr><tr><td>drinking</td><td>2</td><td>C0001948</td><td></td><td></td><td>1946</td><td>?</td></tr><tr><td>drinking</td><td>2</td><td>C0684271</td><td>1946</td><td></td><td>1946</td><td>?</td></tr><tr><td>eCG</td><td>2</td><td>C0018064</td><td>1989</td><td>1989</td><td>1945</td><td>?</td></tr><tr><td>eCG</td><td>2</td><td>C1623258</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Eels</td><td>2</td><td>C0013671</td><td></td><td></td><td>1951</td><td></td></tr><tr><td>Eels</td><td>2</td><td>C0677644</td><td>2003</td><td>2000</td><td>2004</td><td>2004</td></tr><tr><td>EGG</td><td>2</td><td>C0013710</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>EGG</td><td>2</td><td>C0029974</td><td></td><td></td><td>1945</td><td>1945</td></tr><tr><td>EM</td><td>2</td><td>C0014921</td><td>1973</td><td>1972</td><td>1975</td><td>1975</td></tr><tr><td>EM</td><td>2</td><td>C0026019</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>EMS</td><td>2</td><td>C0013961</td><td></td><td></td><td>1967</td><td></td></tr><tr><td>EMS</td><td>2</td><td>C0015063</td><td>1974</td><td>1971</td><td>1975</td><td>1975</td></tr><tr><td>Epi</td><td>2</td><td>C0014563</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Epi</td><td>2</td><td>C0014582</td><td></td><td></td><td>1980</td><td>1980</td></tr><tr><td>ERP</td><td>2</td><td>C0008310</td><td>1978</td><td>1977</td><td>1978</td><td>1978</td></tr><tr><td>ERP</td><td>2</td><td>C0015214</td><td></td><td></td><td>1956</td><td></td></tr><tr><td>ERUPTION</td><td>2</td><td>C0015230</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>ERUPTION</td><td>2</td><td>C1533692</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Erythrocytes</td><td>2</td><td>C0014772</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Erythrocytes</td><td>2</td><td>C0014792</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Exercises</td><td>2</td><td>C0015259</td><td>1945</td><td></td><td>1945</td><td>?</td></tr><tr><td>Exercises</td><td>2</td><td>C0452240</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>FA</td><td>2</td><td>C0015625</td><td></td><td></td><td>1946</td><td>1975</td></tr><tr><td>FA</td><td>2</td><td>C0016410</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Fe</td><td>2</td><td>C0302583</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Fe</td><td>2</td><td>C0376520</td><td>1995</td><td>1992</td><td>1946</td><td>1996</td></tr><tr><td>Fish</td><td>2</td><td>C0016163</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Fish</td><td>2</td><td>C0162789</td><td>1990</td><td>1988</td><td>1953</td><td>1992</td></tr><tr><td>Follicle</td><td>2</td><td>C0018120</td><td></td><td></td><td>1949</td><td>?</td></tr><tr><td>Follicle</td><td>2</td><td>C0221971</td><td></td><td></td><td>1949</td><td>?</td></tr><tr><td>Follicles</td><td>2</td><td>C0018120</td><td></td><td></td><td>1949</td><td>?</td></tr><tr><td>Follicles</td><td>2</td><td>C0221971</td><td></td><td></td><td>1949</td><td>?</td></tr><tr><td>FTC</td><td>2</td><td>C0041713</td><td></td><td></td><td>1982</td><td></td></tr><tr><td>FTC</td><td>2</td><td>C0206682</td><td>1992</td><td>1989</td><td>1993</td><td>1993</td></tr><tr><td>GAG</td><td>2</td><td>C0017346</td><td>1988</td><td>1986</td><td>1982</td><td>1989</td></tr><tr><td>GAG</td><td>2</td><td>C0017973</td><td></td><td></td><td>1949</td><td></td></tr><tr><td>Ganglion</td><td>2</td><td>C0017067</td><td>1946</td><td></td><td>1946</td><td></td></tr><tr><td>Ganglion</td><td>2</td><td>C1258666</td><td>2006</td><td>1946</td><td>1946</td><td>2002</td></tr><tr><td>Gas</td><td>2</td><td>C0016204</td><td>1945</td><td></td><td>1945</td><td>?</td></tr><tr><td>Gas</td><td>2</td><td>C0017110</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Glycoside</td><td>2</td><td>C0007158</td><td></td><td></td><td>1946</td><td>?</td></tr><tr><td>Glycoside</td><td>2</td><td>C0017977</td><td>1946</td><td>1946</td><td>1946</td><td>?</td></tr><tr><td>Haemophilus ducreyi</td><td>2</td><td>C0007947</td><td></td><td></td><td>1977</td><td></td></tr><tr><td>Haemophilus ducreyi</td><td>2</td><td>C0018481</td><td></td><td></td><td>1978</td><td>1978</td></tr><tr><td>HCl</td><td>2</td><td>C0020259</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>HCl</td><td>2</td><td>C0023443</td><td>1975</td><td>1959</td><td>1954</td><td>1976</td></tr><tr><td>Hemlock</td><td>2</td><td>C0242872</td><td>2004</td><td>2002</td><td>2002</td><td>?</td></tr><tr><td>Hemlock</td><td>2</td><td>C0949851</td><td></td><td></td><td>2002</td><td>?</td></tr><tr><td>Heregulin</td><td>2</td><td>C0626201</td><td></td><td></td><td>1994</td><td>1994</td></tr><tr><td>Heregulin</td><td>2</td><td>C0752253</td><td></td><td></td><td>1992</td><td></td></tr><tr><td>HGF</td><td>2</td><td>C0021760</td><td></td><td></td><td>1984</td><td>?</td></tr><tr><td>HGF</td><td>2</td><td>C0062534</td><td></td><td></td><td>1984</td><td>?</td></tr><tr><td>Hip</td><td>2</td><td>C0019552</td><td></td><td></td><td>1946</td><td>?</td></tr><tr><td>Hip</td><td>2</td><td>C0022122</td><td></td><td></td><td>1947</td><td>?</td></tr></tbody></table>

continued on next column or page
<table><thead><tr><th>Target</th><th>N</th><th>CUI</th><th>ET</th><th>LRET</th><th>FYO</th><th>MS</th></tr></thead><tbody><tr><td>HIV</td><td>2</td><td>C0019682</td><td></td><td></td><td>1985</td><td></td></tr><tr><td>HIV</td><td>2</td><td>C0019693</td><td>1987</td><td>1985</td><td>1987</td><td>1987</td></tr><tr><td>HPS</td><td>2</td><td>C0079504</td><td></td><td>1996</td><td>2000</td><td>2000</td></tr><tr><td>HPS</td><td>2</td><td>C0242994</td><td></td><td></td><td>1994</td><td></td></tr><tr><td>HR</td><td>2</td><td>C0010343</td><td>1947</td><td>1950</td><td>1992</td><td></td></tr><tr><td>HR</td><td>2</td><td>C0018810</td><td></td><td></td><td>1947</td><td></td></tr><tr><td>IA</td><td>2</td><td>C0021487</td><td>1946</td><td>1946</td><td>1946</td><td>1946</td></tr><tr><td>IA</td><td>2</td><td>C0022037</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>Ice</td><td>3</td><td>C0020746</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>Ice</td><td>3</td><td>C0025611</td><td>1946</td><td>1946</td><td>1946</td><td>1946</td></tr><tr><td>Ice</td><td>3</td><td>C0534519</td><td>1990</td><td>1990</td><td>1991</td><td>1991</td></tr><tr><td>INDO</td><td>2</td><td>C0021246</td><td>1961</td><td>1959</td><td>1963</td><td>1963</td></tr><tr><td>INDO</td><td>2</td><td>C0021247</td><td></td><td></td><td>1949</td><td></td></tr><tr><td>Ion</td><td>2</td><td>C0022023</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Ion</td><td>2</td><td>C0022024</td><td>1945</td><td>1945</td><td>1946</td><td>1946</td></tr><tr><td>IP</td><td>2</td><td>C0021069</td><td>2000</td><td>1997</td><td>1989</td><td>2001</td></tr><tr><td>IP</td><td>2</td><td>C0021171</td><td></td><td></td><td>1986</td><td></td></tr><tr><td>Iris</td><td>2</td><td>C0022077</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Iris</td><td>2</td><td>C1001362</td><td></td><td>1945</td><td>1946</td><td>2001</td></tr><tr><td>JP</td><td>2</td><td>C0022341</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>JP</td><td>2</td><td>C0031106</td><td>1946</td><td>1946</td><td>1947</td><td>1983</td></tr><tr><td>LABOR</td><td>2</td><td>C0022864</td><td>1945</td><td>1945</td><td>1945</td><td>1945</td></tr><tr><td>LABOR</td><td>2</td><td>C0043227</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Lactation</td><td>2</td><td>C0006147</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Lactation</td><td>2</td><td>C0022925</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Language</td><td>2</td><td>C0023008</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>Language</td><td>2</td><td>C0033348</td><td>1986</td><td>1954</td><td>1958</td><td>1985</td></tr><tr><td>Laryngeal</td><td>2</td><td>C0023078</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Laryngeal</td><td>2</td><td>C0023081</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Lawsonia</td><td>2</td><td>C0752045</td><td></td><td></td><td>2000</td><td></td></tr><tr><td>Lawsonia</td><td>2</td><td>C1068388</td><td></td><td></td><td>2002</td><td>2002</td></tr><tr><td>Leishmaniasis</td><td>2</td><td>C0023281</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Leishmaniasis</td><td>2</td><td>C1548483</td><td>2005</td><td>1996</td><td>1947</td><td>2000</td></tr><tr><td>lens</td><td>3</td><td>C0023308</td><td>1951</td><td>1948</td><td>1952</td><td>1978</td></tr><tr><td>lens</td><td>3</td><td>C0023317</td><td></td><td>1945</td><td>1945</td><td></td></tr><tr><td>lens</td><td>3</td><td>C0023318</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Lupus</td><td>3</td><td>C0024131</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Lupus</td><td>3</td><td>C0024138</td><td>1945</td><td>1945</td><td>1946</td><td>1946</td></tr><tr><td>Lupus</td><td>3</td><td>C0024141</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>lymphogranulomatosis</td><td>2</td><td>C0019829</td><td></td><td></td><td>1945</td><td>1945</td></tr><tr><td>lymphogranulomatosis</td><td>2</td><td>C0036202</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>MAF</td><td>2</td><td>C0079786</td><td></td><td></td><td>1980</td><td></td></tr><tr><td>MAF</td><td>2</td><td>C0919482</td><td>2001</td><td>1994</td><td>1998</td><td>1998</td></tr><tr><td>Malaria</td><td>2</td><td>C0024530</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Malaria</td><td>2</td><td>C0206255</td><td>1991</td><td>1988</td><td>1945</td><td>1992</td></tr><tr><td>MBP</td><td>2</td><td>C0014063</td><td></td><td></td><td>1973</td><td></td></tr><tr><td>MBP</td><td>2</td><td>C0065661</td><td>1999</td><td>1998</td><td>1984</td><td>2001</td></tr><tr><td>MCC</td><td>2</td><td>C0007129</td><td></td><td></td><td>1988</td><td></td></tr><tr><td>MCC</td><td>2</td><td>C0162804</td><td>1990</td><td>1989</td><td>1991</td><td>1993</td></tr><tr><td>Medullary</td><td>2</td><td>C0001629</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>Medullary</td><td>2</td><td>C0025148</td><td></td><td></td><td>1947</td><td>1947</td></tr><tr><td>MHC</td><td>2</td><td>C0024518</td><td></td><td></td><td>1978</td><td></td></tr><tr><td>MHC</td><td>2</td><td>C0027100</td><td>1991</td><td>1986</td><td>1994</td><td></td></tr><tr><td>Milk</td><td>2</td><td>C0026131</td><td>1945</td><td>1945</td><td>?</td><td></td></tr><tr><td>Milk</td><td>2</td><td>C0026140</td><td></td><td>1945</td><td>?</td><td></td></tr><tr><td>Moles</td><td>2</td><td>C0027960</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>Moles</td><td>2</td><td>C0324740</td><td>1946</td><td>1946</td><td>1974</td><td>1974</td></tr><tr><td>MRS</td><td>2</td><td>C0024487</td><td>1959</td><td>1961</td><td>1961</td><td></td></tr><tr><td>MRS</td><td>2</td><td>C0025235</td><td></td><td></td><td>1950</td><td></td></tr><tr><td>NBS</td><td>2</td><td>C0027819</td><td></td><td></td><td>1947</td><td></td></tr><tr><td>NBS</td><td>2</td><td>C0398791</td><td>2003</td><td>2002</td><td>2002</td><td>2006</td></tr><tr><td>NEUROFIBROMA...</td><td>2</td><td>C0085113</td><td></td><td></td><td>1990</td><td></td></tr><tr><td>NEUROFIBROMA...</td><td>2</td><td>C0162678</td><td>1990</td><td>1990</td><td>1991</td><td>1991</td></tr><tr><td>NM</td><td>2</td><td>C0025033</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>NM</td><td>2</td><td>C0027972</td><td>1963</td><td>1962</td><td>1946</td><td>1946</td></tr><tr><td>NPC</td><td>2</td><td>C0028587</td><td></td><td></td><td>1998</td><td></td></tr><tr><td>NPC</td><td>2</td><td>C0220756</td><td>2005</td><td>2002</td><td>2006</td><td>2006</td></tr><tr><td>Nurse</td><td>2</td><td>C0006147</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Nurse</td><td>2</td><td>C0028661</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Nursing</td><td>2</td><td>C0006147</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Nursing</td><td>2</td><td>C0028677</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>OCD</td><td>2</td><td>C0028768</td><td></td><td></td><td>1975</td><td></td></tr><tr><td>OCD</td><td>2</td><td>C0029421</td><td>1983</td><td>1980</td><td>1984</td><td>1984</td></tr><tr><td>OH</td><td>2</td><td>C0028905</td><td></td><td></td><td>1946</td><td>?</td></tr><tr><td>OH</td><td>2</td><td>C0063146</td><td></td><td></td><td>1946</td><td>?</td></tr><tr><td>Orf</td><td>2</td><td>C0013570</td><td></td><td></td><td>1980</td><td></td></tr></tbody></table>

continued on next column or page

<table><thead><tr><th>Target</th><th>N</th><th>CUI</th><th>ET</th><th>LRET</th><th>FYO</th><th>MS</th></tr></thead><tbody><tr><td>Orf</td><td>2</td><td>C0079941</td><td>1986</td><td>1985</td><td>1982</td><td>1982</td></tr><tr><td>ORI</td><td>2</td><td>C0206601</td><td></td><td></td><td>1993</td><td></td></tr><tr><td>ORI</td><td>2</td><td>C0242961</td><td>1993</td><td>1993</td><td>1993</td><td>1993</td></tr><tr><td>PAC</td><td>2</td><td>C0033036</td><td></td><td></td><td>1995</td><td></td></tr><tr><td>PAC</td><td>2</td><td>C0949780</td><td></td><td></td><td>1997</td><td>2001</td><td>2001</td></tr><tr><td>PAF</td><td>2</td><td>C0032172</td><td></td><td></td><td>1979</td><td></td></tr><tr><td>PAF</td><td>2</td><td>C0037019</td><td></td><td></td><td>1980</td><td>1980</td><td>1980</td></tr><tr><td>Parotitis</td><td>2</td><td>C0026780</td><td>1945</td><td>1945</td><td>?</td><td>?</td></tr><tr><td>Parotitis</td><td>2</td><td>C0030583</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>PCA</td><td>5</td><td>C0030131</td><td>1972</td><td>1971</td><td>1974</td><td>1974</td></tr><tr><td>PCA</td><td>5</td><td>C0030625</td><td></td><td></td><td>1957</td><td></td></tr><tr><td>PCA</td><td>5</td><td>C0078944</td><td>1987</td><td>1986</td><td>1989</td><td>1989</td></tr><tr><td>PCA</td><td>5</td><td>C0149576</td><td>1957</td><td>1957</td><td>1957</td><td>1957</td></tr><tr><td>PCA</td><td>5</td><td>C0429865</td><td>1999</td><td>1998</td><td>1960</td><td>2001</td></tr><tr><td>PCB</td><td>2</td><td>C0032447</td><td></td><td></td><td>1971</td><td>?</td></tr><tr><td>PCB</td><td>2</td><td>C0033223</td><td></td><td></td><td>1971</td><td>?</td></tr><tr><td>PCD</td><td>2</td><td>C0022521</td><td></td><td></td><td>1971</td><td></td></tr><tr><td>PCD</td><td>2</td><td>C0162638</td><td></td><td></td><td>1988</td><td>1991</td><td>1991</td></tr><tr><td>PCP</td><td>2</td><td>C0030855</td><td></td><td></td><td>1972</td><td>?</td></tr><tr><td>PCP</td><td>2</td><td>C0031381</td><td></td><td></td><td>1972</td><td>?</td></tr><tr><td>PEP</td><td>2</td><td>C0031642</td><td></td><td></td><td>1971</td><td></td></tr><tr><td>PEP</td><td>2</td><td>C0135981</td><td>1978</td><td>1976</td><td>1980</td><td>1980</td></tr><tr><td>PHA</td><td>2</td><td>C0030779</td><td>2002</td><td>2007</td><td>1976</td><td>1976</td></tr><tr><td>PHA</td><td>2</td><td>C0031858</td><td></td><td></td><td>1975</td><td>1975</td></tr><tr><td>Pharmaceutical</td><td>2</td><td>C0013058</td><td>1963</td><td>1962</td><td>1963</td><td>1963</td></tr><tr><td>Pharmaceutical</td><td>2</td><td>C0031336</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Phosphorus</td><td>2</td><td>C0031705</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Phosphorus</td><td>2</td><td>C0080014</td><td>1945</td><td>1945</td><td>?</td><td>?</td></tr><tr><td>Phosphorylase</td><td>2</td><td>C0017916</td><td></td><td></td><td>1971</td><td></td></tr><tr><td>Phosphorylase</td><td>2</td><td>C0917783</td><td>2005</td><td>1998</td><td>1973</td><td>2001</td></tr><tr><td>pI</td><td>2</td><td>C0022171</td><td></td><td></td><td>1975</td><td>?</td></tr><tr><td>pI</td><td>2</td><td>C0812425</td><td></td><td></td><td>1975</td><td>?</td></tr><tr><td>Plague</td><td>2</td><td>C0032064</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Plague</td><td>2</td><td>C0032066</td><td>1959</td><td>1957</td><td>1946</td><td>1960</td></tr><tr><td>Plague</td><td>2</td><td>C0011389</td><td></td><td></td><td>1950</td><td>?</td></tr><tr><td>Plague</td><td>2</td><td>C0333463</td><td></td><td></td><td>1950</td><td>?</td></tr><tr><td>Platelet</td><td>2</td><td>C0005821</td><td>1945</td><td>1945</td><td>?</td><td>?</td></tr><tr><td>Platelet</td><td>2</td><td>C0032181</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Pleuropneumonia</td><td>2</td><td>C0026934</td><td>1945</td><td>1945</td><td>?</td><td>?</td></tr><tr><td>Pleuropneumonia</td><td>2</td><td>C0032241</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>POL</td><td>2</td><td>C0017360</td><td></td><td></td><td>1986</td><td>1989</td><td>1989</td></tr><tr><td>POL</td><td>2</td><td>C0032356</td><td></td><td></td><td>1946</td><td>1947</td><td>1946</td></tr><tr><td>posterior pituitary</td><td>2</td><td>C0032009</td><td></td><td></td><td>1946</td><td>1945</td><td>1945</td></tr><tr><td>posterior pituitary</td><td>2</td><td>C0032017</td><td></td><td></td><td>1946</td><td>1945</td><td>1945</td></tr><tr><td>Potassium</td><td>2</td><td>C0032821</td><td></td><td></td><td>1989</td><td>1948</td><td>1992</td></tr><tr><td>PR</td><td>2</td><td>C0162800</td><td>1990</td><td>1989</td><td>1948</td><td>1945</td></tr><tr><td>PR</td><td>2</td><td>C0034833</td><td>1972</td><td>1972</td><td>1973</td><td>1973</td></tr><tr><td>Projection</td><td>2</td><td>C0016538</td><td>1970</td><td>1970</td><td>?</td><td>?</td></tr><tr><td>Projection</td><td>2</td><td>C0033363</td><td></td><td></td><td>1970</td><td>?</td></tr><tr><td>PVC</td><td>2</td><td>C0032624</td><td></td><td></td><td>1974</td><td></td></tr><tr><td>PVC</td><td>2</td><td>C0151636</td><td></td><td></td><td>1988</td><td>1992</td></tr><tr><td>RA</td><td>3</td><td>C0002893</td><td>1945</td><td>1946</td><td>?</td><td>?</td></tr><tr><td>RA</td><td>3</td><td>C0003873</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>RA</td><td>3</td><td>C0034625</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Radiation</td><td>2</td><td>C0851346</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Radiation</td><td>2</td><td>C1522449</td><td></td><td></td><td>1946</td><td>1946</td></tr><tr><td>RB</td><td>2</td><td>C0035335</td><td></td><td></td><td>1947</td><td></td></tr><tr><td>RB</td><td>2</td><td>C0035930</td><td>1947</td><td>1951</td><td>1951</td><td>1951</td></tr><tr><td>RBC</td><td>2</td><td>C0014772</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>RBC</td><td>2</td><td>C0014792</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>rDNA</td><td>2</td><td>C0012931</td><td></td><td></td><td>1976</td><td></td></tr><tr><td>rDNA</td><td>2</td><td>C0012933</td><td>1980</td><td>1978</td><td>1981</td><td>1981</td></tr><tr><td>Respiration</td><td>2</td><td>C0035203</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Respiration</td><td>2</td><td>C0282636</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Retinal</td><td>2</td><td>C0035298</td><td>1945</td><td>1945</td><td>?</td><td>?</td></tr><tr><td>Retinal</td><td>2</td><td>C0035331</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Root</td><td>2</td><td>C0040452</td><td>1945</td><td>1945</td><td>?</td><td>?</td></tr><tr><td>Root</td><td>2</td><td>C0242726</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>RSV</td><td>2</td><td>C0035236</td><td>1957</td><td>1960</td><td>1960</td><td>1960</td></tr><tr><td>RSV</td><td>2</td><td>C0086943</td><td></td><td></td><td>1955</td><td></td></tr><tr><td>SARS</td><td>2</td><td>C1175175</td><td></td><td></td><td>2002</td><td></td></tr><tr><td>SARS</td><td>2</td><td>C1175743</td><td>2002</td><td>2002</td><td>2002</td><td>2002</td></tr><tr><td>SARS-assoc...</td><td>2</td><td>C1175175</td><td></td><td></td><td>2002</td><td>2002</td></tr><tr><td>SARS-assoc...</td><td>2</td><td>C1175743</td><td>2002</td><td>2002</td><td>2002</td><td>2002</td></tr><tr><td>SCD</td><td>2</td><td>C0002895</td><td></td><td></td><td>1946</td><td></td></tr></tbody></table>

continued on next column or page
<table><thead><tr><th>Target</th><th>N</th><th>CUI</th><th>ET</th><th>LRET</th><th>FYO</th><th>MS</th></tr></thead><tbody><tr><td>SCD</td><td>2</td><td>C0085298</td><td>1988</td><td>1987</td><td>1950</td><td>1989</td></tr><tr><td>Schistosoma...</td><td>2</td><td>C0036319</td><td></td><td></td><td>1971</td><td></td></tr><tr><td>Schistosoma...</td><td>2</td><td>C0036330</td><td></td><td>1981</td><td>1977</td><td>1985</td></tr><tr><td>SLS</td><td>2</td><td>C0037231</td><td></td><td>1987</td><td>1991</td><td>1991</td></tr><tr><td>SLS</td><td>2</td><td>C0037506</td><td></td><td></td><td>1971</td><td></td></tr><tr><td>Sodium</td><td>2</td><td>C0037473</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Sodium</td><td>2</td><td>C0037570</td><td>1945</td><td>1945</td><td>1945</td><td>1945</td></tr><tr><td>SPR</td><td>2</td><td>C0164209</td><td></td><td></td><td>1981</td><td></td></tr><tr><td>SPR</td><td>2</td><td>C0597731</td><td>1996</td><td>1994</td><td>1998</td><td>1998</td></tr><tr><td>SS</td><td>2</td><td>C0039101</td><td></td><td></td><td>1948</td><td></td></tr><tr><td>SS</td><td>2</td><td>C0085077</td><td>1990</td><td>1960</td><td>1964</td><td>1990</td></tr><tr><td>Staph</td><td>2</td><td>C0038160</td><td></td><td></td><td>1945</td><td>1945</td></tr><tr><td>Staph</td><td>2</td><td>C0038170</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>STEM</td><td>2</td><td>C0162731</td><td></td><td></td><td>1992</td><td></td></tr><tr><td>STEM</td><td>2</td><td>C0242767</td><td></td><td>1992</td><td>1994</td><td>1994</td></tr><tr><td>Sterilization</td><td>2</td><td>C0038280</td><td></td><td>1945</td><td>1945</td><td>?</td></tr><tr><td>Sterilization</td><td>2</td><td>C0038288</td><td></td><td>1945</td><td>1945</td><td>?</td></tr><tr><td>Strep</td><td>2</td><td>C0038395</td><td></td><td>1945</td><td>1945</td><td>1945</td></tr><tr><td>Strep</td><td>2</td><td>C0038402</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Synapsis</td><td>2</td><td>C0039062</td><td></td><td></td><td>1950</td><td></td></tr><tr><td>Synapsis</td><td>2</td><td>C0598501</td><td>1998</td><td>1950</td><td>1951</td><td>1951</td></tr><tr><td>TAT</td><td>3</td><td>C0017375</td><td>1988</td><td>1985</td><td>1989</td><td>1989</td></tr><tr><td>TAT</td><td>3</td><td>C0039341</td><td>1983</td><td>1982</td><td>1985</td><td>1985</td></tr><tr><td>TAT</td><td>3</td><td>C0039756</td><td></td><td></td><td>1975</td><td></td></tr><tr><td>Tax</td><td>2</td><td>C0039371</td><td></td><td></td><td>1975</td><td></td></tr><tr><td>Tax</td><td>2</td><td>C0144576</td><td>1992</td><td>1989</td><td>1983</td><td>1993</td></tr><tr><td>TEM</td><td>2</td><td>C0040975</td><td></td><td></td><td>2004</td><td></td></tr><tr><td>TEM</td><td>2</td><td>C0678118</td><td></td><td></td><td>2002</td><td></td></tr><tr><td>THYMUS</td><td>3</td><td>C0040112</td><td>1948</td><td>1946</td><td>1949</td><td>1949</td></tr><tr><td>THYMUS</td><td>3</td><td>C0040113</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>THYMUS</td><td>3</td><td>C1015036</td><td></td><td>1946</td><td>1946</td><td></td></tr><tr><td>TLC</td><td>2</td><td>C0008569</td><td></td><td>1959</td><td>1959</td><td>?</td></tr><tr><td>TLC</td><td>2</td><td>C0040509</td><td>1974</td><td>1972</td><td>1959</td><td>?</td></tr><tr><td>TMJ</td><td>2</td><td>C0039493</td><td></td><td></td><td>1946</td><td>?</td></tr><tr><td>TMJ</td><td>2</td><td>C0039496</td><td></td><td></td><td>1946</td><td>?</td></tr><tr><td>TMP</td><td>2</td><td>C0040079</td><td></td><td>1972</td><td>1975</td><td>1975</td></tr><tr><td>TMP</td><td>2</td><td>C0041041</td><td></td><td></td><td>1970</td><td></td></tr><tr><td>TNC</td><td>2</td><td>C0076088</td><td>1983</td><td>1982</td><td>1985</td><td>1985</td></tr><tr><td>TNC</td><td>2</td><td>C0077400</td><td></td><td></td><td>1980</td><td></td></tr><tr><td>TNT</td><td>2</td><td>C0041070</td><td></td><td></td><td>1982</td><td>1982</td></tr><tr><td>TNT</td><td>2</td><td>C0077404</td><td></td><td></td><td>1981</td><td></td></tr><tr><td>Tolerance</td><td>2</td><td>C0013220</td><td></td><td></td><td>1946</td><td>?</td></tr><tr><td>Tolerance</td><td>2</td><td>C0020963</td><td></td><td>1946</td><td>1946</td><td>?</td></tr><tr><td>tomography</td><td>2</td><td>C0040395</td><td></td><td></td><td>1947</td><td>?</td></tr><tr><td>tomography</td><td>2</td><td>C0040405</td><td></td><td></td><td>1947</td><td>?</td></tr><tr><td>Torula</td><td>2</td><td>C0010414</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Torula</td><td>2</td><td>C0010415</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>TPA</td><td>2</td><td>C0032143</td><td>1983</td><td>1982</td><td>1982</td><td>1985</td></tr><tr><td>TPA</td><td>2</td><td>C0039654</td><td></td><td></td><td>1975</td><td></td></tr><tr><td>TPO</td><td>2</td><td>C0021965</td><td>1974</td><td>1974</td><td>1975</td><td>1975</td></tr><tr><td>TPO</td><td>2</td><td>C0040052</td><td></td><td></td><td>1974</td><td></td></tr><tr><td>TRF</td><td>2</td><td>C0021759</td><td></td><td></td><td>1980</td><td>1980</td></tr><tr><td>TRF</td><td>2</td><td>C0040162</td><td></td><td></td><td>1968</td><td></td></tr><tr><td>TSF</td><td>2</td><td>C0021756</td><td>1976</td><td>1974</td><td>1977</td><td>1977</td></tr><tr><td>TSF</td><td>2</td><td>C0040052</td><td></td><td></td><td>1974</td><td></td></tr><tr><td>TYR</td><td>2</td><td>C0041484</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>TYR</td><td>2</td><td>C0041485</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>US</td><td>2</td><td>C0041618</td><td>1971</td><td>1964</td><td>1945</td><td>1966</td></tr><tr><td>US</td><td>2</td><td>C0041703</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>Ventricles</td><td>2</td><td>C0007799</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Ventricles</td><td>2</td><td>C0018827</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>veterinary</td><td>2</td><td>C0042615</td><td></td><td></td><td>1945</td><td></td></tr><tr><td>veterinary</td><td>2</td><td>C0206212</td><td></td><td>1959</td><td>1963</td><td>1993</td></tr><tr><td>Wasp</td><td>2</td><td>C0043041</td><td></td><td></td><td>1975</td><td></td></tr><tr><td>Wasp</td><td>2</td><td>C0258432</td><td>1993</td><td>1991</td><td>1994</td><td>1994</td></tr><tr><td>WBS</td><td>2</td><td>C0004903</td><td></td><td></td><td>1982</td><td></td></tr><tr><td>WBS</td><td>2</td><td>C0175702</td><td>1994</td><td>1991</td><td>1995</td><td>1995</td></tr><tr><td>WT1</td><td>2</td><td>C0027708</td><td></td><td></td><td>1946</td><td></td></tr><tr><td>WT1</td><td>2</td><td>C0148873</td><td>1991</td><td>1989</td><td>1991</td><td>1991</td></tr><tr><td>Yellow Fever</td><td>2</td><td>C0043395</td><td></td><td></td><td>1945</td><td>?</td></tr><tr><td>Yellow Fever</td><td>2</td><td>C0301508</td><td></td><td></td><td>1945</td><td>?</td></tr></tbody></table>
**A** For every submission:

☑ A1. Did you describe the limitations of your work?
*Section 7*

☐ A2. Did you discuss any potential risks of your work?
*Not applicable. Left blank.*

☑ A3. Do the abstract and introduction summarize the paper's main claims?
*Abstract and Section 1*

☑ A4. Have you used AI writing assistants when working on this paper?
*Left blank.*

**B** ☑ Did you use or create scientific artifacts?

*Section 1 and 4 and 5*

☑ B1. Did you cite the creators of artifacts you used?
*Section 4 and 5*

☑ B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
*Section 4*

☑ B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)?
*Section 4 and 5*

☐ B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it?
*Not applicable. The data used in this research is a secondary data which was previously published. The data source files were taken from NML and is made of biomedical scientific publications.*

☑ B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.?
*Section 7*

☑ B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be.
*Section 4*

**C** ☑ Did you run computational experiments?

*Section 4 and 5*

☑ C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used?
*Section 4 and 7*

<footnote>The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.</footnote>
<figure><img src="image_5.png" /></figure>

C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values?
*Section 4*

<figure><img src="image_6.png" /></figure>

C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run?
*Section 5*

<figure><img src="image_7.png" /></figure>

C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)?
*Section 4 and 5*

D

<figure><img src="image_8.png" /></figure>

☑ Did you use human annotators (e.g., crowdworkers) or research with human participants?

*Left blank.*

<figure><img src="image_9.png" /></figure>

D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.?

*Not applicable. Left blank.*

<figure><img src="image_10.png" /></figure>

D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)?

*Not applicable. Left blank.*

<figure><img src="image_11.png" /></figure>

D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used?

*Not applicable. Left blank.*

<figure><img src="image_12.png" /></figure>

D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?

*Not applicable. Left blank.*

<figure><img src="image_13.png" /></figure>

D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?

*Not applicable. Left blank.*