Infinite SCAN: An Infinite Model of Diachronic Semantic Change
Seiichi Inoue1 , Mamoru Komachi1 , Toshinobu Ogiso2 , Hiroya Takamura3
and Daichi Mochihashi41Tokyo Metropolitan University 2 The National Institute for Japanese Language andLinguistics 3 The National Institute of Advanced Industrial Science and Technology
4
The Institute of Statistical Mathematics
inoue-seiichi@ed.tmu.ac.jp komachi@tmu.ac.jp
togiso@ninjal.ac.jp takamura.hiroya@aist.go.jp daichi@ism.ac.jp
Abstract
In this study, we propose a Bayesian modelthat can jointly estimate the number of senses
of words and their changes through time. The
model combines a dynamic topic model on
Gaussian Markov random fields (Frermann and
Lapata, 2016) with a logistic stick-breaking
process that realizes the Dirichlet process. In
the experiments, we evaluated the proposedmodel in terms of interpretability, accuracy inestimating the number of senses, and tracking their changes using both artificial data andreal data. We quantitatively verified that themodel behaves as expected through evaluation
using artificial data. Using the CCOHA corpus, we showed that our model outperforms thebaseline model and investigated the semanticchanges of several well-known target words.

1

Introduction

Words exhibit a range of senses depending on thecontext in which they are used. These senses canalso change over time (Blank and Koch, 1999;
Aitchison, 2001). For example, the word cute appeared in the early 18th century, which originallymeant clever or keen-witted. By the late 19th century it signified cunning, and today, cute means
attractive, pretty, or sweet (Stevenson, 2010; Frermann and Lapata, 2016). Automatically capturing these semantic changes is an academic contribution to the fields of lexicology and linguistics(Voyles, 1973; Williams, 1976).
In recent years, many methods have been proposed for the detection of semantic changes using distributional methods (Kutuzov et al., 2018).
They include word embedding-based methods withalignment of word embedding spaces at different
times (Kim et al., 2014; Kulkarni et al., 2015;
Hamilton et al., 2016b), without alignment (Yaoet al., 2018; Dubossarsky et al., 2019; Aida et al.,2021), and using probabilistic frameworks (Bamler and Mandt, 2017; Rudolph and Blei, 2018).

Word embedding-based methods describe semanticchange by changes of surrounding words in semantic space. However, the learned embeddings
themselves cannot account for the existence of eachsense and its relative importance. In contrast, several methods have addressed these issues using a
topic model architecture (Frermann and Lapata,
2016; Emms and Jayapal, 2016). These probabilistic models estimate the latent senses explicitly
and consider their changes, unlike word embedding models. However, these models have a critical problem in that the number of senses is given
and fixed, even though it will vary for each word,
which harms the modeling of semantic change. In
addition, the number of senses is rarely apparent
beforehand, thus it is difficult to set it a priori. A recent method of clustering contextualized word embeddings obtained from BERT can similarly track
sense changes (Giulianelli et al., 2020), but it does
not take time evolution into account, and cannot estimate the number of senses and semantic changes
jointly.
Therefore, to address these limitations, we propose a model that can automatically estimate thenumber of senses and simultaneously capture semantic changes by extending the model proposedby Frermann and Lapata (2016). To this end, we
combined a dynamic topic model on GaussianMarkov random fields (GMRF) with a logistic
stick-breaking process (Ren et al., 2011) to realize a Dirichlet process in latent Euclidean space.1
Here, our work can answer the question of how
many senses the word has in the context of modeling semantic change, which can be applicable
to lexicography. In our experiments, we verified
the performance of the proposed model in termsof the estimation accuracy of the number of senses
and sense change on artificial data. Then, we evaluated the model performance using real data and1
Source code is available at https://github.com/
seiichiinoue/iscan .

YearExampleSnippet

1853
1900
1949
2003

The driver made room for the trunk on the top of the coach.
The chair passed the coach, the horses proceeding at a walk.
Tell him if I start coaching, it’ll be as a head coach at a top school.
Football coach and other top school officials have been interviewed.

{driver, make, room, trunk}{chair, pass, horse, proceed, walk}{tell, start, coach, head, top, school}{football, top, school, official, interview}

Figure 1: Example snippets for input to SCAN for the target word coach. The snippets were obtained from thepreprocessing step described in Section 6.1.

analyzed the semantic changes of several words.
The contributions of this study can be summarized
as follows:• We combine a dynamic topic model on GMRF
with a Dirichlet process to propose a modelthat can jointly estimate the number of senses
and semantic change of words.
• We quantitatively and qualitatively show thatthe proposed model can correctly estimate thenumber of word senses and semantic changes
and outperforms baseline models.

2

Background

2.1

Dynamic Bayesian model of sense change

Frermann and Lapata (2016) proposed a dynamic
Bayesian model that captures the diachronic word
Sense ChANge (SCAN). In the SCAN framework,
one model is constructed for each target word w.
The input is a set of snippets, i.e. short documents,
consisting of context words cd = {cd,1 , . . . cd,I }with length I of a sentence containing the target
word w, and time label of the year in which eachsentence appeared. An example of snippets isshown in Figure 1.
In SCAN, the set of snippets at time t ∈{1 . . . T } is modeled by unigram mixtures at eachtime point:• K-dimensional multinomial distribution ϕt(sense distribution) over the senses.
• V -dimensional multinomial distribution ψt,k(sense–word distribution) over the words foreach word sense k.
Also, a Gaussian distribution is assumed for eachprior distribution. ϕ is obtained by transforming asampled vector as follows:
1. Draw a K-dimensional vector α from the multivariate Gaussian distribution.
2. Project this vector to a K − 1-dimensional
simplex, using P
the softmax transformation
ϕk = exp(αk )/ K
k=1 exp(αk ).
ψ can be obtained in a similar way. Note that K is
assumed to be given (a parameter to be set a priori),

Figure 2: A linear chain iGMRF.

which is a severe problem in practice. Then, they
define the first-order intrinsic Gaussian Markov
random field (iGMRF) (Rue and Held, 2005) forthe prior distribution so that the sense distribution ϕand sense–word distribution ψ change through time.
The iGMRF is a prior distribution such that thevalue at any location is similar to that of neighbors(graphically shown in Figure 2).2 For a real vectorx = (x1 , x2 , . . . , xT ) and Gaussian distribution
N (µ, σ 2 ), the iGMRF is defined as follows:1
1
xt | x−t , κ ∼ N(xt−1 + xt+1 ),
, (1)
2
κ
where x−t is a set of x except for xt and κ is a precision parameter. The Gaussian distribution, which
is a prior distribution of the sense distribution andthe sense–word distribution, has precision parameters κϕ and κψ to control the degree of change,
respectively. In particular, the precision parameterof the sense distribution, κϕ , should be estimatedfrom the data because the “speed” of sense change
varies depending on the target word w.
Based on the above definition, the generative model of SCAN is described as follows,
where Ga(a, b) denotes the gamma distribution andMult(θ) denotes the multinomial distribution.
1. Draw κϕ ∼ Ga(a, b)
2. For time interval t = 1 . . . T(a) Draw a sense distribution
i. αt | α−t , κϕ∼N

−1
1
2 (αt−1 + αt+1 ), κϕ



ii. ϕt = Softmax(αt )(b) For sense k = 1 . . . K
i. Draw a sense–word distribution
2
iGMRF can be viewed as a special case of a Gaussian
process in which the kernel is restricted to adjacent times.

A. βt,k | β
−t , κψ∼N

−1
1
2 (βt−1,k + βt+1,k ), κψ

B. ψt,k = Softmax(βt,k )(c) For snippet d = 1 . . . D
i. Draw a sense zd ∼ Mult(ϕt )ii. For context position i = 1 . . . I
A. Draw a word cd,i ∼ Mult(ψt,zd )2.2

Logistic stick-breaking process

The Dirichlet process (Ferguson, 1973; Antoniak,
1974) is an infinite-dimensional generalization ofDirichlet distribution and generates an infinitedimensional multinomial distribution. The stickbreaking process (SBP) (Sethuraman, 1994) is an
example of its realization. In the SBP representation, a probability distribution G that follows the
Dirichlet process DP(α, G0 ) is generated as follows:k−1
Y
πk = vk(1 − vj ), vk ∼ Be(1, α) (2)
G=

j=1
∞
X

πk δ(θk ), θk ∼ G0 ,

(3)

k=1

where Be(1, α) denotes a beta distribution. First,
the probability for the k-th category πk is determined by recursively breaking a stick of lengthone, which is the sum of the probabilities. Then,the delta function δ(θk ) is set at a location θk sampled from the base measure G0 .
Ren et al. (2011) proposed the logistic stickbreaking process (LSBP), which realizes a Dirichlet process in the same way as the original SBP
by transforming a random variable with logistic
function σ(x) = 1/(1 + e−x ), where each classis associated with a certain covariate3 x ∈ R. Let
xk be a random variable that follows a Gaussian
distribution for each category; the LSBP generates
the probability distribution Gx as follows:π(xk ) = σ(xk )
Gx =

∞
X

k−1
Y

(1 − σ(xj )) ,

(4)

j=1

π(xk )δ(θk ).

(5)

k=1

3

Proposed Method

3.1

Infinite SCAN

We propose an infinite model of diachronic semantic change: Infinite SCAN that extends the archi3

In the case of SCAN, the random variables that follow a
Gaussian distribution are associated with each sense and eachword in the sense.

Figure 3: Graphical model of Infinite SCAN for threetime steps {t−1, t, t+1}. Observations are shown asgray nodes, latent variables as clear nodes, and constants
as dashed nodes. Adapted from Frermann and Lapata(2016) with the number of senses being infinite.

tecture of SCAN (introduced in Section 2.1) using
LSBP (introduced in Section 2.2) to automaticallyestimate the number of word senses for each target
word. The graphical model of Infinite SCAN isshown in Figure 3. The proposed model extends
iGMRF over the time direction in the semantic
distribution with LSBP that realizes a Dirichlet process. This makes the sense distribution practicallyinfinite-dimensional, and the number of senses, Sw ,appropriate for the target word w can be automatically estimated from the corpus. Here, we note that
our idea is similar to the model linking Gaussian
process and Dirichlet process for spatial modeling(Duan et al., 2007) and the method using Gaussian
process and Pitman-Yor process for image segmentation (Sudderth and Jordan, 2008).
In Infinite SCAN, as in SCAN, one model isconstructed for each target word w. The input isa set of snippets consisting of context words cd ={cd,1 , . . . cd,I } of a sentence containing the target
word w, and time label of the year in which eachsentence appeared. The set of snippets appearing
at time t is represented by the sense distribution ϕtand sense–word distributions ψt,k , each following
a first order iGMRF:1−1
αt | α−t , κϕ ∼ N(αt−1 + αt+1 ), κϕ . (6)2Here, we modify the generative process of the sense
distribution ϕt using LSBP as follows, so that the
number of senses Sw , which depends on the target
word w, can be automatically estimated from thecorpus:k−1
Yϕt,k = σ(αt,k )(1 − σ(αt,j ))(7)
j=1(k = 1, . . . , K),3.2

Figure 4: LSBP transformation of random variables
following a Gaussian distribution in a sense distribution.

where K is the maximum number of senses considered. Here, LSBP can generate the infinitedimensional multinomial distribution. However, inpractice, if the dimension of the word sense is sufficient to represent the data, using high-dimensional
distributions is not necessary. Thus, in this study,
we set the maximum number of senses K = 8 byreferring to the previous study (Frermann and Lapata, 2016).4 Figure 4 illustrates the LSBP transformation of the sense distribution. The horizontal axis denotes time, and the vertical axis denotes the scale of the random variable following
a Gaussian distribution. The probability of eachword sense {ϕt,1 , . . . , ϕt,K } is obtained by LSBP
transformation over the set of random variables{αt,1 , . . . , αt,K }.
In SCAN, the precision parameter κϕ of thesense distribution is shared across all senses. Thisis because the sense distribution ϕt at time t is constructed by a softmax transformation, which normalizes the distribution by considering all of thesenses, so that the variance of all senses is withina certain scale. By contrast, in Infinite SCAN, thesense distribution ϕt at time t is constructed by
LSBP transformation. In the LSBP transformation,
sense k is transformed by a sigmoid function independently of the other senses, such that the scaleof the Gaussian random variable corresponding toeach sense will differ. Therefore, in the proposedmodel, κϕ should not be shared across all sensesk ∈ {1 . . . K} unlike SCAN. Instead, we assume(k)and estimate a different precision κϕ for each
Gaussian random variable corresponding to sense
αk .
4Preliminary experiments indicated that there are very few
words that have more than eight senses.

Markov Chain Monte Carlo (MCMC)
estimation

To estimate the parameters of Infinite SCAN, weused a blocked Gibbs sampler. The parameters tobe estimated in Infinite SCAN are (a) the sense assignment z for each snippet, (b) the parameters defined by iGMRF for the semantic distribution α (unnormalized ϕ), (c) sense–word distributions β (unnormalized ψ), and (d) the precision parameter κϕof the sense distribution following the gamma distribution. In the model estimation, each parameteris sampled from its posterior distribution given theother parameters. The pseudo-code of the MCMC
algorithm is shown in Appendix A. Each parameterbasically follows Frermann and Lapata (2016), but
we changed some parameters; see Section 4 fordetails.
Sense assignments of snippet The sense assignments of the d-th snippet, zd , are sampled from thefollowing posterior distribution under the current
model parameters ϕ and ψ:p(zd | w, t, ϕ, ψ) ∝ p(zd | t)p(w | t, zd )Y(t,zd )= ϕ(t)
ψw
.
(8)
zd
w∈w

Sense distribution Because the sense distribution follows a Gaussian distribution, it is not conjugate to the multinomial distribution. Thus, straightforward parameter sampling, such as Dirichletmultinomial, does not apply. Linderman et al.
(2015) proposed a Gibbs sampling for parametersof a multinomial distribution, modeled with a Gaussian prior and the LSBP transformation, by usinga Pólya-gamma auxiliary variable (Polson et al.,2013). This approach is used in this study. The
posterior distribution of α is computed as follows:p(αt | z, α−t , ω) 
∝ N ω −1 f (c) | αt N αt | α−t , κ−1ϕ∝ N αt | µ̃, κ̃−1.
(9)ϕHere, ω is an auxiliary variable that is sampled from Pólya-gamma distribution ω | z, αt ∼PG(N (ck ), αt ), where ck denotes the number ofsnippetbelongingto k-th sense and N (ck ) =
P
P
c
−
c
.
Also, the mean and precik kj<k jsion of the posterior distribution are computed asµ̃ = (f (ck ) + µk κϕ )/κ̃ϕ and κ̃ϕ = ωk + κϕ , wheref (ck ) = ck − N (ck )/2.

Sense–word distribution The sense–word distribution, as the sense distribution, follows Gaussian distribution; thus, cannot be applied to such
Dirichlet-multinomials. Mimno et al. (2008) proposed a Gibbs sampling for parameters of multinomial distribution modeled with a Gaussian priorand softmax transformation; we estimate the parameters using this approach. Let the number ofsnippets be D and the snippet length be Nd ; theposterior distribution of β is as follows:p(βt | z, β−t , κ−1
ψ )∝
D
Y
d=1

Nd
Y
n=1

(10)
!

(t,z )

exp(βwn d )PV(t,zd )

v=1 exp(βv)

N (βt | β−t , κ−1ψ ).

Precision parameter The precision parameter of
the Gaussian distribution given its mean follows
a gamma posterior distribution. With the shapeparameter of the gamma distribution as a and thescale parameter as b, the posterior distribution ofκϕ is as follows:(k)p(κϕ | αk , a, b)!
T
T
1X= Ga a + , b +(αt,k − ᾱk ) (11)
2
2
t=1

P
where ᾱk = 1/T t αt,k is the mean of α corresponding to sense k.

4

Experimental Settings

In the following experiments, we split the data (i.e.
set of snippets) into period of time slice 1 to T .
The time interval for year-labeled data was set to∆t = 20 years. We used vocabulary with frequency
larger than 10.
As model settings, we set the maximum number of senses to K = 8, the initial sense precision(k)parameter as κϕ = 4 for each sense k, and the
gamma parameters as a = 7 and b = 3. We set a relatively large value for the word precision parameter,with κψ = 100.0, following Perrone et al. (2019).
This is because we want to capture the sense changeof the target word as much as possible in terms of a
“shift of the sense distribution” rather than a “shiftof the sense–word distribution.” 5 Finally, we ran
5
In the former case, sense change is explained only by
the shift of the sense–word distribution, resulting in incorrect
detection of sense change, making it difficult to estimate thenumber of senses.

#Senses
1
2
3
4
5

SCAN
K =5 K =8
1.523
0.335
0.216
0.212
0.004

Infinite SCAN

1.997
0.578
0.735
0.150
0.017

0.468
0.039
0.030
0.061
0.004

Table 1: Kullback-Leibler divergence (lower is better)between actual sense distribution and sense distribution
estimated by each model for the artificial data.

the Gibbs sampler for 2,000 iterations6 and resam(k)pled κϕ for each sense k after every 50 iterations,starting from iteration 150.

5

Experiments using Artificial Data

Prior to the experiments using real data, we evaluated the proposed model on artificial data to validate the model for correctly estimating arbitrary
changes and the number of senses.
5.1

DatasetWhen generating artificial data, we first sampled
the curve of sense change from a Gaussian process
and then transformed it by the LSBP at each time
point to obtain the multinomial sense distribution,
as shown in Figure 4. Next, we used a Zipfian
distribution to generate the sense–word distribution,
i.e.,
1/k s
(12)
f (k | s, N ) = PNsn=1 1/nto reproduce Zipf’s law of texts (Zipf, 1945) observed in real data. Finally, we randomly generated
a set of artificial snippets using these sense distributions and sense–word distributions. In thisexperiment, we fixed the number of time points
in the artificial data at T = 10, the original vocabulary size7 at V = 5,000, the snippet length at
I = 10, and changed the number of word senses8
from Sw = 1 to 5. The following example shows
the generated snippets with sense k = 0 and 3.
k = 0: a e y y ag cy et gw xa ahy
k = 3: dkj dkj dkj dkp dlq dms dmy dpy eaj esv

Here, words are actually expressed as integers from0 to 5,000, but we use alphabet (base-26 numbers)
for interpretability in this example.
6

The computational time is proportional to sample size,
and it took 5 minutes to converge on data with 10,000 samples.
7
The mode vocabulary size of the corpus used in the experiments was approximately 5,000.
8
Most polysemous words have five or less senses (Biemannand Nygaard, 2010).

Figure 5: Actual distribution and estimated distribution
for the artificial data for the number of senses Sw = 5.
Senses of the estimated distribution are sorted according
to the actual distribution for interpretability.

5.2

Results

Table 1 shows a comparison of the estimation results of SCAN and Infinite SCAN using artificial
data with the number of senses ranging from Sw = 1
to 5. To quantitatively measure whether the sense
distribution is correctly represented, we used the
Kullback-Leibler divergence on the space RT ×K
between the estimated and actual sense distributions as an indicator. The results show that Infinite
SCAN outperforms SCAN that does not automatically estimate the number of word senses.
Figure 5 shows an example of the actual distribution and the estimation results of Infinite SCAN for
the artificial data with the number of word senses
Sw = 5. Here, the sense of the most dominant wordin the estimated sense-word distribution for eachsense k is shown in the legend for simplicity.
The figure shows that the proposed model captures the sense change almost precisely and estimates the true number of senses correctly.

6

Experiments using Real Data

In the experiments on real data, we firstly evaluatedinterpretability of the model output in the samemanner as topic models (Section 6.3). We further
evaluated the quality of the estimation results ofthe proposed model in terms of the estimation of
the number of senses (Section 6.4) and the sensechange (Section 6.5).
6.1

DatasetWe used the Clean Corpus of Historical American English (CCOHA) (Alatrash et al., 2020), a
large collection of texts from various genres covering the years 1810–2009. As a preprocessing,
we tokenized, lemmatized, and removed stopwords.
Moreover, we performed part-of-speech tagging
using the Natural Language Toolkit (NLTK) (Birdet al., 2009) and extracted only nouns, verbs, andadjectives. After the above preprocessing, we created the target word-specific input corpora, i.e.
snippets, for our models. They consisted of a set ofcontext words cd before and after the point where
the target word w appeared in the corpus, with asymmetric window width of ±5 words.
For quantitative evaluation (Sections 6.3 and
6.4), out of the 4,193 sense-tagged words (nounand verbs), we randomly selected 120 words withfive or less senses 9 from OntoNotes (Hovy et al.,2006). The statistics of the randomly selectedwords are shown in Appendix B. For qualitativeevaluation (Section 6.5), we selected the following three words: coach (Aida et al., 2021), record(Hamilton et al., 2016b), and power (Frermann
and Lapata, 2016), based on previous studies. The
statistics of these words are shown in Appendix C.
6.2

Models

We compared the proposed model with the following previous methods in addition to SCAN.
HDP-LDA We used a Bayesian nonparametricversion of LDA (Blei et al., 2003) using hierarchical Dirichlet process (HDP-LDA) (Teh and Jordan,
2010) that can automatically estimate the numberof topics as one of the baseline models. Unlike
the proposed model, HDP-LDA does not model
the temporal evolution of texts. In addition, sincea document is represented by a mixture of topics
rather than one document with one topic, we used“the set of snippets at time t” instead of “a snippet”
as the input unit to estimate the number of senses
and semantic changes. The number of topics was
initially set to K = 8, but estimated adaptively.
BERT + clustering We also compared our modelwith BERT (Devlin et al., 2019), a method that uses
contextualized word embeddings.10 In line withthe previous study (Giulianelli et al., 2020), we
first obtained contextualized word embeddings forevery sentence containing the target word. Then,we estimated the number of senses and semanticchanges by clustering the contextualized word embeddings across time periods. For the clustering
methods, we used both k-means (Lloyd, 1982) and
DBSCAN (Ester et al., 1996).
9

Most polysemous words have five or less senses (Biemannand Nygaard, 2010) as noted in Section 5.
10
We used base-uncased version of the pre-trained model
available at https://github.com/huggingface/transformers.

ModelCoherence

Diversity

HDP-LDA

0.125

0.821

SCAN (K = 5)
SCAN (K = 8)
Infinite SCAN

0.178
0.171
0.181

0.716
0.654
0.885

Model

Table 2: Sense coherence and diversity (higher is better)computed with the baseline models and Infinite SCAN
for 120 target words randomly selected from OntoNotes.

6.3

Evaluation of interpretability

Accuracy

Pearson Corr.

HDP-LDABERT + k-means
BERT + DBSCAN

0.258
0.217
0.125

0.019
0.026
−0.070SCAN (K = 5)
SCAN (K = 8)
Infinite SCAN

0.158
0.000
0.358

0.141
0.087
0.474

Table 3: Prediction results for the number of senses
computed with the baseline models and Infinite SCAN
for 120 target words randomly selected from OntoNotes.

Metrics For the evaluation of interpretability ofmodel output, we use sense coherence and diversity
to compare the baseline models with the proposedmodel, following Dieng et al. (2020). Sense coherence C is defined as the average similarity between
two words in representative words of each sense:
C=

10
10
K
X
η X X
k=1

45

(k)

(k)

f (wi , wj ),(13)

i=1 j=i+1(k)where wi is the i-th most probable word in sensek, and η is the normalization constant of the word
sense. Although Dieng et al. (2020) simply set η =1/K, in this study, we set η = p(k) using the sense
probability p(k) of each sense k to legitimately
evaluate the sense–word distribution of the senseof very small probability. f (w, w′ ) denotes thesimilarity of words in the semantic space; we use
the cosine similarity calculated through word2vec11(Mikolov et al., 2013). We define sense diversity
as the proportion of words with no overlap among
the top 10 words in all senses. Diversity close to
0 indicates redundancy, and diversity close to 1
indicates less overlap.
Results Table 2 shows the scores calculated using the estimation results of HDP-LDA, SCAN,
and Infinite SCAN12 for the target words. Here,
we note that the initial number of senses is fixed
for both HDP-LDA and SCAN to match the settingof the proposed model estimating without knowing the number of senses. The results show that
Infinite SCAN outperforms the baseline models
on both metrics. An improvement in coherencemeans that the semantic consistency of representative words of the estimated sense–word distribution
is high, indicating a high degree of interpretabilityof estimated sense. Regarding the improvement in
11

We used the pre-trained model available at https://code.google.com/archive/p/word2vec/.
12
BERT is not a probabilistic generative model and cannot automatically extract representative words, so it was notcomparable in this experiment.

Figure 6: Correlation between actual and expectation of
the number of senses computed with Infinite SCAN for
120 target words randomly selected from OntoNotes.

diversity, semantic overlap of representative words
in the estimated sense–word distribution is smaller,meaning that the number of senses is estimated at
an appropriate granularity.
6.4

Evaluation of the number of senses

Metrics For evaluating the number of senses, we
calculated the accuracy and Pearson correlation
coefficient (PCC) using the number of senses registered with OntoNotes as the gold standard. For
SCAN and Infinite SCAN, we calculated the effective number of senses as an expectation of thesense distribution as follows: PE(Sw ) = exp − K
ϕ
log
ϕ
,
(14)
k
k
k=1
P
where ϕk = 1/T Tt=1 ϕt,k is a marginal sense
probability. Here, the term within the exponential is an entropy of the sense distribution, meaningthat E(Sw ) is the perplexity of this distribution. For
example, when ϕ = (0.5, 0.5, · · · , 0), E(Sw ) is 2
and even when ϕ = (0.49, 0.01, 0.5, · · · , 0), E(Sw )becomes also approximately 2. We used this expectation to calculate PCC, and used its floor value forthe accuracy. For HDP-LDA, we directly used theestimated number of topics as the number of senses.
For BERT + k-means, we determined the numberof senses by selecting the number of clusters thatmaximizes the silhouette score (Rousseeuw, 1987),
following Giulianelli et al. (2020). For BERT +Infinite SCAN
SCAN(a) coach (Aida et al., 2021)(b) record (Hamilton et al., 2016b)(c) power (Frermann and Lapata, 2016)

Figure 7: Estimated senses for the target words coach, record, and power. Each bar shows the proportion of eachsense and is labeled with the start year of the respective time interval. Senses are shown as most high NPMI words.

DBSCAN, we used the estimated number of clusters as the number of senses, with hyperparameters
epsilon= 5 and min_samples= 2.

larity of manually-annotated meanings, since there
are some gold labels that have not appeared in the
CCOHA corpus we used for estimation.

Results Table 3 shows prediction results of thenumber of senses for the baseline models and Infinite SCAN. We can see that Infinite SCAN outperforms the other models in terms of accuracy and
PCC between the number of gold and estimatedsenses. Since SCAN has no mechanism for automatically estimating the number of senses, boththe accuracy and PCC are quite low. Even though
HDP-LDA and BERT+clustering have an architecture for determining the number of senses, PCC isquite low and these methods do not capture trends
in the number of senses that differs depending on
a word. Here, compared to k-means, DBSCAN isworse because it ignores some examples as noise,
which results in sparse clustering and an overestimation of the number of senses. In contrast, Infinite SCAN can estimate the number of senses
more appropriately, because it generates a sense
distribution in the stick-breaking architecture that
automatically estimates the number of senses fromdata.
Figure 6 shows that the correlation between theactual and expectation of the number of senses, indicating that the proposed model can capture thetendency of the number of senses. Here, we notethat it is obviously difficult to estimate the granu-

6.5

Evaluation of sense change

Methodology We qualitatively evaluated thetracking of sense change by visualizing the sense
distribution for three target words coach, record,and power estimated using Infinite SCAN and
SCAN. We also show several words with highNormalized pointwise mutual information (NPMI)(Bouma, 2009)
P in the marginalized sense–word distribution Tt=1 ψt,k for each sense k.
Results Figure 7 shows the estimation results forthree targets; Infinite SCAN acquires senses withmore appropriate granularity (coarse-grained) andcaptures sense transitions more interpretably compared to SCAN. Figure 7(a) shows the results on
the target word coach. Infinite SCAN captures twosenses, and also indicates changes, with the sense
vehicle (blue) becoming narrower and the senseteach (orange) becoming more dominant, which isconsistent with the analysis by Aida et al. (2021).
SCAN also captures these senses, but there are
overlaps in captured sense (e.g., orange and purple), making it difficult to capture the spread ofsenses. For the target word record in Figure 7(b),three senses emerge: audio record (blue), document or history (orange), and achievement (green).

According to Hamilton et al. (2016b), the new
sense, which is similar to words such as music
and tape, emerged around 1920; they are captured
more clearly by Infinite SCAN. For the word powerin Figure 7(c), our model captures the senses including mental power (blue), authority (orange),
legal power (green), and energy (red). The latter
is an example of “sense birth” (Mitra et al., 2014),
described by Frermann and Lapata (2016), and ourmodel captures such a trend. By contrast, in SCAN,
the sense energy is divided into two senses (red andgray), making it difficult to identify the correctchange or birth of the sense.

7

Conclusion

In this study, we proposed a statistical model thatcan jointly estimate the number of senses and semantic change of words by combining a dynamic
topic model on GMRF with a Dirichlet process. In
our experiments, we demonstrated that the proposed model correctly estimates the number ofword senses and semantic changes in detail, andshowed that the proposed model outperforms baseline models.
In the future, we would like to enhance the modelby incorporating linguistic knowledge on semanticchange (Ghanbarnejad et al., 2014; Feltgen et al.,2017). Furthermore, we would like to work on analyzing semantic change using the proposed method
such as classification of change patterns (Hamilton
et al., 2016a).

8

Limitations

8.1

Dataset limitation

The proposed model assumes continuous time shift(i.e. iGMRF) and existence of time-continuous
corpus, although few languages have a large scalediachronic corpus. Because the proposed model isa Bayesian model, the unigram mixtures (ϕt andψt ) at a time point t can theoretically be estimatedeven if a small amount of data exists at time t.
Samples

1

2

1,000
2,500
5,000
10,000

0
0
N/A
N/A

5
5
5
0

#Senses
3
4
10
5
5
0

50
30
30
5

5
N/A
40
30
10

Table 4: Sufficient threshold of low-frequency words
for the model to correctly estimate the number of senses
and sense change for different sample sizes.

However, if there is a time point with no data at all,the estimation is likely to fail because it violates
the assumptions of the model. One solution is toadopt a relatively large value for the parameter ∆t,which controls the granularity of time shift, but fora more detailed analysis, it is necessary to use alarge scale time-continuous dataset.
8.2

Model limitation

We investigated the conditions under which themodel can correctly estimate the number of senses
and the sense change of words on artificial data
generated by different conditions. We showed that
Infinite SCAN can estimate the sense change under
the hyperparameter conditions for artificial data
generation outlined in Section 5. However, estimation does not always work well under all the conditions. To examine these conditions, we searched
for a threshold of low-frequency words at which
the model works correctly on artificial data for different numbers of senses and samples.13
Table 4 shows the sufficient threshold of lowfrequency words at which the model works correctly on artificial data for different number ofsenses and samples (i.e. when the vocabulary size
was fixed at V = 5, 000 and the number of samples varied from D = 1, 000 to 10, 000 for different
numbers of senses.) Note that N/A in the tableindicates that the estimation fails no matter what
the threshold value is. These results indicate thatthe smaller the sample size is, the larger the required threshold becomes. This is because the data
becomes sparser as the number of samples is reduced, and that more low-frequency words must be
truncated to capture senses correctly. Additionally,
the threshold for low-frequency words increases
with the number of senses since data with more
senses accelerate data sparsity. Therefore, data
must be prepared with a sample size of at least half
the vocabulary size, and the threshold must be set
appropriately to stabilize the estimation.
These limitations are also present in real data
where it is difficult to estimate the number of senses
and semantic changes for words with a large number of senses or for data with small sample sizes.
This can be solved by appropriately modifying thedata distribution (i.e. vocabulary) by thresholding.
We would like to address the formulation of theseheuristics in the future.
13
The evaluation of the model estimation was performed
manually by visualizing the sense distribution.

Acknowledgements
This research was supported by the NINJAL collaborative research project and NINJAL Diachronic
Corpus project at the National Institute for Japanese
Language and Linguistics, Japan.

References
Taichi Aida, Mamoru Komachi, Toshinobu Ogiso, Hiroya Takamura, and Daichi Mochihashi. 2021. A
comprehensive analysis of PMI-based models formeasuring semantic differences. In Proceedings ofthe 35th Pacific Asia Conference on Language, Information and Computation, pages 21–31, Shanghai,
China. Association for Computational Lingustics.

Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages
4171–4186, Minneapolis, Minnesota. Association for
Computational Linguistics.
Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei.
2020. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439–453.
Jason A Duan, Michele Guindani, and Alan E Gelfand.
2007. Generalized spatial Dirichlet process models.
Biometrika, 94(4):809–825.

Jean Aitchison. 2001. Language change: Progress or
decay? Cambridge university press.

Haim Dubossarsky, Simon Hengchen, Nina Tahmasebi,
and Dominik Schlechtweg. 2019. Time-out: Temporal referencing for robust modeling of lexical semantic change. In Proceedings of the 57th Annual
Meeting of the Association for Computational Linguistics, pages 457–470, Florence, Italy. Association
for Computational Linguistics.

Reem Alatrash, Dominik Schlechtweg, Jonas Kuhn, andSabine Schulte im Walde. 2020. CCOHA: Clean corpus of historical American English. In Proceedingsof the 12th Language Resources and Evaluation Conference, pages 6958–6966, Marseille, France. European Language Resources Association.

Martin Emms and Arun Kumar Jayapal. 2016. Dynamic
generative model for diachronic sense emergence detection. In Proceedings of COLING 2016, the 26th
International Conference on Computational Linguistics: Technical Papers, pages 1362–1373, Osaka,
Japan. The COLING 2016 Organizing Committee.

Charles E Antoniak. 1974. Mixtures of dirichlet processes with applications to bayesian nonparametricproblems. The annals of statistics, pages 1152–1174.

Martin Ester, Hans-Peter Kriegel, Jörg Sander, andXiaowei Xu. 1996. A density-based algorithm fordiscovering clusters in large spatial databases withnoise. In Proceedings of the Second International
Conference on Knowledge Discovery and Data Mining, pages 226–231.

Robert Bamler and Stephan Mandt. 2017. Dynamic
word embeddings. In Proceedings of the 34th International Conference on Machine Learning, pages
380–389. PMLR.
Chris Biemann and Valerie Nygaard. 2010. Crowdsourcing wordnet. In The 5th International Conference of the Global WordNet Association (GWC2010).
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text
with the natural language toolkit. O’Reilly Media,
Inc.
Andreas Blank and Peter Koch. 1999. Historical Semantics and Cognition. 13. Walter de Gruyter.
David M Blei, Andrew Y Ng, and Michael I Jordan.
2003. Latent dirichlet allocation. Journal of machine
Learning research, 3(Jan):993–1022.
Gerlof Bouma. 2009. Normalized (pointwise) mutual
information in collocation extraction. In Proceedingsof the German Society for Computational Linguistics
and Language Technology, volume 30, pages 31–40.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of
the North American Chapter of the Association for

Quentin Feltgen, Benjamin Fagard, and J-P Nadal. 2017.
Frequency patterns of semantic change: corpus-based
evidence of a near-critical dynamics in languagechange. Royal Society open science, 4(11):170830.
Thomas S Ferguson. 1973. A bayesian analysis of some
nonparametric problems. The annals of statistics,
pages 209–230.
Lea Frermann and Mirella Lapata. 2016. A Bayesianmodel of diachronic meaning change. Transactionsof the Association for Computational Linguistics,
4:31–45.
Fakhteh Ghanbarnejad, Martin Gerlach, José M Miotto,
and Eduardo G Altmann. 2014. Extracting information from s-curves of language change. Journal of
The Royal Society Interface, 11(101):20141044.
Mario Giulianelli, Marco Del Tredici, and Raquel Fernández. 2020. Analysing lexical semantic changewith contextualised word representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3960–
3973, Online. Association for Computational Linguistics.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky.
2016a. Cultural shift or linguistic drift? comparing two computational measures of semantic change.

In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages
2116–2121, Austin, Texas. Association for Computational Linguistics.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky.
2016b. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
1489–1501, Berlin, Germany. Association for Computational Linguistics.
Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance
Ramshaw, and Ralph Weischedel. 2006. OntoNotes:
The 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 57–60, NewYork City, USA. Association for Computational Linguistics.
Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde,
and Slav Petrov. 2014. Temporal analysis of language through neural language models. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages
61–65, Baltimore, MD, USA. Association for Computational Linguistics.
Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, andSteven Skiena. 2015. Statistically significant detection of linguistic change. In Proceedings of the 24thinternational conference on world wide web, pages
625–635.
Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski,
and Erik Velldal. 2018. Diachronic word embeddings
and semantic shifts: a survey. In Proceedings of the27th International Conference on Computational Linguistics, pages 1384–1397, Santa Fe, New Mexico,
USA. Association for Computational Linguistics.
Scott Linderman, Matthew J Johnson, and Ryan P
Adams. 2015. Dependent multinomial models madeeasy: Stick-breaking with the Pólya-gamma augmentation. Advances in Neural Information Processing
Systems, 28.
Stuart Lloyd. 1982. Least squares quantization in pcm.
IEEE transactions on information theory, 28(2):129–
137.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey
Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of International
Conference on Learning Representations.
David Mimno, Hanna Wallach, and Andrew McCallum.
2008. Gibbs sampling for logistic normal topic models with graph-based priors. In NIPS Workshop on
Analyzing Graphs, volume 61.
Sunny Mitra, Ritwik Mitra, Martin Riedl, Chris Biemann, Animesh Mukherjee, and Pawan Goyal. 2014.
That’s sick dude!: Automatic identification of word

sense change across different timescales. In Proceedings of the 52nd Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 1020–1029, Baltimore, Maryland. Association for Computational Linguistics.
Valerio Perrone, Marco Palma, Simon Hengchen,
Alessandro Vatri, Jim Q. Smith, and Barbara
McGillivray. 2019. GASC: Genre-aware semanticchange for Ancient Greek. In Proceedings of the
1st International Workshop on Computational Approaches to Historical Language Change, pages 56–
66, Florence, Italy. Association for Computational
Linguistics.
Nicholas G Polson, James G Scott, and Jesse Windle.
2013. Bayesian inference for logistic models using
Pólya-gamma latent variables. Journal of the American statistical Association, 108(504):1339–1349.
Lu Ren, Lan Du, Lawrence Carin, and David B Dunson.
2011. Logistic stick-breaking process. Journal of
Machine Learning Research, 12(1).
Peter J Rousseeuw. 1987. Silhouettes: a graphical aid
to the interpretation and validation of cluster analysis.
Journal of computational and applied mathematics,
20:53–65.
Maja Rudolph and David Blei. 2018. Dynamic embeddings for language evolution. In Proceedings of the2018 World Wide Web Conference, pages 1003–1011.
Havard Rue and Leonhard Held. 2005. GaussianMarkov random fields: theory and applications.
Chapman and Hall/CRC.
Jayaram Sethuraman. 1994. A constructive definitionof Dirichlet priors. Statistica sinica, pages 639–650.
Angus Stevenson. 2010. The Oxford English Dictionary.
Oxford University Press.
Erik Sudderth and Michael Jordan. 2008. Shared segmentation of natural scenes using dependent PitmanYor processes. Advances in neural information processing systems, 21.
Yee Whye Teh and Michael I Jordan. 2010. Hierarchical
Bayesian nonparametric models with applications.
Bayesian nonparametrics, 1:158–207.
Joseph B Voyles. 1973. Accounting for semanticchange. Lingua, 31(2-3):95–124.
Joseph M Williams. 1976. Synaesthetic adjectives: A
possible law of semantic change. Language, pages
461–478.
Zijun Yao, Yifan Sun, Weicong Ding, Nikhil Rao, andHui Xiong. 2018. Dynamic word embeddings forevolving semantic discovery. In Proceedings of theeleventh acm international conference on web searchand data mining, pages 673–681.
George Kingsley Zipf. 1945. The meaning-frequency
relationship of words. The Journal of general psychology, 33(2):251–256.

A

Pseudo-code of MCMC algorithm

Algorithm 1 shows the MCMC algorithm for theestimation of Infinite SCAN. In practice, we sam(k)ple z, ϕ and ψ for 2,000 iterations, and sample κϕfor each sense k after every 50 iterations, startingfrom iteration 150.

B

Statistics of snippets used for theevaluation of the number of sense

Table 5 shows the statistics of snippets used for theevaluation of the number of senses (Sections 6.3
and 6.4). This table lists the number of words ineach sense, the average number of samples, and thevocabulary size.

Algorithm 1: MCMC algorithm(k)Initialize κϕ = 4.0 (for all k)
2 Initialize κψ = 100.0
3 Initialize a = 7.0, b = 3.0
4 for t = 1 . . . T do
5
Initializeαt ∼ N 21 (αt−1 + αt+1 ), κ−1
ϕ
1

6
7
8

Set ϕt = LSB(αt )for k = 1 . . . K do
Initialize βt,k ∼
N 21 (βt−1,k + βt+1,k ), κ−1ψSet ψt,k = Softmax(βt,k )end
11 end12 for j = 1 . . . J do
13
Sample z according to Eq. (8)14for t = 1 . . . T do
15
Sample ϕ according to posterior in
Eq. (9)16
Sample ψ according to posterior in
Eq. (10)17
end
18
Sample κϕ according to posterior in Eq.
(11)
19 end9

10#Senses#WordsSamples#Vocab

1
2
3
4
5

30
33
30
17
10

15,922
17,085
17,471
18,32818,17814,40315,014
15,86816,388
16,645

Table 5: Snippet statistics of target words randomly
selected from OntoNotes. Sample size and vocabulary
size are shown as averages.

C

Statistics of snippets used for theevaluation of sense change

Table 6 shows the statistics of snippets used for theevaluation of sense change (Section 6.5). The tablelists the years, sample size, and vocabulary size foreach target word.
Word

Years

Samples#Vocabcoach
recordpower

1811–2009
1815–2009
1810–2009

9,758
33,992
142,527

11,962
23,88642,932

Table 6: List of target words and snippet statistics.