Inﬁnite SCAN: An Inﬁnite Model ofDiachronic Semantic Change
Seiichi Inoue,1 Mamoru Komachi,1 Toshinobu Ogiso,2 Hiroya Takamura,3and Daichi Mochihashi4
1Tokyo Metropolitan University
2The National Institute for Japanese Language and Linguistics
3
The National Institute of Advanced Industrial Science and Technology
4
The Institute of Statistical Mathematics

Sense Modeling with a Topic Modeltarget word: “coach ”

1Creating Inputs

Diachronic Corpus

Usage with Time Stamp
1853
1900
1949
2003

The driver made room for the trunk on the top of thecoach.
The chair passed the coach, the horses proceeding at a walk.
Tell him if I start coaching, it’ll be as a head coach at a top school.
Football coach and other top school officials have been interviewed.

⋮

Snippets of Contextual Information
1853
1900
1949
2003

[driver, make, room, trunk][chair, pass, horse, proceed, walk][tell, start, coach, head, top, school][football, top, school, official, interview]⋮2

Sense Modeling with a Topic Model2

Sense Modeling3

Analysis

Snippets (inputs)
1853
1900
1949
2003

[driver, make, room, trunk][chair, pass, horse, proceed, walk][tell, start, coach, head, top, school][football, top, school, official, interview]

⋮…
1810s

…
1910s

1990s
3

Preliminaries: SCAN
SCAN: Bayesian Model of Sense Change [Frermann+, 2016]
For each time t :Sense 1

Sense 3

Sense 2

Snippets at time t

Sense 1

φ: Sense Distribution

Sense 2

Sense 3

ψ: Word Distribution per Sense

4

Preliminaries: SCAN
SCAN: Bayesian Model of Sense Change [Frermann+, 2016]
For each time t :Sense 1

Sense 3

Change with time

Sense 2

Snippets at time t

Sense 1

φ: Sense Distribution

Sense 2

Sense 3

ψ: Word Distribution per Sense

5Preliminaries: SCAN
SCAN: Bayesian Model of Sense Change [Frermann+, 2016]
For each time t :Sense 1

Sense 3How many senses ?

Sense 2

Snippets at time t

Sense 1

φ: Sense Distribution

Sense 2

Sense 3

ψ: Word Distribution per Sense

6Proposal: Inﬁnite SCAN
Inﬁnite SCAN: Inﬁnite Model of Diachronic Semantic Change
Automatically estimate the number of senses using Dirichlet process (Stick-breaking process)Sense 1Snippets at time t

Sense 1

Sense 2

φ: Sense Distribution

Sense 2

ψ: Word Distribution per Sense

7Proposal: Inﬁnite SCAN
Stick-breaking transformation of Gaussian sense distribution
Logistic normal priors on
Gaussian Markov Random Field

Projecting to K-1 simplex using Logistic
stick-breaking process (not softmax)
8Setup: Experiments on Artiﬁcial Data
“Ideal” distribution
Random prior

Draw

Estimate &Compare●●

Sense distribution: Draw fromGaussian process
Sense-word distribution: Drawfrom Zipﬁan distribution

Estimated distribution

9Experiments on Artiﬁcial Data
For artiﬁcial data with ranging the number of senses 1 to 5:●●

Correctly estimate the number of senses except for S = 1
Properly estimate the actual distribution: KL (Actual || Estimated) ≃ 0.0
Example for the number of senses: S = 5

Kullback-Leibler divergence (Actual || Estimated) = 0.004

10Setup: Experiments on Real Data

Diachronic Corpus
CCOHA [Alatrash+, 2020]
Extract

Estimated distribution
EvaluateSnippets

Gold number ofsenses

Known
Sense change

Sense inventory ofOntoNotes [Hovy+, 2006]

Previous work

[Giulianelli+, 2020;
Frermann+, 2016]11

Experiments on Real Data
Can the model correctly estimate the number of senses?
For 120 target words randomly selected from OntoNotes:ModelHDP-LDA
BERT + K-means
BERT + DBSCANSCAN (K = 5)
SCAN (K = 8)Inﬁnite SCAN

Accuracy
0.258
0.217
0.125
0.158
0.000
0.358

PCC
0.019
0.026
-0.070
0.141
0.087
0.474

Prediction results for the number of senses.

Correlation between actual and estimatednumber of senses by Inﬁnite SCAN.

12

Experiments on Real Data: “coach ”
Can the model properly describe the semantic change?
Inﬁnite SCAN

SCAN
Trainer

Carriage

Semantic shift: “carriage” → “trainer”

13

Experiments on Real Data: “power ”
Can the model properly describe the semantic change?
Inﬁnite SCAN
Legal

EnergySCAN

Authority
MentalSense birth: “energy”
14Conclusions / Future Work
● Inﬁnite SCAN: An inﬁnite model of diachronic semanticchange○ based on dynamic topic model○ combining Gaussian Markov random ﬁelds with a Dirichlet
process● Outperformed baseline models○ interpretability of model output○ estimation of the number of senses● Captured semantic changes in line with previous studies● In future, working on classiﬁcation of change patterns
15

Appendices: Setup of experiments on Artiﬁcial DataGeneration of artiﬁcial data●
●Sense distribution: Draw from Gaussian
process with the number of senses K
Sense-word distribution: Draw from
Zipﬁan distribution with the vocabularysize VParameter settings
●
●
●
●
●

Number of time points: T = 10
Original vocabulary size: V = 5,000
Snippet length: I = 10
Number of senses: S = 1 ~ 5Sample size: N = 10,000

Metric●Kullback-Leibler distance between actualand estimated distribution

16

Appendices: Setup of experiments on Real Data
Historical corpus
Clean Corpus of Historical American English (CCOHA) [Alatrash+, 2020]

Target words with gold number of senses
randomly selected 120 words (noun, verbs) with S ≦ 5 from OntoNotes [Hovy+,2006]

Target words with semantic change description●●“coach “ [Giulianelli+, 2020; Aida+, 2021]“power “ [Frermann+, 2016]

Baseline models●●●

HDP-LDA [Teh+, 2010]
BERT (base-uncased) [Devlin+, 2019] + K-means, DBSCANSCAN [Frermann+, 2016]17