A Word Sense Distribution-based approach for Semantic Change Prediction
Authors: Xiaohang Tang
Yi Zhou
Taichi Aida
Procheta Sen
Danushka Bollegala
Venue: EMNLP 2023 Findings (2023)
Keywords: Semantic Change Detection
Temporal Semantics
Sense Embeddings
TL;DR: We propose a sense embedding-based method for detecting whether the meaning of a given word has changed from one corpora to another.

Abstract: Semantic Change Detection of words is an important task for various NLP applications that must make time-sensitive predictions.
Some words are used over time in novel ways to express new meanings, and these new meanings establish themselves as novel senses of existing words.
On the other hand, Word Sense Disambiguation (WSD) methods associate ambiguous words with sense ids, depending on the context in which they occur.
Given this relationship between WSD and SCD, we explore the possibility of predicting whether a target word has its meaning changed between two corpora collected at different time steps, by comparing the distributions of senses of that word in each corpora.
For this purpose, we use pretrained static sense embeddings to automatically annotate each occurrence of the target word in a corpus with a sense id.
Next, we compute the distribution of sense ids of a target word in a given corpus.
Finally, we use different divergence or distance measures to quantify the semantic change of the target word across the two given corpora.
Our experimental results on SemEval 2020 Task 1 dataset show that word sense distributions can be accurately used to predict semantic changes of words in English, German, Swedish and Latin.

word Can Word Sense Distribution Detect Semantic Changes of Words?
Xiaohang Tang'
Yi Zhou
Taichi Aida Procheta Sen Danushka Bollegala
University of Liverpool, United Kingdom.
Cardiff University, United Kingdom.
Tokyo Metropolitan University, Japan. Amazon
xiaohangtango1 @gmail. com zhouy1 31 31@cardiff wtt.ddifff.c..ukk
aida-taichi@ed. tmu. ... jp {procheta. sen, danushka (atrsshkk@@liveepoooo wliveepool............
Abstract
using static snapshot. Moreover, Su al. (2022) showed that the temporal generalisation of LLMs is closely related to their ability to detect semantic variations of words.
Semantic Change Detection (SCD) of words is important task for various NLP applica- tions that must make time-sensitive predictions. Some words are used over time novel ways to express new meanings, and these new mean- ings establish themselves as novel senses of existing words. On the other hand, Word Sense Disambiguation (WSD) methods associate am- biguous words with sense ids, depending on context in which they occur. Given this relation- ship between WSD and SCD, we explore the possibility of predicting whether a target word has its meaning changed between two corpora collected at different time steps, by comparing the distributions of senses of that word in each corpora. For this purpose, we use pretrained static sense embeddings to automatically an- notate each occurrence of the target word in a corpus with a sense id. Next, we compute the distribution of sense ids of target word in given corpus. Finally, vergence or distance measures to quantify the semantic change of the target word across the two given corpora. Our experimental results on SemEval 2020 Task dataset show that word sense distributions predict semantic changes of words in English, German, Swedish and Latin.
A word is often associated with multiple senses as listed in dictionaries, corresponding to its dif- ferent meanings. Polysemy (i.e. coexistence of several possible meanings for word) has been shown to statistically correlate with the rate of semantic change in prior work (Bréal, 1897; Ullmann, 1959; Magué, 2005). For example, consider the word cell, which has the following three noun senses according to the WordNer!: (a) cell % 1:03:00- - the basic structural and functional unit of all organisms, (b) cell % 1 1:06:04 a hand-
held mobile radiotelephone for use in an area di- vided into small sections, and (c) cell % 1:06:01 room where a prisoner is kept. Here, the WordNet sense ids for each word sense are shown in boldface font. Mobile phones were first produced in the late 1970s and came into wider circulation after 1990. Therefore, the sense (b) is considered as a more recent association compared to (a) and (c). Given two sets of documents, one sampled before 1970 and one after, would expect encounter (b) more frequently in the latter Likewise, articles on biology are likely contain (a). As seen from this example the sense distributions two corpora provide useful information about its possible meaning changes over time.
different di-
be accurately used to
1 Introduction
SCD of words over time has provided important insights for diverse fields such as linguistics, lex- icography, sociology, and information retrieval (IR) (Traugott and Dasher, 2001; Cook and Steven- son, 2010; Michel et al., 2011; Kutuzov et al., 2018). For example, in IR one must know the sea- sonal association of keywords used in user queries provide relevant results pertaining to a particu- lar time period. Moreover, has been shown that the performance of publicly available pretrained LLMs declines over time when applied to emerg- ing data (Su et al., 2022; Loureiro et al., 2022; Lazaridou et al., 2021) because they are trained
Given this relationship between the two tasks SCD and WSD, a natural question arises is the word sense distribution indicative of seman- tic changes of words? To answer this question, we design and evaluate an unsupervised SCD method that uses only the word sense distributions to pre- dict whether the meaning associated with a target word w has changed from one text corpora C₁ to another Cz. For the case of future references, we name this method Sense-based Semantic Change
htttp::/wwordeet.rriiee www.cetoo.ednnnu

---

Score (SSCS). Given target word w, we first disambiguate each occurrence of w in each cor- pus. For this purpose, we measure the similar- ity between the contextualised word embedding of wc obtained from a pre-trained Masked Language Model (MLM), from the contexts containing w with each of the pre-trained static sense embed- dings corresponding to the different senses of w. Next, compute the sense distribution of in each corpus separately. Finally, we use multiple distance/divergence measures to compare the two sense distributions of w to determine whether its meaning has changed from C to C2.
degree or presence of semantic changes for given larget word between two given corpora, sampled from different time periods. Various methods have been proposed to map vector spaces from differ- ent time periods, such as initialisation (Kim et al., 2014), alignment (Kulkarni et al., 2015; Hamilton et al., 2016), and joint learning (Yao et al., 2018; Dubossarsky et al., 2019; Aida et al., 2021).
Existing SCD methods can be broadly cate- gorised into two groups: (a) methods that com- pare word/context clusters (Hu et al., 2019; Giu- lianelli et al., 2020; Montariol et al., 2021), and (b) methods that compare embeddings of the tar- get words computed from different corpora sam- pled at different time periods (Martinc et al., 2020; Beck, 2020; Kutuzov and Giulianelli, 2020; Rosin et al., 2022). Rosin and Radinsky (2022) recently proposed a temporal attention mechanism, which achieves SoTA performance for SCD. However, their method requires additional training of the en- tire MLM with temporal attention, which is compu- tationally expensive for large MLMs and corpora. The change of the grammatical profile (Kutuzov et al., 2021; Giulianelli et al., 2022) of word, cre- ated using its universal dependencies obtained from UDPipe (Straka and Straková, 2017), has shown to correlate with the semantic change of that word, However, the accuracy of the grammatical pro- file depends on the accuracy of the parser, which can be low for resource poor languages and noisy texts. Sabina Uban et al. (2022) used polysemy as a feature for detecting lexical semantic change discovery. The distribution of the contextualised embeddings a word over its occurrences (aka. sibling embeddings) in a corpus has shown to be an accurate representation of the meaning of that word in corpus, which can be used to compute various semantic change detection scores (Kutu- zov et al., 2022; Aida and Bollegala, 2023). XL- LEXEME (Cassotti et al., 2023) is supervised SCD method where a bi-encoder model is trained using WiC (Pilehvar and Camacho-Collados, 2019) dataset to discriminate whether a target word ap- pears in different senses in a pair of sentences. XL- LEXEME reports SoTA SCD results for English, German, Swedish and Russian.
To evaluate the possibility of using word senses for SCD, we compare the performance of SSCS against previously proposed SCD methods us- ing the SemEval-2020 Task 1 (unsupervised lex- ical SCD) (Schlechtweg et al., 2020) benchmark dataset. This task has two subtasks: (I) a binary classification task, where for a set of target words, we must decide which words lost or gained sense(s) from C1 to C2. and (2) ranking task, where we must rank target words according to their degree of lexical semantic change from C1 to C2. We upply SSCS on two pre-trained static sense embeddings, and six distance/divergencee measures. Despite the computationally lightweight and unsupervised na- ture of SSCS, our experimental results show that it surprisingly outperforms most previously pro- posed SCD methods for English, demonstrating the effectiveness of word sense distributions for SCD. Moreover, evaluations German, Latin and Swedish show that this effectiveness holds in other languages well, although not to the same levels as in English. We hope our findings will motivate future methods for SCD to explicitly incorporate word sense related information. Source code imple- mentation for reproducing our experimental results is publicly available.?
2 Related Work
Semantie Change Detection: SCD is modelled in the literature as the unsupervised task of detect- ing words whose meanings change between two given time-specific corpora (Kutuzov et al., 2018; Tahmasebi et al., 2021). In recent years, several shared tasks have been held (Schlechtweg et al., 2020; Basile et al., 2020; Kutuzov and Pivovarova, 2021), where participants required to predict the
Sense embedding learning methods represent different senses of an ambiguous word with different vectors. The concept of multi- prototype embeddings to represent word senses was introduced by Reisinger and Mooney (2010).
Sense Embeddings:
"https://gitte"" wwwptttuuu..ommiiiPPP... iwNLP/ Lense-based -aaeed--aaattt---aaeeeeeeeiiiiii Prediction

---

This idea was further extended by Huang et al. (2012), who combined both local and global con- texts their approach. Clustering is used in both works to categorise contexts of word that belong to the same meaning. Although the number of senses a word can take depends on that word, both approaches assign predefined fixed number of senses to all words. To address this limitation, Nee- lakantan al. (2014) introduced non-parametric model, which able dynamically estimate the number of senses for each word.
w
M
|C|
Figure 1: Plate diagram showing the dependencies among the sentences 8 in the corpus C, the target word w, and its sense in s.
Although clustering- baased approaches can allo- cate multi-prototype embeddings to a word, they still suffer from the fact that the embeddings gener- ated this way are not linked to any sense invento- ries Coalacho-oollaoos and Pilchvar, 2018). On the other hand, knowledge based methods obtain sense embeddings by extracting sense-specific in- formation from external sense inventories, such as the WordNet (Fellbaum and Miller, 1998) or the BabelNet (Navigli and Ponzetto, 2012). Chen et al. (2014) extended word2vec (Mikolov et al., 2013) to learn sense embeddings using WordNet synsets. Rothe and Schütze (2015) made use of the semantic relationships in WordNet to embed words into shared vector space. Iacobacci al. (2015) used the definitions of word senses BabelNet and conducted WSD to extract contextual information that is unique to each sense.
target word in a corpus. Second, in $3.2, we use different distance (or divergence) measures to com-
pare the sense distributions computed for the same target word from different corpora to determine whether its meaning has changed between the two corpora.
3.1 Computing Sense Distributions
We represent the meaning expressed by target word in corpus C by the distribution of w's word senses, p( wu, C). As explained in $ our working hypothesis is that if the meaning of w has changed from C₁ to C2, then the correspond- ing sense distributions of w. p(zwlu w, C1) will be different from p(z C2)). Therefore, we first es- timate p(zor |w, C) according to the probabilistic model illustrated in the plate diagram in Figure 1. We consider the corpus, C to be collection of [C] sentences from which sentence s randomly sampled according to p(s[C). Next, for each word w in vocabulary V that appears in 8, we randomly sample a sense Znr from its set of sense ids Z As shown in Figure 1, we assume the sense that a word takes in sentence to be independent of the other sentences in the corpus, which enables us factorise p(2/uc, C) as in (1).
Recently, contextualised embeddings produced by MLMs have been used to create sense embed- dings. To achieve this, Loureiro and Jorge (2019) created LMMS sense embeddings by averaging over the contextualised embeddings of the sense annotated tokens from SemCor (Miller et al., 1993). Scarlini et al. (2020a) proposed SenST (Sense Embedded BERT), which makes use of the lexical-semantic information in BabelNet to cre- ate sense embeddings without relying on sense- annotated data. ARES (context-AwaRe Embed- dinS) (Scarlini et al., 2020b) is a knowledge-based method for generating BERT-based embeddings of senses by means of the lexical-semantic informa- tion available in BabelNet and Wikipedia. ARES and LMMS embeddings are the current SoTA sense embeddings.
p(zu C) = Σ
p(zul wizlwwssppss/ (1)
MEC(m)
Here, C(w) is the subset of sentences in C where w occurs. We assume p(s(C) to be uniform and set it to be 1//C|, where |C) is the number of sentences in C.
3 Sense-based Semantic Change Score
Following the prior works that static sense embeddings for conducting WSD, the similarity between the pre-trained static sense embedding z of the sense 2u of w, and the contextualised word embedding f(w, s) of w in (obtained from
SSCS consists of two steps. First, in $3.1, we com- pute the distribution of word senses associated with

---

a pre-trained MLM) can be used as the confidence score for predicting whether uo takes the sense Pw in s. Specifically, the LMMS and ARES sense em- beddings we use in our experiments are computed using BERT (Devlin et al., 2019) as the back-end, producing aligned vector spaces where we can com- pute the confidence scores using the inner-product given by (2).
binary classification and ranking. In the classifica- tion subtask, the words in the evaluation set must be classified as to whether they have semantically changed over time. Classification Accuracy (i.e. percentage of the correctly predicted words in the set of test target words) is used as the evaluation metric for this task.
To predict whether target word w has its meaning changed, we use Bayesian opti- misation to find a threshold on the distance, d(p(zwluu wo C1), p(zul (2)),, between the sense distributions of w computed from C₁ and C2. Specifically, we use the Adaptive Experimentation Platform* to find the threshold that maximises the classification accuracy on a randomly selected held- out portion of words from the SemEval dataset, served for validation purposes. We found that using Bayesian optimisation is more efficient than con- ducting a linear search over the parameter space. We repeat this threshold estimation process five times and use the averaged parameter values in the remainder of the experiments.
(Zws f(w, 8))
p( Zwr (w, Ee'eez (z'ws f(w, s))
(2)
In WSD, an ambiguous word typically as- sumed to take only a single sense in a given context. Therefore, WSD methods assign the most probable sense z to w in s, where = arg max m22p s). However, not all meanings Cu of word might necessarily map to a single word sense due to the incompleteness of sense invento- ries. For example, novel use of an existing word might be a combination of multiple existing senses of that word rather than a novel sense. Therefore, is important to consider the sense distribution over word senses, p(zul w, s), instead of only the most probable sense. Later in $5.1, we experimentally study the effect of using top-k senses of word in a given sentence.
In the ranking subtask, the words in the evalu- ation set must be sorted according the degree of semantic change. Spearman's rank correlation coefficient (p E [-1, 1)) between the human-rated gold scores and the induced ranking scores is used as the evaluation metric for this subtask. Higher, p values indicate better SCD methods.
3.2 Comparing Sense Distributions
Statistics of the data used in our experiments are shown in Table 4 in Appendix B. The En- glish dataset includes two corpora from different centuries extracted from CCOHA (Alatrash et al., 2020). Let us denote the corpora collected from the early 1800s and late 1900s to early 2000s respec- tively by C1 and Cz. For each language, its test set has 30-48 target words that are selected to indicate whether they have undergone semantic change between the two time periods. These words are annotated by native speakers indicating whether their meanings have changed over time and if so the degree of the semantic change.
Following the procedure described in $3.1, we in- dependently compute the distributions p(z1| C1) and p(z wu, C2) respectively from C₁ and Cz. Next, we compare those two distributions using different distance measures, d(p(zwlv (, C1), p(z lus, C2)). to determine whether the meaning of w has changed between C1 and Cz. For this purpose, we use five distance measures (i.e. Cosine, Cheby- shev. Canberra. Bray-Curtis, Euclidean) and two divergence measures (i.e. densen-Shannon (JS), Kullback-Leibler (KL)) in our experiments. For computing distance measures, we consider each sense , as a dimension in a vector space where the corresponding value is set to p(zul wu C). The defi- nitions of those measures are given in Appendix A.
Sense Embeddings: We use two pre- trained sense embeddings in our experi- ments: LMMS (Loureiro and Jorge, 2019) and ARES (Scarlini et al., 2020b). For English mono- lingual experiments we use the 2048-dimensional
4 Experiments
Data and Evaluation Metries: We use the 2emEval-2020 Task dataset (Schlechtweg et al., 2020) to evaluate SCD of words over time for En- glish, German, Swedish and Latin in two subtasks:
bittps://ax.det

---

code information that is not relevant to the meaning of the target word in the given context. Therefore, there some benefit of not strictly limiting the sense distribution only to the top-1 ranked sense, but to consider k( 1) senses for a target word in given context. However, when we increase k, we will consider less likely senses of w in a given context 8, thereby introducing some noise in the computation of p( (w,CC.
Spearman Accuracy
0.70
Perforati
0.65
0.60
0.33
0.50
1
Top-k
We study the effect of using more than one sense in given context compute the sense distribution of a target word. Specifically, in the descending order of Zuul w. s), and select the top-k ranked senses to represent w in s. Set- ting & 1 corresponds to considering the most probable sense, i.e. 22 max p(z w, s). In Figure 2, we plot the accuracy (for the binary clas- sification subtask) and (for the ranking subtask) obtained using LMMS sense embeddings and JS divergence measure against k. For this experiment, randomly held-out set of English words from the SemEval dataset. From Figure 2, we see that both accuracy and p increase initially with & up to a maximum at = 2, and then start to drop. Following this result, we limit the sense distribu- tions to the top 2 senses for the remainder of the experiments reported in this paper.
Figure 2: Spearman's p and Accuracy on SemEval En- glish held-out data when using the top-k senses in the sense distribution with JS as the divergence.
sort the senses
LMMS and ARES embeddings computed using cart-large-cased" which use WordNet sense ids. For the multilingual experiments, we use the 768-dimensional ARES embeddings computed which uses BabelNet
using mul multilingual-bert sense ids. As a baseline method that does not use sense embeddings, we use the English WSD implementation in NLTKº to predict WordNet sense-ids for the target words, when computing the sense distributions, p(z (s) in (I).
Hardware and Hyperparameters: We used a single NVIDIA RTX A6000 and 64 GB RAM in experiments. It takes ca. 7 hours compute semantic change scores for all target words in the 15.3M sentences in the SemEval datasets for all languages we consider. The only hyperparame- ter in SSCS is the threshold used in the binary classification task, which is tuned using Bayesian Optimisation. The obtained thresholds for the dif- ferent distance measures are shown in Table 5 in Appendix B.
5.2 English Monolingual Results
Aida and Bollegala (2023) showed that the perfor- mance of an SCD method depends on the metric used for comparisons. Therefore, in Table 1 we show SCD results for English target words with dif- ferent distance/divvrgencee measures using LMMS, ARES sense embeddings against the NLTK WSD baseline. We see that LMMS sense embeddings coupled with JS divergence report the best perfor- mance for both the binary classification and ranking subtasks across all settings, whereas ARES sense embeddings with IS divergence obtain similar ac- curacy for the binary classification subtask. Com- paring the WSD methods, we see that NLTKK is not performing as well as the sense embedding- methods (i.e. LMMS and ARES) in terms of Spear- man's 's for the ranking subtask. Although NLTK matches the performance of ARES on the classi- fication subtask, it is still below that of LMMS. Moreover, the best performance for NLTK for the classification subtask is achieved with multiple met- rics such as Cosine, Bray- -urtis, Euclidean and KL. Therefore, we conclude that the sense embedding-
5
Results
5.1 Effect of Top-k Senses
The correct sense of a word might not necessarily be ranked as the top-1 because of two reasons: (a) the sense embeddings might not perfectly encode all sense related information, and (b) the contex- to compute
tualised word embeddings that the inner-product with sense embeddings might en-
Phttps: pttttp://githuu..omm. htt:://tttbb..comm/moo//////sss/
LMMS ACL19
"http:/ hhttp::/ssenseebert..or org/
hhttps://huggiigffee watpo//c..cbbbtttbeeeeeee co.bbert-aarge-casee
"https://huggiiggfacs
hhttps://wwal nl tk. org arg/howto/wsd.hrg html

---

Metric Spearman's
Accuracy
proposed by Rosin al. (2022) that fine-tunes pretrained BERT-base models using time tokens. BERT + APD was proposed by Kutuzov and Giu- lianelli (2020) that uses the averages pairwise co- sine distance. Based on this insight, Aida and Bol- legala (2023) evaluate the performance of BERT + Tokens + Cosine with the average pair- wise cosine distance computed using pre-trained BERT-base the MLM. BERT+DSCD is the sib- ling Distribution-
Cosine
0.007
0.595
Chebyshev
0.301
0.541
Canberra
0.423
0.568
NLTK
Bray-Curtis
0.175
0.595
Euclidean
0.257
0.595
IS
0.302
0.514
KL
0.351
0.595
Cosine
0.149
0.595
SCD (DSCD) method pro- posed by Aida and Bollegala (2023).
Chebyshev
0.080
0.568
Canberra
0.447
0.649 A Temporal Attention mechanism was pro- posed by Rosin and Radinsky (2022) where they add trainable temporal attention matrix to the pre- trained BERT models. Because their two proposed methods (fine-tuning with time tokens and tempo- ral attention) are independent, Rosin and Radinsky (2022) proposed to use them simultaneously, which denoted by BERT + Time Tokens Temporal Attention. Yüksel et al. (2021) extended word2vec to create Gaussian embeddings (Vilnis and Mc- Callum, 2015) for target words independently from each corpus
ARES
Bray-Curtis
0.485
0.622
Euclidean
0.277
0.568
JS
0.529
0.730
KL
0.233
0.622
Cosine
0.274
0,622
Chebyshev
0.124
0.568
LMMS
Canberra
0.502
0.405
Bray-Curtis
0.541
0.676
Euclidean
0.245
0.568
JS
0.589
0.730
KL
0.329
0.676
Rother et al. (2020) proposed Clustering on Manifolds of Contextualised Embeddings (CMCE) where they use mBERT embeddings with dimen- sionality reduction to represent target words, and then apply clustering algorithms to find the dif- ferent sense clusters. CMCE is the current SoTA for the binary classification task. Asgari al. (2020) proposed Embedl which uses fasttext (Bojanowski et al., 2017) create word embeddings from each corpora separately, and mea- sures the cosine similarity between a word and a fixed set of pivotal words to represent a word in a corpus using the distribution over those pivots.
Table I: Semantic Change Detection performance SemEval 2020 Task English dataset.
based sense distribution computation is superior to that the NLTK baseline.
Among the different distance/divergence mea- sures used, we see that IS divergence measure per- forms better than the other measures. In particular, for both ARES and LMMS, JS outperforms other measures for both subtasks, and emerges the overall best metric for SCD. On the other hand, Cosine distance, which has been used as a base- line in much prior work on semantic change detec- tion (Rosin et al., 2022) performs poorly for the ranking subtask although it does reasonably well on the classification subtask. Rosin et al. (2022) predicted semantic changes by thresholding the cosine distance. They used peak detection meth- ods (Palshikar, 2009) to determine this threshold, whereas we use Bayesian optimisation methods.
UWB (Pražák et al., 2020) learns separate word embeddings for target word from each cor- pora and then Canonical Correlation Analysis (CCA) to align the two vector spaces. UWB ranked 1st for the binary classification subtask the official SemEval 2020 Task competition.
In Table 2, we compare our SSCS with JS using LMMS sense embeddings (which reported the best performance according to Table 1) against prior work. For prior SCD methods, report perfor- mance from the original publications, without running those methods. However, not all prior SCD methods evaluate on both binary classification and ranking subtasks as we do in this paper, which is indicated by N/A (not available) in Table 2. XL-
5.3 English SCD Results
We compare SSCS against the following prior SCD methods the SemEval-2020 Task English data. Due to space limitations, further details of those methods are given Appendix C.
BERT + Time Tokens + Cosine the method

---

Model
Accuracy
Spearman
BERT-base TimeTokkens + Cosine (Rosin et al., 2022)
N/A
0.467
BERT-base APD (Kutuzov and Giulianelli, 2020)
N/A
0.479
BERT-base Temporal Attention (Rosin and Radinsky, 2022)
N/A
0.520
BERT-base TimeTTokens + DSCD (Aida and Bollegala, 2023)
N/A
0.529
BERT-base TimeTookes + Temporal Attention (Rosin and Radinsky, 2022)
N/A
0.548
Gaussian Embeddings (Yüksel et al., 2021)
0.649
0.400
CMCE (Rother et al., 2020)
0.730
0.440
EmbedLexChange (Asgari et al., 2020)
0.703
0.300
UWE (Pražák et al., 2020)
0.622
0.365
XL-LEXEME (Cassotti et al., 2023) (supervised)
N/A
0.757
SSCS (LMMS JS)
0.730
0.589
Table 2: Comparison against previously proposed SCD methods on the English data in SemEval-20200 Task
LEXEME supervised SCD method that is the current SoTA on this dataset.
based LLMs. On the other hand, Temporal At- tention increases the number of parameters the transformer as uses an additional time- specific weight matrix. Interestingly, from Ta- ble 2 we see that SSCS outperforms both those methods convincingly despite not requiring any fine-tuning/eetraiiing of the sense embeddings nor MLMs, which is computationally attractive.
From Table 2, we see that SSCS obtains compet- itive results for both binary classification and rank- ing subtasks on the SemEval-2020 Task English dataset, showing the effectiveness of word sense in- formation for SCD. It matches the performance of CMCE for the binary classification subtask, while outperforming Temporal attention with Time To- ken fine-tuning (Rosin and Radinsky, 2022) on the ranking subtask.
SSCS (which is unsupervised) does not outper- form XL-LEXEME (which is trained on WiC data) for the ranking subtask. In particular, nificant performance gap between XL-LEXEME and the rest of the unsupervised methods, indicat- ing that future work on SCD should explore the possibility of incorporating some form a supervi- sion to further improve performance. Although SSCS, we used pre-trained static sense embeddings without any further fine-tuning, we could have used WiC data to select the classification threshold. Dur- ing inference time, XL-LEXEME computes the average pair-wise cosine distance between the em- beddings of sentences that contain the target word (which we are interested predicting whether its meaning has changed over time), selected from each corpora. However, as already discussed in 5.2, JS divergence outperforms cosine distance for SCD. Therefore, it would be an interesting fu- ture research direction would be to incorporate the findings from unsupervised SCD to further improve performance in supervised SCD methods.
a sig-
Although models such as CMCE and EmbedLex- Change have good performance for the binary clas- sification subtask, their performance the ranking subtask is poor. Both of those methods learn static word embeddings for a target word independently from each corpus. Therefore, those methods must first learn comparable distributions before a dis- tance measure can be used to calculate a semantic change score for a target word. As explained above, CMCE learns CCA-based vector space alignments, while EmbedLexChane ity over set of fixed pivotal words selected from the two corpora. Both vector space alignments and pivot selection are error prone, and add ad- ditional noise to SCD. On the other hand, SSCS uses the same MLM and sense embeddings on both corpora when computing the sense distributions, thus obviating the need for any costly vector space alignments.
uses the cosine similar-
Both Token and Temporal Attention meth- ods require retraining transformer model (i.e. BERT models are used in the original papers). Token prepends each sentence with a times- tamp, thereby increasing the input length, which results in longer training times with transformer-
5,4 Multilingual SCD Results
To evaluate the effectiveness of word sense distributions for detecting semantic change of words in other languages, we use the 768- dimensional ARES multilingual sense embed-

---

obtained from mBERT has poor coverage for Latin. Although more language-specialised MLMs are available such as GermanBERT' LatinBERT, and SwedishBERT we must have compatible sense embeddings to compute the sense distribu- tions. Learning accurate multilingual sense embed- dings is an active research area (Rezaee et al., 2021; Upadhyay et al., 2017) on its own and is beyond the scope of this paper which focuses SCD.
0.4
Speermen
0.3
0.1
5.5 Qualitative Analysis
0.0
English
German
Latin
Swedish sh
Oosine
Canberrs
Euclidean
Figure 4 shows example sense distributions and the corresponding JS divergence scores for the words plane (a word that has changed meaning according to SemEval annotators, giving a rating of 0.882) and pin (a word that has not changed its meaning, with a rating of 0.207) from the SemEival English binary classification subtask. We see that the two distributions for plane are significantly different from each other (the second peak at sense-id 5 vs. 6, respectively in C1 and C2), as indicated by high JS divergence (i.e. 0.221). On the other hand, the sense distributions for pin are similar, resulting in a relatively smaller (i.c. 0.027) JS divergence. This result supports our claim that sense distributions provide useful clues for SCD of words.
Osabyshev
Bar-Ciitii
J5
Figure 3: Multilingual semantic change detection of SSCS for ranking task in English, German, Latin and Swedish.
dings," 10 trained using BabelNet concept ids, We use bert-base-multt icauell as the multilingual MLM in this evaluation because it is compatible with the ARES multilingual sense embeddings. For evaluations, we use the ranking subtask data in the SemEval 2020 Task for Ger- man, Swedish and Latin.
In Figure 3 we compare values obtained us- ing different distance/divergenee measures. We see that JS performs best for English, KL for German and Latin, whereas Canberra for Swedish. Overall, the divergence-bart based measures (i.e. KL and JS) report better results than distance-based measures across languages, except in Swedish. Various fac- tors affect the performance such as the coverage and sparseness of sense distributions, size of the corpora in each time period, and the number of test target words in each evaluation dataset. There- fore, it is difficult attribute the performance dif- ferences across languages purely the different distance/divergeence measures used to compare the sense distributions.
In Table 3, we show the top- and bottom-8 ranked words according their semantic change scores the SemEval English dataset. We compare the ranks assigned to words according to SSCS against the NLTK baseline (used Table 1) and DSCD (Aida and Bollcgala, 2023). From Table 3 we see that for 6 (i.e. plane, tip, graft, record, stab, head) out of the top-8 ranked words with a se- mantic change between the corpora, SSCS assigns equal or lower ranks than either of NLTK or DSCD. Moreover, see that SSCS assigns lower ranks to words that have not changed meaning across corpora.
As an error analysis, let us consider risk, which is assigned a higher rank (8) incorrectly by SSCS, despite not changing its meaning. Further inves- tigations (see Appendix D) reveal that the sense distributions for risk computed from the cor- pora are indeed very similar, except that C2 has two additional senses not present in C₁. However, those additional senses are highly similar to ones
The performance for non- non-English languages is much lower compared to that for English. This is due to three main reasons: (a) the limited sense coverage in BabelNet for non-English languages (especially Latin and Swedish in this case), (b) the accuracy of ARES sense embedding for German and Latin being lower compared to that for English, and (c) the multilingual contextualised embeddings
>Phttps://uugiingface co/ bert-base-gerrnnn-cce cased
10 htttp://sensemberrs /tr.rr/rrr/rrecceearrreee embedding. .ar.gz
/#nttps://github. com/ htw.dbbamnn/laiinbbrrt
htttps: http//ghuggggaacgeeee/e waaggiingacee..o.. bert- Bert-base-mull www.emulltiinnnlllccccm
"nttpss//huggiiggaae co/KB/

---

0.5
0.5
0.
0.4
0.4
0.3
9.3
0.
01
(a) plane in C1
(h) plane in C2
(c) pin in Ci
(d) pin in C2
Figure 4: Sense distributions of pin and plane in the corpora in SemEval 2020 Task I English dataset. In each subfigure, probability (in y-axis) is shown against the sense ids (in rraxis). The sense distributions of plane have changed across corpora, while 0.882 and 0.207, indicating that plane has changed its meaning between the two corpora, while pin has not. The JS divergence between the two sense distributions for plane is 0.221, while that for pin is 0.027.
of pin remain similar. The human ratings for plane and pin are respectively
Gold rank A
NLTK
SSCS
DSCD rank
mEval 2020 Task dataset. The experimental sults highlight the effectiveness of using word sense distribution detect semantic changes words in different languages.
Word
rank
rank
plane
d
2
15
tip prop
/
17
v
24
17
graft
4
23
4
36
record
7
14
Acknowledgements
stab
Y
11
11
11
bit
8
15
Danushka Bollegala holds concurrent appoint- ments Professor at University of Liverpool and Amazon Scholar. This paper deseribes work performed the University of Liverpool and not associated with Amazon.
head
V
10
28
multitude
30
X
26
23
35
savage
31
X
22
29
26
contemplation
32
X
13
35
37
tree
33
X
12
27
30
relationship
34
X
37
31
34
fiction
35
X
35
33
29
7 Limitations
chairman
36
X
27
34
33
risk
37
X
31
21
An important limitation in our proposed sense distribution-based SCD method is its reliance on sense labels. Sense labels could be obtained ing WSD tools (as demonstrated by the of NLTK baseline in our experiments) or by perform- ing WSD using pre-trained static sense embeddings with contextualised word embeddings, obtained from pre-trained MLMs (as demonstrated by the use of LMMS and ARES in our experiments) Even if the WSD accuracy is not high for the top- pre- dicted sense, SSCS can still accurately predict SCD because it uses the sense distribution for target word, and just the top-1 predicted sense. More- over, SSCS uses the sense distribution of target word over the entire corpus and not for a single sen- tence. Both WSD and sense embedding learning are active research topics in NLP (Bevilacqua and Navigli, 2020). We can expect the performance of WSD tools and sense embeddings improve further in the future, which will further improve the SCD accuracy of SSCS.
Spearman
1.000 0.442
0.589
0.529
Table 3: Ablation study on the top-8 semantically changed (A = v) words with the highest degree of semantic change and the bottom-8 stable words (A X) with the lowest degree of semantic change. NLTK baseline performs WSD using NLTK's WSD functional- ity and uses KL to compare sense distributions. DSCD is proposed by Aida and Bollegala (2023) and approxi- mates sibling embeddings using multivariate Gaussians. SSCS is our proposed method, which uses LMMS sense embeddings and JS the distance metric.
present C and imply the semantic invariance of risk. Explicitly incorporating sense similarity into SSCS could further improve its performance.
6 Conclusion
We proposed, SSCS, sense distribution bbasdd method for predicting the semantic change of word from one corpus to another. SSCS obtains good performance among the unsuupervised meth- ods for both binary classification and ranking sub- tasks for the English unsupervised SCD on the Se-
Although we evaluated the performance of SSCS in German, Latin and Swedish in addition to En- glish, this still a limited set of languages. How-

---

ever, conducting large-scale multilingual evalua- tion for SCD remains a formidable task due to the unavailability of human annotated semantic change scores/labels for words in many different languages. As demonstrated by the difficulties in data anno- tation tasks during the SemEval 2020 Task 1 on Unsupervised SCD (Schlechtweg et al., 2020), it is difficult to recruit native speakers for all languages of interest. Indeed for Latin it has been reported that each test word was annotated by only a single expert because was not possible to recruit native speakers via crowd-sourcing for Latin, which is not a language in active usage. Therefore, we can ex- pect similar challenges when evaluating SCD per- formance for rare or resource poor languages, with limited native speakers. Moreover, providing clear annotation guidelines for the semantic changes of a word in a corpus is difficult, especially when the semantic change happens gradually over longer period of time.
Having said that, we would like to point out that we are using pre-trained MLMs and static sense embeddings in SSCS. MLMs are known to encode unfair social biases such as gender- or race-related biases (Basta et al., 2019) 1)). Moreover, Zhou et al. (2022) showed that static sense embeddings also encode unfair social biases. Therefore, it is unclear how such biases would affect the SCD performance of SSCS. On the other hand, some gender-related words such as gay have changed their meaning over the years (e.g. offering fun and gaiety vs. someone who is sexually attracted to persons of the same sex). The ability to correctly detect such changes will be important for NLP models to make fair and unbiased decisions and generate unbiased responses when interacting with human users in real-world applications.
References
The semantic changes of words considered in SemEival 2020 Task dataset span relatively longer time periods, such as 50-200 years. Although it is possible to evaluate the performance of SCD methods for detecting semantic changes of words that happen over a longer period, it is unclear from the evaluations on this dataset whether SSCS can detect more short term semantic changes. For ex- ample, the word corona gained novel meaning in 2019 with the wide spread of COVID-19 pandemic compared to its previous meanings (e.g. sun's corona rings and a beer brand). We believe that it is important for an SCD method to accurately detect such short term semantic changes, especially when used in applications such as information retrieval, where keywords associated with user interests vary over a relatively shorter period of time (c.g. sea- sonality related queries can vary over a few weeks to few months).
Taichi Aida and Danushka Bollegala. 2023. Unsuper- vised semantic variation prediction using the distri- bution of sibling embeddings. In Proc. of the 61st Annual Meeting of the Association for Computational Linguistics (Findings of ACL 2023).
Taichi Aida, Mamoru Komachi, Toshinobu Ogiso, Hi- roya Takamura, and Daichi Mochihashi. 2021. A comprehensive analysis of PMI-based models for measuring semantic differences. In Proceedings of the 35th Pacific Asia Conference on Language, In- formation and Computation. Association for Compu- tational Lingustics, Shanghai, China, pages 21-31. https:/tcclnttioogg..or/02 wttp://0/111.....cc.....
Reem Alatrash, Dominik Schlechtweg. Jonas Kuhn, and Sabine Schulte im Walde. 2020. CCOHA: Clean corpus of historical American English. In Proceed- ings of the Twelfth Language Resources and Eval- uation Conference. European Language Resources Association, Marseille, France, pages 6958-6966. http://iclattholeyy.gg2220.... 1....
Ehsaneddin Asgari, Christoph Ringlstetter, and Hinrich Schütze. 2020. EmbLLeXhaange SemFval-2020 task I: Unsupervised embeddinn-based detection of lexical semantic changes. In Proceedings of the Fourteenth Workshop on Semantic Evaluation. Inter- national Committee for Computational Linguistics, Barcelona (online), pages 201-207.
8 Ethical Considerations
We considered the problem of SCD of words across corpora, sampled at different points in time. To evaluate our proposed method, SSCS, against pre- viously proposed methods for SCD we use the pub- licly available SemEval 2020 Task 1 datasets. We are unaware of any social biases other ethical is- sues reported regarding this dataset. Moreover, we did not collect, annotate, or distribute any datasets as part of this work. Therefore, we do not foresce any ethical concerns regarding our work.
Pierpaolo Basile, Annalina Caputo. Tommaso Caselli, Pierluigi Cassotti, and Rossella Varvara. 2020. Diacr- ita @ evalita2020: Overview of the evalita2020 di- achronic lexical semantics (diacr-ita) task. In Va- lerio Basile, Danilo Croce. Maria Di Maro, and Lucia C. Passaro, editors, Proceedings of the Sev- enth Evaluation Campaign of Natural Language Pro- cessing and Speech Tools for Italian. Final Work- shop (EVALITA 2020). CEUR Workshop Proceedings

---

(CEUR-WS org. Evaluation Campaign of Natural Language Processing and Speech Tools for Italian, EVALITA 2020; Conference date: 17-12-2020.
the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers). Asso- ciation for Computational Linguistics, Minneapolis, Minnesota, pages 4171-4186.
Christine Basta, Marta R. Costa-jussă, and Noe Casas. 2019. Evaluating the underlying gender bias in contextualized word embeddings. In Proceedings of the First Workshop on Gender Bias in Natu- ral Language Processing. Association for Compu- tational Linguistics, Florence, Italy, pages 33-39. https://do..org/100 1tt//0000.8668/00 18653/v1/W19-3000
Haim Dubossarsky, Simon Hengchen, Nina Tah- masebi, and Dominik Schlechtweg. 2019. Time- out: Temporal referencing for robust modeling of lexical semantic change. In Proceedings of the 57th Anmual Meeting of the Association for Com- putational Linguistics. Association for Computa- tional Linguistics, Florence, Italy, pages 457-470. https://do..or//10.. 10. 1/P11////1111111111
Christin Beck. 2020. DiaSense at SemEval 2onEvlll2020 task 1: Modeling sense change via pre-trained BERT embed- dings. Proceedings of the Fourteenth Workshop on Semantic Evaluation, International Committee for Computational Linguistics, Linguitts,,,,,,uusiiisssss,, Barcelona (online), pages 1ttp////...13///////.//
Christiane Fellbaum and George Miller. 1998. Word- Net: An electronic lexical database. MIT press.
50-58. https://doi.org// 00. 18533 1/0000.semeaa 1.4.
https//ieeepplore..eee.o/////o/p ora.eeo.rgg/boo//663399.
Michele Bevilacqua and Roberto Navigli. 2020. Break- ing through the 80% glass ceiling: Raising the state of the art word sense disambiguation by incorpo- rating knowledge graph information. In Proceedings of the 58th Annual Meeting of the Association for Computational I. Linguistics. Association for Compu- tational Linguistics, Online, pages 2854-2864.
Mario Giulianelli, Marco Del Tredici, and Raquel Feránndzz. 2020. Analysing lexical semantic change with contextualised word representations. In Pro- ceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics. Associa- tion for Computational Linguistics. Online, pages 3960-3973. https://doi.org/ 10. 18.5 main.365.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Associa- tion for Computational Linguistics 5:135-146.
Mario Giulianelli, Andrey Kutuzov, and Lidia Pivo- varova. 2022. Do not fire the linguist: Grammatical profiles help language models detect semantic change. In Proceedings of the 3rd Workshop Computa- rional Approaches to Historical Language Change. Association for Computational Linguistics, Dublin, Ireland, pages 54-67.
Michel Bréal. 1897. Exsai de Sémantique: (science des significations). Slatkine Reprints.
Jose Gamacho-Collader and Mohammad Taher Pilehvar. 2018. From Word To Sense Embeddings: A Survey on Vector Representations of Meaning. JAIR 63:743- 788. https://doi.org// 10.1111..11111 1613/jair... 1259..
William L. Hamilton, Jure Leskovec. and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 1501. https://doi.oo orr.//...rr/..... I0. 36653/1//P16 16-111111 141.
Pierluigi Cassotti, Lucia Siciliani, Marco DeGemmis, Giovanni Semeraro, and Pierpaolo Basile. 2023. XL- LEXEME: WiC pretrained model for -ingull LEXical sEMantic changE. In Proceedings of the st Annual Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers). Associa- tion for Computational Linguistics, Toronto, Canada, pages 1577-1585.
Renfen Hu, Shen Li, and Shichen Liang. 2019. Di- achronic sense modeling with deep contextualized word embeddings: An ecological view. In Proceed- ings the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computa- tional I. Linguistics, Florence, Italy, pages 3899-3908. https//doi.orr// 10. 18653/v 1/P19-1379.
Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014. unified model for word sense representation and disambiguation. In Proceedings of EMNLP, pages 1025-1035. htts:///clanttooog.nn//l/111111 p0..f..
Eric H Huang, Richard Socher, Christopher D Man- ning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of ACL. pages 873-882. hutps//accanthology..r//// 10922.pd
Paul Cook and Suzanne Stevenson. 2010. Automati- cally identifying changes the semantic orientation of words. In Proceedings of the Seventh International Conference Language Resources and Evaluation (L) European Language Resources Associa- tion (ELRA), Valletta, Malta.
Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2015. Sensemhed: Learning sense embeddings for word and relational simi- larity. Proceedings of ACL. pages 95-105. https//cccaathooog...g.1/-0-...0
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of

---

Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal analysis of lan- guage through neural language models. In Proceed- ings of the ACL 2014 Workshop on Language Tech- nologies and Compulational Social Science. Associa- tion for Computational Linguisties, Baltimore, MD, USA, pages 61-65. https:/doii. http://oooorg//O... 1rg/1 15/1/1141 2517.
Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose camaaho-coolados 2022. MimeIMs: Diachronic language models from Twitter. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics: System Demonstrations. Association for Computa- tional Linguisties, Dublin, Ireland, pages 251-260. https://doi.or/// http://0...r//0....o/ 1/25///
Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015, Statistically significant detec- tion of linguistic change. In WWW 2015. pages 625- 635.
Daniel Loureiro and Alipio Jorge. 2019. Language Modelling Makes Sense: Propagating Represen- tations through WordNet for Full-Coverage Word Sense Disambiguation. In Proceedings of the 57th Anmual Meeting of the Association for Computa- tional Linguistics, Florence, Italy, pages 5682-5691. https://cclatttooeg..r//////
Andrey Kutuzov and Mario Giulianelli. 2020. UiO- UvA SemEval-2020 task 1: Contextualised em- beddings for lexical semantic change detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation. International Committee for Computa- tional Linguistics. Barcelona (online), pages 126- 134. https://doi.org/ wttpg//......////1 1/22000se000 1.14.
Jean- Philippe Magué. 2005. Changements sémantiques el cognition: disférentes méthodes pour différentes échellex lemporelles. Ph.D. thesis, Université Lumière-Lyon II.
Matej Martine, Petra Kralj Novak, and Senja Pol- lak. 2020. Leveraging contextual embeddings for detecting diachronic semantic shift. In Proceed- ings of the Twelfth Language Resources and Eval- uation Conference. European Language Resources Association, Marseille, France, pages 481 1-4819. http:///eeaathooogggg///00Z0e/
Andrey Kutuzov, Lilja Ovrelid, Terrence Szymanski, and Erik Velldal. 2018. Diachronic word embeddings and semantic shifts: a survey. In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguis- ties, Santa Fe, New Mexico, USA, pages 1384-1397. http:///allaatoooooooogg/CC/--/
Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K. Gray, null null. Joseph P. Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, Steven Pinker, Mar- tin A. Nowak, and Erez Lieberman Aiden. 2011. Quantitative analysis of culture using millions of digitized books. Science 331 16--188. https://doi.org/// 126/science. 1999444
Andrey Kutuzov and Lidia Pivovarova. 2021. RuShiftE- val: a shared task on semantic shift detection for Russian. In Computational linguistics and intellec- tual technologies: P the annual conference Dialogue.
Andrey Kutuzov, Lidia Pivovarova, and Mario Giu- lianelli. 2021. Grammatical profiling for semantic change detection. In Proceedings of the 25th Confer- ence on Computational Natural Language Learning. Association for Computational Linguistics, Online, pages 423 434.
Tomas Mikolov, Kai Chen, and Jeffrey Dean. 2013. Effficinnt estimation of word representation in vec- tor space. In Proc. of International Conference on Learning Representations.
Andrey Kutuzov, Erik Velldal, and Lilja Ovrelid. 2022. Contextualized embeddings for semantic change detection: Lessons learned. In North- ern European Journal of Language Technology, Volume 8. Northern European Association of Language Technology, Copenhagen, Denmark. org/https://doi.or////0 1533.2022.3477
George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21- 24, 1993. http://wwww..www.bboo////toooggo 106 1061.pdf.
Syrielle Montariol, Matej Martine, and Lidia Pivo- varova. 2021. Scalable and interpretable seman- tic change detection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Hu- man Language Technologies. Association for Com- putational Linguistics, Online, pages 4642-4652, http://doi.or/11.1885333 http://5ww.11110000m.mmmmm...... mainl-mainn36..
Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gri- bovskaya, Devang Agrawal, Adam Liska, Tay- fun Terzi, Mai Gimenez, Cyprien de Mas- son d'Autume, Tomás Kočiský, Sebastian Ruder. Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021. Mind the gap: As- sessing temporal generalization in neural lan- guage models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Ad- vances in Neural Information Processing Systems. wdte:/foooommmdd333mttC333a
Roberto Navigli and Simone Paolo Ponzetto. 2012. Ba- beinet: The automatic construction, evaluation and application of wide-coverage multilingual semantic network, Artificial intelligence 193:217-250.

---

Arvind Neelakantan, Jeevan Shankar, Alexandre Pas- sos, and Andrew McCallum. 2014. Efficient non- parametric estimation of multiple embeddings per word in vector space. In Proceedings of EMNLP. pages 1059-1069. hupps//achathooogg..rgg/ 11 11...pdf.
David Rother, Thomas Haider, and Steffen Eger. 2020. CMCE at SemEval-2020 task 1: Clustering on man- ifolds of contextualized embeddings to detect his- torical meaning shifts. In Proceedings of the Four- leenth Workshop on Semantic Evaluation. Interna- tional Committee for Computational Linguistics, Barcelona (online), pages 187-193.
G. Palshikar. 2009. Simple algorithms for peak detec- tion in time series. In Proc. of the 1st International Conference on Advanced Data Analysis, Business Analytics and Intelligence. pages 1-13.
Ana Sabina Uban, Alina Maria Cristea, Anca Daniela Dinu. Liviu P Dinu, Simona Georgescu, and Laurentiu Zoicas. 2022. CoToHiLi at LSCDis- covery: the role of linguistic features in predict- ing semantic change. In Proceedings of the 3rd Workshop on Computational Approaches to Histor- ical Language Change. Association for Computa- tional Linguistics. Dublin, Ireland, pages 187-192. https:/dooi.or// 18653/v 1/2022..changee-111
Mohammad Taher Pilchvar and Jose Camacho-Coll Colladd 2019. WiC: the word-in carr--in-coneett dataset for evalu- ating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), Association for Computational Linguistics. Minneapolis, Minnesota, pages 22731273.
Bianca Scarlini, Tommaso Pasini, and Roberto Nav- igli. 2020a. SensEmBERT: Context-Enhancee Sense Embeddings for Multilingual Word Sense Disam- biguation. In Proceedings of AAAI. volume 34(05), pages 8758-8765.
Ondiej Pražák, Pavel Pribáñ, Stephen Taylor, and Jakub Sido. 2020. UWB at Se0Eval-2020 task I: Lexical semantic change detection. Proceedings of the Fourleenth Workshop on Semantic Evaluation. Inter- national Committee for Computational Linguistics, Barcelona (online), pages 254.
Bianca Scarlini, Tommaso Pasini, and Roberto Nav- igli. 2020b. With more contexts comes better per- formance: Contextualized sense embeddings for all-round word sense disambiguation. In Pro- ceedings of EMNLP. Online, pages 3528-3539. htaps//cccaatooooogggg//2Z2222e
285/.
Joseph Reisinger and Raymond Mooney. 2010. Multi- prototype vector-space models of word mean- ing. In Proceedings of NAACL. pages 109-117. https://cclanhhlogg.oooo /rrpooooggooo///N-111110111111
Dominik Schlechtweg. Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tah- masebi. 2020. SemEval-2020 task 1: Unsuper- vised lexical semantic change detection. In Pro- ceedings of the Fourteenth Workshop on Semantic Evaluation. International Committee for Computa- tional I. Linguistics, Barcelona (online), pages 1-23. https://doi. org0 18653/v 111.11661111222.seeeeeeee 1/2020.semevall-L....
Kiamehr Rezaee, Daniel Loureiro, Jose Camacho- Collados, and Mohammad Taher Pilehvar. 2021. On the lross-lngual transferability of contextualized sense embeddings. In Proceedings of the 1st Work- shop on Multilingual Representation Learning. As- sociation for Computational Linguistics, Punta Cana, Dominican Republic, pages 107-115.
Milan Straka and Jana Straková. 2017. Tokenizing, POS tagging, lemmatizing and parsing UD 2.0 with UDPipe. In Proceedings of the CONLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computa- tional Linguisties, Vancouver, Canada, pages 88-99. https://doi.or// 10tp//....../1///../eo
Guy D. Rosin, Ido Guy, and Kira Radinsky. 2022. Time masking for temporal language models. In Proceedings of the Fifteenth ACM Interna- tional Conference on Web Search and Data Min- ing. Association for Computing Machinery. New York, NY. USA, WSDM '22 pages 833-841. http://...rrrg/01.111444448888..
Zhaochen Su, Zecheng Tang. Xinyan Guan, Lijun Wu. Min Zhang, and Juntao Li. 2022. Improving tem- poral generalization of pre-trained language models with lexical semantic change. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. Abu Dhabi, United Arab Emirates, pages 6380-6393. https://claanholloy wrwp:oogggorg//2..mmm.... main.428.
Guy D. Rosin and Kira Radinsky. 2022. Tem- poral attention for language models. In Find- ings of the Association for Computational Linguis- fies: NAACL 2022. Association for Computational Linguistics, Seattle, United States, pages 1498- 1508. https://doi.org/ 1650010065////2222..mmm 112
Nina Tahmasebi, Lars Borina, and Adam Jatowtb. 2021. Survey of computational approaches to lexical se- mantic change detection. Computational approaches to semantic change 6:1.
Sascha Rothe and Hinrich Schütze. 2015. Au- toExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes. In Pro- ceedings of ACL-IJCNLP. pages 1793-1803, http:///alathhooggggggss--/1///
Elizabeth Closs Traugott and Richard B. Dasher. 2001. Prior and current work on seman- tic change, Cambridge University Press, pages
pages

---

51-104. Cambridge Studies in Linguistics.
Bray-Curtis Distance:
Dieg, (pi(2) - ma(a)]
d(pi.p)) P2) Drezn [pi(z) + +
(6)
Stephen Ullmann. 1959. The Principles of Semantics. Blackwell.
Canberra Distance:
Shyam Upadhyay, Kai-Wei Chang, Matt Taddy, Adam Kalai, and James Zou. 2017. Beyond bilingual: Multi-sense word embeddings using multilingual con- text. In Proceedings of the 2nd Workshop on Repre- sentation Learning for NLP. Association for Compu- tational Linguistics, Vancouver, Canada, pages 101- 110.
[1(()) P2(=] [pi(3) + pz(z)
d(r. P2)
(7)
Chebyshev Distance:
d(pı. p2) S22w
max (pi(s) pz(z)
(8)
Luke Vilnis and Andrew McCallum. 2013. Word repre- sentations via gaussian embedding. In Proceedings of the 3rd International Conference on Learning Rep- resentations. San Diego, CA, USA.
Cosine Distance:
Exez, (1(2)pa(2)
=1
v Er p1(z)² / P(())
Zijun Yao, Yifan Sun, Weicong Ding, Nikhil Rao, and Hui Xiong. 2018. Dynamic word embeddings for evolving semantic dis- covery, In WSDM 2018. pages 673-681.
Euclidean Distance:
d(pi.p2) =
Σ [(pi(z) p=(=))²
(10)
Arda Yüksel, Berke Uğurlu, and Aykut Kog. 2021. Semantic change detection with gaussian word embeddings. IEEE/ACM Transactions on Audio, Speech, and Language Processing 29:3349-3361. https://doi.or//10. 109/TASLP.2021.3111
B Experimental Settings
Statistics the SemEval 2020 Task 1 Unsuper- vised Semantic Change Detection Dataset shown in Table 4. Thresholds found with Bayesian op- timisation for the classification task are listed in Table 5. We used random seed of 42 while perform- ing the Bayesian optimisation. From Table 5 we that the classification thresholds that learnt for different divergence/distance metries are largely different.
Yi Zhou, Masahiro Kaneko, and Danushka Bollegala. 2022. Sense embeddings are also biased evaluating social biases in static and contextualised sense em- beddings. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, pages 1924-1935.
A Distance and Divergence Measures
C Previously Proposed SCD Methods
We describe the distance and divergence measures d used in our experiments to compare the sense distributions p1(2) p(zw (w,C)) C1) and (2(2) p(zul N0, C2) computed respectively from C, and C2.
We compare SSCS against the following prior SCD methods on the SemEval-2020 Task English data.
BERT Time Tokens + Cosine Rosin et al. (2022) performed fine-tuning of the published pre- trained BERT-base models using time tokens. They add a time token (e.g. <2023>) the beginning of sentence. In the fine-tuning step, the mod- els use two types masked language modelling objectives: 1) predicting the masked time tokens from given contexts, and 2) predicting the masked tokens from given contexts with time tokens. They make predictions with the average distance of the target token probabilities or the cosine distance of (4) the average sibling embeddings. According to their results, the cosine distance achieves better perfor- mance than the average distance of the probabilities. Therefore, we use cosine distance as the distance (5) metric for this method.
Kullback-Liebles (KL) Divergence:
(1()
KL-pilpp2 (mi
p2(2)
2C2w
Jensen-Shannon (JS) Divergence:
JS lpp))
1 KL (pillg) 111)) KL (pzll4)
2
Here,
((=) (p((2) + +z(x))

---

Language
Time Period
#Targets
#Sentences
#Tokens
#Types
English
1810-1860
254k
6.5M
87k
37
1960-2010
354k
6.7M
150k
German
1800-1899
2.6M
70.2M
1.0M
48
1946-1990
3.5M
72.3M
2.3M
Latin
B.C. 200-0
96k
1.7M
65k
40
0-2000
463k
9.4M
253k
Table 4: Statistics of the SemEval 2020 Task
Unsupervised Semantic Change Detection Dataset.
risk 1
risk 2
0.35
0.30
0.3
0.25
0.20
0.15
0.10
0.05
(a) risk in Ci
(b) risk in C2
Figure 5: Sense distributions computed for the word risk from Ci and C2.
BERT + APD Kutuzov and Giulianelli (2020) report that the average pairwise cosine distance outperforms the cosine distance. Based on this insight, Aida and Bollegala (2023) evaluate the performance of BERT Tokens Cosine with the average pairwise cosine distance computed using pre-trained BERT-base the MLM.
pus. They use the cosine distance following their earlier work (Rosin et al., 2022).
BERT Time Tokens + Temporal Attention Because their two proposed methods (fine-tuning with time tokens and temporal attention) are inde- pendent, Rosin and Radinsky (2022) proposed to use them simultancously. They also use the cosine distance for prediction and report that this model achieves current of-te-of-the-rr performance in mantic change detection.
BERT+DSCD Aida and Bollegala (2023) pro- posed Distribution-barn Semantic Change De- tection (DSCD), considering distributions of sib- ling embeddings (sibling distribution). During pre- diction, they sample an equal number of target word vectors from the sibling distribution (approx- imated by multivariate diagonal Gaussian) for each time period and calculate the average distance. They report that the Chebyshev distance function achieves the best performance.
Gaussian Embeddings: Yüksel et al. (2021) ex- tend word2vec to create Gaussian embeddings (Vil- nis and McCallum, 2015) for target words indepen- dently in each corpus. They then learn mapping between the two vector spaces for computing a semantic change score.
Rother et al. (2020) proposed Cluster- ing Manifolds of Contextualised Embeddings (CMCE) where they use mBERT embeddings with dimensionality reduction to represent target words, and then apply clustering algorithms to find the different sense clusters. CMCE is the current SoTA
CMCE:
Temporal Attention Rosin and Radinsky (2022) proposed temporal attention mechanism, where they add a trainable temporal attention ma- trix to the pretrained BERT models. Subsequently, additional training is performed on the target cor-

---

D Error Analysis
Metric
Threshold
Cosine
0.132
As concrete example of computing semantic change scores using sense distributions where SSCS assigned a low JS divergence value using LMMS incorrectly its meaning, we consider the word risk. The sense distributions, p(=w) C1) and p(=ulw, C2), of risk the two corpora, respectively C₁ and C2 are shown in Figure The word risk is consid- ered as a noun and the annotator-assigned semantic change score is O, indicating that it has not changed meaning between the two corpora. Looking at the sense distributions in Figure 5, see that they are al- most similar with two peaks at sense-ids 2 and 4. However, two additional senses, correspond- ing sense-id Trisk%1:07:005% the probability of being exposed to an infectious agent) and sense-id 5 ( 1:07:01::' expose to chance of loss or damage) can be observed Figure 5b in C2. According to the sense definitions in the WordNet for those two senses, they are very similar (i.e. expressing different types of risks), which could be considered be case where the meaning of risk remains largely the same in the two corpora. Because of this reason the JS diver- gence score between the two distributions tends to be higher at 0.2139, giving it a lower rank (rank 8 among 37 words in Table 3. indicating that risk has changed its meaning). This is typical example where although word might take different senses in different corpora (or within different sentences in the same corpora), some of those senses could be highly similar and could be mapped the same meaning for the purpose of SCD,
Chebyshev
146.808
Canberra
0.829
Bray- Bray-Cuutis
0.046
word that had not changed
Euclidean
63.024
IS
0.0285
KL
0.024
Cosine
0.3238
SemEval 2020 Task dataset,
Chebyshev
212.995
Canberra
0.901
ARES
Bray-Curte
0.036
Euclidean
509.011
JS
0.080
KLL
0.108
Cosine
0.052
risk of exposure,
Chebyshev
206.235
LMMS
Canberra
0
Bray- Prryi-Cutis
0.103
Fuclidean
331.680
JS
0.117
KL
0.065
Table 5: Thresholds found with Bayesian optimisation for the classification task on English datasets.
for the binary classification task.
EmbedLexChange: Asgari et al. (2020) used fasttext (Bojanowski et al.. 2017) to create word embeddings from each corpora separately, and mea- sure the cosine similarity between a word and a selected fixed set of pivotal words to represent word in corpus using the distribution over those pivots. They used KL divergence to measure the similarity between the two distributions associated with target word to compute semantic change score.
UWB: Pražák et al. (2020) learnt separate word embeddings for target word from each cor- pora and then use Canonical Correlation Analysis (CCA) align the two vector spaces. They use the average cosine similarity between the beddings over all target words the threshold for predicting semantic changes of words. UWB was ranked 1st for the binary classification subtask at the official SemEval 2020 Task competition.


Reviews:
[{"review_id": "jB9oDih6t0", "reviewer": "Reviewer_LWUC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**RR1:** *The reliance on sense labels means that the method might be less useful in specific domains, or with new words and senses such as the case of novel slang terms on social media.*\n\n**Response:** Please note that we have discussed this point already in the Limitations section in the paper. Research on WSD is active and on-going and more accurate WSD tools are being developed for low-resource languages and specialised domains [Gogoi and Baruah 2022, Zhang+2022]. \n\nOn the other hand, please note that our proposed method does not depend on actual sense labels but can operate using \"sense clusters\". Specifically, provided with a method that can \"discriminate\" different senses of a word (\"disambiguation\" is not required), we can represent a word by a distribution of sense cluster ids. Moreover, sense clusters could be replaced by “topics” that are automatically generated via a topic modelling method such as LDA [Blei+ 2003], in a “sense induction” manner, thus obviating the need for sense labels at all. The feasibility of these extensions are deferred to future work, which would be inspired by the current paper.\n\n**RR2:** *Results are only shown for a rather large time gap*\n\n**Response:**  Please note that this is not a limitation of our proposed method but rather a limitation of benchmark datasets that would enable us to test the proposed method (as well as all prior work on lexical semantic change detection) for shorter time durations. The SemEval 2020 Task 1 dataset that we use in our experiments (and considered as the standard evaluation benchmark for lexical semantic change detection in much prior work) lists words that change their meaning over a relatively longer period of time.\n\n**RR3:** *An elaborate chain of resources is necessary to carry out this task*\n\n**Response:** Although the proposed method does depend on WSD tools and contextualised word embeddings obtained from MLMs, we disagree that these are hard-to-find resources in NLP nowadays. For example, WSD tools have been developed for many languages [Elayeb 2019, Orlando+2021, Zhang+ 2022, Lendvai 2022, Su+ 2022] and pretrained monolingual and multilingual MLMs are available [Liu+ 2019, Devlin+ 2019, Lan+ 2020, Conneau+ 2020, Barbieri+ 2022]. Moreover, the NLP community is actively working on WSD and MLMs, which means that the availability and quality of these resources will further improve in the future, thus having a positive domino effect on the performance of our proposed method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper contributes a methodological innovation in the area of semantic change detection and beats the state of the art at the time of this paper's submission. It is demonstrated for several languages, and it is methodologically thorough, trying different sense embeddings and taking the top-k senses from the classification rather than just one.\n\nThe method is relatively transparent due to its reliance on word senses - an interested scholar can easily inspect which parts (senses) of the distribution have changed, as is also demonstrated in the paper.", "reasons_to_reject": "The reliance on sense labels means that the method might be less useful in specific domains, or with new words and senses such as the case of novel slang terms on social media.\n\nResults are only shown for a rather large time gap (different centuries). This may limit the practical significance of the work, as the authors note, and it is not clear that the question of temporal generalization raised in the introduction can be addressed.\n\nA rather elaborate chain of resources is necessary to carry out this task, limiting its applicability to under-resourced languages: there should be a reliable and broad sense inventory, pretrained sense embeddings and contextualized embeddings (though apparently just using mBERT performs well).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper tackles the task of semantic change detection (by way of the standard SemEval 2020 Task 1). It introduces a new method that involves word sense annotation, and then comparing the distributions of senses of target word types between corpora from different time periods. The word sense annotation task is done by using reference sense embeddings and comparing contextual embeddings from the corpus to these sense embeddings. Various different distance and divergence measures are tried to compare the distributions, and different sense embeddings are tried. The method beat the state-of-the-art result for this task, and the authors compare the metrics to various other recent approaches to this task. The task is done for English, German and Latin.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The author names in the reference to Tahmasebi et al. (2021) are misspelled."}, "extra_text": {}}, {"review_id": "KqMLueqWdD", "reviewer": "Reviewer_jdV6", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**RR1:** *The paper's novelty appears to be somewhat limited. *\n\n**Response:** We respectfully disagree. As you have already stated in the reasons for acceptance our work is a pioneering effort in incorporating sense embeddings for semantic change detection. Moreover, the proposed method already obtains SoTA for English on SemEval 2020 Task 1 benchmark, although not for other languages. Nevertheless, according to ACL reviewer guidelines [https://2023.aclweb.org/blog/review-acl23/], papers should not be rejected purely based on not being able to obtain SoTA. \n\n**RR2:** *Unsupervised clustering methods for distinguishing word meanings already exists, thus this paper lacks novelty*\n\n**Response:** Please note that “distinguishing word meanings” and “measuring semantic changes of words” are related but vastly different tasks. In the latter (which this paper is about), word sense information has never been used before as a signal, where the novelty of our work lies upon. On the other hand, unsupervised sense induction methods can be further helpful to improve the performance of our proposed method especially when human-created sense inventories are not available for a language of interest. \n\n**RR3:** *Exploring a qualitative comparison between the sense identification made by SSCS and other (un)supervised clustering methods could be extremely valuable.*\n\n**Response:** We agree it would be interesting to see whether we can use unsupervised clustering methods for sense induction (particularly useful when human-compiled sense inventories do not exist for learning sense embeddings for resource poor languages) with the proposed method. However, this is beyond the scope of the current paper and better done in an extended journal version of this 8 page conference paper because our main claim/contribution in this paper is “given a sense embedding, how to compute a semantic change detection score”.\n\n**RR3:** *The abstract states that the method is computationally efficient, but there is no reference in the paper to clarify this statement and demonstrate this feature compared to other approaches.*\n\n**Response:** As stated in Section 3.5 (Comparisons against SoTA), our proposed method does not require training transformer models with specialised temporal attention mechanisms [Rosin and Radinsky 2022] nor require fine-tuning existing MLMs on corpora sampled from the time points [Kuzutov and Guilianelli 2020]. For these reasons, we consider our proposed method to be computationally efficient compared to prior work on semantic change detection.\n\n**RR4:** *The abstract states that the method significantly outperforms state-of-the-art (SOTA) approaches, but it only performs better in English, specifically for the mentioned work, and the improvement is of a small magnitude, i.e. 0.041.*\n\n**Response:** Although the improvement might appear to be small,  the proposed method obtains SoTA for English on SemEval 2020 Task 1 dataset. This should not be used as a reason for rejection according to the ACL reviewer guidelines.\n\n**RR5:** *A comparison against previously proposed SCD methods (as presented in Table 2 for English) should also be provided for the other languages considered.*\n\n**Response:**  This information is already available in prior work, and we can easily include such a Table with the extra page allowed for the camera ready version if accepted.\n\n**RR6:** *The author should provide a more thorough explanation of the intuition behind conducting experiments with various distance/divergence measures.*\n\n**Response:** Different divergence and distance measures have been used already in much prior work on semantic change detection. As shown by Aida and Bollegala (ACL 2023), different distance/divergence measures perform differently under different settings (e.g. MLMs, languages, frequency of the words etc.). Therefore, we consider it would be both important and informative for the readers to learn about the performance of our proposed method when used with different divergence/distance measures. This is our motivation to experiment with different divergence/distance measures in Section 5.4 (Multilingual analysis). We will include this motivation in the final version of this paper.\n\n**Suggestion:** *Evaluations on Spanish and Russian…*\n\n**Response:** Thank you for the suggestions on the evaluation for Spanish and Russian. Although this would be beyond the scope of the conference version of this paper, we plan to conduct evaluations on these languages in an extended journal version to further evaluate the multilingual applicability of the proposed method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is noteworthy for being among the first works to propose the use of Sense Embeddings derived from pretrained contextualized model for detecting lexical semantic changes. The paper is well-written and effectively presents its topic in a clear and understandable manner. The methodology and its underlying setting are clearly presented, the technical quality is appropriate.", "reasons_to_reject": "The paper's novelty appears to be somewhat limited. One compelling reason to consider this paper is the authors' claim of outperforming existing methods (specifically for English). However, the paper's comparison isn't exhaustive, as there are other state-of-the-art approaches that demonstrate superior performance. For instance, when examining the ranking task for the SemEval English dataset:\n- [1] presents results indicating that the smaller the model, the better the performance. In particular, [1] reports scores of .520 with BERT-base, .589 with BERT-small, and .627 with BERT-tiny (thus, the highest result is .627, rather than .520 - as indicated in the proposed paper).\n- [2] and [3] report a score of .605 using ELMo.\n- [11] reports .757 (even if it's supervised)\n- Additionally, other works, although not referenced in this paper, exhibit comparable results such as .571 [4] and .512 [5]. While SSCS achieved a sligltly superior score (i.e. 589), it could be argued that for a relatively small set of target words like in the SemEval English dataset, such marginal differences hold negligible significance.\nFor a comprehensive comparison over different languages, I recommend referring to [6]. \n\nThe major novelty of the paper is the use of sense embeddings in semantic change detection. Instead, it's worth noting that various unsupervised methods already exist, focusing on distinguishing word meanings (e.g. by Unsupervised clustering such as [5] and [7]; or by the use of Supervision through word sense induction [8], or Gloss-based WSD systems [9]) that calculate semantic changes by measuring divergence in sense distributions (typically JSD).\n\nIntroducing SSCS would be more valuable if it leads to improved results. While it's true, as mentioned in the paper, that unsupervised methods based on embedding clustering are not tied to specific sense inventories, it's important to recognize that such sense inventories might not be widely available, particularly for languages with limited resources. This is especially relevant in scenarios involving Historical texts, like those found in the SemEval datasets.\n\nExploring a qualitative comparison between the sense identification made by SSCS and other (un)supervised clustering methods could be extremely valuable. This comparison could serve to clearly highlight the superiority of the SSCS approach and sense embeddings in the realm of Lexical Semantic Change Detection by using word embeddings.\n\nOther minor points:\n- The abstract states that the method is computationally efficient, but there is no reference in the paper to clarify this statement and demonstrate this feature compared to other approaches.\n- The abstract states that the method significantly outperforms state-of-the-art (SOTA) approaches, but it only performs better in English, specifically for the mentioned work, and the improvement is of a small magnitude, i.e. 0.041.\n- A comparison against previously proposed SCD methods (as presented in Table 2 for English) should also be provided for the other languages considered.\n- The author should provide a more thorough explanation of the intuition behind conducting experiments with various distance/divergence measures.\n- The authors claim that the performance in German and Latin is much lower compared to that in English, primarily due to the limited sense coverage in BabelNet for non-English languages. Given that the zero-shot cross-lingual transferability of contextualized embedding models has recently proven to be effective in Semantic Change Detection [9,10], it would be beneficial for the authors to further investigate this statement. For instance, they could explore finetuning the model or comparing their approach to other works, taking into consideration benchmarks for languages like Spanish and Russian\n\nReferences:\n- [1] Guy D. Rosin and Kira Radinsky. 2022. Temporal Attention for Language Models. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1498–1508, Seattle, United States. Association for Computational Linguistics.\n- [2] Andrey Kutuzov, Erik Velldal, and Lilja Øvrelid. 2022. Contextualized embeddings for semantic change detection: Lessons learned. In Northern European Journal of Language Technology, Volume 8, Copenhagen, Denmark. Northern European Association of Language Technology.\n- [3] Andrey Kutuzov and Mario Giulianelli. 2020. UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 126–134, Barcelona (online). International Committee for Computational Linguistics.\n- [4] Severin Laicher, Sinan Kurtyigit, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2021. Explaining and Improving BERT Performance on Lexical Semantic Change Detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 192–202, Online. Association for Computational Linguistics.\n- [5] Francesco Periti, Alfio Ferrara, Stefano Montanelli, and Martin Ruskov. 2022. What is Done is Done: an Incremental Approach to Semantic Shift Detection. In Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change, pages 33–43, Dublin, Ireland. Association for Computational Linguistics.\n- [6] Montanelli, Stefano, and Francesco Periti. \"A Survey on Contextualised Semantic Shift Detection.\" arXiv preprint arXiv:2304.01666 (2023).\n- [7] Matej Martinc, Syrielle Montariol, Elaine Zosa, and Lidia Pivovarova. 2020. Capturing Evolution in Word Usage: Just Add More Clusters? In Companion Proceedings of the Web Conference 2020 (WWW '20). Association for Computing Machinery, New York, NY, USA, 343–349. https://doi.org/10.1145/3366424.3382186\n- [8] Renfen Hu, Shen Li, and Shichen Liang. 2019. Diachronic Sense Modeling with Deep Contextualized Word Embeddings: An Ecological View. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3899–3908, Florence, Italy. Association for Computational Linguistics.\n- [9] Maxim Rachinskiy and Nikolay Arefyev. 2022. GlossReader at LSCDiscovery: Train to Select a Proper Gloss in English – Discover Lexical Semantic Change in Spanish. In Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change, pages 198–203, Dublin, Ireland. Association for Computational Linguistics.\n- [10] Rachinskiy, M., & Arefyev, N. (2021). Zeroshot crosslingual transfer of a gloss language model for semantic change detection. In Computational linguistics and intellectual technologies: Papers from the annual conference Dialogue (Vol. 20, pp. 578-586).\n- [11] Pierluigi Cassotti, Lucia Siciliani, Marco DeGemmis, Giovanni Semeraro, and Pierpaolo Basile. 2023. XL-LEXEME: WiC Pretrained Model for Cross-Lingual LEXical sEMantic changE. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1577–1585, Toronto, Canada. Association for Computational Linguistics.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a novel approach to Lexical Semantic Change Detection called Sense-based Semantic Change Score (SSCS). This approach leverages sense embedding representations derived from pretrained models to differentiate between the multiple meanings a word conveys in two different corpora. Then, it quantifies the semantic change of the word by computing the divergence/distance in the time-specific sense distributions. The reported results indicate that SSCS outperform other state-of-the-art work in detecting semantic change over the SemEval 2020 Task 1 benchmark.", "excitement": 1, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Andrey Kutuzov, Erik Velldal, and Lilja Øvrelid. 2022. Contextualized embeddings for semantic change detection: Lessons learned. In Northern European Journal of Language Technology, Volume 8, Copenhagen, Denmark. Northern European Association of Language Technology.\n- Severin Laicher, Sinan Kurtyigit, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2021. Explaining and Improving BERT Performance on Lexical Semantic Change Detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 192–202, Online. Association for Computational Linguistics.\n- Francesco Periti, Alfio Ferrara, Stefano Montanelli, and Martin Ruskov. 2022. What is Done is Done: an Incremental Approach to Semantic Shift Detection. In Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change, pages 33–43, Dublin, Ireland. Association for Computational Linguistics.\n- Montanelli, Stefano, and Francesco Periti. \"A Survey on Contextualised Semantic Shift Detection.\" arXiv preprint arXiv:2304.01666 (2023).\n- Matej Martinc, Syrielle Montariol, Elaine Zosa, and Lidia Pivovarova. 2020. Capturing Evolution in Word Usage: Just Add More Clusters? In Companion Proceedings of the Web Conference 2020 (WWW '20). Association for Computing Machinery, New York, NY, USA, 343–349. https://doi.org/10.1145/3366424.3382186\n- Maxim Rachinskiy and Nikolay Arefyev. 2022. GlossReader at LSCDiscovery: Train to Select a Proper Gloss in English – Discover Lexical Semantic Change in Spanish. In Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change, pages 198–203, Dublin, Ireland. Association for Computational Linguistics.\n- Rachinskiy, M., & Arefyev, N. (2021). Zeroshot crosslingual transfer of a gloss language model for semantic change detection. In Computational linguistics and intellectual technologies: Papers from the annual conference Dialogue (Vol. 20, pp. 578-586).", "Typos_Grammar_Style_and_Presentation_Improvements": "- row 077: temporal shift -> I would like to suggest adopting a more widely established terminology, as seen in prior research, such as \"lexical semantic change\" or \"semantic shift\".\n- row 135: two groups: (a) methods that compare word/context clusters, and (b) methods that compare embeddings of the target words computed from different corpora sampled at different time periods -> I would like to suggest adopting a more widely established terminology, as seen in prior research, such as sense-based and form-based approaches\n- row 500: esbalishing -> establishing\n- row 512: measures -> measure\n- row 518: adds -> add"}}, {"review_id": "yPpaeItqgR", "reviewer": "Reviewer_dbcW", "summary": "", "questions": "A) Why was Swedish (SemEval 2020 Task 1) omitted from the experiments?\nB) Why did you not use the RuShiftEval dataset in the evaluation, allowing you to evaluate your model against the similar \"GlossReader\" model?\nC) Why have you not tried state-of-the-art WSD models, such as ESC? (ESC: Redesigning WSD with extractive sense comprehension)", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**RR1:** *The use of different measures, such as cosine distance and Jensen-Shannon Divergence (JSD), is not adequately justified. Without a clear explanation of the advantages brought by each measure, the experiments appear arbitrary, resembling a brute-force approach to improve results rather than a well-reasoned methodology. *\n\n**Response:** Different divergence and distance measures have been used already in much prior work on semantic change detection. As shown by Aida and Bollegala (ACL 2023), different distance/divergence measures perform differently under different settings (e.g. MLMs, languages, frequency of the words etc.). Therefore, we consider it would be both important and informative for the readers to learn about the performance of our proposed method when used with different divergence/distance measures. This is our motivation to experiment with different divergence/distance measures in Section 5.4 (Multilingual analysis). We will include this motivation in the final version of this paper.\n\n**RR2:** Moreover, the rationale behind using Bayesian Optimization needs to be better-supported, especially considering that the threshold search problem is relatively simple and involves only one dimension.\n\n**Response:** Bayesian optimisation is a simple yet highly efficient technique for estimating thresholds for classification tasks as popularly used in NLP [https://aclanthology.org/2023.findings-acl.550/] For example, unlike linear search, which requires us to pre-define the range and the step for increment during the search, Bayesian optimisation takes care of all these settings. \n\n**Q1:** *Why was Swedish (SemEval 2020 Task 1) omitted from the experiments?*\n\n**Response:**  We could not find publicly available pretrained sense embeddings for Swedish, which is the reason why we could not evaluate for Swedish.\n\n**Q2:** *Why did you not use the RuShiftEval dataset in the evaluation, allowing you to evaluate your model against the similar \"GlossReader\" model?*\n\n**Response:** We were not aware of this dataset at the time of submission because it has not been used by any prior work on SemEval 2020 Task 1 lexical semantic change detection.  Moreover, this dataset is limited to the Russian language. However, we can easily evaluate our proposed method for Russian semantic change detection using this dataset in the final version, which should further strengthen the multilingual results in this paper.\n\n**Q3:**  *Why have you not tried state-of-the-art WSD models, such as ESC? (ESC: Redesigning WSD with extractive sense comprehension)*\n\n**Response:** Please note that our proposed method does not require WSD tools. It uses only pretrained sense embeddings. WSD is used only as a baseline in the evaluations. One can use any WSD tool, including ESC, for this purpose.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a significant contribution by introducing the application of Word Sense Disambiguation (WSD) tools in the diachronic domain for English, Latin, and German languages. This novel approach opens up exciting avenues for studying sense-aware semantic changes over time, largely unexplored in the literature. The qualitative analysis provided in the paper offers fresh insights and reveals different perspectives from existing research, highlighting the uniqueness of the proposed method. While the paper acknowledges the potential of explainability offered by WSD tools, further exploring this aspect could lead to a more in-depth understanding of the results.", "reasons_to_reject": "The use of different measures, such as cosine distance and Jensen-Shannon Divergence (JSD), is not adequately justified. Without a clear explanation of the advantages brought by each measure, the experiments appear arbitrary, resembling a brute-force approach to improve results rather than a well-reasoned methodology. Additionally, the paper lacks essential references to the model with state-of-the-art results, and other related works still need to be included (See Missing References Section). Moreover, the rationale behind using Bayesian Optimization needs to be better-supported, especially considering that the threshold search problem is relatively simple and involves only one dimension. The paper should have emphasized and highlighted the results obtained maximizing the accuracy, ensuring a fair and comprehensive comparison with other approaches.", "questions_to_authors": "A) Why was Swedish (SemEval 2020 Task 1) omitted from the experiments?\nB) Why did you not use the RuShiftEval dataset in the evaluation, allowing you to evaluate your model against the similar \"GlossReader\" model?\nC) Why have you not tried state-of-the-art WSD models, such as ESC? (ESC: Redesigning WSD with extractive sense comprehension)", "paper_topic_and_main_contributions": "This paper introduces the Sense-based Semantic Change Score (SSCS), an unsupervised method for detecting semantic changes in words over time. By using pre-trained static sense embeddings to annotate occurrences of target words with sense labels and computing their distributions in different corpora, SSCS quantifies semantic change through divergence or distance measures. The sense embeddings are obtained using LMMS and ARES pre-trained embeddings for the English monolingual and ARES multilingual pre-trained embeddings for German and Latin.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Row 122: Typo in the Tahmasebi reference Tahmasebia et al. 2021 -> Tahmasebi et al. 2021"}, "extra_text": {"Missing_References": "Related Works:\nRachinskiy, M., & Arefyev, N. (2021). Zeroshot crosslingual transfer of a gloss language model for semantic change detection. In Computational linguistics and intellectual technologies: Papers from the annual conference Dialogue (Vol. 20, pp. 578-586).\n\nMaxim Rachinskiy and Nikolay Arefyev. 2021. GlossReader at SemEval-2021 Task 2: Reading Definitions Improves Contextualized Word Embeddings. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 756–762, Online. Association for Computational Linguistics.\n\nSOTA Results:\nPierluigi Cassotti, Lucia Siciliani, Marco DeGemmis, Giovanni Semeraro, and Pierpaolo Basile. 2023. XL-LEXEME: WiC Pretrained Model for Cross-Lingual LEXical sEMantic changE. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1577–1585, Toronto, Canada. Association for Computational Linguistics."}}]

Author rebuttal:

Meta-review: The content and results are interesting but there appear to be many missing references.
Decision: Accept-Findings