$\textit{Swap and Predict}$ -- Predicting the Semantic Changes in Words across Corpora by Context Swapping
Authors: Taichi Aida
Danushka Bollegala
Venue: EMNLP 2023 Findings (2023)
Keywords: Computational Semantics
Contextualised Word Embeddings
Semantic Change Detection
TL;DR: We propose a context swapping-based method for semantic change detection.

Abstract: Meanings of words change over time and across domains.
Detecting the semantic changes of words is an important task for various NLP applications that must make time-sensitive predictions.
We consider the problem of predicting whether a given target word, $w$, changes its meaning between two different text corpora, $\mathcal{C}_1$ and $\mathcal{C}_2$.
For this purpose, we propose $\textit{Swapping-based Semantic Change Detection}$ (SSCD), an unsupervised method that randomly swaps contexts between $\mathcal{C}_1$ and $\mathcal{C}_2$ where $w$ occurs.
We then look at the distribution of contextualised word embeddings of $w$, obtained from a pretrained masked language model (MLM), representing the meaning of $w$ in its occurrence contexts in $\mathcal{C}_1$ and $\mathcal{C}_2$.
Intuitively, if the meaning of $w$ does not change between $\mathcal{C}_1$ and $\mathcal{C}_2$, we would expect the distributions of contextualised word embeddings of $w$ to remain the same before and after this random swapping process.
Despite its simplicity, we demonstrate that even by using pretrained MLMs without any fine-tuning, our proposed context swapping method accurately predicts the semantic changes of words in four languages (English, German, Swedish, and Latin) and across different time spans (over 50 years and about five years).
Moreover, our method achieves significant performance improvements compared to strong baselines for the English semantic change prediction task. Source code is available at https://github.com/a1da4/svp-swap .

Swap and Predict - Predicting the Semantic Changes in Words across Corpora by Context Swapping
Taichi Aida
Danushka Bollegala
Tokyo Metropolitan University aida-taichi@ed. tmu.
Amazon, University of Liverpool danushka@liverpooll. ac uk
Abstract
semantic change NO
semantic change YES
Corpur
Corpus
Corpue
Corpus?
Mcanings of words change over time and across domains. Detecting the semantic changes of words is important task for various NLP ap- plications that must make time-sensitive predic- tions. We consider the problem of predicting whether a given target word, changes its meaning between two different text corpora, C1 and C2. For this purpose, we propose Swapping- based Semantic Change Detection (SSCD), an unsupervised method that randomly swaps con- texts between C1 and C2 where occurs. We then look at the distribution of contextualised word embeddings of w, obtained from a pre- trained masked language model (MLM), rep- resenting the meaning of w in its occurrence contexts in and Cz. Intuitively, if the mean- ing of v does not change between C1 and C2, we would expect the distributions of contex- tualised word embeddings of remain the same before and after this random swapping process. Despite its simplicity, we demonstrate that by using pretrained MLMs without any fine-tuning, our proposed context swap- ping method accurately predicts the semantic changes of words in four languages (English, German, Swedish, and Latin) and across dif- ferent time spans (over 50 years and about five years). Moreover, our method achieves sig- nificant performance improvements compared strong baselines for the English semantic change prediction task.'
Dg
Dg
1,5wap
Dg
Dy momp
Dg
Figure Overview of SSCD. Left: The contextualised word embedding distributions, 1), and 1)2 of a word which has not changed its meaning between the two corpora. Two random samples of equal number of sen- tences containing the target word, 8 and 82, are taken respectively from D; and D: and swapped between the corpora. Here, embedding distributions after swapping (í.e. Diswap and D2 swap) are similar those before, thus preserv- ing the distance between distributions. Right: For word that has different meanings the two corpora, swapping process pushes both distributions become similar, thus reducing the distance between the swapped versions smaller to that between the original ones.
that the contextualised word
linguistics, lexicography, sociology, and informa- tion retrieval (Traugott and Dasher, 2001; Cook and Stevenson, 2010; Michel al., 2011; Kutuzov et al., 2018). For example, e-commerce, a user might use the same keyword (e.g. scarf) to search for different types of products different seasons (e.g. silk scarves in spring vs. woollen scarves in winter). The performance of pretrained Large Language Models (LLMs) is shown to decline over time (Loureiro et al., 2022; Lazaridou al., 2021) because they are trained on static snapshots. If we can detect which words have their meanings changed, we can efficiently fine-tune LLMs to re- flect only those changes (yu Su et al., 2022).
1 Introduction
Meaning of word is dynamic concept that evolves over time (Tahmasebi al., 2021). For example, the meaning of the word gay has trans- formed from happy to homosexual, whereas cell has included cell phone to its previous meanings of prison and biology. Automatic detection of words whose meanings change over time has pro- vided important insights for diverse fields such as
Source code available at https://ggtte com/ al swapspp-swap.

---

Detecting whether a word has its meaning changed between two given corpora, sampled at different points in time, a challenging task due to several reasons. First, a single (polysemous) word can take different meanings in different contexts even within the same corpus. Therefore, creating a representation for the meaning of word across an entire corpus is a challenging task compared to document. Prior work
An important limitation of the above-described approach is that it provides only a single estimate of the semantic change from a given pair of cor- pora, which is likely to be unreliable. This is es- pecially problematic when the corpora are small and noisy. To overcome this limitation, SSCD ran- domly samples sentences 81 and that contain w, respectively from C1 and C2 and swaps the two sam- ples between C1 and C2- The intuition behind this swapping step is visually explained in Figure 1. On average, a random sample of distribution will be similar to the sampling distribution (Dekking et al., 2005). Therefore, if u has similar meanings in C₁ and C2, the distance between sibling distributions before and after the swapping step will be similar. On the other hand, if w's meaning has changed be- tween the corpora, the distributions after swapping will be different from the original ones, thus having different distances between them. SSCD conducts this sampling and swapping process multiple times to obtain a more reliable estimate of the semantic changes for a target word.
that in a single sentence have averaged static (Kim et al., 2014; Kulkarni et al., 2015; Hamilton al., 2016; Yao et al., 20;8; Dubossarsky et al., 2019; Aida et al., 2021) or con- textualised (Martinc et al., 2020; Beck, 2020; Kutu- zov and Giulianelli, 2020; Rosin et al., 2022; Rosin and Radinsky, 2022) word embeddings for this purpose, which is suboptimal because averaging conflates multiple meanings of word into single vector. Second, large corpora or word lists labelled for semantic changes of words do not exist, thus requiring semantic change detection (SCD) to be approached as an unsupervised task (Schlechtweg et al., 2020).
We evaluate SSCD against previously proposed SCD methods on two datasets: SemEval-2020 Task I (Schlechtweg et al., 2020) and Liverpool FC (Del Tredici et al., 2019), which cover four languages (English, German, Swedish, and Latin) and two time periods (some spanning longer than fifty years to as less than ten years). Experimental results show that SSCD achieved significant perfor- mance improvements compared to strong baselines. Moreover, SSCD outperforms the permutation test proposed by Liu et al. (2021) on both datasets and in three languages, showing the generalisability of SSCD across datasets and languages. Moreover, we show that there exists a trade-off between the percentage of sentences that can be swapped be- tween two corpora (i.e. swap rate) and the perfor- mance of SSCD, and propose a fully unsupervised method that does not require labelled data to deter- mine the optimal swap rate.
To address the Above-mentioned challenges, we propose Swapping- Semantic Change De- tection (SSCD). To explain SSCD further, let us assume that we are interested in detecting whether a target word w has changed its meaning from a corpus Cj to another corpus C2. First, to represent the meaning of w in corpus, SSCD uses the set of contextualised word embeddings (aka. sibling em- beddings) of w in all of its contexts in the corpus, computed using a pre-trained Masked Language Model (MLM). According to the distributional hy- pothesis (Harris, 1954), if w has not changed its meaning from C₁ to C2. w will be represented by similar distributions in both Cj and Cg. Prior work has shown that contextualised word embedding of a word encodes word-sense related information that is useful for representing the meaning of the word in its occurring context (Zhou and Bollegala, 2021; Loureiro et al., 2022). Unlike the point es- timates made in prior work (Liu et al., 2021) by averaging word embeddings across corpus (thus conflating different meanings), we follow Aida and Bollegala (2023) and represent a word by mul- tivariate Gaussian distribution that captures both mean and variance of the sibling distribution. Vari- ous distance/diverggenee measures can then be used to measure the distance between the two sibling distributions of w, computed independently from C and Cz.
Related Work
The phenomenon of diachronic semantic change of words has been extensively studied in linguis- tics (Traugott and Dasher, 2001) but has recently at- tracted interest in the NLP community well. Un- supervised SCD is mainly conducted using word embeddings, and various methods have been pro- posed that use static word embeddings such as ini- tialisation (Kim et al., 2014), 4), alignment (Kulkarni

---

al., 2015; Hamilton et al., 2016), and joint learn- ing (Yao al., 2018: Dubossarsky et al., 2019; Aida et al., 2021). In recent years, with the ad- vent of pretrained MLMs, word embeddings can be obtained per context rather than per corpus, and the set of contextualised word embeddings (sibling embeddings) can be used for the semantic change analysis (Hu et al., 2019; Giulianelli et al., 2020)
to consider also its variance, obtain performance comparable the previous SoTA. Consequently, we use sibling embeddings to represent the distri- bution of a word in corpus in SSCD.
However, existing methods make predictions only once for given target word. Such point es- timates of semantic change scores unreliable especially when a target word is rare in a corpus. To overcome this unreliability. Liu et al. (2021) use context swapping to make multiple predictions for the semantic change of a target word. First, the degree of semantic change is calculated by the cosine distance of the average sibling embeddings between time periods, and the reliability of the prediction is validated by context swapping-based tests (permutation test and false discovery rate). Next, words with low reliability (p 0.005) are excluded from the prediction results, and only the words with high reliability are evaluated. However, this means that it is not possible to assess the seman- tic change of less reliable words, especially those that are less frequent. In particular, it is desirable to be able to utilise pretrained models without ad- ditional training and still be able to detect semantic changes appropriately. As discussed later in 4.1,, SSCD successfully overcomes those limitations in Liu et al. (2021), and consistently outperforms the latter multiple SCD tasks.
and detection (Martinc al., 2020; Beck, 2020;
Kutuzov and Giulianelli, 2020; Rosin et al., 2022;
Rosin and Radinsky, 2022; Aida and Bollegala, 2023; Cassotti et al., 2023).
Recent research has mainly focused on MLMs, which embed useful information related to the meaning of words in their embeddings (Zhou and Bollegala, 2021). While prior work use pretrained or fine-tuned MLMs without considering temporal information, Rosin et al. (2022) proposed a fine- tuning method by adding temporal tokens (such as <2023>) at the beginning of a sentence. In the fine- tuning step, MLMS optimise two masked language modelling objectives: I) predicting the masked time tokens from given contexts, and 2) predicting the masked tokens from given contexts with time tokens. This method has been shown to outper- form the previously unbeatable static word embed- dings in the SCD benchmark, SemEval 2020 Task (Schlechtweg al., 2020). Moreover, Rosin and Radinsky (2022) proposed adding time-aware at- tention mechanism within the MLMs. In the train- ing step, they conduct additional training on the MLM and the temporal attention mechanism on the target data. This model also achieves signifi- cant performance improvements in unsupervised SCD benchmarks.
Recently, a supervised SCD method called XL-LEXEME (Cassotti et al., 2023), which fine- tunes sentence embeddings produced from an MLM on the Word-in-Context (WiC) (Pilehvar and Camacho-Collados, 2019) dataset has achieved state-of-the-are (SoTA) performance for SCD, Dur- ing training, they fine-tune the pretrained MLLM to minimisc the contrastive loss calculated from the sentence pairs in WiC instances such that the distinct senses of a target word can be correctly discriminated. However, it has been shown that these external deta-dependent methods are difficult to apply languages that are not presented in the fine-tuning data (e.g. Latin). Our focus in this pa- per is on unsupsevised SCD methods that do not require such sense-labelled resources. Therefore, we do not consider such supervised SCD methods.
Yu Su et al. (2022) applied SCD to the tempo- ral generalisation of pretrained MLMs. Pretrained MLMs perform worse the further away in time from the trained timestamps, and require additional training (Lazaridou et al., 2021; Loureiro et al., 2022). They show that an SCD method can ef- fectively improve the performance of pretrained MLMs, because the additional training can be lim- ited to those words whose meanings change over time. Aida and Bollegala (2023) introduced method that represents sibling embeddings at each time period using multivariate Gaussians, thus en- abling various divergence and distance metrics to be used to compute semantic change scores. Ex- perimental results show that instead of using only the mean of the sibling embeddings, it is important
3 Swapping-based Semantic Change Detection
3.1 Definition
Problem setting. Given two text corpora C1 and C2, we would like to predict whether a target word

---

Algorithm 1 Context Swapping
Algorithm 2 SSCD
Input: target word w, swap rate r, sentences in which the target word appears $1"; S2
Input: target word swap rate r, time-specific
corpora C2, masked language model M,
Output: swapped sentences swapp 2,swap
Sw
divergence/distance function d
1: NI < (en(S)), N < len(S2)
Oulput: semantic change score ue, C1)
2: Nw
swap < min(rNV r
1: S <-
3: Si" < random_sane (aaioommsmmleeS77 ST" Nawap)
2: S2 f ittain_sentences((( C2)
4: 82 < random_sample S2
Nw
3: DY < [M(w, [[] C ST)
swap
5: ST. I.wap f (S" ( U 2
4: DY + {M ()]s E ST)
6: S
2,swap <- (S S S
5: Coriginal <- ,
7: return S L.swap 2.swap
S obtain swapped sentences S Iswwap 2.wwap from Algorithm 1
SH. Disswap - (M.,,)) saap Dswaap -- [M, 8)|s E SW swapp
w has its meaning changed from C1 C2. Here. we assume that Ci is sampled from an earlier time period than C2. Let the set of sentences containing to in C₁ be ST" and that in C2 be Sy According to the distributional hypothesis, if the distribution of words that co-occur with w in SY and that in Sy are different, we predict w to have its meaning changed from Ci to C2.
9: Eswap & d(Di,, I.wwap' 2swapp
10: return Coriiinal Eywapi
in C1 and C2, denoted respectively by SY and Sy. We then randomly select subsets si E ST and si E Sz", containing exactly Nawap number of sentences, determined by the number of sentences in the smaller set between ST" and ST", and the swap rate r(E [0, 1]) (Lines 2-4). Next, we exchange s and sy between C1 and C2 and obtain swapped sets of sentences Si.swap and S2 (Lines 5 and 6). The context swapped corpora next used to com- pute a semantic change score for w by SSCD described in Algorithm 2.
Meaning modelling. Given a sentence 8 and a target word w that occurs in s, we obtain the con- textualised token embedding M (w,, s) of w in s, produced by an MLM M. We refer to Mww, s) sibling embedding of w. Given set 8 of sentences containing w, we define the set of sibling embeddings by T {M [W, s) s }. Let the mean and the covariance of D be respectively 1t and 전"
If the sets of the mean and the covariance are different between the two time periods (i.e. H * ull and Σ o ΣT),, a non-zero dis- tance will remain between the two distributions (i.e. Coriginal > 0). In this case, the context swap- ping process described in Algorithm 1 will produce sibling distributions D swa and Dig swasw that are different to the corresponding original (i.e. prior to swapping) distributions DI and Dy Therefore, the distance Gswap between Di.swap and D 2.swap will be different from Coriginal between DD and D producing a large |Coriginal repeatedly computes eswap using multiple random samples (20 repetitions are used in the experiments)
Based on prior work that show contextual word embeddings to encode useful information that de- termine the meaning of a target word (Zhou and Bollegala, 2021), we form the following working assumption.
Assumption. The set of sibling embeddings D represents the meaning of a word w in a corpus C.
We can reformulate this assumption as null hy- pothesis the meaning of w has not changed from C₁ to Cz, (unless the corresponding sibling distribu- tions have changed), for the purpose of computing the probability that wo has its meaning changed from C1 to Cz-
eswapl- SSCD
to obtain reliable estimate for (Coriginal eswapl Therefore, the null hypothesis could be rejected with high probability, concluding that to have changed its meaning between C1 and C2.
To validate this hypothesis, SSCD uses the con- text swapping process described in Algorithm 1. We first find the set of sentences that contain w
On the other hand, if the sets of the mean and the covariance remain similar between the two tar- get time periods (i.c. n²"" s3 μD and Σ 23 Eg), the original semantic distance Coriginal (Line in
FFllowing Aida and Bollegala (2023), we use diagonal covariance matrices, which are shown to be numerically more stable and accurate for approximating sibling distributions.

---

Algorithm 2) will be close to zero, Moreover, the distance between Coriginal and eswap (Line 10) will also be close to zero because swapping would not change the shape of the sibling distributions (i.e. Coriginal R Gswap). Therefore, (original - Gswapl will be smaller in this case, hence the null hypothe- sis cannot be rejected for w, and SSCD will predict to be semantically invariant between C1 and C2. Similar to Liu et al. (2021), SSCD can also be extended to calculate the confidence interval for rejecting the null hypothesis using boostrap- ping (Efron and Tibshirami, 1994; et al.. 2012). This is particularly useful for the binary classification subtask in the SemEval 2020 Task 1 for unsupervised SCD, where we must clas- sify a given word as to whether its meaning has changed between the two given corpora. However, for our evaluation on the ranking subtask we re- quire only the semantic change score.
by Aida and Bollegala (2023) to measure the distance between two sibling distributions. We randomly sample equal number of target word vectors from the two Gaussians Mpr "1r and N Mp Σ²), and calculate the average pairwise distance among those vectors. We use the seven distance metrics mentioned above for this purpose.
Experiments
4.1 Effectiveness of Context Swapping
In this section, we evaluate the effectiveness of SSCD by comparing it against the method that uses context swapping for the reliability of prediction proposed by Liu et al. (2021). The concern with their method is that not all words will be evalu- ated, as already discussed in $ 2. Moreover, they require fine-tuning MLMs, which can be computa- tionally costly for large corpora and for every target language of interest. In this experiment, we show that SSCD successfully overcomes those issues as follows: 1) SSCD uses context swapping for cal- culating the degree of semantic change and makes appropriate predictions for all words; 2) SSCD uses pretrained multilingual BERT without addi- tional architectural modifications such as temporal attention nor fine-tuning. Following previous ex- periments (Liu et al., 2021) and findings (Laicher et al., 2021), we obtain token embeddings from the last four layers of the MLM. We use the metrics described in $ 3.2 for computing distances.
3.2 Metries
Prior work has shown that the metries used for predictions are also important and the best perfor- mance for different languages is reported by differ- ent metrics (Kutuzov and Giulianelli, 2020; Aida and Bollegala, 2023). Our proposed SSCD can be applied with various divergence/distance metrics (Lines 5 and 9 in Algorithm as we describe next.
Divergence measures: We use Kullback-Leible (KL) divergence and Jeffrey's divergence." 3 Fol- lowing previous work (Aida and Bollegala, 2023), we approximate D by Gaussian NV NN",,, E"). Based this setting, two divergence functions have closed-form formulae, computed from the mean and variance of the two multivariate Gaus- sians N (uTY, E)") and (H Σ/") Since the KL divergence is asymmetric and Jeffrey's 's divergence is symmetric, we calculate two versions for the KL divergence (KL((C)|C2)) and KL(C2 (C)),, and one for the Jeffrey's divergence (e(f())ll(2)).
To evaluate the performance of SCD methods, we use two benchmark datasets: SemEval-2020 Task (Schlechtweg et al., 2020) and Liverpool FC (Del Tredici et al., 2019), which cover four languages (English, German, Swedish, and Latin) for longer (spanning over 50 years) and shorter (spanning less than ten years) time periods. In both datasets, method under evaluation is required to rank given word list according to the degree of semantic change, which is subsequently compared against assigned ranks using the Spearman rank correlation coefficient (p c 1, 1), where higher values indicate better agreement with the human ratings.
Distance function: We use seven distance met- ries as follows: Bray-Curtis, Canberra, Chebyshev, City Block, Correlation, Cosine, and Euclidean." In this setting, we calculate the distance berween the two mean vectors 1 and fel from D and DY, respectively.
Results of SSCD against the method proposed by Liu et al. (2021) are shown in Table 1. Duc to space limitations, we report SSCD results in the setting with the highest average p values taken
DSCD: We use the Distribution- basribution--bsed Seman- tie Change Detection (DSCD) method proposed
'We use pert-base-mltilinngaall published on Hugging Face https://huggingfaee co/ bert-base b-s--aase-------- meeeemmlliilll wtt.linggalllceeeet
model
DDefiittions are provided in the Appendix A. 'Definitions are provided in Appendix B.

---

SemEval
Liverpool FC English
Model
English
German
Swedish
Latin
Liu et al. (2021)
MLMuned
0.331
0.302
0.141
0.512
0.536
Permutation Test
0.341
0.304
0.162
0.502
0.561
+ False Discovery Rate
0.339
0.304
0.162
0.502
0.478
SSCD
MLMpre. Divergence
0.209
0,547
0.127
0,460
0,470
MLMpre pre. Distance (mean only)
0.383
0.597
0.234
0.433
0,492
MLM Distance (DSCD)
0.364
0,476
0.199
0,410
0.364
Cassotti et al. (2023)
XL-LEXEME (supervised)
0.757
0.877
0.754
-0.056
N/A
Table Comparison against prior work SemEval-2020 Task and Liverpool DSCD indicates the average distance calculated vectors sampled from the distributions of time-specific sibling embeddings (Aida and Bollegala, 2023).
over 20 rounds for each metric. 6 These results reveal that even with the pretrained MLM (no fine- tuning), SSCD outperforms the SCD method pro- posed by Liu et (2021), which uses fine-tuned MLMs, on three out of the five datasets. can be seen that SSCD accurately detects the seman- tic changes of words in four languages (English, German, Swedish, and Latin).
ing time tokens (Rosin et al., 2022). They add time token (such as <2023>) the beginning of sentence. In the fine-tuning step, the models use two types of MLL objectives: 1) predicting the masked time tokens from given contexts, and 2) predicting the masked tokens from given contexts with time tokens.
While the above methods unsupervised, also include in Table the results of the supervised SCD method, XL-LEXEME (Cassotti et al., 2023). XL-LEXEME achieves SoTA performance in En- glish and German, which are included the fine- tuning data. Moreover, this model also achieves SoTA in Swedish which does not exist in the la- belled data, but Danish, quite similar to Swedish is included. On the other hand, this model performs significantly worse in Latin which is not included in the sense-labelled data. This indicates the difficulty of applying supervised SCD methods in languages that are not in the fine-tuning data.
Rosin et al. (2022) make predictions with the average distance of the target token probabil- ities the cosine distance of the average sibling embeddings. According to their results, the co- sine distance achieves better performance than the average & distance between the probability distri- butions.
Cosine:
APD: Kutuzov and Giulianelli (2020) report that the average pairwise cosine distance outperforms the cosine distance. Based on this insight, Aida and Bollegala (2023) evaluate the performance MLM with the average pairwise cosine dis- tance.
4.2 Comparison against Strong Baselines
DSCD: Aida and Bollegala (2023) proposed distribution-bast based SCD, considering the distribu- tions of the sibling embeddings (sibling distribu- tion). During prediction, they sample an equal num- ber of target word vectors from the sibling distri- bution (approximated by Gaussians) for each time
Building upon the results of the previous section, we compare SSCD with multiple strong baselines. Here, we take MLMemmp very powerful unsu- pervised model for SCD (Rosin et al., 2022), starting point. MLM is the fine-tuned version the published pretrained BERT-base models? us-
and Radinsky, 2022), the results are only for English and have not been verified in the other languages. Hence, experiments will be conducted using the BERT-base model, which is widely used.
"Full results are shown $ D.I.
'Although previous rescarch has shown that reducing the model size down to BERT-tiny improves performance (Rosin

---

period and calculate the average distance. They report that Chebyshev distance measure achieves the best performance.
Swap rate Opt.
Metric
Spearman
Est.
Divergence functions
Temp. Att.: Rosin and Radinsky (2022) proposed a temporal attention mechanism, where they add trainable temporal attention matrix the pretrained BERT models. Because their two proposed meth- ods (fine-tuning with time tokens and temporal at- tention) are independent, they proposed to use them simultaneously. Subsequently, additional training is performed on the target corpus. They use the cosine distance following their earlier work (Rosin al., 2022).
KL(Ciil(2))
0.552
0.4
0.6
KL(C2 (1)
0.516
0.4
0.6
Jeff(Ci|l(2)
0.534
0.4
0.6
Distance functions
Bray-Curtis
0.423
0.4
0.6
Canberra
0.345
0.4
0.6
Chebyshev
0.372
0.4
0.6
City Block
0.443
0.4
0.6
Correlation
0.471
0.4
0.6
Cosine
0.471
0.4
0.6
In our evaluations, we will compare using the results published in the original papers, without re-running them. For SSCD, we use the fine- tuned BERT model MLM in line with previous work (Rosin and Radinsky, 2022; Aida and Bolle- gala, 2023) and make predictions using the same three types of metrics described in 4.1. We use the SemEval-2020 Task English benchmark for this evaluation.
Euclidean
0.453
0.4
0.6
Table 2:
Best performance (measured using p) obtained using SSCD with different divergence/distance mea- sures. The swap rate at which the best performance is obtained (Swap Opt.) and the optimal swap rate esti- mated using the unsupervised method (Swap Est.) are shown. All results are averaged over runs. We see that Swap Opt. and Swap Est. are similar and are inde- pendent of the measure being used.
Prediction metrics within SSCD. Before com- paring the performance with strong baselines, we employ various divergence/distance functions pre- sented in 3.2 in SSCD and compare their per- formance. Results within our method are shown in Table 2.8 Table 2 shows that divergence measures perform better than the distance-based metrics." 9
overlapping for the most part). The optimal swap rates for all metries are shown in Table and are or around 0.4, which are sufficiently closer the f estimated by minimising éswap for those metrics.
Context Sampling Strategies for SSCD. The version of SSCD presented in Algorithm 1 ran- domly selected (in Lines 3 and 4) contexts for swapping, we refer as SSCD However, be- sides random sampling, different criteria can be considered when selecting the subsets of contexts for swapping. Considering that word w would be considered to have its meaning changed, if the furthest meanings of w in C₁ and C2 are dissimilar, we propose a distance-basee based deterministic context selection method. Specifically, we sort each sibling embedding of w in SY in descending order of the distance the centroid of the sibling embeddings in S We then select the top (i.c. furthest) Ngwap number of sibling embeddings from S/r as s/ as the candidates for swapping in Algorithm 1. Like- wise, we select sy considering the centroid of SY: Let us denote this version of SSCD by SSCD
Unsupervised hyperparameter search. Swap rate is the only hyperparameter in SSCD, which must be specified in both Algorithms and 2. How- ever, benchmarks for SCD tasks do not have dedi- cated development sets, which is problematic for hyperparameter tuning. To address this problem, we propose an unsupervised method to determine the optimal swap rate as follows. Recall that con- text swapping minimising the distance between the sibling distributions computed independently for w from the corpora. Therefore, we consider eswap (computed in Line 9 in Algorithm 2) ob- jective function for selecting the swap rate. Specifi- cally, we plot eswap against the swap rate in Figure and find the swap rate that minimises Gswap- We see that eawap is minimised at swap rate of 0.6 for all three divergence functions (all three curves are
FFul results are shown $ D.2
As shown in Table 3, SSCDais when used with DSCD as the distance metric, obtains a value 0.563, clearly outperforming all other metrics used
'Although we cannot list the standard deviation due to space limitations here, the mean and standard deviation (taken over 20 runs) are shown in D.2.

---

30
Method
Spearman
Divergence (C) swap || C2 swan)
kL
klc2_cl
Unsupervised
25
Cosine (Rosin et al., 2022)
0,467
APD (Kutuzov and Giulianelli, 2020)
0,479
20
DSCD (Aida and Bollegala, 2023)
0,529
Temp. Att. (Rosin and Radinsky, 2022)
0,548
SSCD KT((C)||C2)
0.552
15
SSCD DSCD
0,563
Supervised
10-
XI-LEXEME (Cassotti et al., 2023)
0.757
0.1
0.2
0.3
0.4
0.5 0.G
0.7
0.8
0.9
Table 3: Comparison against strong baselines in
Swap rate
SemEval-2020 Task English. All methods except XL-LEXEME (Cassotti al., 2023) start from the same model, MI.M (Rosin et al., 2022).
Figure 2: Average divergence of the two distributions after context swapping in all target words. Each plot and error bar shows the mean and standard deviation in 20 seeds, respectively.
4.3 Qualitative Analysis
We conduct an ablation study further study the effect of context swapping and the swap rate. For this purpose, we use the following variants of our SSCD: 1) without context swapping (swap rate = 0.0), and 2) context swapping with extremely high or low swap rate (swap rate = 0.1 vs 1.0). Following Aida and Bollegala (2023), select (a) words with the highest degree of semantic change (and labelled semantically changed), and (b) words with the lowest degree of semantic change (and labelled as stable) by the annotators in the ScmEval-2000 2020 Task for English SCD.
10 with random sampling- based SSCD rand.
Main comparison. The main result are shown Table from which we see that both SSCDrond and SSCDaisr outperform all other the strong base- lines. Moreover, although both the previous best method (Temp. Att.) and our SSCD use the same fine-tuned model (MLM) the former requires additional training for the temporal attention mech- anism, whereas SSCD require no additional train- ing. This is particularly attractive from computa- tional time and cost saving ofint-of-view. How- ever, from Table 3 we see that there a significant performance gap between the best unsupervised SCD method and the supervised XL-LEXEME. Although XL-LEXEME uses sense-labelled WiC data for fine-tuning sentence encoders, during in- ference only single point estimates of SCD scores are made. Therefore, it would be interesting future research direction to explore the possibility using SSCD at inference time with pre-trained XL-LEXEME, where multiple sets of sentences containing a target word is used to obtain a more reliable estimate of its SCD score.
As shown in Table 4, we see that the use of context swapping improves the underestimation (tip) and overestimation (fiction) that occurs when context swapping is not used (i.e. swap rate 0.0). Moreover, there is also further improvement (head, realtionship,fictinn swap rate (swap rate- = 0.4*).
by using the optimum
However, we also see that SSCD cannot detect words which are rare (i.e. relatively low frequency of occurrence in the corpus) with novel obso- lete meanings (i.e. bit), or words used in different senses (chairman and risk). This because SSCD assumes only one set of sibling embeddings per time period, which means that it can only roughly detect semantic changes in words. As mentioned in Aida and Bollegala (2023), we believe that sep- arating the sets of sibling embeddings into sense levels (c.g. assuming mixed Gaussian distributions) will further improve the performance of SSCD. Al- though most of the target words in this benchmark are nouns and verbs, detecting words used wider contexts, such as chairman, risk and adverbs, re- mains an interesting open problem for future work.
WWe also considered variants of SSCD that takes into ac- count the number sentences Ci and C2 because the imbalance the corpus size might affect context swapping process. Specifically, we normalise N1 and N2" respectively by Ci and [Czl (i.e. the total numbers of sentences in each corpus), prior to computing aawp Algorithm However, in our preliminary investigations, we did not observe a signifi- cant improvement of performance due to this down sampling and believe it could be due to the fact that the datasets we used for evaluations are carefully sampled to have approx- imately equal numbers of sentences covering cach point in time. Further results are shown in $ D.2.

---

Gold rank
Swap rate
Limitations
Word
0.1
0.4"
In this paper, we show that our method can prop- erly detect semantic changes in four languages (En- glish, German, Swedish, and Latin) across two time spans (over 50 years and about five years). However, We not conduct detecting seasonal se- mantic changes, which occur periodically and over extremely shorter time spans (e.g. few months). e-commerce, keywords such as scarf used by users may refer to different products in different seasons. To detect and evaluate seasonal semantic changes, training dataset and a list of words for evaluation need to be annotated in the future work.
plane
1
1
tip
25
7
2
prop
3
20
graft
2
9
36
record
stab
12
bit
29
head
33
multitude
30
X
15
33
36
23
savage
31
X
28
36
35
22
contemplation
X
21
37
37
33
tree
33
X
35
30
27
29
relationship
17
35
21
fiction
chairman
4
26
risk
14
Ethics Statement
Spearman
1.000
0.130
0.596
0.627
0.164
The goal of this paper to detect semantic changes of words based on context swapping. For this pur- pose, we use publicly available SemEval 2020 Task I dataset, and do not collect annotate any addi- tional data by ourselves. Morcover, aware of any social biases in the SemEval dataset that we use our experiments. However, we also use pretrained ML.Ms our experiments, which are known to encode unfair social biases such as racial or gender-related biases (Basta et al.. 2019). Considering that the sets of sibling embeddings used in our proposed method are obtained from MLMMs that may contain such social biases, the sen- sitivity of our method to such undesirable social biases needs be further evaluated before it can be deployed in users.
Table 4: Ablation study the words with the highest/lowest degree of semantic change labelled as changed/stable. A indicates the word is semantically changed (V) stable (X). Swap rate 0.4* is the opti- mal swap rate in this setting.
not
5 Conclusion
We proposed SSCD, method that swaps con- texts between two corpora for predicting seman- tic changes of words. Experimental results show that SSCD outperforms a previous method that uses context swapping for improving the reliability SCD. Morcover, SSCD even with using only pretrained models, and not requiring fine-tuning achieves significant performance improvements compared to strong baselines for the unsupervised English SCD, We also discussed two perspectives our SSCD: 1) We showed that modifying the context swapping method improves performance compared to random context swapping. 2) We proposed an unsupervised method to find the op- timal context swapping-rate, and showed that is relatively independent of the distance/divergeece measure used in SSCD. As in future work, we will apply our method for Lime-dependen tasks such temporal generalisation (Lazaridou et al., 2021) and temporal question answering (Dhingra al., 2022). Furthermore, as our method and supervised methods are independent of each other, we are con- sidering applying our method to the supervised SCD method.
applications used by human
A Acknowledgements
This work was supported by JST, the establish- ment of university fellowships towards the creation of science technology innovation, Grant Number JPMJFS2139, Danushka Bollegala holds concur- rent appointments as Professor at University of Liverpool and as an Amazon Scholar. This paper describes work performed at the University of Liv- erpool and is not associated with Amazon.
References
Taichi Aida and Danushka Bollegala. 2023. Unsuper- vised semantic variation prediction using the distri- bution of sibling embeddings. In Findings of the As- sociation for Computational Linguisties: ACL 2023, pages 6868-6882 6682, Toronto, Canada. Association for Computational Linguistics.

---

Taichi Aida, Mamoru Komachi, Toshinobu Ogiso, Hi- roya Takamura, and Daichi Mochihashi. 2021. A comprehensive analysis of PMI-based models for measuring semantic differences. In Proceedings of the 35th Pacific Asia Conference on Language, In- formation and Computation, pages 21-31, Shanghai, China. Association for Computational Lingustics.
Haim Dubossarsky, Simon Hengehen, Nina Tahmasebi. and Dominik Schlechtweg. 2019. Time-out: Tem- poral referencing for robust modeling of lexical se- mantic change. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 457-470, Florence, Italy. Association for Computational Linguistics.
Christine Basta, Marta R. Costa-jussà, and Noe Casas. 2019. Evaluating the underlying gender bias con- textualized word embeddings. In Proceedings of the First Workshop Gender Bias in Natural Language Processing, pages 33-39, Florence, Italy, Association for Computational Linguistics.
Bradley Efron and Robert J. Tibshirani. 1994, An intro- duction to the bootstrup. CRC Press.
Mario Giulianelli. Marco Del Tredici, and Raquel Fer- nández. 2020. Analysing lexical semantic change with contextualised word representations. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 3960- 3973, Online. Association for Computational Lin- guistics.
Christin Beck. 2020. DiaSense at SemEval-2200 -022 task I: Modeling sense change via pre- trained BERT embed- dings. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 50-58, Barcelona (online), Interaatoonl Committee for Computational Linguistics.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statisti- cal laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 1489-1501, Berlin, Germany, Association for Com- putational Linguistics.
Taylor Berg- Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical sig- nificance in nlp. In Proceedings of the 2012 Joint Conference on Empiricul Methods in Natural Lan- guage Processing and Computational Natural Lan- Jeju Island, Korea. Association for Computational Linguistics.
guage Learning, pages
Zellig Harris. 1954. Distributional structure. Word, 10623):146-16..
Pierluigi Cassotti, Lucia Siciliani, Marco DeGemmis, Giovanni Semeraro, and Pierpaolo Basile. 2023. XI. LEXEME: WiC pretrained model for cross- longul LEXical sEMantic ChangE. In Proceedings of the st Annual Meeting of the Association for Compu- tational Linguistics ( (Volume 2: Short Papers). pages 1577-1585, Toronto, Canada. Association for Com- putational Linguistics.
Renfen Hu, Shen Li, and Shichen Liang. 2019. Di- achronic sense modeling with deep contextualized word embeddings: An ecological view. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguisties. pages 3899-3908. Flo- rence, Italy, Association for Computational Linguis- tics.
Paul Cook and Suzanne Stevenson. 2010. Automati- cally identifying changes in the semantic orientation of words. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), Valletta, Malta. European Language Re- sources Association (ELRA).
Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal analysis of lan- guage through neural language models. In Proceed- ings of the ACL 2014 Workshop on Language Tech- nologies and Computational Social Science, pages 61-65, Baltimore, MD, USA. Association for Com- putational Linguistics.
Frederik Michel Dekking, Cornelis Kraaikamp, Hen- drik Paul Lopuhaä, and Ludolf Erwin Meester. 2005. A Modern Introduction to Probubility and Statistics: Understanding why and how, volume 488. Springer.
Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically significant detec- tion of linguistic change. In WWW 2015, pages 625- 635.
Marco Del Tredici, Raquel Fernández, and Gemma Boleda. 2019. Short-term meaning shift: distri- butional exploration. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2075, Minneapolis, Minnesota. Association for Computational Linguistics.
Andrey Kutuzov and Mario Giulianelli. 2020. UiO- UvA at SemEval-2020 task 1: Contextualised em- beddings for lexical semantic change detection. Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 126-134 Barcelona (online). Inter- national Committee for Computational Linguistics.
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos. Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language mod- els as temporal knowledge bases. Transactions of the Association for Computational Linguisties, 10:257- 273.
Andrey Kutuzov, Lilja Ovrelid, Terrence Szymanski, and Erik Velldal. 2018. Diachronic word embeddings and semantic shifts: a survey. In Proceedings of the 27th International Conference on Computational Lin- guistics, pages 1384-1397, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

---

Severin Laicher, Sinan Kurtyigit, Dominik Schlechtweg. Jonas Kuhn, and Sabine Schulte im Walde. 2021. Ex- plaining and improving BERT performance on lex- ical semantic change detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistiex: Student Research Workshop, pages 192-202, Online. Associ- ation for Computational Linguistics.
Guy D. Rosin and Kira Radinsky. 2022. Temporal at- tention for language models. In Findings of the Asso- ciation for Computational Linguistics: NAACI. 2022, pages 1498-1508, Seattle, United States. Association for Computational Linguisties.
Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020. SemEval-2020 task 1: Unsupervised lexical semantic change detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1-23, Barcelona (online). International Committee for Computational Linguistics.
Angeliki Lazaridou, Adhiguma Kuncoro, Elena Gri- bovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Tomáš Kočiský, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021. Mind the gap: Assessing temporal generalization in neural language models. In Advances in Neural Information Processing Systems.
Nina Tahmasebi, Lars Borina, and Adam Jatowtb, 2021. Survey of computational approaches to lexical mantic change detection. Computational approaches to semantic change, 6:1.
Yang Liu, Alan Medlar, and Dorota Glowacka. 2021. Statistically significant detection of semantic shifts using contextual word embeddings. In Proceedings of the 2nd Workshop on Evaluation and Compari- son of NLP Systems, pages 104-113, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Elizabeth Closs Traugott and Richard Dasher. 2001. Prior and current work on semantic change, Cam- bridge Studies in Linguisties, page 51-104. Cam- bridge University Press.
Zijun Yao, Yifan Sun, Weicong Ding, Nikhil Rao, and Hui Xiong, 2018. Dynamic word embeddings for evolving semantic discovery. In WSDM 2018, page 673-681.
Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose comacho-colladod 2022. Mi:eLMs: Diachronic language models from Twitter. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguisties: System Demonxtrations, pages 251-260, Dublin, Ire- land. Association for Computational Linguistics.
Zhao yu Su, Zecheng Tang. Xinyan Guan, Juntao Li, Lijun Wu, and M. Zhang. 2022. Improving tem- poral generalization of pre-trained language mod- els with lexical semantic change. Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 6380-6399 Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Matej Martine, Petra Kralj Novak, and Senja Pollak. 2020. Leveraging contextual embeddings for detect- ing diachronic semantic shift. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 4811-481111 1-4819, 19, Marseille, France. European Language Resources Association.
Yi Zhou and Danushka Bollegala. 2021. Learning sense- specific static embeddings using contextualised word embeddings as proxy. In Proceedings of the 35th Pacific Asia Conference on Language, Information and Computation, pages 493-502, Shanghai, China. Association for Computational Lingusties.
Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K. Gray, null null, Joseph P. Pickett, Dale Hoiberg, Dan Clancy, Pe- ter Norvig, Jon Orwant, Steven Pinker, Martin A. Nowak, and Erez Licberman Aiden. 2011. Quanti- tative analysis of culture using millions of digitized books. Science, (30(4)1):177 182.
A Divergence Functions
Mohammad Taher Pilehvar and Jose Comacho-Collados, 2019. WiC: the word-in-contexx dataset for evalu- ating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers). pages 1267-1273, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
We elaborate on two divergence functions. For simplicity, we denote two d- d-variate Gaussian distri- butions N (HH) Enr) and A/ N( Σg) as N and N3T, respectively.
Kullback-Leibler
Guy D. Rosin, Ido Guy, and Kira Radinsky, 2022. Time masking for temporal language models. In Proceed- ings of the Fifteenth ACM International Conference on Web Search and Data Mining, WSDM '22, pages 833-841, New York, NY, USA. Association for Com- puting Machinery.
KL(N
det (E
tr Ey
'DIT d - log
det Eny') (1)
2
(1
11 w Σし ( n4

---

Jeffrey's
D Full Results
D.1 Effectiveness of Context Swapping
2 (<L(MI1A + -
In $ 4.1, we evaluate the performance of pretrained multilingual BERT with SSCD in the Semlival- 2020 Task 1 and the Liverpool FC benchmarks. Tables 6-9 show the full results.
4 (I(E0 - + tr (>1 pm-1 Σタ
2d (2)
+ (11 Mi) D [12 mi)
D.2 Comparison against Strong Baselines
(uf - p2 プリ [n u1)
In $ 4.2, we compare the performance of our SSCD against strong baselines in SemEval-2020 Task English benchmark. Full results are shown in Ta- ble 10. In this setting, Table 11 shows the average and the standard deviation of the 20 seeds.
B Distance Functions
We elaborate on seven distance functions. In this part, wii) denotes the i-dimensional value of the word vector w and to denotes the vector subtracted by the average of all dimensional values.
After that, we consider the context sampling strategies for our SSCD. Table 12 shows all the results of distance-based context swapping (SSCDais)). Moreover, Tables 13 and 14 are the full results of considering the ratio of corpus size before context swapping methods SSCD and SSCD dist-
Bray-Curtis
Eied Iw (1) (i)
( ((w,ww)
Lied /1 (2) + (2)))
Canberra
w w2
(w11, w2)
(4)
Wi (i) + 22 (i)
Chebyshev
(ili, w2) = max wi (i) wi)
(5)
City Block
(w1,, w2) 1t (i) (22(1)
Σ
(6)
sed
Correlation
((w1, w2) =1
W1 W2
(7)
|2
[W22 12
Cosine
(ww,, (2) =1 ||w
W1 W2
(8)
[w2
Euclidean
(wi, w2) =
||w] wall
(9)
C Data Statistics
Data statistics are presented in Table 5. For the Liv- erpool FC benchmark, the statisties resulting from the pre-processing are shown. This data contains a variety of information, such as user names, ids, and timestamps, as well as the text. We extracted only the body text and used the NLTK library" 11 to determine sentence boundaries and tokenise words.
"https: //wwww nltk.orf org/

---

Dataset
Language
Time Period
#Targets
#Sentences
#Tokens
#Types
1810-1860
254k
6.5M
87k
English
37
1960-2010
354k
6.7M
150k
1800-1899
2.6M
70.2M
1.0M
SemEval
German
48
1996-1990
3.5M
72.3M
2.3M
1790-1830
3.4M
71.0M
1.9M
Swedish
30
1895-1903
5.2M
110.0M
3.4M
2011-2013
576k
9.5M
137k
Liverpool FC
English
97
2017
1.0M
15.7M
146k
Table 5: Statistics of datasets.
Swap rate
Metric
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Divergences
Cl)
0.209
0.202
0.190
0.176
0.165
0.144
0.118
0.076
0.011
KL(C2l C1)
0.136
0.115
0.112
0.101
0.088
0.062
0.045
0.005
0.045
Jeff(C |c2
0.170
0.148
0.145
0.133
0.126
0.103
0.072
0.033
0.018
Distance functions
Bray-Curtis
0.185
0.190
0.186
0.161
0.140
0.090
0.018
0.057
0.147
Canberra
0.223
0.245
0.246
0.217
0.189
0.117
0.010
0.091
0.156
Chebyshev
0.294
0.361
0.351
0.383
0.370
0.334
0.323
0.291
0.251
City Block
0.258
0.256
0.256
0.247
0.230
0.180
0.124
0.052
0.055
Correlation
0.183
0.171
0.162
0.161
0.155
0.131
0.104
0.061
0.002
Cosine
0.183
0.171
0.162
0.161
0.156
0.131
0.104
0.061
0.002
Euclidean
0.272
0.269
0.273
0.264
0.244
0.195
0.136
0.066
0.045
DSCD (Aida and Bollegala, 2023)
Bray-Curtis
0.293
0.333
0.354
0.359
0.364
0.350
0.333
0.272
0.161
Canberra
0.183
0.177
0.224
0.246
0.282
0.258
0.262
0.199
0.103
Chebyshev
0.080
0.120
0.178
0.180
0.196
0.196
0.177
0.104
0.101
City Block
0.212
0.261
0.285
0.312
0.301
0.295
0.284
0.264
0.211
Correlation
0.154
0.234
0.266
0.277
0.291
0.287
0.268
0.233
0.173
Cosine
0.229
0.254
0.319
0.331
0.336
0.305
0.283
0.258
0.175
Euclidean
0.293
0.313
0.340
0.356
0.334
0.327
0.310
0.283
0.216
Table 6: Results within MLMp,. with SSCD aod in SemEval-2020 Task English. All values averages over 20
seeds.

---

Swap rate
Metric
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Divergences
CL)
0,511
0.536
0.547
0.536
0.537
0.539
0.533
0.520
0,490
[L(C2| C1)
0.510
0.526
0.542
0.539
0.537
0.535
0.526
0.509
0.477
Jeff(Ci||
0.515
0.533
0.547
0.540
0.540
0.538
0.531
0.517
0.484
Distance functions
Bray-Curtis
0.486
0.522
0.560
0.576
0.580
0.597
0.583
0.529
0.434
Canberra
0.370
0.431
0.474
0.499
0.512
0.550
0.534
0.459
0.332
Chebyshev
0.236
0.288
0.355
0.385
0.398
0.447
0,462
0.462
0.439
City Block
0.337
0.358
0.381
0.386
0.388
0,414
0,417
0.428
0.369
Correlation
0.563
0.577
0.591
0.592
0.586
0.586
0.577
0.554
0.525
Cosine
0.563
0.577
0.591
0.592
0.586
0.586
0.577
0.554
0.525
Fuclidean
0.348
0.364
0.392
0.394
0.397
0.423
0.422
0.436
0.377
DSCD (Aida and Bollegala, 2023)
Bray-Curtis
0.388
0.432
0.473
0.476
0.471
0.473
0.466
0.441
0.407
Canberra
0.284
0.360
0.420
0.429
0.424
0.435
0,429
0.403
0.359
Chebyshev
0.006
0.062
0.126
0.172
0.197
0.204
0.201
0.180
0.158
City Block
0.338
0.348
0.381
0.396
0.381
0.390
0.400
0.389
0.366
Correlation
0.373
0.427
0.457
0.460
0.443
0.437
0.435
0.408
0.375
Cosine
0.262
0.338
0.400
0.420
0.413
0.406
0.406
0.378
0.344
Euclidean
0.274
0.315
0.357
0.383
0.369
0.378
0.389
0.383
0.356
Table Results within MLM with SSCD nand SemEval-2020 Task
German. All values are averages over
seeds.

---

Swap rate
Metric
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Divergences
CL
0.104
0.083
0.072
0.058
0.038
0.019
0.004
0.029
0.127
[L(C2| C1)
0.110
0.083
0.088
0.081
0.063
0.037
0.035
0.014
0.114
Jeff(Ci
0.112
0.092
0.087
0.079
0.060
0.030
0.014
0.025
0.125
Distance functions
Bray-Curtis
0.002
0.020
0.014
0.015
0.022
0.022
0.020
0.019
0.006
Canberra
0.006
0.004
0.003
0.008
0.016
0.047
0.069
0.109
0.105
Chebyshev
0.107
0.094
0.071
0.084
0.087
0.106
0.109
0.142
0.116
City Block
0.154
0.152
0.146
0.155
0.167
0.180
0.229
0.227
0.230
Correlation
0.036
0.042
0.054
0.061
0.064
0.095
0.101
0.068
0.036
Cosine
0.036
0.042
0.054
0.061
0.064
0.095
0.101
0.068
0.036
Fuclidean
0.151
0.146
0.137
0.145
0.163
0.173
0.229
0.232
0.234
DSCD (Aida and Bollegala, 2023)
Bray-Curtis
0.068
0.015
0.015
0.018
0.037
0.050
0.059
0.057
0.054
Canberra
0.013
0.011
0.018
0.048
0.067
0.082
0.105
0.119
0.095
Chebyshev
0.162
0.137
0.143
0.176
0.197
0.199
0.171
0.140
0.160
City Block
0.094
0.121
0.115
0.117
0.113
0.111
0.116
0.123
0.119
Correlation
0.127
0.112
0.103
0.102
0.108
0.097
0.077
0.081
0.093
Cosine
0.090
0.072
0.075
0.068
0.088
0.076
0.050
0.061
0.066
Euclidean
0.109
0.093
0.106
0.093
0.090
0.092
0.084
0.089
0.060
Table Results within MLMpt with SSCD yond in 2e0Eval-2020 Task Swedish. All values are averages over
seeds.

---

Swap rate
Metric
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Divergences
CL(
0.052
0.198
0.268
0.291
0.305
0.460
0.464
0.470
0.462
K CD
0.047
0.157
0.235
0.247
0.259
0.396
0.394
0.394
0.393
Jeff(Ci]]
0.051
0.172
0.260
0.277
0.291
0.438
0.438
0.439
0.440
Distance functions
Bray-Curti
0.027
0.186
0.261
0.278
0.293
0.441
0.448
0.462
0.476
Canberra
0.018
0.135
0.202
0.228
0.245
0.375
0.391
0.405
0.436
Chebyshev
0.025
0.085
0.130
0.166
0.177
0.334
0.343
0.362
0.384
City Block
0.000
0.161
0.248
0.273
0.294
0.465
0.467
0.473
0.489
Correlation
0.065
0.217
0.298
0.309
0.317
0.475
0,477
0,480
0.488
Cosine
0.065
0.217
0.298
0.309
0.317
0.475
0.477
0.480
0.488
Euclidean
0.001
0.161
0.249
0.275
0.294
0.466
0.469
0.470
0.492
DSCD (Aida and Bollegala, 2023)
Bray-Curtin
0.116
0.173
0.201
0.231
0.213
0.342
0.344
0.352
0.354
Canberra
0.098
0.149
0.208
0.227
0.215
0.328
0.338
0.350
0.358
Chebyshev
0.150
0.163
0.189
0.210
0.221
0.279
0.264
0.268
0.291
City Block
0.174
0.181
0.210
0.229
0.226
0.364
0.358
0.353
0.357
Correlation
0.087
0.158
0.177
0.212
0.209
0.358
0.345
0.340
0.349
Cosine
0.093
0.157
0.185
0.214
0.207
0.354
0.342
0.340
0.344
Euclidean
0.170
0.199
0.218
0.236
0.233
0.362
0.356
0.350
0.353
Table 9; Results within MLMpp with SSCD aund in Liverpool FC, All values averages of 20 seeds.

---

Swap rale
Metric
0.1
0.2
0.4
0.5
0.6
0.7
0.8
0.9
Divergences
KL(Ci] C2)
0.461
0.536
0.549
0.552
0.543
0.526
0.483
0.396
0.239
KL(C2] (1)
0,433
0.503
0.509
0.516
0.514
0.497
0.439
0.369
0.234
Jeff(C (2)
0.450
0.522
0.528
0.534
0.529
0.516
0.460
0.383
0.239
Distance functions
Bray-Cume
0.267
0.392
0.395
0.423
0.391
0.335
0.259
0.140
0.020
Canberra
0.194
0.283
0.296
0.345
0.292
0.229
0.148
0.063
0.073
Chebyshev
0.158
0.273
0.317
0.372
0.341
0.351
0.276
0.197
0.081
City Block
0.282
0.413
0.414
0.443
0.407
0.349
0.273
0.148
0.007
Correlation
0.355
0.441
0.445
0.471
0.471
0.466
0.437
0.350
0.153
Cosine
0.355
0.441
0.445
0.471
0.471
0.466
0,437
0.350
0.153
Euclidean
0.296
0.419
0.419
0.453
0.416
0.358
0.272
0.148
0.004
DSCD (Aida and Bollegala, 2023)
Bray-Curtis
0.149
0.331
0.356
0.367
0.336
0.293
0.214
0.134
0.057
Canberra
0.233
0.341
0.385
0.378
0.354
0.318
0.267
0.202
0.111
Chebyshev
0.029
0.026
0.030
0.120
0.112
0.118
0.118
0.122
0.146
City Block
0.245
0.336
0.411
0.400
0.377
0.335
0.250
0.195
0.120
Correlation
0.185
0.315
0.352
0.365
0.339
0.301
0.224
0.160
0.089
Cosine 0.212
0.332
0.383
0.375
0.356
0.312
0.253
0.212
0.125
Euclidean
0.088
0.237
0.315
0.313
0.298
0.253
0.210
0.154
0.074
Table 10: Results within MLM with SSCD rand in SemEval-2020 Task English. All values are averages over 20
seeds.
Divergences
Swap rate
KL(C1||C2)
KL(C2||C1)
Jeff(C|//C2)
0.1
0.461+0.080
0.433+000800
0.450+:0.07
0.2
0.536:10.04
0.503+0.0071
0.522+0.077
0.3
0.549+0.064
0.509=00.055
0.52810.063
0.4
0.55210.037
0.516=0.041
0.53410.037
0.5
0.543+0.037
0.514+0.032
0.529+0.032
0.6
0.52610.021
0.497+0.026
0.516:1..024
0.7
0.483+0.027
0.439+000000
0.460+00.030
0.8
0.396
0.369+00.049
0.383++0.047
0.9
0.23940.035
0.234+0.043
0.239+0.037
Table 11; Full results of divergence functions within MLM with SSCD
aned in SemEval-2020 Task English.

---

Swap rate
Metric
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Divergences
CL)
0.501
0.507
0.490
0.325
0.124
0.022
0.069
0.147
0.214
KL(Ca] C1)
0.463
0.466
0.427
0.235
0.014
0.144
0.218
0.039
0.048
Jeff(Ci (2)
0.475
0.496
0.474
0.291
0.051
0.101
0.079
0.003
0.117
Distance functions
Bray-Curtis
0.378
0.396
0.355
0.179
0.004
0.241
0.229
0.166
0.046
Canberra
0.304
0.420
0.249
0.172
0.015
0.261
0.306
0.223
0.086
Chebyshev
0.268
0.492
0.307
0.088
0.121
0.269
0.315
0.336
0.243
City Block
0.392
0.395
0.363
0.201
0.003
0.241
0.232
0.178
0.032
Correlation
0.454
0.434
0.430
0.226
0.010
0.228
0.112
0.088
0.037
Cosine
0.454
0,434
0.430
0.226
0.010
0.228
0.112
0.088
0.037
Fuclidean
0.401
0.398
0.356
0.162
0.020
0.251
0.145
0.125
0.073
DSCD (Aida and Bollegala, 2023)
Bray-Curtis
0.242
0.298
0.174
0.004
0.028
0.253
0.394
0.444
0.474
Canberra
0.349
0.557
0.253
0.163
0.084
0.328
0.226
0.395
0.426
Chebyshev
0.266
0.106
0.003
0.110
0.073
0.054
0.114
0.267
0.263
City Block
0.259
0.264
0.210
0.099
0.030
0.268
0.303
0.414
0.372
Correlation
0.334
0.427
0.220
0.054
0.051
0.231
0.362
0.563
0.492
Cosine
0.528
0.490
0.219
0.011
0.102
0.295
0.342
0.499
0.489
Euclidean
0.402
0.310
0.233
0.051
0.073
0.111
0.312
0.447
0.467
Table 12: Results within MLM with SSCD
dis in SemEval-2020 Task English. All values are averages over
seeds.

---

Swap rate
Metric
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Divergences
CL)
0.445
0.493
0.507
0.514
0.513
0.523
0.518
0.512
0.488
[L(C2| C1)
0.411
0.467
0.475
0.488
0.486
0.492
0,485
0.472
0.460
leff(Cil]Cg
0.428
0.484
0.490
0.505
0.501
0.512
0.504
0,490
0.470
Distance functions
Bray-Curtis
0.282
0.376
0.370
0.400
0.407
0.409
0.351
0.293
0.195
Canberra
0.238
0.287
0.271
0.330
0.334
0.321
0.239
0.191
0.083
Chebyshev
0.208
0.290
0.334
0.403
0.387
0.412
0.366
0.304
0.261
City Block
0.299
0.392
0.382
0.411
0.422
0.423
0.366
0.303
0.204
Correlation
0.365
0.434
0.444
0.453
0.467
0.469
0.472
0.452
0.396
Cosine
0.365
0,434
0.444
0.453
0.467
0.469
0.472
0.452
0.396
Fuclidean
0.309
0.400
0.391
0.421
0.429
0.426
0.371
0.310
0.214
DSCD (Aida and Bollegala, 2023)
Bray-Curtis
0.375
0.477
0.457
0.471
0.437
0.436
0.410
0.366
0.351
Canberra
0.007
0.209
0.258
0.293
0.324
0.293
0.270
0.242
0.181
Chebyshev
0.158
0.054
0.005
0.114
0.119
0.101
0.150
0.146
0.146
City Block
0.205
0.336
0.365
0.403
0.371
0.374
0.327
0.316
0.279
Correlation
0.263
0.396
0.401
0.440
0.411
0.381
0.356
0.329
0.298
Cosine
0.309
0.438
0.436
0.448
0.431
0.412
0.379
0.355
0.329
Euclidean
0.193
0.351
0.366
0.387
0.385
0.377
0.347
0.337
0.305
Table 13: Results within MLM with normalized SSCD ramd in SemEval-2020 Task English. All values are averages over 20 seeds.

---

Swap rate
Metric
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Divergences
CL)
0.442
0,482
0.538
0.507
0.420
0.223
0.151
0.079
0.237
[L( C1)
0.415
0.449
0.485
0.420
0.232
0.060
0.048
0.037
0.019
Jeff(Cil/C)
0.434
0.473
0.516
0.500
0.345
0.128
0.034
0.041
0.145
Distance functions
Bray-Curtis
0.326
0.408
0.403
0.340
0.246
0.053
0.094
0.113
0.107
Canberra
0.259
0.405
0.313
0.246
0.153
0.036
0.167
0.212
0.141
Chebyshev
0.289
0.436
0.499
0.255
0.009
0.242
0.287
0.312
0.237
City Block
0.336
0.413
0.409
0.338
0.243
0.046
0.104
0.112
0.105
Correlation
0.385
0.424
0.461
0.428
0.282
0.049
0.018
0.021
0.083
Cosine
0.385
0,424
0.461
0.428
0.282
0.049
0.018
0.021
0.083
Fuclidean
0.346
0.405
0.413
0.349
0.232
0.031
0.076
0.049
0.163
DSCD (Aida and Bollegala, 2023)
Bray-Curtis
0.475
0.377
0.419
0.217
0.057
0.159
0.254
0.370
0.470
Canberra
0.305
0.449
0.271
0.217
0.097
0.042
0.209
0.375
0.313
Chebyshev
0.190
0.105
0.042
0.022
0.211
0.259
0.157
0.402
0.251
City Block
0.336
0.394
0.304
0.265
0.089
0.006
0.186
0.441
0.463
Correlation
0.414
0,440
0.344
0.186
0.120
0.158
0.287
0.480
0.443
Cosine
0.348
0.346
0.244
0.114
0.050
0.176
0.236
0.369
0.424
Euclidean
0.302
0.430
0.244
0.089
0.016
0.277
0.377
0.327
0.451
Table 14: Results within MLM with normalized SSCDass SemEval-2020 Task averages over 20 seeds.
English. All values are


Reviews:
[{"review_id": "hPfUBnf1yW", "reviewer": "Reviewer_8L1o", "summary": "", "questions": "A) Why have you excluded Latin (SemEval 2020 Task 1) from the experiments? \n\nB) Why did you not consider Mahalanobis distance which is usually used to compare Gaussian distributions?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort to provide insightful feedback on our paper. Below, we provide a summary of the main points and our corresponding responses.\n\nQ1: *the experiments disproportionately focus on distance and divergence metrics*  \nA1: In this paper, we do not propose distance/divergence metrics, but simply use in the experiments what has already been proposed in previous studies. This imbalance arises because the previous studies happened to use more distance metrics compared to divergence measures.\n\nQ2: *Contrary to the authors' assertion of achieving SOTA results for English, their comparison with [2] under the same setting without the SSCD approach yields a result of 0.571 [4].*   \nA2: Our experiments were conducted with uniformly lemmatised data in all languages. The high score in [2] is due to using the original data, which is 0.493 when compared with the lemmatised dataset under the same conditions, and our method achieves a higher performance. As mentioned in [2], our method is expected to perform even better if it is able to accurately extract usages from original data for any language, taking into account their utilisation. We will include this discussion in the camera ready version, if accepted.\n\nQ3: *Why have you excluded Latin (SemEval 2020 Task 1) from the experiments?*  \nA3: In our evaluations, we focused on languages that are currently being actively used.\n\nQ4: *Why did you not consider Mahalanobis distance which is usually used to compare Gaussian distributions?*  \nA4: Please note that our proposed swapping-based semantic change detection method is independent of any particular distance function. Our main contribution of this paper is to propose and analyse the swapping method, and we do not propose any distance/divergence metrics. Instead, we follow Aida+Bollegala [ACL findings 2023] for the distance/divergence measures. As for the specific question about Mahalanobis distance, it can also be used as a distance measure with our proposed SSCD. However, please note that we approximate the covariance matrices of Gaussians to be diagonal in our experiments, which makes Mahalanobis distance somewhat similar to the Euclidean in this case.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper offers a compelling contribution to the field of lexical semantic change detection while drawing parallels to the work proposed by [1], introducing an element of innovation to the field and bringing forth a new way to evaluate the significance of detected semantic shifts.", "reasons_to_reject": "Due to several significant concerns, the paper's experimental setting needs more convincing evidence to support its claims. While the proposed approach is independent of specific models or measures of lexical semantic change, the experiments disproportionately focus on distance and divergence metrics. It is essential to assess the approach independently of specific models to gauge its potential to enhance or diminish the performance of state-of-the-art (SOTA) models such as [2] or [3]. Contrary to the authors' assertion of achieving SOTA results for English, their comparison with [2] under the same setting without the SSCD approach yields a result of 0.571 [4]. This incongruity raises questions about the reliability of the proposed method's effectiveness, underscoring the need for a more comprehensive and accurate experimental design.", "questions_to_authors": "A) Why have you excluded Latin (SemEval 2020 Task 1) from the experiments? \n\nB) Why did you not consider Mahalanobis distance which is usually used to compare Gaussian distributions?", "paper_topic_and_main_contributions": "The presented research introduces Swapping-based Semantic Change Detection (SSCD), an unsupervised method designed to predict shifts in word meaning between distinct text corpora. By swapping contexts between two corpora and analyzing the distribution of contextualized word embeddings from a pre-trained masked language model (MLM), SSCD determines whether a target word's semantics change. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Haim Dubossarsky, Daphna Weinshall, and Eitan Grossman. 2017. Outta Control: Laws of Semantic Change and Inherent Biases in Word Representation Models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1136–1145, Copenhagen, Denmark. Association for Computational Linguistics.\n\n[2] Severin Laicher, Sinan Kurtyigit, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2021. Explaining and Improving BERT Performance on Lexical Semantic Change Detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 192–202, Online. Association for Computational Linguistics.\n\n[3] Pierluigi Cassotti, Lucia Siciliani, Marco DeGemmis, Giovanni Semeraro, and Pierpaolo Basile. 2023. XL-LEXEME: WiC Pretrained Model for Cross-Lingual LEXical sEMantic changE. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1577–1585, Toronto, Canada. Association for Computational Linguistics.\n\n[4] Francesco Periti, Alfio Ferrara, Stefano Montanelli, and Martin Ruskov. 2022. What is Done is Done: an Incremental Approach to Semantic Shift Detection. In Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change, pages 33–43, Dublin, Ireland. Association for Computational Linguistics."}}, {"review_id": "LDeOfXKlU3", "reviewer": "Reviewer_UWZX", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort to provide insightful feedback on our paper. Below, we provide a summary of the main points and our corresponding responses.\n\nQ1: *When analyzing the ranking task for the SemEval English dataset, it is important to consider and mention other results, such as:\nFinetuned/Trained models: [4] reports .757\nPretrained models: Other works exhibit superior results such as .571 [5]*  \nA1: First, please note that [4] uses many external datasets in its model. This makes it difficult to adapt to languages that are not present in the external datasets. On the other hand, we show that our method can be adapted to any language model, and that it is sufficient to detect semantic change even in the pre-trained multilingual model.\nSecond, our experiments were conducted with uniformly lemmatised data in all languages. The high score in [2] is due to using the original data, which is 0.493 when compared with the lemmatised dataset under the same conditions, and our method achieves a higher performance. As mentioned in [2], our method is expected to perform even better if it is able to accurately extract usages from original data for any language. This discussion will be added to the camera ready.\n\nQ2: *Specifically, the experiments might benefit from a more explicit explanation of the specific advantages offered by each measure, along with the underlying intuition behind the methods that achieved the most favorable result.*  \nA2: As already stated in Lines 465-467 (Table 2), divergence-based measures outperform distance-based metrics for SCD. Among the distance-based metrics, Cosine and Correlation outperform the others. We believe this phenomenon is due to the fact that by swapping sentences in two time periods, the set of vectors after the swap becomes sparse, so that the average part of the set alone cannot be used to properly calculate the similarity of the swapped sentences. We also consider that this is the same as the example of “cell” in Aida+Bollegala [ACL findings 2023], where the set of vector increases by acquiring a new meaning of “cell phone“, while retaining the old meanings of “prison” and “biology,” and the mean component of the vector set alone cannot detect the semantic change. Due to the 8 page limit, we could not include the above details in the paper, but with the extra 1 page allowed, we can include this discussion in the camera ready paper if accepted.\n\nQ3: *row 226 regarding the frequency of words, should be better motivated, as the impact of frequency is not discussed throughout the paper.*  \nA3: In the method of Liu et al. [Eval4NLP 2021], low-frequency words are removed from the prediction results. In order to make appropriate predictions for infrequent words, we use context swapping to calculate the degree of semantic change. In experiments, on benchmarks with words with a frequency of less than 10 times, such as Liverpool FC, we achieved comparable performance on the pre-trained MLM against the finetuned MLM of Liu et al. [Eval4NLP 2021].\n\nQ4: *A comparison between mBERT and XLM-R (or monolingual BERT) may potentially provide valuable insights and enhance the overall contribution of the paper.*  \nA4: By using a multilingual MLM we can use the same model for multiple languages, without having to train separate models for each language. To investigate the feasibility of this approach, we used mBERT as the base MLM in these evaluations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "While the proposed method is relatively straightforward, its originality advances research in the field of semantic change detection. It prompts an exploration into the reliability and robustness of state-of-the-art approaches, thereby complementing the use of the permutation test to validate prediction reliability. However, although the proposed methods outperform the compared approach, they do not surpass the state of the art, as mentioned. A more in-depth discussion of the results is required.", "reasons_to_reject": "The paper's comparison is not exhaustive, as there exist other state-of-the-art approaches that exhibit superior performance. While it is fair to exclude comparisons with approaches that involve special training or fine-tuning (which should be presented separately, e.g. [1]), it is essential to include comparisons with other approaches that use embeddings from pre-trained or fine-tuned (through continued pretraining) models. In this context, the paper does not beat the state-of-the-art methods, as indicated in the abstract (row 024), since there are other approaches that outperform it (row 184, i.e. [1]). Instead, it outperforms the compared method (row 232, 397).\n\nWhen analyzing the ranking task for the SemEval English dataset, it is important to consider and mention other results, such as: \n\nFinetuned/Trained models:\n- [1] presents results indicating that the smaller the model, the better the performance. In particular, [1] reports scores of .520 with BERT-base, .589 with BERT-small, and .627 with BERT-tiny\n- [2,3] report a score of .605 using ELMo\n- [4] reports .757\n\nPretrained models:\n- Other works exhibit superior results such as .571 [5] \n\nAlthough SSCD achieved a slightly inferior score (.563), it cannot be argued that it outperformed the state of the art. For a comprehensive comparison across various languages, I suggest referring to [6]. \n\nMoreover, it would be beneficial to provide a more comprehensive justification and discussion regarding the use of different distances and context sampling strategies (the sampling strategies has the potential to be very interesting if further investigated). Specifically, the experiments might benefit from a more explicit explanation of the specific advantages offered by each measure, along with the underlying intuition behind the methods that achieved the most favorable result.\n\nOther statements, such as the one in row 226 regarding the frequency of words, should be better motivated, as the impact of frequency is not discussed throughout the paper. If it refers to the potential use of pre-trained models instead of training a new model to handle low-frequent word, the reference to the previous approach by Liu et al. is not suitable, given that they also employ pre-trained models.\n\nAs a minor point:\nLastly, it would be beneficial if the authors could provide some justification for their choice to use multilingual BERT (mBERT). Typically, multilingual models are commonly used for the following reasons: i) addressing the absence of monolingual models in HuggingFace for Latin, ii) leveraging the models' cross-lingual transferability property, iii) leveraging the power of XLM-R. However, this paper does not test the Latin benchmark, nor does it exploit cross-lingual transferability or discuss the use of XLM-R. Thus, it might be worth considering to use monolingual models that may potentially yield higher results in this specific context. A comparison between mBERT and XLM-R (or monolingual BERT) may potentially provide valuable insights and enhance the overall contribution of the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors address the problem of identifying lexical semantic changes, i.e. words that change their meaning over time. They propose an original and straightforward approach called Swapping based Semantic Change Detection (SSCD), which involves randomly swapping contexts of a target word between the two corpora and analyzing the distribution of contextualized word embeddings obtained from a pretrained BERT model. SSCD proved to be effective over the SemEval-2020 Task1 benchmark for English, German, and Swedish.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- [1] Guy D. Rosin and Kira Radinsky. 2022. Temporal Attention for Language Models. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1498–1508, Seattle, United States. Association for Computational Linguistics.\n- [2] Andrey Kutuzov, Erik Velldal, and Lilja Øvrelid. 2022. Contextualized embeddings for semantic change detection: Lessons learned. In Northern European Journal of Language Technology, Volume 8, Copenhagen, Denmark. Northern European Association of Language Technology.\n- [3] Andrey Kutuzov and Mario Giulianelli. 2020. UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 126–134, Barcelona (online). International Committee for Computational Linguistics.\n- [4] Pierluigi Cassotti, Lucia Siciliani, Marco DeGemmis, Giovanni Semeraro, and Pierpaolo Basile. 2023. XL-LEXEME: WiC Pretrained Model for Cross-Lingual LEXical sEMantic changE. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1577–1585, Toronto, Canada. Association for Computational Linguistics.\n- [5] Severin Laicher, Sinan Kurtyigit, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2021. Explaining and Improving BERT Performance on Lexical Semantic Change Detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 192–202, Online. Association for Computational Linguistics.\n- [6] Montanelli, Stefano, and Francesco Periti. \"A Survey on Contextualised Semantic Shift Detection.\" arXiv preprint arXiv:2304.01666 (2023).", "Typos_Grammar_Style_and_Presentation_Improvements": "- row 011: occur -> occurs\n- row 056: challening -> challenging\n- row 089: distribitions -> distributions\n- row 114: distribition -> distribution\n- row 135: establisning -> establishing\n- row 145: unsupervided -> unsupervised\n- Table 2: unsupervied -> unsupervised\n- Table 2: averagd -> averaged\n- row 572: semanctic -> semantic\n- row 488: the curves are not overlapping, but identical. Might there be an error? Otherwise you should explain better this behavior.\n- Table 4: Is the value of .627 correct? Should it instead be .563? If this is not the case, please provide a better explanation and discussion of the value"}}, {"review_id": "My86ovZFCQ", "reviewer": "Reviewer_Wzo4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort to provide insightful feedback on our paper. Below, we provide a summary of the main points and our corresponding responses.\n\nQ1: *The method would not work for changes that happened in the past.*  \nA1: Although the model performance depends on the MLM being used, this is orthogonal to the main technical contributions we make in this paper, which is a swapping-based sampling method for semantic change detection (SCD). If we know the duration within which we must detect semantic changes of words, we can pre-train/fine-tune MLMs on historical data from that time period to obtain an MLM that is better suited for the SCD task.\n\nQ2: *There is also little intrinsic exploration of the correctness of the meaning representations of these contextual embeddings in the paper.*  \nA2: According to the distributional hypothesis, we assume that contextual embeddings to reflect word meanings. If there are more accurate models for word meaning representation, we can use those instead of contextualised word embeddings to further improve the accuracy of SCD with our proposed method.\n\nQ3: *As the authors note, the method is only applied to change detection over relatively long periods of time, in which the task is easier than e.g. recent temporal generalization.*  \nA3: Please note that we not only conduct long-term SCD such as over 50 years, but also short-term SCD of around 5 years, as in the Liverpool FC benchmark.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper contributes a methodological innovation in the area of semantic change detection and beats the state of the art at the time of this paper's submission. It is demonstrated for several languages, and appears to work well also for low-frequency words which is important in specific domains. Classification decisions appear to be statisticlly significant.\n", "reasons_to_reject": "While the performance looks good, the method seems to depend on contextual embeddings derived from models pretrained on contemporary language data. It seems to me that these contextual embeddings should be less accurate, the further away we get from the training data. It might thus be the case that this method would not work well if we are not dealing with a change from the relatively recent past from the present. How would the approach perform on semantic changes between 1800-1850 rather than 1970-2020? Probably a lot worse, but this is not really explored. Of course, there is no gold standard for this, but the limitation could have been discussed or perhaps qualitatively explored. There is also little intrinsic exploration of the correctness of the meaning representations of these contextual embeddings in the paper.\n\nAlthough the state-of-the-art is beaten and there is methodological innovation, this methodological innovation is limited in scope, relying on some other recent work. The swapping had been done before, just not for low-frequency words and without model tuning.\n\nAs the authors note, the method is only applied to change detection over relatively long periods of time, in which the task is easier than e.g. recent temporal generalization", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper tackles the task of semantic change detection (by way of the standard SemEval 2020 Task 1). It introduces a new method that involves obtaining contextual embeddings for the target word in corpora from different times, and comparing their distributions. Various different distance and divergence measures are tried to compare the distributions. The method beat the state-of-the-art result for this task, and the authors compare the metrics to various other recent approaches to this task. The task is done for English, German and Swedish, and an advantage of the method is that there is no need to tune a model.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "518-519: esbalises a nove SOTA"}, "extra_text": {}}, {"review_id": "tgWxDqrblQ", "reviewer": "Reviewer_SSFX", "summary": "", "questions": "* The statement on line 113 is not precise. The authors should specify which chapter and section of Dekking et al. (2005) they're referring to.\n\n* In the experiments where the swapping rate is optimized (Fig 2), why doesn't the divergence reach 0? Is this related to random noise and differences in size corpora and other factors for which the swapping rate compensates? How is the swapping rate related to permutations of combined corpora as is standard in permutation significance tests?\n\n* I don't understand the experiment with swap-rate = 0. Do you still use Algorithm 2 for Table 4 or the distance $e_{original}$ only?\n", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort to provide insightful feedback on our paper. Below, we provide a summary of the main points and our corresponding responses.\n\nQ1: *The sampling distribution cannot be used unless multiple samples can be drawn from the distribution.*  \nA1: Even though only two snapshots C1 and C2 might be given, we can still obtain subsamples (with replacement) to make a more reliable estimate, as done by our proposed method SSCD. Therefore, we disagree with the statement \"this is the only option if C1 and C2 are seen as samples from some distribution\".\n\nQ2: *from line 214, it is not clear how this is achieved by SSCD*  \nA2: This is already described in Lines 311-322 in the paper. We can perform a bootstrapping test using the value $|e_{original} - e_{swap}|$ if we would like to compute p-values. However, note that this is not necessary for the graded semantic change detection tasks we conduct in the paper.\n\nQ3: *This score is interpreted as a degree of semantic shift and used only to rank words.*  \nA3: Please see the response to the previous (Q2) comment/question.\n\nQ4: *the argument about efficiency made in section 4.1 is not valid*  \nA4: Fine-tuning of large MLMs is indeed very expensive. Parameter-efficient fine-tuning methods such as LoRA [Hu et al., ICLR 2022] have been proposed specifically due to this efficiency reason. Therefore, we believe it is an important distinction especially when using semantic change detection (SCD) methods at scale using large MLMs.\n\nQ5: *The superiority of the proposed approach in (some of) the tasks described in the introduction is yet to be demonstrated.*  \nA5: It is not clear what are these \"some of tasks\" you are referring to. We have conducted evaluations on two different datasets and on three different languages, which is over and above the evaluations conducted in prior work on this topic.\n\nQ6: *However, in my opinion, detecting new meanings of words using paradigmatic models is not a completely automated process.*  \nA6: We do not claim SCD is a fully automatic process either. As you point out, although distributional changes are usually associated with semantic changes of words, there could be other reasons as well. On the other hand, the SSCD scores that we compute will help humans (linguists for example) to filter candidate words that might have undergone a semantic change for their manual investigation, which will save their time and effort.\n\nQ7: *The statement on line 113 is not precise. The authors should specify which chapter and section of Dekking et al. (2005) they're referring to.*  \nA7: We refer to section 17.2 in Dekking et al. (2005). We will include this information in the camera ready version as well, if accepted.\n\nQ8: *In the experiments where the swapping rate is optimized (Fig 2), why doesn't the divergence reach 0?*  \nA8: Yes, this is exactly because of the random noise in the data.\n\nQ9: *I don't understand the experiment with swap-rate = 0. Do you still use Algorithm 2 for Table 4 or the distance $e_{original}$ only?*  \nA9: Yes, we use $e_{original}$ only when swap rate = 0.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The idea is simple and somewhat novel up to my knowledge.\n* It performs well on two datasets.", "reasons_to_reject": "* The motivation given between lines 104 and 125 needs some clarification. The authors write that \"previous approach provides only a single estimate\" based on corpora C1 and C2. But this is the only option if C1 and C2 are seen as samples from some distribution. The sampling distribution cannot be used unless multiple samples can be drawn from the distribution.\n\n* The proposed SSCD seems more like a test for the difference between two samples similar to the permutation test for statistical significance used by Liu et al. (2021) but without relying on a predefined p-value. While the authors point to these similarities and discuss the limitations to overcome starting from line 214, it is not clear how this is achieved by SSCD. The authors only point to increased performance on SemEval but no insight is given.\n\n* The authors discuss rejecting (or not) the null hypothesis in section 3.1 based on the value of $|e_{original} - e_{swap}|$ but no further details are given. This score is interpreted as a degree of semantic shift and used only to rank words.\n\n* While Liu et al. (2021) fine-tune the LM before computing word embeddings, fine-tuning can be skipped. On the other hand, the authors can also test SSCD with fine-tuned LM. Therefore, the argument about efficiency made in section 4.1 is not valid unless some principled reason is provided for the choice of fine-tuning or lack thereof. Liu et al. (2021) and Martinc et al. (2020) also don't discuss this question.\n\n* The superiority of the proposed approach in (some of) the tasks described in the introduction is yet to be demonstrated.\n\n* I understand that distributional models are models of word meaning, see, for instance, Sahlgren (2008) for a discussion. However, in my opinion, detecting new meanings of words using paradigmatic models is not a completely automated process. It often requires human validation and domain knowledge. There are instances where shifts in distributional properties of words are not necessarily related to new meanings but rather reflect other factors or linguistic phenomena. These factors include trends and use in pop culture, regional variations, slang, neologisms, acronyms, lexical borrowing or simply noise in the used corpora. I think the paper can benefit from a discussion (and analysis) of this issue.\n\n\nReference:\nThe distributional hypothesis, Magnus Sahlgren, Italian Journal of Linguistics, 2008\n", "questions_to_authors": "* The statement on line 113 is not precise. The authors should specify which chapter and section of Dekking et al. (2005) they're referring to.\n\n* In the experiments where the swapping rate is optimized (Fig 2), why doesn't the divergence reach 0? Is this related to random noise and differences in size corpora and other factors for which the swapping rate compensates? How is the swapping rate related to permutations of combined corpora as is standard in permutation significance tests?\n\n* I don't understand the experiment with swap-rate = 0. Do you still use Algorithm 2 for Table 4 or the distance $e_{original}$ only?\n", "paper_topic_and_main_contributions": "This paper proposes a method (SSCD) to measure semantic shift for individual words between two given corpora. It relies on the distance between sibling distributions (Aida and Bollegala, 2023) to compare words and on a context-swapping test similar to the permutation significance test used by Liu et al. (2021). The experiments show that SSCD sometimes correlates on par or better than previous approaches with human ranking of words by their degrees of semantic shifts on two datasets.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The methodology is fairly clear and appears to offer interesting results for the intended taks. The linguistic or cognitive significance are however not explicitly highlighted.
Decision: Accept-Findings