A Semantic Distance Metric Learning approach for Lexical Semantic Change Detection (2024)

Abstract: Detecting temporal semantic changes of words is an important task for various NLP applications that must make time-sensitive predictions. Lexical semantic change detection (SCD) task involves predicting whether a given target word, w, changes its meaning between two different text corpora, C1 and C2. For this purpose, we propose a supervised two-staged SCD method that uses existing Word-in-Context (WiC) datasets. In the first stage, for a target word w, we learn two sense-aware encoders that represent the meaning of w in a given sentence selected from a corpus. Next, in the second stage, we learn a sense-aware distance metric that compares the semantic representations of a target word across all of its occurrences in C1 and C2. Experimental results on multiple benchmark datasets for SCD show that our proposed method achieves strong performance in multiple languages. Additionally, our method achieves significant improvements on WiC benchmarks compared to a sense-aware encoder with conventional distance functions.1

heading: 1 Introduction
text: The notion of word meaning is a dynamic one, and evolves over time as noted by Tahmasebi et al. (2021). For example, the meaning of the word cell has changed over time to include cell phone to its previous meanings of prison and related to biology. Detection of such semantic changes of words over time remains a challenging, yet an important task for lexicography, sociology, and information retrieval (Traugott and Dasher, 2001; Cook and Stevenson, 2010; Michel et al., 2011; Kutuzov et al., 2018). For example, in E-commerce, a user might use the same keyword, such as scarf, to search for different types of products based on seasonal variations, such as silk scarves in
1Source code is available at https://github.com/ LivNLP/svp-sdml .
spring versus woollen scarves in winter. Recently, a decline in the performance of pretrained Large Language Models (LLMs) over time has been observed, and attributed to their training on static snapshots (Loureiro et al., 2022; Lazaridou et al., 2021). Recognising the words with changed meanings enables efficient fine-tuning of LLMs to incorporate only those with specific semantic shifts (yu Su et al., 2022).
Detecting whether a word has its meaning changed between two given corpora, sampled at different points in time, requires overcoming two important challenges, which we name as the representational and measurement challenges.
Representational Challenge: A word can take different meanings in different contexts even within the same corpus. Therefore, creating a representation for the meaning of a word across an entire corpus is a challenging task compared to that in a single sentence or a document. Prior work has averaged static (Kim et al., 2014; Kulkarni et al., 2015; Hamilton et al., 2016; Yao et al., 2018; Du-
7570
bossarsky et al., 2019; Aida et al., 2021) or contextualised (Martinc et al., 2020; Beck, 2020; Kutuzov and Giulianelli, 2020; Rosin et al., 2022; Rosin and Radinsky, 2022) word embeddings for this purpose, which is suboptimal because averaging conflates multiple meanings of a word into a single vector.
Measurement Challenge: We require a distance metric that can accurately measure the semantic change of a target word between two given corpora, using the learnt representations of the target word from each corpus. This challenge has been addressed in prior work using parameter-free distance functions due to the lack of labelled training data to provide any supervision for SCD (Kutuzov and Giulianelli, 2020; Card, 2023; Cassotti et al., 2023; Aida and Bollegala, 2023b,a; Tang et al., 2023).
To address those challenges, we propose Semantic Distance Metric Learning (SDML), a supervised two-staged SCD method. To solve the representational challenge, we learn a senseaware2 encoder from sense-level supervision that represents the meaning of a target word in a sentence. Word sense information has shown to be useful for SCD (Rachinskiy and Arefyev, 2021; Arefyev et al., 2021; Cassotti et al., 2023; Tang et al., 2023) and outperform word representations that encode only time (Rosin et al., 2022; Rosin and Radinsky, 2022). Following Cassotti et al. (2023), we use WiC (Pilehvar and CamachoCollados, 2019) data, annotated for word sense discrimination for training the sense-aware encoder.
To solve the measurement challenge, we propose a method to learn a sense-aware distance metric to compare two sense-aware embeddings for a target word in sentences selected from the two corpora. Specifically, the distance metric is trained such that it returns a smaller value between two sentences where the target word takes the same meaning compared to that between two sentences which express different meanings of the target word. We learn a distance metric that satisfies this criteria, using the weak-supervision provided by existing WiC datasets.
2In this paper, the term ‘sense’ refers not to strictly defined senses as found in dictionaries or WordNet (Fellbaum and Miller, 1998), but rather to the nuances of word meanings derived from the distributional hypothesis (Harris, 1954).
Experimental results show that SDML achieves exceptional performance in the SCD task. Remarkably, SDML obtains performance improvements of 2-5% over the current strong baselines. While our focus is on the SCD task, we also explore the applicability of our SDML to the WiC task. In WiC benchmarks, SDML significantly enhances performance in multiple languages. These results demonstrate the effectiveness of learning both a sense-aware encoder and a sense-aware distance metric.

heading: 2 Related Work
text: The diachronic semantic changes of words have been extensively investigated by linguists to explore how meanings evolve over time (Traugott and Dasher, 2001). In recent years, the advent of diachronic corpora and advancements in representing word meanings have paved the way for automated SCD in NLP (Tahmasebi et al., 2021). To detect semantically changed words in time-separated corpora, unsupervised SCD methods compare word embeddings trained on the target corpora. Many methods exist that align vector spaces over time spanned by static word embeddings, such as initialisation (Kim et al., 2014), alignment (Kulkarni et al., 2015; Hamilton et al., 2016), and joint learning (Yao et al., 2018; Dubossarsky et al., 2019; Aida et al., 2021). Likewise, methods for comparing sets of target word representations computed as contextualised word embeddings have also been proposed. Such methods include comparing the average (Martinc et al., 2020; Beck, 2020; Kutuzov and Giulianelli, 2020; Laicher et al., 2021; Giulianelli et al., 2022; Rosin et al., 2022; Rosin and Radinsky, 2022) or each pair of embeddings (Kutuzov and Giulianelli, 2020; Laicher et al., 2021). Additionally, Aida and Bollegala (2023b) and Nagata et al. (2023) have proposed methods that consider the variance in the sets of embeddings. These unsupervised SCD methods have been evaluated using unsupervised SCD tasks (Del Tredici et al., 2019; Schlechtweg et al., 2020; Kutuzov and Pivovarova, 2021).
Due to the lack of manually-labelled data for lexical semantic change, prior work on SCD have been limited to unsupervised approaches. A notable exception is XL-LEXEME (Cassotti et al., 2023), a supervised SCD method that uses senselevel supervision. They focused on the WiC task (Pilehvar and Camacho-Collados, 2019) de-
signed to detect semantic differences of a target word in a given pair of sentences. XL-LEXEME fine-tunes a pretrained multilingual Masked Language Model (MLM) across multilingual WiC datasets, and achieves strong performance in some SCD tasks. Recent work show SCD is a challenging task for LLMs such as GPT-3.5 (Sorensen et al., 2022; Periti et al., 2024), reporting low performance even with carefully designed prompts.

heading: 3 Semantic Distance Metric Learning
text: Given a target word w and two corpora C1 and C2 sampled at distinct time points, the goal in SCD is to predict a score for w that indicates the degree of semantic change undergone by w between C1 and C2. For this purpose, we propose, SDML, which overcomes the two challenges introduced in § 1. Specifically, to address the representational challenge, we first learn a sense-aware encoder in § 3.1. Next, to overcome the measurement challenge, we learn a sense-aware distance metric between two sense-aware representations of a target word computed from C1 and C2, as described in § 3.2.

heading: 3.1 Learning Sense-Aware Encoder
text: Following prior work on SCD that show contextualised word embeddings to outperform the static word embeddings (Aida and Bollegala, 2023b; Rosin and Radinsky, 2022), we represent the meaning of a word w in a sentence s by the ddimensional token embedding,3 f(w, s; θ)(∈ Rd), obtained from a pretrained MLM parametrised by θ. Although contextualised word embeddings capture the contextual information of a word in a sentence (Zhou and Bollegala, 2021), explicitly encoding word sense information has been shown to improve performance in SCD tasks (Tang et al., 2023).
For this purpose, we follow Cassotti et al. (2023) and fine-tune the MLM on WiC. WiC contains tuples (w, s1, s2, y), where the label y = 1 if w has the same meaning in both sentences s1 and s2, and y = 0 otherwise. We use a Siamese biencoder approach (Reimers and Gurevych, 2019), which uses two encoders to produce respective embeddings for pairs of sentences. This architecture enables us to capture the semantic relationship between the pairs of sentences. In this
3When a word has been tokenised into multiple subtokens, we compute the average of the subtoken embeddings to create the token embedding
paper, we use the same MLM encoder to obtain two representations, w1 = f(w, s1; θ) and w2 = f(w, s2; θ) for the meaning of w in s1 and s2. We then update θ such that the contrastive loss, ℓc (Hadsell et al., 2006) given by (1) is minimised.
ℓc(θ) = 1
2
( yδ2 + (1− y)max(0,m− δ)2 ) (1)
Here, we set the margin m = 0.5 and δ = 1 − (w1⊤w2)/ ||w1|| ||w2|| is the cosine distance. Note that the same encoder is used to compute both w1 and w2 here. We use AdamW (Loshchilov and Hutter, 2019) as the optimiser to minimise the contrastive loss in (1) with the initial learning rate set to 10−5 and the weight decay coefficient set to 0.01.

heading: 3.2 Learning Sense-Aware Distance Metrics
text: Armed with the sense-aware encoder trained in § 3.1, we are now ready to learn a sense-aware distance metric h(w1,w2;A) given by (2) that measures the semantic distance between two senseaware embeddings w1 = f(w, s1; θ) and w2 = f(w, s2; θ), for w in two sentences s1 and s2, respectively.
h(w1,w2;A) = (w1 −w2)⊤A(w1 −w2) (2)
Here, A ∈ Rd×d is a Mahalanobis matrix (assumed to be positive definite) that defines a (squared) distance metric, Mahalanobis distance. Unlike conventional distance functions such as Euclidean distance, Mahalanobis distance weights each dimension according to its variance and accounts for cross-dimensional correlations.
We use the WiC training data instances to learn A in (2). Specifically, we consider two types of constraints that must be satisfied by h as follows. For instances where y = 1, the distance between w1 and w2 is required to be greater than a distance lower bound ℓ such that, h(w1,w2) ≥ ℓ. Likewise, for instances where y = 0, the distance between w1 and w2 is required to be lower than a distance upper bound u such that, h(w1,w2) ≤ u. There exists a bijection (up to a scaling function) between the set of Mahalanobis distances and the set of equal mean (denoted by µ) multivariate Gaussian distributions given by p(w;A) = 1Z exp ( −12h(w,µ;A) ) , where Z is a normalising constant and A−1 is the covariance of the distribution. We can use this bijection to measure the distance between two Mahalanobis distance functions parametrised by positive definite
matrices A0 and A using the relative entropy (KLdivergence) between the corresponding multivariate Gaussians, given by (3).
KL(p(w;A0)||p(w;A)) = ∫
p(w;A0) log p(w;A0) p(w;A) dw
(3)
Relative entropy is a non-negative and convex function in A. The overall optimisation problem is then given by (4).
min A
KL(p(w;A0)||p(w;A))
subject to h(w1,w2) ≤ u y = 1, (4) h(w1,w2) ≥ ℓ y = 0.
The optimisation problem given in (4) can be expressed as a particular type of Bregman divergence, which can be efficiently solved using the Bregman’s method (Censor and Zenios, 1997). We use Information-Theoretic Metric Learning (ITML) (Davis et al., 2007) for learning a Mahalanobis matrix A that satisfies those requirements. Further details of ITML are described in Appendix A.

heading: 3.3 Measuring Temporal Semantic Change
text: Using the sense-aware distance metric h and the sense-aware encoder f learnt as described in the previous sections, we can now compute the semantic change score, α(w,C1, C2), of w between C1 and C2 as the average pairwise distance computed over S1(w) and S2(w), given by (5).
1
n1n2
∑
s1∈S1(w) s2∈S2(w)
h(f(w, s1; θ),f(w, s2; θ);A) (5)
Here, we denote the set of sentences where w occurs in Ci to be Si(w) for i = 1, 2, and the corresponding number of sentences in each set to be ni = |Si(w)|. Unlike much prior work in SCD, which first computes a single vector (often by averaging) for a target word w from a corpus, thereby conflating its different meanings, (5) computes representations per each occurrence of w and compares the average over all distances. Although the total number of summations, n1n2, in (5) could become large for frequent w, semantic change scores can be efficiently computed by pre-computing and indexing the sense-aware embeddings for all occurrences of w in each corpus. Moreover, the summation in (5) can be computed in parallel over different batches of sentences to obtain a highly efficient Map-Reduce version (Dean and Ghemawat, 2004).

heading: 4 Experiments

heading: 4.1 Setting
text: To learn the sense-aware encoder and the distance metric described in § 3, we use WiC datasets covering multiple languages as shown in Table 1: XLWiC (Raganato et al., 2020), MCL-WiC (Martelli et al., 2021), and AM2iCo (Liu et al., 2021).4
We use the sense-aware encoder released by Cassotti et al. (2023) as f ,5 which is based on XLM-RoBERTalarge (Conneau et al., 2020) for the remainder of the experiments with SDML reported in this paper. We use the metric learn6 package to train a sense-aware distance metric with ITML. The slack parameter γ in ITML weights the margin violations, and is searched from the 11 values in {10−5, 10−4, ..., 100, ..., 104, 105}, according to the best performance measured on the development data in each WiC dataset.
After that, we evaluate the performance on SCD tasks. In this paper, we use the two benchmark datasets – SemEval-2020 Task 1 (Schlechtweg et al., 2020) (covering English (En), German (De), Swedish (Sv) and Latin (La)) and RuShiftEval (Kutuzov and Pivovarova, 2021) (covering Russian (Ru)), which have also been used in prior work on SCD (Kutuzov et al., 2021; Giulianelli
4XL-WiC and MCL-WiC are licensed under a Creative Commons Attribution-NonCommercial 4.0 License, and AM2iCo is licensed under a Creative Commons Attribution 4.0 International Public license.
5This model is available at https://huggingface.co/ pierluigic/xl-lexeme
6https://github.com/scikit-learn-contrib/ metric-learn
et al., 2022; Cassotti et al., 2023).7 Statistics of those datasets are summarised in Table 2.
For English, German and Russian, there exist WiC training data splits that we can use to train SDML separately for each of those languages. However, no such training data are available for Latin and Swedish languages. Therefore, when evaluating SDML for Latin, we train it on the WiC training data available for Italian and French, which are in the same Romance language family. Likewise, when evaluating SDML for Swedish, we train it on the WiC training data available for German, which is in the same Germanic language family. Creating WiC datasets for languages that do not have such resources is beyond the scope of this paper and is deferred to future work.
Before presenting the results of the SCD task, we first look at the performance of our SDML on the WiC task for languages corresponding to the SCD benchmarks. As a baseline, we use a sense-aware encoder fine-tuned to distinguish the different meanings of a target word by optimising (1). For SDML, the sense-aware distance metric is learned via ITML using WiC datasets, classification boundaries are also obtained. Therefore, we can use these boundaries for the prediction of the WiC tasks.
7SemEval-2020 Task 1 and RuShiftEval are licensed under a Creative Commons Attribution 4.0 International License and a GNU General Public License version 3.0, respectively.
Table 3 shows that the combination of the sense-aware encoder and the sense-aware distance metric (our SDML) constantly outperforms the baseline. Interestingly, SDML achieves significant improvements at the 95% confidence interval computed using Bernoulli trials in all languages. These results support our hypothesis: even in the task of detecting semantic differences at the same time period, it is better to use the combination of the sense-aware encoder and the sense-aware distance metric than the sense-aware encoder only. More comprehensive results can be found in § 4.3.

heading: 4.2 Evaluating Semantic Changes of Words
text: We compare the performance of SDML against prior work on several SCD benchmarks. In this evaluation, given a set of target words, an SCD method under evaluation is required to predict scores that indicate the degree of semantic changes undergone by each word in the set. The Spearman’s rank correlation coefficient r (∈ [−1, 1]) between those predicted semantic change scores and that assigned by the human annotators in each benchmark dataset (i.e. gold ratings) is computed as the evaluation metric. An SCD method that reports a high Spearman’s r value indicates better agreement with human ratings, and is considered to be desirable for detecting the semantic changes of words over time.
We compare the proposed methods against strong baselines as described next.
Baselines: Kutuzov and Giulianelli (2020) and Laicher et al. (2021) showed that the average pairwise cosine distance of the sets of contextualised embeddings perform well (APD). Aida and Bollegala (2023a) proposed swapping-based SCD, conducting context-swapping across target corpora to obtain more reliable scores (SSCD). Card (2023) proposed a token replacement-based JS divergence metric, to mitigate the influence of token frequency by replacing tokens with neighbouring words (ScaledJSD). Rosin and Radinsky (2022) proposed temporal attention, additional time-aware attention mechanism to MLMs. SCD scores are calculated by the average cosine distance (w/ TA + CD). Giulianelli et al. (2022) introduced an ensemble method combining the average cosine distance from MLMs with the cosine distance between morpho-syntactic features labelled from universal dependencies (CD & UD + CD).
Sense-Aware Methods: Here, we introduce sense-aware methods that use sense-level supervision. DeepMistake (Arefyev et al., 2021) and GlossReader (Rachinskiy and Arefyev, 2021) both leverage data from the WiC and Word Sense Disambiguation tasks to fine-tune MLMs. Subsequently, both methods make predictions using linear regression trained on the provided training data. XL-LEXEME is fine-tuned on WiC data across multiple languages. In our method, SCD is performed using the average pairwise distance. For SDML, we predict the semantic change scores for the target words using (5).
Results are shown in Table 4.8 Our method performs comparable or superior to previous senseaware methods using a sense-aware encoder only. Moreover, SDML accomplishes performance improvements of 2-5% over the sense-aware methods9, provided that training data in the corresponding languages is available. In SemEval Sv and La, no training data for the corresponding languages
8In SemEval De, Sv, and La, we have several models due to multiple WiC datasets in the corresponding/related languages, but we report the highest performance.
9In SCD tasks, there is no statistical significance across ALL methods due to the lack of target words when we use the Fisher transformation.
exist, but our method performed equally well for SemEval Sv and showed a drastic performance improvement for SemEval La.
However, the performance in Latin is lower than pretrained MLMs in Table 4. We attribute these results to the composition of the data used during MLM pretraining and SDML training, respectively. Firstly, while Latin is included in the pretraining of XLM-RoBERTa, the dataset for pretraining is only a fraction (1/10 to 1/100) of the size of the datasets for the other languages evaluated in the SCD benchmarks. Secondly, Latin is also absent from the WiC dataset used to train our SDML. Therefore, we believe that the poor information on Latin, which was scarce even during the pretraining phase, was further diluted by the training of the SDML, resulting in a low performance. In the future, as the WiC datasets are expanded and include more languages, it is expected that the performance of models for a wider range of languages will improve.

heading: 4.3 Applying to WiC Prediction
text: Our main focus in this paper has been Lexical Semantic Change Detection – detecting whether a target word, w, has its meaning changed from
one corpus, C1, to another, C2. However, we use WiC datasets for training the sense-aware encoder (described in § 3.1) as well as the senseaware distance metric (described in § 3.2) because there does not exist sufficiently large manually annotated datasets for training temporal SCD methods. Although none of the WiC datasets we used are sampled from temporally distinct corpora, they provide a convenient alternative for training models that discriminate the meaning of a word in two given sentences. Therefore, by evaluating our proposed SDML on benchmark datasets for WiC, we will be able to sanity check whether SDML can indeed detect words that have same/different meanings in two given sentences, even if the two sentences might not be sampled from temporally distinct corpora. In this WiC task, a model is required to predict whether a target word takes the same meaning in two given sentences. This is modelled as a binary classification task and classification accuracy is used as the evaluation metric. A random prediction baseline would report an accuracy of 50% on WiC test datasets, which are balanced.
Baselines: As baselines for comparison, use report the accuracy of binary classifiers that have been trained using different MLMs for the WiC task as follows. Given an instance (w, s1, s2, y)
from a WiC training dataset, the token embeddings f(w, s1) and f(w, s2) are concatenated to represent the meaning of w in s1 and s2. Next, a binary logistic regression classifier is trained on the training data from a WiC dataset. Note that the parameters of the MLMs are not updated during this process. For these baselines, we use the multilingual MLMs such as mBERTbase and XLM-RoBERTabase/large as well as target language-specific BERT models (Lang. BERT) to train the baselines in XL-WiC such as BERTbase-german-cased10 for German, CamemBERTlarge11 for French, and BERT-base-italian-xxlcased12 for Italian. Rather than re-implementing or re-running these methods, we use the results presented in the corresponding benchmarks.
Sense-Aware Methods: Unlike the MLMs used in the above-mentioned baselines, XL-LEXEME uses a sense-aware encoder fine-tuned to discriminate the different meanings of a target word. XL-LEXEME is already fine-tuned using WiC datasets and does not require a binary classifier to
10https://huggingface.co/dbmdz/ bert-base-german-cased
11https://huggingface.co/almanach/ camembert-large
12https://huggingface.co/dbmdz/ bert-base-italian-xxl-cased
be trained, and predicts cosine distances less than the margin m (set to 0.5) to be instances where the target word takes the same meaning in both sentences. For SDML, we use the sense-aware distance metric learned via ITML to predict whether a target word expresses the same meaning in the two given sentences in WiC datasets. Since ITML also learns classification boundaries (i.e. the upper and lower bounds denoted by respectively u and l in Algorithm 1), which can be used to make binary predictions. Note that however, we do not have two different corpora in this WiC-based evaluation, thus not requiring to average the distances as done in (5), and instead can perform a single point estimate considering the two sentences in a single WiC test instance.
Table 5 shows the results for MCL-WiC and XL-WiC datasets. XL-WiC has in-vocab (IV) test sets and out-of-vocab (OOV) test sets extracted from the test set, in addition to the vanilla test set. The results show that our SDML consistently outperforms all of the baselines. It shows 90% accuracy in English, which is a significant improvement compared to XLLEXEME, using the sense-aware distance metric. In German and French, SDML outperforms XLM-RoBERTalarge, which is the same model as XL-LEXEME, as well as language-specific language models (Lang. BERT). Interestingly, even in Italian, where there are only a few thousand training data instances (Table 1), the performance was significantly improved from XL-LEXEME. Moreover, statistical tests reveal that our SDML achieves significant improvements over all other methods on multiple test sets, with statistical significance observed at the 90% or 95% for binomial proportion confidence intervals.
Similarly, in the AM2iCo dataset, Table 6 also demonstrates the performance improvements achieved by SDML. In particular, according to Table 6, SDML achieves the best performance on all test sets. Statistical tests demonstrate the significant performance enhancements of our SDML compared to the other methods on most test sets, with significance observed at the 90% or 95% for binomial proportional confidence intervals. While regular multilingual MLMs and XL-LEXEME do not perform well for low-resource language (Eu) or languages not similar to English (Ja, Zh, Ar, and Ko), sense-aware distance metrics can dramatically improve performance for even just a few
thousand instances.

heading: 5 Ablation Study
text: Motivated by the superior performance reported in § 4.2 by SDML, we conduct an ablation study to understand the importance of (i) using senseaware distance metric, and (ii) considering crossdimensional correlations by Mahalanobis matrix.
Firstly, we conduct a quantitative comparison between our SDML (sense-aware sentence encoder and sense-aware distance metric using full components of Mahalanobis matrix, SDMLfull) and two variants; (i) sense-aware sentence encoder only (Baseline), and (ii) sense-aware sentence encoder and sense-aware distance metric using diagonal components of Mahalanobis matrix (SDMLdiag). Results on SCD tasks are shown in Table 7. While our method achieves comparable or superior performance to the baseline even with the diagonal components of the Mahalanobis matrix (SDMLdiag), using the full Mahalanobis matrix (SDMLfull) yields further improvements. These results indicate the importance of considering inter-dimensional information by the full components of the sense-aware Mahalanobis matrix.
Secondly, we perform a qualitative analysis using gold labels. Following Aida and Bollegala (2023a,b), we pick up the (i) top-8 semantically changed words, and (ii) top-8 semantically stable words in the SemEval En. Additionally, due to the sense-aware method, we also evaluated the relationship between the SCD performance and two factors; polysemy and frequency as described in Hamilton et al. (2016). We count the number of senses (synsets defined in WordNet) and frequency of the evaluation set of words in given corpora. From Table 8, we can see that our method SDMLfull slightly improves prediction. Moreover, polysemous words tend to change their meanings (the law of innovation), but frequent words, contrary to the law of conformity, show no correlation with semantic change, as described in (Hamilton et al., 2016). The same trends are in Table 9. From this table, SDMLdiag has a higher/lower correlation with frequency/polysemy than the baseline, which degrades the performance in SCD. However, in SDMLfull, while a correlation with frequency/polysemy is slightly weaker than the baseline, they contribute to the performance improvement in SCD. We conclude that reliev-
13http://wordnetweb.princeton.edu/perl/webwn
ing/enhancing the effect of frequency/polysemy will likely lead to further improvements in SCD performance.
Much prior work on SCD use the Euclidean distance (or cosine similarity) for measuring semantic change scores of words (Laicher et al., 2021; Giulianelli et al., 2022; Rosin et al., 2022; Rosin and Radinsky, 2022; Cassotti et al., 2023), which (a) weights all dimensions equally, and (b) does not consider cross-dimensional correlations. However, our experimental results show that Euclidean distance is a suboptimal choice for this purpose and learning a Mahalanobis distance met-
ric is more appropriate.

heading: 6 Conclusion
text: We proposed a supervised two-staged SCD method to address two challenges in the SCD tasks: 1) models must obtain sense-aware embeddings of target words over time, and 2) due to the lack of labelled training data, SCD is often an unsupervised task using conventional distance metrics. For the first challenge, we propose to learn a sense-aware encoder. Next, we address the second challenge by learning a sense-aware distance metric to compare sense-aware embeddings. In both stages, we used WiC datasets to provide senselevel supervision. Experimental results show that our proposed method, SDML, achieves strong performance in four SCD benchmarks. Moreover, SDML achieves significant performance improvement in WiC benchmarks. Our findings highlight the importance of learning both a sense-aware encoder and a sense-aware distance metric.
Limitations
We show that our method can achieve strong performance in three languages (English, German, and Russian). However, due to the limited training datasets, our method cannot perform well in other languages such as Swedish and Latin. A potential solution to address this limitation and to further improve our method for languages not covered by existing WiC training datasets, is to explore the possibility of using cross-lingual language transfer methods.
Ethical Considerations
In this paper, we focus on detecting semantic changes of words over time. We did not create new datasets and used existing datasets for WiC and SCD for training and evaluation. To the best of our knowledge, no ethical issues have been reported related to those datasets. However, we used publicly available and pretrained MLMs in this paper, and some of those MLM are known to encode unfair social biases such as gender or race (Basta et al., 2019). It is possible that some of those social biases will be present (and possibly have been amplified) during the sense-aware encoder training process. Therefore, we consider it to be an important and necessary task to evaluate the senseaware encoder that we trained for any social biases before it is used in downstream tasks.

heading: A Details of ITML
text: In this section, we describe the method we use to learn a sense-aware distance metric, h(w1,w2;A), that measures the semantic distance between two sense-aware embeddings w1 = f(w, s1;θ) and w2 = f(w, s2;θ), for w in two sentences s1 and s2, respectively. For the ease of reference, we re-write (2) below as (6), defining this distance metric.
h(w1,w2;A) = (w1 −w2)⊤A(w1 −w2) (6)
Here, A ∈ Rd×d is a Mahalanobis matrix (assumed to be positive definite) that defines a (squared) distance metric.
To learn the Mahalanobis distance matrix A in (6), we use training instances (w, s1, s2, y) from a WiC dataset for a target word w and two sentences s1 and s2, where y = 1 indicates that w has the same meaning in both s1 and s2, whereas y = 0 indicates that w takes different meanings.
We consider two types of constraints that must be satisfied by h as follows. For instances where y = 1, the distance between w1 and w2 is required to be greater than a distance lower bound ℓ such that, h(w1,w2) ≥ ℓ. Likewise, for instances where y = 0, the distance between w1 and w2 is required to be lower than a distance upper bound u such that, h(w1,w2) ≤ u. Recall that there exists a bijection (up to a scaling function) between the set of Mahalanobis distances and the set of the equal mean (denoted by µ) multivariate Gaussian distributions given by p(w;A) = 1 Z exp ( −12h(w,µ;A) ) , where Z is a normalising constant and A−1 is the covariance of the distribution. We can use this bijection to measure the distance between two Mahalanobis distance functions parametrised by A0 and A using the relative entropy ( KL-divergence) between the correspond-
ing multivariate Gaussians, given by (7).
KL(p(w;A0)||p(w;A)) = ∫
p(w;A0) log p(w;A0) p(w;A)
(7)
Relative entropy is a non-negative and convex function in A. The overall optimisation problem is given by (8).
min A
KL(p(w;A0)||p(w;A))
subject to h(w1,w2) ≤ u y = 1, (8) h(w1,w2) ≥ ℓ y = 0.
The optimisation problem given in (8) can be expressed as a particular type of Bregman divergence, which can be efficiently solved using the Bregman’s method (Censor and Zenios, 1997). We use the ITML algorithm proposed by Davis et al. (2007) to learn A, which is described in Algorithm 1. We arrange the sense-aware representations f(w, s1) and f(w, s2) for each instance (w, s1, s2, y) as columns to create the input matrix X ∈ Rd×2n, where d is the dimensionality of the sense-aware embeddings produced by the encoder trained in § 3.1, and n is the total number of training instances in the WiC dataset. The initial value of the distance matrix, A0 is set to the identity matrix, which corresponds to computing the Euclidean distance. The upper bound, u, is set to the distance that covers the top 5% of the distances between positive instances (i.e. y = 1), while the lower bound, l, is set to the distance that covers the bottom 5% of the distances between the negative instances (y = 0).

heading: B Data Statistics
text: We provide the full data statistics for the SCD benchmarks in Table 4.
Algorithm 1 Information Theoretic Metric Learning (ITML) Input: input matrix X, labels y, distance thresholds [l, u],
input Mahalanobis matrix A0, slack parameter γ Output: Mahalanobis matrix A 1: # Initialise A, λ, and ξ 2: A← A0 3: for i = 1 to n do 4: λi ← 0 5: ξi ← u if yi = 1 else l 6: end for 7: # Optimise A 8: repeat 9: for i = 1 to n do 10: obtain i-th instance (w1,w2, yi) from X and y 11: d← h(w1,w2;A) in (6) 12: δ ← 1 if yi = 1 else −1 13: α← min(λi, δ(1/d− γ/ξi)/2) 14: ξi ← γξi/(γ + δαξi) 15: λi ← λi − α 16: β ← δα/(1− δαd) 17: A← A + βα(w1 −w2)(w1 −w2)⊤A 18: end for 19: until convergence 20: return A

heading: Future Work
text: In the future, as the WiC datasets are expanded and include more languages, it is expected that the performance of models for a wider range of languages will improve