Definition generation for lexical semantic change detection (2024)

Abstract: We use contextualized word definitions generated by large language models as semantic representations in the task of diachronic lexical semantic change detection (LSCD). In short, generated definitions are used as ‘senses’, and the change score of a target word is retrieved by comparing their distributions in two time periods under comparison. On the material of five datasets and three languages, we show that generated definitions are indeed specific and general enough to convey a signal sufficient to rank sets of words by the degree of their semantic change over time. Our approach is on par with or outperforms prior non-supervised sensebased LSCD methods. At the same time, it preserves interpretability and allows to inspect the reasons behind a specific shift in terms of discrete definitions-as-senses. This is another step in the direction of explainable semantic change modeling.

heading: 1 Introduction and related work
text: Lexical semantic change detection (LSCD) methods up to now have mostly been based on token embeddings produced by large language models. While efficient, when measured on the existing benchmarks like diachronic word usage graphs (Schlechtweg et al., 2021), these methods are largely non-interpretable and produce rather abstract ‘change scores’. On the other hand, historical linguistics usually deals with semantic change in terms of discrete and interpretable senses being lost or gained (or changing their frequency).
Recently, a number of works were published which made an attempt to bridge this gap. In particular, Tang et al. (2023) proposed a sense distribution based LSCD method. Basically, they perform word sense disambiguation (WSD) on every occurrence of a target word in two diachronic corpora, using pre-trained sense embeddings (based on WordNet and BabelNet). Once all the occurrences
are assigned a sense, the sense frequency distributions are compared between two time periods to quantify the semantic change. This approach preserves the possibility to interpret these shifts, e.g., by analyzing which sense is ‘responsible’ for the shift.
We argue that while such methods constitute a significant advance for LSCD, they are inherently limited by their reliance on a pre-defined sense inventory. Even the best ontologies like BabelNet can miss important senses, especially when dealing with chronologically recent text data. For many languages, good ontologies simply do not exist.
Thus, we propose to replace retrieving a fitting sense for a given target word usage from an external ontology by generating a dictionary-like contextualized definition for this specific occurrence, using a large language model (LLM). These definitions serve as semantic representations of target word usages in the LSCD pipeline. The usage of generated definitions as semantic representations in LSCD was first proposed by Giulianelli et al. (2023) and further developed by Kutuzov et al. (2024), but they did not conduct comprehensive empirical evaluations for semantic change detection per se. In this paper, we fill in this gap and actually test definitions as representations on the existing diachronic semantic change benchmarks. We show that our method yields competitive results, often outperforming Tang et al. (2023), without relying on any manually created lexical database, but at the same time preserves interpretability via human-readable definitions of senses.
The contributions of this paper are as follows:
1. Contextualized definitions generated by LLMs can be used to rank words by the degree of their diachronic semantic change, with competitive performance.
2. Using definition embeddings with classical LSCD methods gives better results than using
5712
contextualized token embeddings as in prior work. However, this approach makes it less convenient to interpret and analyze semantic shifts.
3. Using generated definitions as text strings (with some merging based on their form) yields slightly lower results in comparison, but allows inspecting the nature of a semantic shift: e.g., what senses appeared, disappeared, or changed their frequency significantly.
All our code is available at https://github.com/ltgoslo/ Definition-generation-for-LSCD.

heading: 2 Data
text: We experiment on English, Norwegian and Russian benchmarks, since for these languages, definition generators were already created by Kutuzov et al. (2024). However, scaling to other languages is comparatively easy and requires only a small dataset of contextualized definitions (see §3).
To evaluate the performance of a semantic change detection system, we used existing LSCD datasets (diachronic corpora and gold scores for the target words): the English part of the SemEval’20 Task 11 (Schlechtweg et al., 2020), NorDiaChange2 (Kutuzov et al., 2022a) for Norwegian, and RuShiftEval3 (Kutuzov and Pivovarova, 2021b) for Russian. NorDiaChange actually contains two datasets and RuShiftEval contains three datasets, with different time period pairs under comparison (for Norwegian, the sets of target words are also different). The Russian datasets feature the highest number of target words (99, as compared to 37 in English and Norwegian datasets).
Note that SemEval’20 Task 1 included two subtasks: binary classification of words (changed or not changed) and ranking the words by the degree of their change. In this work, we focus only on the ranking task: 1) because the Russian dataset does not include binary labels, and 2) because even in the English and Norwegian datasets the binary labels are in many ways derivatives of the change scores.4
1https://www.ims.uni-stuttgart.de/en/research/ resources/corpora/sem-eval-ulscd/
2https://github.com/ltgoslo/nor_dia_change 3https://github.com/akutuzov/rushifteval_
public/tree/main 4In contrast to the English and Norwegian datasets which contain change scores, the Russian datasets contain similarity
It is also important to note that the RuShiftEval dataset was used in a shared task of the same name (Kutuzov and Pivovarova, 2021a). However, the scores in its leaderboard or in Cassotti et al. (2023) are not directly comparable to the scores in this work, since in the shared task, the dataset was split into development and test parts, so that the participants were able to tune their systems on the development set. In this paper, we focus on unsupervised approaches, aiming to avoid the necessity of tuning hyperparameters and leaving this for future work.

heading: 2.1 Preprocessing
text: We use the lemmatized versions of the SemEval2020 English corpora when reproducing Tang et al. (2023)’s Lesk baseline. No preprocessing of the Norwegian and Russian corpora has been done, except for lower-casing when running the Lesk baselines (see the details in the section 4) and taking lemmas of the target words into account when sampling usage examples for both Lesk and definition generation methods. Since frequent words may have more than 100 000 occurrences in the Norwegian and Russian corpora, we sampled randomly no more than 1000 usages (sentences) for each target word from every diachronic corpus.
This resulted in sub-corpora of total 58 000 word usages for English, 47 000 for Norwegian-1, 51 000 for Norwegian-2, and 164 000, 183 000 and 168 000 for Russian-1, Russian-2 and Russian-3 correspondingly.5

heading: 3 Definition generation methods
text: Our general pipeline of generating definitions from an LLM (‘DefGen’) is similar to Giulianelli et al. (2023). We use the definition generation models presented by Kutuzov et al. (2024). They are based on mT0-xl (Muennighoff et al., 2023) and were fine-tuned on WordNet (Ishiwatari et al., 2019), Oxford (Gadetsky et al., 2018) and CoDWoE (Mickus et al., 2022) for English, CoDWoE for Russian and Bokmålsordboka6 for Norwegian. The models are extensively described and evaluated in Kutuzov et al. (2024), but we provide the most important details in Appendix A.
scores. The obtained correlations are thus negative. We flip the sign when reporting these numbers to improve readability.
5We use only examples no longer than 350 subword tokens in all our experiments.
6https://ordbokene.no/
All CoDWoE datasets originally come from Wiktionary, so it is straightforward to extend this method to any major language. As a prompt for the LLM, we used the original example usage followed by the question ‘What is the definition of TARGETWORD?’ (in English, Norwegian or Russian).
Additionally, we conducted a series of experiments with different generation strategies. Prior work used only basic greedy decoding, while we experimented with alternative strategies such as multinomial sampling, beam search, and diverse beam search (Vijayakumar et al., 2018).
Note that these experiments do not deal with LSCD – they only evaluate the capability of the models to generate definitions similar to the gold ones. The results of our experiments for English, Russian and Norwegian, evaluated with BLEU, RougeL and BertScore, are shown in Table 1. We used the default implementations of these metrics from the Evaluate library7 with the only change of using whitespace tokenizer in RougeL for all languages (instead of the default one aimed at English). For English and Russian, we evaluated on the CoDWoE trial sets (about 200 instances each); for Norwegian, we used our own test set of about 7000 instances.8
The performance scores are consistent across all three languages: the default mode of greedy decoding turned to be a hard-to-beat baseline. However, using beam search with 5 beams (or its diverse version with diversity penalty of 0.5) does outperform greedy decoding according to all three metrics.
In the experiments below, we use definitions generated with all three approaches: greedy decoding, beam search and diverse beam search, to explore to what extent the definition generation performance translates to LSCD performance.
7https://huggingface.co/docs/evaluate/ 8Generating definitions with our models for 1000 example
usages takes about 2 minutes on an NVIDIA A100 GPU.

heading: 4 Semantic change detection with generated definitions
text: In this section, we describe the methods we used in our experiments, starting with the XLM-R and sense embeddings baselines from prior work. Next, we present our own approach based on definition embeddings and the standard LSCD aggregation algorithms: PRT (prototype embeddings), APD (average pairwise distance), and their arithmetic mean (AM) (Kutuzov et al., 2022b). Finally, we move on to our interpretable ‘definitions-as-senses‘ method which employs definitions directly (as text strings) and merges similar definitions together using Levenshtein edit distance.

heading: 4.1 Baselines
text: Table 2 shows the results from previous studies that we use as our baselines, as well as the best results of our definition-based systems.
The first baseline is a state-of-the-art unsupervised LSCD method based on contextualized XLMR token embeddings. Its results are taken from Giulianelli et al. (2022) (best scores). This approach is not interpretable. Note that we do not report any other methods also using contextualized word embeddings (e.g. those surveyed in Periti and Tahmasebi (2024)) either because they are supervised (somehow tuned on development sets or parts of test sets) and thus their results are not directly comparable to ours, or because they do not constitute state-of-the-art.
As a second, interpretable baseline, we follow Tang et al. (2023) and use the Lesk WSD algorithm (Lesk, 1986) with WordNet definitions (this method is called ‘NLTK’ in the Table 1 of their paper). Their result for English, as well as our extensions to Norwegian and Russian, are shown in the lower part of Table 2. We were able to reproduce their Lesk results with only small fluctuations9. Since no
9Probably due to the fact that we used top 1 sense in all our experiments, while Tang et al. (2023) experimented with top
open WordNet-like databases exist for Norwegian or Russian10, we used the aforementioned Bokmålsordboka and CoDWoE/Wiktionary as sources of Norwegian and Russian sense definitions.
We also experimented with adding part-ofspeech information to the Lesk algorithm (that is, restricting Lesk WSD search to only the synsets corresponding to the desired part-of-speech of the target word). The English SemEval’20 dataset explicitly specifies parts-of-speech for the target words, while the Norwegian and Russian datasets contain nouns only. The two variants of Lesk yield identical results on the Russian datasets since the target words are not PoS-ambiguous. For English and Norwegian-2, Lesk outperforms the XLM-R token embeddings and comes close to our approach based on definition embeddings.
However, Tang et al. (2023)’s main results are based on ARES and LMMS sense embeddings. Unfortunately, these embeddings are not publicly available anymore due to link rot. The LMMS download link11 leads to a private file storage, and the ARES embeddings also cannot be found at the provided URL. We contacted the authors but got no answer by the time of writing. Thus, we can
k highest ranked senses on a held-out set and found k = 2 to perform best. However, we focus on unsupervised approaches to the task and leave hyperparameter tuning on development sets for future work.
10The Open Multilingual WordNet allows searching for words in other languages than English, but the synset definitions remain in English.
11https://github.com/danlou/LMMS
only quote the performance scores from Tang et al. (2023).

heading: 4.2 Using definition embeddings
text: Generated definitions can be easily vectorized by using any sentence embedding model. We embedded the generated definitions for every target word usage with DistilRoBERTa12. After that, it becomes possible to use the standard LSCD methods like PRT, APD, and their arithmetic mean (AM) (Kutuzov et al., 2022b). The only difference to the standard setup is that instead of token embeddings, we feed contextualized definition embeddings into the algorithm.
The intuition here is that by measuring the average or pairwise distances between definitions of one and the same target word in two historical corpora, one can quantify the degree of semantic change for this word between two time periods. As can be seen in the Table 3, this is indeed the case. Our definition embeddings outperform the contextualized XLM-R token embeddings from Giulianelli et al. (2022) on five of the six evaluated datasets.
Note in this context that using token embeddings from a masked LM requires the knowledge of the exact position of the target token in the input sentence (with additional issues in case of the target word being split into multiple sub-words). In our approach, adding the ‘What is the definition of X?’
12https://huggingface.co/sentence-transformers/ all-distilroberta-v1
prompt to the input sentence is completely decoupled from the location of X within the sentence.
The decoding strategy does not seem to make a significant difference in terms of LSCD performance. Greedy decoding is a reasonable default choice despite its slightly lower scores in Table 1.
On English, the APD method on definition embeddings also outperforms the best senseembedding-based approaches from Tang et al. (2023) by a large margin (see Table 2). Note, however, that using definition embeddings in this case still yields a non-interpretable result: we do not know what exact senses are responsible for a high degree of semantic change. For this reason, we propose to use the generated definitions directly in the next section.

heading: 4.3 Merging definitions together
text: The definitions generated by a DefGen system can be used directly for LSCD. In this case, each unique definition is considered a separate word sense, and the sense distributions of the two time periods can be compared in the same way as in Tang et al. (2023). This approach is straightforward and already results in competitive performance (see the “No merging” section in Table 413).
However, it obviously suffers from too granular senses. As an example, for almost 1000 oc-
13Table 4 reports results after using two different distance metrics: cosine and Jensen-Shannon divergence (JSD). JSD is superior in most cases, but not always.
currences of the word ‘plane’ in the SemEval’20 English dataset, more than 200 unique definitions were generated, most with only one occurrence. This list includes definitions obviously belonging to one and the same sense: for example, ‘An aircraft, especially one designed for military use’ and ‘An aircraft, especially a military aircraft’. This leads to noise and – even worse – to reduced interpretability. It is easy to observe that definitions belonging to the same sense are often similar in their surface form. Thus, in this subsection, we describe our experiments with merging similar definitions together.
Any decision about what word usages belong to one sense is inherently arbitrary (Kilgarriff, 1997). The same applies to definitions: in order to decide whether two definitions represent one and the same sense, one has to find a way to quantify their similarity. In order to preserve interpretability, we decided to use surface string similarity metrics (as opposed to, e.g., cosine similarity between definition embeddings).
We remind again that the top part of Table 4 shows the performance scores on our datasets with no merging involved: every unique definition is considered to be a separate sense on its own and we simply compare the distribution of these ‘senses’ across two time periods. In addition to that, we introduce two merging strategies which we dub ‘minimalist’ and ‘full-fledged’ merging. The intuition behind them is that one replaces some of the
generated definitions for a target word with another similar definition generated for the same target word, thus reducing the total number of unique definitions per word and making it closer to a realistic number of senses.
First, we filter out punctuation marks from all definitions. Second, every time period (out of two) is processed separately. We also tried joint processing of both time periods to make the resulting definitions-as-senses more comparable. However, it consistently resulted in worse performance: the most probable reason being that it makes the sense distributions too close to each other, eliminating meaningful differences. It also can bias the predictions if one time period is represented by a larger corpus than another.
The processing is done as follows. For every target word, we sort the generated definitions by their frequency and loop over them, starting from the top (most frequent) ones, representing the dominant senses of the word. For every step in this loop (let’s designate it as ‘hub definition’), we loop again over the remaining definitions, calculating the edit distance14 between them and the hub definition. If the edit distance is lower than the pre-defined threshold, the current definition is replaced with the hub definition (we assume they belong to one
14We use Levenshtein distance (Levenshtein, 1966); other edit distance formulations are possible. In addition, only definitions containing 4 words and more could serve as hubs, to prevent merging together very short definitions with low edit distances.
sense). With the ‘minimalist strategy’, only the first (most frequent) definition can be the hub (and have other definitions replaced by it), the loop is stopped after it is compared to all other definitions. With the ‘full-fledged’ strategy, the loop continues, and other (less frequent) definitions also get a chance to become hubs, if they were not subsumed by another definition before. The ‘full-fledged’ strategy naturally results in even stronger reduction on the number of unique senses (see Figure 1). The datasets with replaced definitions are used for LSCD in the usual way.
The value of the edit distance threshold is a hyperparameter. One can tune it on a designated
development set, but in this study, we tried to avoid the supervised setup. Thus, after studying the data, we only tested two intuitively sensible threshold values of 10 and 50. It turned out that the value of 10 is optimal for Norwegian and Russian, while the value of 50 (more merging) is optimal for English (changes in the number of senses after merging are more obvious for English on Figure 1 because of the more aggressive Levenshtein threshold).
As shown in Table 4, merging definitions indeed improves the performance in the LSCD ranking task for all the languages and benchmarking datasets, in comparison to no merging at all. It still does not reach the level of APD on definition embeddings (Table 3), but outperforms one of two sense embedding approaches from Tang et al. (2023). For two Russian datasets out of three, our merged definitions-as-senses outperform previous best unsupervised results (Giulianelli et al., 2022).
Curiously, only for Norwegian the best performance is achieved using the full-fledged merging strategy (and is still comparatively low); for English and Russian, the minimalist strategy (only merging with the dominant sense) gives the best scores. Thus, merging should be done cautiously: merging too much can degrade the performance. Another interesting finding is that greedy decoding again turned out to be the best generation strategy for LSCD. We hypothesize that using beam search often results in too diverse a set of definitions, which prevents efficient comparisons between their diachronic distributions.
Overall, using surface forms of generated definitions directly is outperformed by using vectorized definitions and APD or PRT methods for semantic change detection (see 4.2), which is consistent with findings in Giulianelli et al. (2023). They found that cosine similarity between vectorized definitions better approximates human similarity judgments than surface form similarity metrics like edit distance or BLEU. However, using definitions in their textual form has a clear advantage of being interpretable. In the next section, we illustrate how semantic change can be explained and analyzed by comparing the generated (and merged) definitions.

heading: 5 Qualitative analysis
text: When predicting semantic change on the basis of the distribution of senses (or definitions-as-senses), it becomes possible to analyze and interpret this change, by simply looking at the distribution of
entries (senses) which contribute most to the difference.
Let’s consider the top performing set of English definitions (generated with greedy decoding and merged in the minimalist approach with the edit distance threshold of 50). For the word ‘ball’ in the SemEval’20 time periods, the JSD metric yields a high change score of 0.83. After looking at the list of top frequent definitions-as-senses for this word, it becomes clear that its dominant sense has changed: while in time period 1 (19th century), more than 82% of all usage examples were given the definition ‘A spherical object especially one that is round in shape’ with ‘A party’ being the next most frequent sense, in time period 2 (20th century), 80% of ‘ball’ usages were defined as
‘The object hit in a game’, with similar definitions following this one in the top-frequent list. This is a clear evidence of the ‘dancing party’ sense for the word ‘ball’ becoming obsolete in the 20th century, with the sports-related sense taking the dominant position.
For the noun ‘attack’, the system predicts a medium change degree of 0.34. Again, it is straightforward to find the reasons. While in both periods the dominant sense is the same (‘An instance of military action against an enemy’), in the time period 2 its ratio drops down from 87% to 80%, and we observe the appearance of a new rare but not unique sense of ‘An instance of sudden violent activity of a bodily organ or system especially the heart’. This is a linguistically plausible explanation of a semantic shift, much more useful to a lexicographer than a raw change score. See Appendix B for more examples in English, Norwegian and Russian.

heading: 6 Conclusion
text: In this paper, we showed how contextualized dictionary-like definitions generated by a finetuned large language model can be used for the practical downstream task of semantic change detection (in particular, ranking words by the degree of their diachronic semantic change).
Following Giulianelli et al. (2023), we treat generated definitions as semantic representations of the target words. These definitions (and their frequency distributions) can be used ‘as is’, following Tang et al. (2023)’s method, or after embedding them in a dense vector space using any available sentence embedding model. The second method yields results which are empirically better (con-
sidering existing benchmarks for three different languages), but the first method makes it much easier to interpret and explain semantic change, since it operates directly on generated definitions in their textual forms.
We consider this study a small step towards more explainable semantic change modeling, which can be closer to linguistically plausible discrete ‘senses’, while still retaining empirical performance. In the future, we plan to explore to what extent it is possible to improve our results by tuning hyperparameters on development sets (where available). Another direction for future research is using more advanced string distance metrics like weighted Levenshtein distance, Longest Common Subsequence Ratio, or Word Mover’s Distance (Kusner et al., 2015), in the hope that it will allow to handle more nuanced similarities and dissimilarities between generated definitions.
Ethical impact
For fine-tuning our definition generators, only open and publicly available datasets, mostly dictionaries were used. However, some of them (especially Wiktionary) are crowd-sourced, and thus can (and do) contain inappropriate phrases. In addition, the foundational mT0 language model on which we base our pipeline, was trained among other data on web-crawled texts, also far from being clean. Thus, generated definitions are not guaranteed to be free from swearing, discriminative passages and other inappropriate content.
Limitations
This work is limited to only three languages (English, Norwegian and Russian), while the standard SemEval’20 ‘LSCD suite’ contains four languages (English, German, Latin, Swedish). Also, we did not experiment with hyperparameter tuning or different ways of training definition generators. It should also be noted that Spearman rank correlation may not be accurate for samples the size of LSCD benchmarks: we use it to preserve compatibility and comparability with prior work.
Finally, we have not yet empirically evaluated how useful in practice the definition-based explanations of semantic change will be for historical linguists and lexicographers (although what we see after manual inspection of the system predictions is promising).

heading: A Definition generator models
text: Contextual definitions in this work are created with fine-tuned large language models, using the method proposed by Giulianelli et al. (2023) and further developed in Kutuzov et al. (2024): an encoder-decoder language model is fine-tuned on a dataset of target word usages and the corresponding definitions. Then, definitions are conditionally generated for every example in the test set. Giulianelli et al. (2023) used Flan-T5 (Chung et al., 2022) as the underlying language model. However, it was trained predominantly on English and lacks the capability to properly encode or generate texts in languages with significantly different writing systems (especially true for Russian, but Norwegian characters ‘å’, ‘ø’ and ‘æ’ are also not processed by the Flan-T5 tokenizer). Because of that, Kutuzov et al. (2024) and this work are using mT0-xl (Muennighoff et al., 2023) which is essentially a multilingual version of Flan-T5. Fine-tuning was done in a standard text-to-text setup, for every language (English, Norwegian, Russian) separately, so that in the end we had three languagespecific models: https://huggingface. co/ltg/mt0-definition-en-xl, https:// huggingface.co/ltg/mt0-definition-no-xl and https://huggingface.co/ltg/ mt0-definition-ru-xl.

heading: B Examples of merged definitions
text: Tables 5, 6 and 7 show examples of most frequent definitions-as-senses for some of the target words in our English, Norwegian and Russian benchmarks. All the definitions are generated with the best strategy for the specific dataset (see Table 4).

heading: Future Work
text: In the future, we plan to explore to what extent it is possible to improve our results by tuning hyperparameters on development sets (where available). Another direction for future research is using more advanced string distance metrics like weighted Levenshtein distance, Longest Common Subsequence Ratio, or Word Mover’s Distance (Kusner et al. , 2015), in the hope that it will allow to handle more nuanced similarities and dissimilarities between generated definitions. Ethical impact
For fine-tuning our definition generators, only open and publicly available datasets, mostly dictionaries were used. However, some of them (especially Wiktionary) are crowd-sourced, and thus can (and do) contain inappropriate phrases. In addition, the foundational mT0 language model on which we base our pipeline, was trained among other data on web-crawled texts, also far from being clean. Thus, generated definitions are not guaranteed to be free from swearing, discriminative passages and other inappropriate content. Limitations
This work is limited to only three languages (English, Norwegian and Russian), while the standard SemEval’20 ‘LSCD suite’ contains four languages (English, German, Latin, Swedish). Also, we did not experiment with hyperparameter tuning or different ways of training definition generators. It should also be noted that Spearman rank correlation may not be accurate for samples the size of LSCD benchmarks: we use it to preserve compatibility and comparability with prior work. Finally, we have not yet empirically evaluated how useful in practice the definition-based explanations of semantic change will be for historical linguists and lexicographers (although what we see after manual inspection of the system predictions is promising)