Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis
Year: 2023

Abstract:
We propose using automatically generated natural language definitions of contextualised word usages as interpretable word and word sense representations. Given a collection of usage examples for a target word, and the corresponding data-driven usage clusters (i.e., word senses), a definition is generated for each usage with a specialised Flan-T5 language model, and the most prototypical definition in a usage cluster is chosen as the sense label. We demonstrate how the resulting sense labels can make existing approaches to semantic change analysis more interpretable, and how they can allow users—historical linguists, lexicographers, or social scientists—to explore and intuitively explain diachronic trajectories of word meaning. Semantic change analysis is only one of many possible applications of the ‘definitions as representations’ paradigm. Beyond being human-readable, contextualised definitions also outperform token or usage sentence embeddings in word-in-context semantic similarity judgements, making them a new promising type of lexical representation for NLP.

heading: 1 Introduction
text: Accurate semantic understanding in language technologies is typically powered by distributional word representations and pre-trained language models (LMs). Due to their subsymbolic nature, however, such methods lack in explainability and interpretability, leading to insufficient trust in end users. An example application which requires capturing word meaning with its nuanced contextdetermined modulations is lexical semantic change analysis, a task which consists in detecting whether a word’s meaning has changed over time, for example by acquiring or losing a sense. Modern semantic change detection systems rely on static and contextualised word representations, LMbased lexical replacement, grammatical profiles, supervised word sense and word-in-context disambiguation (Kutuzov et al., 2018; Tahmasebi et al.,
2021). But the main potential end users of these technologies—historical linguists, lexicographers, and social scientists—are still somewhat reluctant to adopt them precisely because of their lack of explanatory power. Lexicographers, for instance, are not satisfied with detecting that a word has or hasn’t changed its meaning over the last ten years; they want descriptions of old and new senses in humanreadable form, possibly accompanied by additional layers of explanation, e.g., specifying the type of semantic change (such as broadening, narrowing, and metaphorisation) the word has undergone.
Our work is an attempt to bridge the gap between computational tools for semantic understanding and their users. We propose to replace blackbox contextualised token embeddings produced by large LMs with a new type of interpretable lexical semantic representation: automatically generated contextualised word definitions (Gardner et al., 2022). In this paradigm, the usage of the word ‘apple’ in the sentence ‘She tasted a fresh green apple’ is represented not with a dense highdimensional vector but with the context-dependent natural language definition ‘EDIBLE FRUIT’. With an extended case study on lexical semantic change analysis, we show that moving to the more abstract meaning space of definitions allows practitioners to obtain explainable predictions from computational systems, while leading to superior performance on semantic change benchmarks compared to state-ofthe-art token-based approaches.
This paper makes the following contributions.1
1. We show that word definitions automatically generated with a specialised language model, fine-tuned for this purpose, can serve as interpretable representations for polysemous words (§5). Pairwise usage similarities between contextualised definitions approximate human semantic similarity judgements better
1All the code we used can be found at https:// github.com/ltgoslo/definition_modeling.
3130
than similarities between usage-based word and sentence embeddings.
2. We present a method to obtain word sense representations by labelling data-driven clusters of word usages with sense definitions, and collect human judgements of definition quality to evaluate these representations (§6). We find that sense labels produced by retrieving the most prototypical contextualised word definition within a group of usages consistently outperform labels produced by selecting the most prototypical token embedding.
3. Using sense labels obtained via definition generation, we create maps that describe diachronic relations between the senses of a target word. We then demonstrate how these diachronic maps can be used to explain meaning changes observed in text corpora and to find inconsistencies in data-driven groupings of word usages within existing lexical semantic resources (§7).

heading: 2 Related Work

heading: 2.1 Definition Modelling
text: The task of generating human-readable word definitions, as found in dictionaries, is commonly referred to as definition modelling or definition generation (for a review, see Gardner et al., 2022). The original motivation for this task has been the interpretation, analysis, and evaluation of word embedding spaces. Definition generation systems, however, also have practical applications in lexicography, language acquisition, sociolinguistics, and within NLP (Bevilacqua et al., 2020). The task was initially formulated as the generation of a natural language definition given an embedding—a single distributional representation—of the target word, or definiendum (Noraset et al., 2017). Word meaning, however, varies according to the context in which a word is used. This is particularly true for polysemous words, which can be defined in multiple, potentially very different ways depending on their context. The first formulation of definition
modelling was therefore soon replaced by by the task of generating a contextually appropriate word definition given a target word embedding and an example usage (Gadetsky et al., 2018; Mickus et al., 2022). When the end goal is not the evaluation of embedding spaces, generating definitions from vector representations is still not the most natural formulation of definition modelling. Ni and Wang (2017) and Mickus et al. (2019) treat the task as a sequence-to-sequence problem: given an input sequence with a highlighted word, generate a contextually appropriate definition. In the current work, we follow this approach. Table 1 shows an example of a contextualised definition generated by our model (see §4) for the English word ‘draftee’.
Methods Methods that address this last formulation of the task are typically based on a pre-trained language model deployed on the definienda of interest in a natural language generation (NLG) setup (Bevilacqua et al., 2020). Generated definitions can be further improved by regulating their degree of specificity via specialised LM modules (Huang et al., 2021), by adjusting their level of complexity using contrastive learning training objectives (August et al., 2022), or by supplementing them with definitional sentences extracted directly from a domain-specific corpus (Huang et al., 2022). We will compare our results to the specificity-tuned T5-based text generator proposed by Huang et al. (2021).
Evaluation Generated definitions are typically evaluated with standard NLG metrics such as BLEU, NIST, ROUGE-L, METEOR or MoverScore (e.g., Huang et al., 2021; Mickus et al., 2022), using precision@k on a definition retrieval task (Bevilacqua et al., 2020), or measuring semantic similarity between sentence embeddings obtained for the reference and the generated definition (Kong et al., 2022). Because reference-based methods are inherently flawed (for a discussion, see Mickus et al., 2022), qualitative evaluation is almost always presented in combination with these quantitative metrics. In this paper, we evaluate
generated definitions with automatic metrics and by collecting human judgements.

heading: 2.2 Semantic Change Detection
text: Words in natural language change their meaning over time; these diachronic processes are of interest both for linguists and NLP practitioners. Lexical semantic change detection (LSCD) is nowadays a well represented NLP task, with workshops (Tahmasebi et al., 2022) and several shared tasks (e.g., Schlechtweg et al., 2020; Kurtyigit et al., 2021). LSCD is usually cast either as binary classification (whether the target word changed its meaning or not) or as a ranking task (ordering target words according to the degree of their change). To evaluate existing approaches, manually annotated datasets are used: so-called DWUGs are described below in §3.
An important issue with current LSCD methods is that they rarely describe change in terms of word senses, which are extremely important for linguists to understand diachronic meaning trajectories. Instead, systems provide (and are evaluated by) perword numerical ‘change scores’ which are hardly interpretable; at best, a binary ‘sense gain’ or ‘sense loss’ classification is used. Even approaches that do operate on the level of senses (e.g., Mitra et al., 2015; Homskiy and Arefyev, 2022) do not label them in a linguistically meaningful way, making it difficult to understand the relations between the resulting ‘anonymous’ types of word usage.

heading: 3 Data

heading: 3.1 Definitions Datasets
text: To train an NLG system that produces definitions (§4), we use three datasets containing a humanwritten definition for each lexicographic sense of a target word, paired with a usage example. The WordNet dataset is a collection of word definitions and word usages extracted by Ishiwatari et al. (2019) from the WordNet lexical database (Miller, 1995). The Oxford dataset (also known as CHA in prior work) consists of definitions and usage ex-
Dataset Entries Lemmas Ratio Usage length Definition length WordNet 15,657 8,938 1.75 4.80 ± 3.43 6.64 ± 3.77 Oxford 122,318 36,767 3.33 16.73 ± 9.53 11.01 ± 6.96 CoDWoE 63,596 36,068 2.44 24.04 ± 21.05 11.78 ± 8.03
Table 2: Main statistics of the datasets of definitions. Ratio is the sense-lemma ratio: the number of entries over the number of lemmas.
amples collected by Gadetsky et al. (2018) from the Oxford Dictionary. Definitions are written by experts and usage examples are in British English. The CoDWoE dataset (Mickus et al., 2022) is based on definitions and examples extracted from Wiktionary.2 It is a multilingual corpus, of which we use the English portion. Table 2 reports the main statistics of these datasets. Further statistics, e.g. on the size of the different splits, are provided by Huang et al. (2021) as well as in Appendix A.3

heading: 3.2 Diachronic Word Usage Graphs
text: We showcase interpretable word usage (§5) and sense representations (§6 and 7) using a dataset where target lemmas are represented with diachronic word usage graphs (DWUGs, Schlechtweg et al., 2021). A DWUG is a weighted, undirected graph, where nodes represent target usages (word occurrences within a sentence or discourse context) and edge weights represent the semantic proximity of a pair of usages. DWUGs are the result of a multi-round incremental human annotation process, with annotators asked to judge the semantic relatedness of pairs of word usages on a 4-point scale. Based on these pairwise relatedness judgements, word usages are then grouped into usage clusters (a data-driven approximation of word senses) using a variation of correlation clustering (Bansal et al., 2004; Schlechtweg et al., 2020).
DWUGs are currently available in seven
2https://www.wiktionary.org 3Note that in theory, a definition dataset could be also be extracted from the SemCor corpus (Miller et al., 1993). However, we do not anticipate it will contribute much to training or evaluation since SemCor does not contain any new definitions with respect to WordNet: only more examples for the same word-definition pairs. This can be investigated in future work.
languages.4 In this paper, we use the English graphs, which consist of usage sentences sampled from the Clean Corpus of Historical American English (Davies, 2012; Alatrash et al., 2020) and belonging to two time periods: 1810-1860 and 1960-2010. There are 46 usage graphs for English, corresponding to 40 nouns and 6 verbs annotated by a total of 9 annotators. Each target lemma has received on average 189 judgements, 2 for each usage pair. Figure 1 shows an example of a DWUG, with colours denoting usage clusters (i.e., data-driven senses): the ‘blue’ and ‘orange’ clusters belong almost entirely to different time periods: a new sense of the word has emerged. We show how our approach helps explain such cases of semantic change in §7.

heading: 4 Definition Generation
text: Our formulation of the definition generation task is as follows: given a target word w and an example usage s (i.e., a sentence containing an occurrence of w), generate a natural language definition d that is grammatical, fluent, and faithful to the meaning of the target word w as used in the example usage s. A definition generator is a language process that maps words and example usages to natural language definitions. As a generator, we use Flan-T5 (Chung et al., 2022), a version of the T5 encoder-decoder Transformer (Raffel et al., 2020) fine-tuned on 1.8K tasks phrased as instructions and collected from almost 500 NLP datasets. FlanT5 is not trained specifically on definition generation but thanks to its massive multi-task instruction fine-tuning, the model exhibits strong generalisation to unseen tasks. Therefore, we expect it to produce high-quality definitions. We extensively test three variants of Flan-T5 of different size and compare them to vanilla T5 models (Table 4 and Table 12, Appendix C.2); based on our results, we recommend using the largest fine-tuned Flan-T5 model whenever possible.
To obtain definitions from Flan-T5, we use natural language prompts consisting of an example usage preceded or followed by a question or instruction. For example: ‘s What is the definition of w?’ The concatenated usage example and prompt are provided as input to FlanT5, which conditionally generates definitions (Ta-
4https://www.ims.uni-stuttgart.de/en/ research/resources/experiment-data/wugs/
ble 1 shows an example instance).5 We choose greedy search with target word filtering as a simple, parameter-free decoding strategy. Stochastic decoding algorithms can be investigated in future work.
Prompt selection In preliminary experiments, we used the pre-trained Flan-T5 Base model (250M parameters) to select a definition generation prompt among 8 alternative verbalisations. Appending the question ‘What is the definition of w?’ to the usage example consistently yielded the best scores.6 We use this prompt for all further experiments.

heading: 4.1 Evaluating Generated Definitions
text: Before using its definitions to construct an interpretable semantic space—the main goal of this paper—we perform a series of experiments to validate Flan-T5 as a definition generator. We use the target lemmas and usage examples from the corpora of definitions presented in §3, conditionally generate definitions with Flan-T5, and then compare them to the gold definitions in the corpora using reference-based NLG evaluation metrics. We report SacreBLEU and ROUGE-L, which measure surface form overlap, as well as BERT-F1, which is sensitive to the reference and candidate’s semantics. As mentioned in §2.1, reference-based metrics are not flawless, yet designing and validating a reference-free metric for the definition generation task is beyond the scope of this paper. We will later resort to correlations with human judgements and expert human evaluation to assess the quality of generated definitions.
We evaluate the Flan-T5 XL (3B parameters) in three generalisation tests: 1) in distribution, 2) hard domain shift, and 3) soft domain shift.7 We use these tests to choose a model to be deployed in further experiments. For reference, we report the BLEU score of the definition generator by Huang et al. (2021); ROUGE-L and BERT-F1 are not reported in their paper.
In distribution We fine-tune Flan-T5 XL on one corpus of definitions at a time, and test it on a held-out set from that same corpus (except
5This is a simpler workflow in comparison to prior work (Bevilacqua et al., 2020; Almeman and Espinosa Anke, 2022) where inputs are encoded as ‘target word - context’ pairs.
6Further details in Appendix B. 7Tests defined following the GenBench generalisation taxonomy (Hupkes et al., 2022). We also include a fourth setup, zero shot (task shift), where we directly evaluate the pretrained Flan-T5 XL. Results (including other models) are presented in Appendix C.1-C.2, and an evaluation card in Appendix C.3.
CoDWoE which does not provide train-test split). The quality of the definitions increases substantially with fine-tuning, in terms of both their lexical and semantic overlap with gold definitions (Table 3). We find significantly higher scores on Oxford, which may be due to the larger size of its training split and to the quality of the WordNet examples, which sometimes are not sufficiently informative (Almeman and Espinosa Anke, 2022).
Hard domain shift We fine-tune Flan-T5 XL on WordNet and test it on Oxford, and vice versa. These tests allow us to assess the model’s sensitivity to the peculiarities of the training dataset. A model that has properly learned to generate definitions should be robust to this kind of domain shift. The quality of the definitions of Oxford lemmas generated with the model fine-tuned on WordNet (see the Oxford column in Table 3) is lower than for the model fine-tuned on Oxford itself (same column, see row ‘In-distribution’). Instead, for outof-domain WordNet definitions, all metrics surprisingly indicate higher quality than for in-distribution tests (WordNet column). Taken together, our results so far suggest that the quality of a fine-tuned model depends more on the amount (and perhaps quality) of the training data than on whether the test data is drawn from the same dataset.
Soft domain shift We finally fine-tune Flan-T5 XL on a collection of all three definition datasets: WordNet, Oxford, and CoDWoE. Our previous results hint towards the model’s preference for more training examples, so we expect this setup to achieve the highest scores regardless of the soft shift between training and test data. Indeed, on WordNet, our fine-tuned model marginally surpasses the state-of-the-art upper bound in terms of BLEU score (Table 3), and it achieves the highest scores on the other metrics. Oxford definitions generated with this model are instead still below Huang et al.’s upper bound; this may be due to Oxford being generally more difficult to model
than WordNet, perhaps because of longer definitions and usages (see Figures 4-5 in Appendix A). We consider the observed model performance sufficient for the purposes of our experiments, in particular in view of the higher efficiency of finetuned Flan-T5 with respect to the three-module system of Huang et al. (2021). We therefore use this model throughout the rest of our study.
The Flan-T5 models fine-tuned for definition generation are publicly available through the Hugging Face model hub.8

heading: 5 Definitions are Interpretable Word Representations
text: We propose considering the abstract meaning space of definitions as a representational space for lexical meaning. Definitions fulfil important general desiderata for word representations: they are human-interpretable and they can be used for quantitative comparisons between word usages (i.e., by judging the distance between pairs of definition strings). We put the definition space to test by applying it to the task of semantic change analysis, which requires capturing word meaning at a finegrained level, distinguishing word senses based on usage contexts. We use our fine-tuned Flan-T5 models (XL and other sizes) to generate definitions for all usages of the 46 target words annotated in the English DWUGs (ca. 200 usages per word; see §3.2).9 These definitions (an example is provided in Table 1) specify a diachronic semantic space.

heading: 5.1 Correlation with Human Judgements
text: We construct word usage graphs for each lemma in the English DWUGs: we take usages as nodes and assign weights to edges by measuring pairwise similarity between usage-dependent definitions. We
8Model names: ltg/flan-t5-definition-en-base, ltg/flan-t5definition-en-large, ltg/flan-t5-definition-en-xl.
9The training datasets used in §4 contain nouns, verbs, adjectives and adverbs. The English DWUGs contain only nouns and verbs.
compute the similarity between pairs of definitions using two overlap-based metrics, SacreBLEU and METEOR, as well as the cosine similarity between sentence-embedded definitions. We then compare our graphs against the gold DWUGs, where edges between usage pairs are weighted with human judgements of semantic similarity, by computing the Spearman’s correlation between human similarity judgements and similarity scores obtained for pairs of generated definitions. We compare our results to DWUGs constructed based on two additional types of usage-based representations: sentence embeddings obtained directly for usage examples, and contextualised token embeddings. Sentence embeddings (for both definitions and usage examples) are SBERT representations (Reimers and Gurevych, 2019) extracted with mean-pooling from the last layer of a DistilRoBERTa LM finetuned for semantic similarity comparisons.10 For tokens, we extract the last-layer representations of a RoBERTa-large model (Liu et al., 2019) which correspond to subtokens of the target word (following Giulianelli et al., 2020) and use mean-pooling to obtain a single vector. While we report string-overlap similarities for definitions, these are not defined for numerical vectors, and thus similarities for example sentences and tokens are obtained with cosine only.
Pairwise similarities between definitions approximate human similarity judgements far better than similarities between example sentence and word embeddings (Table 4). This indicates that definitions are a more accurate approximation of contextualised lexical meaning. The results also show that similarity between definitions is best captured by their embeddings, rather than by overlap-based
10DistilRoBERTa (sentence-transformers/alldistilRoBERTa-v1) is the second best model as reported in the official S-BERT documentation at the time of publication (https://www.sbert.net/docs/ pretrained_models.html). For a negligible accuracy reduction, it captures longer context sizes and is ca. 50% smaller and faster than the model that ranks first.
metrics like SacreBLEU and METEOR.

heading: 5.2 Definition Embedding Space
text: We now examine the definition embedding space (the high-dimensional semantic space defined by sentence-embedded definitions), to identify properties that make it more expressive than usage-based spaces. Figure 2 shows the T-SNE projections of the DistilRoBERTa embeddings of all lemmas in the English DWUGs, for the three types of representation presented earlier: generated definitions, tokens, and example sentences.11 The definition spaces exhibit characteristics that are more similar to a token embedding space than an example sentence embedding space, with definitions of the same lemma represented by relatively close-knit clusters of definition embeddings. This suggests that definition embeddings, as expected, represent the meaning of a word in context (similar to token embeddings), rather than the meaning of the whole usage example sentence in which the target word occurs.
For each target word, we also measure (i) the variability in each embedding space and (ii) the inter-cluster and intra-cluster dispersion (Caliński and Harabasz, 1974) obtained when clustering each space using k-means. This allows us to quantitatively appreciate properties exhibited by datadriven usage clusters that are obtained from different representation types. To cluster the embedding spaces, we experiment with values of k ∈ [2, 25], and select the k which maximises the Silhouette score. Our results are summarised in Table 5. We observe that the clusters in the definition spaces have on average the lowest intra-cluster dispersion, indicating that they are more cohesive than the clusters in the token and example sentence spaces. While, on average, token spaces exhibit higher inter-cluster dispersion (indicating better cluster separation), the ratio between average separation and cohesion is highest for the definition spaces. These findings persist for the gold clusters determined by the English DWUGs (Table 14).
In sum, this analysis shows that definition embedding spaces are generally suitable to distinguish different types of word usage. In the next section, we will show how they can indeed be used to characterise word senses.
11T-SNE projections for RoBERTa-large are in Appendix G.

heading: 6 Labelling Word Senses With Definitions
text: For generated definitions to be useful in practice, they need to be able to distinguish word senses. For example (ignoring diachronic differences and singleton clusters), there are three main senses of the word ‘word’ in its DWUG, which we manually label as: (1) ‘WORDS OF LANGUAGE’, (2) ‘A RUMOUR’, and (3) ‘AN OATH’. Manual inspection of the generated definitions indicates that they are indeed sense-aware:
1. ‘A communication, a message’, ‘The text of a book, play, movie’, etc.
2. ‘Information passed on, usually by one person to another’, ‘communication by spoken or written communication’, etc.
3. ‘An oath’, ‘a pronouncement’, etc.
But let’s again put ourselves in the shoes of a historical linguist. Sense clusters are now impractically represented with multitudes of contextualised definitions. Cluster (1) for ‘word’, e.g., features 190 usages, and one must read through all of them (otherwise there will be a chance of missing something) and generalise – all to formulate a definition that covers the whole sense cluster (a sense label). We now show how DWUGs can be automatically augmented with generated sense labels, vastly improving their usability.
Selecting sense labels From n definitions, generated for n word usages belonging to the same DWUG cluster, we use the most prototypical one
as the sense label—with the aim of reflecting the meaning of the majority of usages in the cluster. We represent all definitions with their sentence embeddings (cf. §5.1) and select as prototypical the definition whose embedding is most similar to the average of all embeddings in the cluster. Clusters with less than 3 usages are ignored as, for these, prototypicality is ill-defined. As a sanity check, these are the sense labels obtained by this method for the DWUG clusters of ‘word’; they correspond well to the sense descriptions provided earlier.
1. ‘A SINGLE SPOKEN OR WRITTEN UTTERANCE’
2. ‘INFORMATION; NEWS; REPORTS’
3. ‘A PROMISE, VOW OR STATEMENT’
We compare these sense labels to labels obtained by generating a definition for the most prototypical usage (as judged by its token embedding), rather than taking the most prototypical definition, and we evaluate both types of senses labels using human judgements. Examples of labels can be found in Appendix D.
Human evaluation Five human annotators (fluent English speakers) were asked to evaluate the quality of sense labels for each cluster in the English DWUGs, 136 in total. Each cluster was accompanied by the target word, two labels (from definitions and from usages) and five example usages randomly sampled from the DWUG. The annotators could select one of six judgements to indicate overall quality of the labels and their relative ranking. After a reconciliation round, the categorical judgements were aggregated via majority voting. Krippendorff’s α inter-rater agreement is 0.35 on the original data and 0.45 when the categories are reduced to four. Full guidelines and results are reported in Appendix E.12
We find that our prototypicality-based sense labelling strategy is overall reliable. Only for 15% of the clusters, annotators indicate that neither
12There exist no established procedures for the collection of human quality judgements of automatically generated word sense labels. The closest efforts we are aware of are those in Noraset et al. (2017), who ask annotators to rank definitions generated by two systems, providing as reference the gold dictionary definitions. In our case, (1) generations are for word senses rather than lemmas, (2) we are interested not only in rankings but also in judgements of ‘sufficient quality’, (3) dictionary definitions are not available for the DWUG senses; instead (4) we provide annotators with usage examples, which are crucial for informed judgements of sense definitions.
of the labels is satisfactory (Figure 9). When comparing definition-based and usage-based labels, the former were found to be better in 31% of the cases, while the latter in only 7% (in the rest of the cases, the two methods are judged as equal). We also analysed how often the labels produced by each method were found to be acceptable. Definition-based labels were of sufficient quality in 80% of the instances, , while for usage-based labels this is only true for 68% of the cases.
In sum, prototypical definitions reflect sense meanings better than definitions of prototypical usage examples. We believe this is because definitions are more abstract and robust to contextual noise (the same definition can be assigned to very different usages, if the underlying sense is similar). This approach takes the best of both worlds: the produced representations are data-driven, but at the same time they are human-readable and naturally explanatory. After all, ‘senses are abstractions from clusters of corpus citations’ (Kilgarriff, 1997). In the next section, we demonstrate how automatically generated definition-based sense labels can be used to explain semantic change observed in diachronic text corpora.

heading: 7 Explaining Semantic Change with Sense Labels
text: Word senses in DWUGs are collections of example usages and they are only labelled with numerical identifiers. This does not allow users to easily grasp the meaning trajectories of the words they are interested in studying. Using sense labels extracted from generated definitions, we can produce a fully human-readable sense dynamics map—i.e., an automatically annotated version of a DWUG which displays synchronic and diachronic relations between senses (e.g, senses transitioning one into another, splitting from another sense, or two senses merging into one). One can look at sense dynamics maps as reproducing the work of Mitra et al. (2015) on the modern technological level and, importantly, with human-readable sense definitions.
Given a target word, its original DWUG, and its semi-automatic sense clusters, we start by assigning a definition label to each cluster, as described in §6. Then, we divide each cluster into two subclusters, corresponding to time periods 1 and 2 (for example, sub-cluster c21 contains all usages from cluster 1 occurring in time period 2).13 We
13Note that the labels are still time-agnostic: that is, sub-
compute pairwise cosine similarities between the sentence embeddings of the labels (their ‘definition embeddings’), thereby producing a fully connected graph where nodes are sub-clusters and edges are weighted with sense label similarities. Most edges have very low weight, but some sub-cluster pairs are unusually similar, hinting at a possible relation between the corresponding senses. We detect these outlier pairs by inspecting the distribution of pairwise similarities for values with z-score higher than 1 (similarities more than 1 standard deviation away from the mean similarity). Sub-cluster pairs connected with such edges form a sense dynamics map.
As an example, the noun ‘record’ has only one sense in time period 1 but it acquires two new senses in time period 2 (Figure 3; as before, we ignore clusters with less than 3 usages). The sense clusters defined by the DWUG are anonymous collection of usages, but with the assigned sense labels (also shown in Figure 3) they can be turned into a proto-explanation of the observed semantic shift:
• A novel sense 2 of ‘record’ in time period 2 (‘A PHONOGRAPH OR GRAMOPHONE CYLINDER CONTAINING AN AUDIO RECORDING.’) is probably an offshoot of a stable sense 0 present in both time periods (‘A DOCUMENT OR OTHER MEANS OF PROVIDING INFORMATION ABOUT PAST EVENTS.’).
It becomes now clear that sense 2 stems from the older general sense 0 of ‘record’—arguably representing a case of narrowing (Bloomfield, 1933)— while the second new sense (1: ‘THE HIGHEST SCORE OR OTHER ACHIEVEMENT IN THE GAME’) is not related to the others and is thus independent.
Sense dynamics maps can also help in tracing potentially incorrect or inconsistent clustering in DWUGs. For instance, if different sense clusters are assigned identical definition labels, then it is likely that both clusters correspond to the same sense and that the clustering is thus erroneous. Using our automatically produced sense dynamics maps, DWUGs can be improved and enriched (semi-)automatically.
An interesting case is ‘ball’ (see Appendix F for another example regarding the word ‘chef ’).
clusters c11 and c21 have the same label. This is done for simplicity and because of data scarcity, but in the future we plan to experiment with time-dependent labels as well. We use two time periods as only two periods are available in Schlechtweg et al.’s English DWUGs (2021), but the same procedure can be executed on multi-period datasets.
Although none of its sense labels are identical, its sense cluster c0 is very close to cluster c2 (similarity of 0.70), while c2 is close to c3 (similarity of 0.53); all three senses persist throughout both time periods, with sense 3 declining in frequency. The generated definitions for the ‘ball’ clusters are: 0: ‘A SPHERE OR OTHER OBJECT USED AS THE OBJECT OF A HIT’ (the largest cluster), 2: ‘A ROUND SOLID PROJECTILE, SUCH AS IS USED IN SHOOTING’, and 3: ‘A BULLET’. This case demonstrates that similarity relations are not transitive: the similarity between c0 and c3 is only 0.50, below our outlier threshold value. This is in part caused by inconsistent DWUG clustering: while the majority of usages in c12 are about firearm projectiles, c22 contains mentions of golf balls and ball point pens. This shifts sense 2 from ‘BULLET’ to ‘ROUND SOLID PROJECTILE’, making it closer to sense 0 (general spheres) than it should be. Ideally, all the ‘BULLET’ usages from c2 should have ended up in c3, with the rest joining the general sense 0.
Besides suggesting fixes to the DWUG clustering, the observed non-transitivity also describes a potential (not necessarily diachronic) meaning trajectory of ‘ball’: from any spherical object, to spherical objects used as projectiles, and then to any projectiles (like bullets), independent of their form. Our generated sense labels and their similarities help users analyse this phenomenon in a considerably faster and easier way than by manually inspecting all examples for these senses.

heading: 8 Conclusion and Future Work
text: In this paper, we propose to consider automatically generated contextualised word definitions as a type of lexical representation, similar to traditional
word embeddings. While generated definitions have been already shown to be effective for word sense disambiguation (Bevilacqua et al., 2020), our study puts this into a broader perspective and demonstrates that modern language models like Flan-T5 (Chung et al., 2022) are sufficiently mature to produce robust and accurate definitions in a simple prompting setup. The generated definitions outperform traditional token embeddings in word-in-context similarity judgements while being naturally interpretable.
We apply definition-based lexical representations to semantic change analysis and show that our approach can be used to trace word sense dynamics over time. Operating in the space of humanreadable definitions makes such analyses much more interesting and actionable for linguists and lexicographers—who look for explanations, not numbers. At the same time, we believe the ‘definitions as representations’ paradigm can also be used for other NLP tasks in the area of lexical semantics, such as word sense induction, idiom detection, and metaphor interpretation.
Our experiments with diachronic sense modelling are still preliminary and mostly qualitative. It is important to evaluate systematically how well our predictions correspond to the judgements of (expert) humans. Once further evidence is gathered, other promising applications include tracing cases of semantic narrowing or widening over time (Bloomfield, 1933) by analysing the variability of contextualised definitions in different time periods and by making cluster labels time-dependent. Both directions will require extensive human annotation, and we leave them for future work.
Limitations
Data in this work is limited to the English diachronic word usage graphs (DWUGs). Our methods themselves are language-agnostic and we do not anticipate serious problems with adapting them to DWUGs in other languages (which already exist). At the same time, although Flan-T5 is a multilingual LM, we did not thoroughly evaluate its ability to generate definitions in languages other than English. Again, definition datasets in other languages do exist and technically it is trivial to fine-tune Flan-T5 on some or all of them.
Generated definitions and mappings between definitions and word senses can contain all sorts of biases and stereotypes, stemming from the underlying language model. Filtering inappropriate character strings from the definitions can only help as much, and further research is needed to estimate possible threats.
In our experiments with Flan-T5, the aim was to investigate the principal possibility of using this LM for definition modelling. Although we did evaluate several different Flan-T5 variants, we leave it for the future work to investigate the impact of model size and other experimental variables (such as decoding algorithms).
The cases shown in §7 are hand-picked examples, demonstrating the potential of using generated definitions for explainable semantic change detection and improving LSCD datasets. In the future, we plan to conduct a more rigorous evaluation of different ways to build sense dynamics map.

heading: Acknowledgements
text: This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 819455). The computations were performed on resources provided through Sigma2—the national research infrastructure provider for High-Performance Computing and large-scale data storage in Norway.

heading: A Preliminary Analysis of Usage Examples
text: In Section 3.1 of the main paper, we present three corpora of human-written definitions and report their main statistics in Table 2, including mean and standard deviation of usage example length. Because the length of usage examples has been shown to affect the quality of generated definitions (Almeman and Espinosa Anke, 2022), in a preliminary analysis, we compare the length distributions of usage examples in the corpora of definitions as well as in the English DWUGs (Schlechtweg et al., 2021). Figures 4-7 show the length distributions of the four datasets. We also measure the correlation between definition quality (BertScore, BLEU, NIST) and (i) the length of usage examples, (ii) the absolute position of the target word in the examples, and (iii) the target word’s relative position in the examples. Tables 6 and 7 show the correlation coefficients.

heading: B Prompt Selection
text: As briefly discussed in Section 4, in preliminary experiments, we use the pretrained Flan-T5 Base model (250M parameters; Chung et al., 2022) to select a definition generation prompt among 8 alternative verbalisations. These are a combination of four different instruction strings (‘Define w’, ‘Define the word w’, ‘Give the definition of w’, ‘What is the definition of w?) and two ways of concatenating instructions to usage examples – i.e., either prepending them or appending them. Tables 8-11 show the results of our experiments. In the tables, the strings ‘pre’ and ‘post’ refer to the concatenation method (prepending or appending the instruction), the numbers 128, 256, and 512 refer to the maximum length of the usage examples provided to Flan-T5 (in sub-words), and ‘filter’ refers to the decoding strategy of always avoiding the target word (definiendum).

heading: C Additional Results
text: C.1 Zero-Shot Evaluation of Flan-T5 (Task Shift)
Here we directly evaluate Flan-T5 XL on the WordNet and Oxford test sets, without any fine-tuning nor in-context learning.14 Table 3 in the main paper shows low BLEU and ROUGE-L scores but rather high BERT-F1. Overall, the model does not exhibit consistent task understanding (e.g. it generates ‘SKEPTICISM’ as a definition for ‘healthy’ as exemplified in the phrase ‘healthy skepticism’). A qualitative inspection, however, reveals that the generated definitions can still be often informative (e.g., ‘A WORKWEEK THAT IS LONGER THAN THE REGULAR WORKWEEK’ is informative with respect to the meaning of ‘overtime’ although the ground truth definition is ‘BEYOND THE REGULAR TIME’). The two surface-overlap metrics cannot capture this, but the relatively high BERT-F1 confirms that the semantic content of generations is largely appropriate. There are indeed also many good zero-shot definitions. For example ‘INTENSE’ for ‘fervent’ as in ‘the fervent heat’, or ‘A CONVERSATION’ for ‘discussion’ in ‘we had a good discussion’.
C.2 Other Models and Model Variants
We evaluate T5 (base and XL) and Flan-T5 (base, large, and XL) under the same generalisation conditions presented for Flan T5 XL in the main paper
14We only condition generation on the usage examples and the task prompt. We do not provide full instances (i.e., usage examples, task prompts, and definitions) in the context, as one would do in a few-shot setup.
(Section 4.1) and above in Appendix C.1. Results for FlanT5-XL are reported in the main paper (Table 3); here, in Table 12, we report results for all models and model variants.
C.3 Evaluation Cards
In Table 13, we provide an evaluation card to clarify the nature of the generalisation tests performed on definition generators.15 In-distribution tests are not included as they do not include any shift between the training and test data distributions (Hupkes et al., 2022). We also register our work in the GenBench evolving survey of generalisation in NLP.16

heading: D Additional Examples of Generated Definitions and Sense Labels
text: Some definitions generated by Flan-T5 XL manage to capture very subtle aspects of the contextual lexical meaning. In the following list, we give the usage and then the contextual definition of ‘word’:
1. ‘There are people out there who have never heard of the Father, Son and Holy Spirit, let alone the Word of God.’: ‘THE BIBLE’
2. ‘Good News Bible Before the world was created, the Word already existed; he was with God, and he was the same as God.’: ‘( CHRISTIANITY ) THE SECOND PERSON OF THE TRINITY ; JE’
3. ‘It was in that basement that I learned the skills necessary to succeed in the difficult thespian world-specifically, get up on stage, say
15https://genbench.org/eval_cards 16https://genbench.org/references
my words, get off the stage-skills...’: ‘THE DIALOGUE OF A PLAY.’
Interesting insights can be drawn from how the embeddings of the generated definitions are located in the vector space. Figure 8 shows PCA projections of definition embeddings for usages of the words ‘chef ’ and ‘lass’ from the English DWUG. Colours represent sense clusters provided in the DWUG, and the legend shows most prototypical definitions for each sense generated by our best system (singleton clusters are ignored). The large star for each sense corresponds to its sense label (as opposed to smaller stars corresponding to other definitions not chosen as the label).
For the word ‘chef ’, there are two sense clusters, for which an identical definition is chosen (‘A COMMANDER’). This most probably means that these clusters should in fact be merged together, or that they are in the process of splitting (see also Section 7). These two senses are (not surprisingly) much closer to each other than to the definitions from the ‘PROFESSIONAL COOK’ sense. For the word ‘lass’, it is interesting how separate is a small bluish group of definitions in the bottom right corner of the plot, where the target form is actually ‘lassi’. The fine-tuned Flan-T5-XL model defined this group as ‘A COLD DRINK MADE FROM MILK
CURDLED BY YOGURT’, which is indeed what ‘lassi’ is (ignoring minor details).

heading: E Human Evaluation Guidelines
text: Figures 9 and 10 show the results of the human evaluation.
‘You are given a spreadsheet with four columns: Targets, Examples, System1 and System2. In every row, we have one target English word in the Targets column and five (or less) example usages of this word in the Examples column. Usages are simply sentences with at least one occurrence of the target word: one usage per line.
Every row is supposed to contain usages where the target word is used in the same sense: this means that for ambiguous words, there will be multiple rows, each corresponding to a particular sense. This division into senses is not always 100% correct, but for the purposes of this annotation effort, we take it for granted. Note that the five example usages in each row are sampled randomly from a larger set of usages belonging to this sense.
System1 and System2 are computational models which produce human-readable labels or definitions for each sense of a target word. They employ different approaches, and your task is to compare and evaluate the labels generated by these two systems. Note that in each row, the names ‘System1’ and ‘System2’ are randomly assigned to the actual generation systems.
The generated sense labels are supposed to be useful for historical linguists and lexicographers. Thus, they must be:
1. Truthful: i.e., should reflect exactly the sense in which the target word is occurring in the example usages. Ideally, the label should be general enough to encompass all the usages from the current row, but also specific enough so as not to mix with other senses (for polysemantic target words).
2. Fluent: i.e., feeling like natural English sentence or sentences, without grammar errors, utterances broken mid-word, etc
You have to fill in the Judgements column with one of six integer values:
• 0: both systems are equally bad for this sense
• 1: System 1 is better, but System 2 is also OK
• 11: System 1 is better, and System 2 is bad
• 2: System 2 is better, but System 1 is also OK
• 22: System 2 is better, and System 1 is bad
• 3: both systems are equally good for this sense
Some rows are already pre-populated with the 3 judgement, because the sense labels generated by both systems are identical. We hypothesise that this most probably means that both labels are equally good. Please still have a look at these identical labels and change 3 to 0 in case you feel that in fact they are equally bad.’

heading: F Sense Dynamics Maps
text: It is easy to find different sense clusters which are assigned identical definition labels. Usage examples from sense clusters c2 and c3 for the word ‘chef ’, to which our system assigned the same label: ‘A COMMANDER’:
• c2: ‘He boasted of having been a chef de brigade in the republican armies of France’, ‘Morrel has received a regiment, and Joliette is Chef d’Escadron of Spahis’, ‘as majorgeneral and chef d’escadron, during the pleasure of our glorious monarch Louis le Grand’
• c3: ‘That brave general added to his rank of chef de brigade that of adjutant general’, ‘I frequently saw Mehevi and several other chefs and warriors of note take part’
Thus, a user can safely accept the suggestion of our system to consider these two clusters as one sense.
Note that ‘A COMMANDER’ practically disappeared as a word sense in the 20th century, replaced by ‘A PROFESSIONAL COOK, USUALLY IN A RESTAURANT’.

heading: G Clustering Embedding Spaces
text: We constructed three types of embedding spaces; (i) contextualised token embeddings, (ii) sentence embeddings, and (ii) definition embeddings. We did so for two language models: RoBERTa-large and DistilRoBERTa. Since we cluster the embedding spaces for each target word individually, we obtain different optimal number of clusters for each target word. Table 5 displays the average results over all target words.
We observe that the optimal number of clusters K is substantially higher for the definition embedding spaces for both RoBERTa-large and DistilRoBERTa. However, this is an artefact of the data: since some distinct usages yield identical definitions for a target word, the definition space oftentimes consist of less distinct data points, which greatly impacts the average silhouette scores. Future work should point out what clustering methods are most applicable to definition embedding spaces. Still, this decrease in data points confirms how the definition embedding space could represent usages at a higher level of abstraction, collapsing distinct usages into identical representations.
Figure 11 displays the T-SNE projections of each of the three embedding spaces of RoBERTA-large. As for Distil-RoBERTa, the definition embedding space appears to have spacial properties that are more similar to contextualised token embedding spaces than to sentence embedding spaces: the definition embeddings are more separated than the sentence embeddings, and are cluttered in a similar manner as the token embeddings.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Unnumbered "Limitations" section after the Conclusion (section 8)
3 A2. Did you discuss any potential risks of your work? Unnumbered "Limitations" section after the Conclusion (section 8)
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3
3 B1. Did you cite the creators of artifacts you used? 3
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The licenses are described in the papers we cite.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We did not collect or use any such data.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? This is described in the papers we cite.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3
C 3 Did you run computational experiments? 4,5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4 and Appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4,5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 6
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 6
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? The human annotator co-authored the paper, so this discussion was not necessary.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? We did not collect any data.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? 6

heading: Future Work
text: Future work should point out what clustering methods are most applicable to definition embedding spaces. Still, this decrease in data points confirms how the definition embedding space could represent usages at a higher level of abstraction, collapsing distinct usages into identical representations. Figure 11 displays the T-SNE projections of each of the three embedding spaces of RoBERTA-large. As for Distil-RoBERTa, the definition embedding space appears to have spacial properties that are more similar to contextualised token embedding spaces than to sentence embedding spaces: the definition embeddings are more separated than the sentence embeddings, and are cluttered in a similar manner as the token embeddings. ACL 2023 Responsible NLP Checklist