Towards more complete solutions for Lexical Semantic Change: an extension to multiple time periods and diachronic word sense induction (2024)

Abstract: Thus far, the research community has focused on a simplified computational modeling of semantic change between two time periods. This simplified view has served as a foundational block but is not a complete solution to the complex modeling of semantic change. Acknowledging the power of recent language models, we believe that now is the right time to extend the current modeling to multiple time periods and diachronic word sense induction. In this position paper, we outline several extensions of the current modeling and discuss issues related to the extensions.

text: Proceedings of the 5th Workshop on Computational Approaches to Historical Language Change, pages 108–119 August 15, 2024 ©2024 Association for Computational Linguistics
Thus far, the research community has focused on a simplified computational modeling of semantic change between two time periods. This simplified view has served as a foundational block but is not a complete solution to the complex modeling of semantic change. Acknowledging the power of recent language models, we believe that now is the right time to extend the current modeling to multiple time periods and diachronic word sense induction. In this position paper, we outline several extensions of the current modeling and discuss issues related to the extensions.

heading: 1 Introduction
text: Lexical Semantic Change (LSC) is the problem of automatically identifying words that change their meaning over time (Periti and Montanelli, 2024; de Sá et al., 2024; Tahmasebi et al., 2021; Kutuzov et al., 2018; Tang, 2018). Conceptually, this problem implicitly involves a fundamental step of diachronic word sense induction to distinguish each individual sense of a word over all the multiple time periods of interest (Periti et al., 2023; Alsulaimani and Moreau, 2023; Alsulaimani et al., 2020; Emms and Jayapal, 2016; Tahmasebi, 2013). However, the computational challenges in handling large corpora and the absence of comprehensive benchmarks have in practice led to a simplified modeling focused on two time periods 𝑡1 and 𝑡2 only. These are either modeled individually 𝑡1, 𝑡2 or in a single time interval ⟨𝑡1, 𝑡2⟩ considering all the data jointly.
Typically, approaches over two time periods are assumed to be directly extendable to real scenarios involving multiple time periods. For example, approaches designed for a single interval ⟨𝑡1, 𝑡2⟩, can be iteratively re-executed across multiple, contiguous intervals ⟨𝑡1, 𝑡2⟩, ⟨𝑡2, 𝑡3⟩, …,
*Authors contributed equally
⟨𝑡𝑛−1, 𝑡𝑛⟩ (Giulianelli et al., 2020). However, multiple re-executions presents a computational challenge that significantly escalates as the number of considered periods increases. Procedures that were initially considered optional steps to expedite modeling in two time periods become fundamental over multiple time periods. For instance, since words can occur thousands of times in a diachronic corpus, it becomes imperative to randomly sample a limited number of occurrences and to leverage hardware components, such as GPU processor units.
Due to the absence of diachronic lexicographic resources (e.g., dictionaries, thesauri), and the gap between a general resource and specific data, the modeling of word sense is commonly approached in an unsupervised manner. Clustering techniques are generally employed to aggregate usages of a specific word into clusters, with the idea that each cluster denotes a specific word meaning that can be recognized in the considered documents. However, clusters of usages (regardless of method of clustering) do not necessarily correspond to precise senses (Martinc et al., 2020), but typically represent noisy projections related to specific context (Periti and Montanelli, 2024). As a result, manual activity is always required to translate the automatically derived clusters into a diachronic sense inventory. This sense inventory is the basis for interpreting the identified semantic change and modeling sense evolution (see Figure 1). While automatic methods, such as keywords extraction (Kellert and Mahmud Uz Zaman, 2022), or generating definitions for word usages (Giulianelli et al., 2023), have been proposed to support cluster interpretation, a reliable interpretation still needs manual supervision. Therefore, when multiple time periods are considered, interpretability challenges increase several orders of magnitude, making the direct re-execution of existing approaches unsuitable for effectively detecting semantic change and the evolution of each individual word meaning (Periti et al., 2023, 2022).
108
We thus argue that the diachronic word sense induction over multiple time periods inherent to LSC requires more careful considerations compared to the simplified modeling currently done. More efforts should be devoted to develop approaches for assisting text-based researchers like linguists, historians and lexicographers as much as possible.
Our original contribution In this paper, we discuss the complexities inherent in modeling semantic change for each word sense individually over multiple time periods. We challenge the general assumption that conventional approaches designed to address LSC over two time periods are easily extendable over multiple time periods. Because currently, contextualized embeddings represents the preferred tool for addressing LSC (Periti and Tahmasebi, 2024), we will use these as an example. Our discussion is however more general, and can be applied regardless of which model is used to represent individual word usages – such as definitions (Giulianelli et al., 2023), co-occurrence vectors (Schütze, 1998), lexical replacements (Periti et al., 2024), or bag-ofsubstitutes (Kudisov and Arefyev, 2022) – or sense clusters in general as in Tahmasebi and Risse, 2017.
We advocate for an alternative modeling of LSC over multiple time periods, and specifically, we present i) five distinct approaches for tracking semantic change and the evolution of word meanings; and ii) three distinct settings for assessing semantic change over time. Our work has significant
implications for both the computational modeling and the creation of benchmarks, contributing to the ongoing discussion presented by Periti and Montanelli (2024); Hengchen et al. (2021); Montariol et al. (2021) on the open challenges associated with modeling semantic change.

heading: 2 Background and related work
text: Since SemEval-2020 (Schlechtweg et al., 2020), there is an established evaluation framework for LSC to compare the performance of various models and approaches. However, given the substantial annotation efforts required to create reliable benchmarks over multiple time periods, the framework is typically adopted to create simplified benchmarks over two time periods, with gold labels for semantic change but without diachronic sense labels (Ling et al., 2023; Chen et al., 2023; Kutuzov et al., 2022a; Zamora-Reina et al., 2022; Kutuzov and Pivovarova, 2021; Basile et al., 2020; Schlechtweg et al., 2020).1 In such benchmarks, the LSC problem is defined as follows.

heading: 2.1 Problem statement over two time periods
text: Given a diachronic corpus  containing a set of documents (e.g., sentences, paragraphs) from two time periods 𝑡1 and 𝑡2, the current modeling of LSC involves the following evaluation tasks:
1Kutuzov and Pivovarova, 2021 introduced a benchmark encompassing two time intervals. However, these intervals have been treated independently, leading to their consideration as two distinct sub-benchmarks over a single time interval.
i) to quantify the semantic change of words (i.e., Graded Change Detection);
ii) to recognize words that change their meaning by either gaining new ones or losing old ones (i.e., Binary Change Detection, Sense Gain Detection, Sense Loss Detection).
Words that change their meanings by means of gaining or loosing senses will have a high degree of (graded) semantic change, while words that have a high degree of graded change do not need to have lost or gained senses.
These tasks inherently involve the modeling of word meanings across 𝑡1 and 𝑡2. However, due to the lack of diachronic sense labels, researchers and practitioners tend to focus on addressing tasks i) and ii) without adequately tackling the challenges associated with modeling sense evolution.

heading: 2.2 State-of-the-art approaches to LSC
text: Thus far, computational approaches to solve the above tasks have followed a standard receipt using a four-step pipeline (Periti and Montanelli, 2024). Given a corpus  spanning two time periods 𝑡1 and 𝑡2, and a target word 𝑤:
1) extraction of the word occurrences from both 𝑡1 and 𝑡2; 2) computational representation of each occurrence (the current standard is to leverage pretrained contextualized embeddings); 3) word sense induction by aggregating embeddings with a clustering algorithm; 4) assessment of semantic change by leveraging a distance measure on the embeddings from 𝑡1 and 𝑡2.
Approaches are typically distinguished in formbased and sense-based. The former does not induce sense (3) but quantifies semantic change using (1,2,4), either as a shift in the dominant meaning of 𝑤 or in its degree of polysemy. There is thus no easy way to discern individual senses from the change score without integrating “close reading” by humans. Sense-based approaches remedies this by relying on all steps (1-4) but generally induce senses (3) in a synchronic way, without considering the temporal nature of the documents (Ma et al., 2024). That is, they consider all the documents from 𝑡1 and 𝑡2 available as a whole and perform a single clustering activity over the entire set of generated embeddings, regardless of their time origin.

heading: 2.3 Modeling senses through clusters
text: The clustering of representations via word sense induction, step (3) above, serves as a tool to operationalize word senses in an unsupervised fashion through unstructured text (Lake and Murphy, 2023). On one hand, this operationalization offers a flexible adaptation to the data under consideration and allows to derive senses that do not necessarily need to be aligned with available static lexicographic resources (Kilgarriff, 1997). For instance, senses derived from youth slang (Keidar et al., 2022), or scientific texts are unlikely to align with a general lexicon meant to cover the whole spectrum of a given language.
On the other hand, as computational models derive information from the contexts surrounding word tokens, sense modeling tends to emphasize word usages rather than word meanings (Tahmasebi and Dubossarsky, 2023; Kutuzov et al., 2022b). Thus, while ideally we would like each cluster to correspond to one, and only one sense, in practice, multiple clusters may correspond to different nuances of the same sense. This effect is further amplified when considering data from diverse time, domains, or genres, where distinct linguistic registers, styles, or co-occurrence patterns may results in different senses.
Additionally, the interpretation of clusters as senses requires a notion of (word) “meaning” that can both differ in the mind of humans according to social or cultural background and age, as well as in the varying usages of a word in context. Thus, the mapping of clusters to senses involves i) identifying commonalities on the usages of each cluster that may be judged differently, as well as ii) mapping these commonalities to word meanings. The outcome results in a sense inventory.

heading: 2.4 Modeling LSC over multiple time periods
text: Modeling LSC involves computationally deriving word senses progressively over time. This entails re-executing the steps (1-4) multiple times. At each execution 𝑖, a set of clusters is generated and humans are needed to identify and update the sense inventory. This involves mapping the clusters generated at the 𝑖-th execution to senses and aligning senses temporally.
The way senses align over time give us important insights into how word meanings change. Classifying types of semantic change has been long studied and different schema have been proposed (Blank,
1997; Bloomfield, 1933; Paul, 1880). Among others, common types of change include broadening of meaning (e.g., dog was used to refer to dogs of specific large and strong breeds), narrowing of meaning (e.g., girl was used to refer to people of either gender), novel senses (e.g., rock as a music genre) and metaphorical extensions (e.g., surfing the web). The result is a diachronic sense inventory with temporal information on the active senses at each time, as well as potential relationships between senses.
To facilitate the interpretation of semantic change and the evolution of word meaning, the current, synchronic modeling of senses can benefit from diachronic modeling encompassing both incremental word sense induction and cluster alignment (Kanjirangat et al., 2020). Aligning clusters computationally will allow the simultaneous interpretation of multiple clusters, thereby reducing the burden of manual supervision at each time period. Clusters aligned over time can potentially suggest the continuation of an active sense, as well as the broadening and narrowing of meanings. In contrast, clusters not aligned over time can reveal both the continuation of different senses, as well as types of semantic change, like metaphoric extension.
Thus far, word meanings have been modeled through conventional clustering algorithms such as Affinity Propagation (Martinc et al., 2020) or K-Means (Kobayashi et al., 2021). However, these algorithms were originally designed for one-time data clustering and are not inherently suited to handle temporal dynamics. Specifically, clusters generated at 𝑡𝑖−1 can become mixed up when reexecuting the algorithm with both previous data and new data points at time ⟨𝑡𝑖−1, 𝑡𝑖⟩. Consequently, objects that were previously clustered together at time 𝑡𝑖−1 may either remain in the same cluster or be reassigned to different clusters based on the updated data at time 𝑡𝑖. This dynamic nature complicates the task of tracking the history of specific clusters across different time periods, and can lead to the creation of noisy clusters, especially when new data points arrive according to a skewed distribution.
Diachronic sense clustering. Conventional unsupervised clustering algorithms do not incorporate the faithfulness properties typical in incremental clustering literature, where clustering activities at any given point in time should remain faithful to the already existing clusters as much as possible (Chakrabarti et al., 2006) while at the same time
be flexible to fit the new data. This would avoid dramatic change in clusters from one time-step to the next that do not derive from semantic change, but from differences in the underlying documents over time (Castano et al., 2024).
To this end, we argue that, for each target word, modeling LSC over time should involve monitoring the evolution of each individual senses across all the time periods under consideration, as well as tracing the types of each change. However, this extension is not straightforward; instead, it requires crucial time series analysis to mitigate potential noise introduced by the predictions of computational approaches (Kulkarni et al., 2015).
Monitoring and tracing word meaning evolution and semantic change require a careful consideration in the current four-step pipeline of sense-based approaches. As for scalability and interpretability issues related to (1-3), suggestions and workaround are discussed in Periti and Montanelli, 2024; Montariol et al., 2021. In this paper, we further discuss the extension of steps (3) and (4) when considering multiple time points. In particular, we discuss diachronic word sense induction in Section 3, and semantic change assessment in Section 4.

heading: 3 Diachronic word sense induction
text: For the sake of simplicity, consider a diachronic corpus  spanning three general, consecutive time periods 𝑡1, 𝑡2, 𝑡3, not necessarily contiguous. This simplification does not lead to any loss of information, but serves to aid the discussion in a clear
and concise fashion. At the same time, three time points are easily extendable to the general case of tens or hundreds of time periods. Word usages, and their corresponding representations, for time period 𝑡1, 𝑡2, and 𝑡3 are denoted with ■, △, ⬖, respectively. From here on, we will use contextualized embeddings as an example for contextualized representations. In the following, we present five different approaches for monitoring the evolution of word meanings and discuss suitability, and drawbacks.

heading: 3.1 Clustering over consecutive time intervals
text: Clustering algorithms used for jointly modeling senses over two time periods 𝑡1 and 𝑡2 can be progressively re-executed over consecutive pairs of time periods ⟨𝑡1, 𝑡2⟩ and ⟨𝑡2, 𝑡3⟩. To facilitate the interpretation of sense evolution, a cluster alignment step is thus required between consecutive reexecutions. For instance, in Figure 2, the clusters generated in step (B) are linked to those generated in step (A) through a cluster alignment step (C) (Deng et al., 2019).
When clustering over consecutive time intervals ⟨𝑡1, 𝑡2⟩,… , ⟨𝑡𝑛−1, 𝑡𝑛⟩, the embeddings from 𝑛 − 2 time periods (all time periods but first and last) are clustered twice. For instance, consider the embeddings △ from 𝑡2 in Figure 2: (A) they are first clustered with the embeddings ■ from 𝑡1, and (B) then re-clustered with the embeddings ⬖ from 𝑡3. When a limited number of word usages is available, this approach can potentially enhance the emergence of certain senses, as patterns of embeddings from 𝑡𝑖−1 are reinforced by additional evidence (if present) from 𝑡𝑖. However, this compromises the
faithfulness property, as embeddings from 𝑡𝑖 can be clustered differently when considered jointly with 𝑡𝑖−1 compared to when considered jointly with 𝑡𝑖+1 (from a past and future perspective respectively).

heading: 3.2 Clustering over consecutive time periods
text: When a substantial number of documents is available for each time period, there is no need to cluster the embeddings of a time interval as a whole. Instead, the embeddings of each time period can be clustered individually, and a cluster alignment algorithm can be applied progressively to link the clusters across time periods (Kanjirangat et al., 2020; Montariol et al., 2021). This approach is represented in Figure 3. Step (A), (B), and (D) represents the application of a conventional clustering algorithm over the embeddings of time period 𝑡1, 𝑡2, 𝑡3, respectively. Step (C) and (E) represents cluster alignment steps to link the clusters generated through step (B) to the cluster generated through step (A), and in turn, the clusters generated through step (D) to the cluster generated through step (B) (Deng et al., 2019).
Clustering over time periods involves a similar number of clustering activities and cluster alignment steps as clustering over time intervals. However, each clustering activity is more scalable, as it involves a smaller number of embeddings.

heading: 3.3 One-time clustering over all time periods
text: Embeddings from all the considered time periods can be clustered jointly in one single execution. For instance, in Figure 4 step (A), embeddings
■, △, ⬖ are clustered together as a whole. This single clustering activity results in clusters that may include embeddings from various combinations of time periods. For example, a cluster may include embeddings from a single, all, or selected time periods. A cluster alignment step (B) can be further executed to enable the modeling of sense evolution and change type.
When dealing with hundreds of time periods and a significant number of embeddings at once, clustering can be unfeasible due to scalability issues. In real scenarios, a diachronic corpus can be dynamic (Periti et al., 2022), where documents from subsequent time periods are not available as a whole but are progressively added (e.g., posts from social networks, Kellert and Mahmud Uz Zaman, 2022; Noble et al., 2021). In such scenarios this approach is thus not suitable as it would require re-execution of the clustering from scratch when new documents are added.
Furthermore, the use of conventional clustering algorithms is generally insensitive to the order of time periods, allowing embeddings of later time periods to influence pattern of the earlier time periods. This risks leading to a global view of word meaning while precluding a local view where smaller and gradual variation of individual senses as well as small sense clusters are missed. These issues can be mitigated by considering the temporal order of documents in the clustering activity (Smyth, 1996).

heading: 3.4 Incremental clustering over time periods
text: Incremental clustering algorithms are designed to effectively address the temporal nature of data (Kulkarni and Mulay, 2013). Thus, they are
a suitable option to model the dynamic nature of language where temporal progression is key. When employed for diachronic word sense induction, they can efficiently and directly update the prior clustering results by processing and assimilating new data into existing clusters. The word usages observed in past time periods are consolidated into a set of clusters that constitute the memory of the word meanings observed thus far (Periti et al., 2022). This memory then serves as a foundation for understanding subsequent word usages in the current time period. Like Figure 4, Figure 5 represents similar steps (A-C) without alignment as clusters generated in step (A-C) are directly and consecutively updated.
Some of the incremental algorithms implement the faithfulness property in an evolutionary way: once a cluster has been created, it can only gain new members (i.e, word usages) but can never loose any members that have already been assigned to it. Meanwhile, the word usages observed in the present must be stratified or integrated over those from the past, that is, either be placed in existing clusters, or create new clusters. Other algorithms implement the faithfulness property in a more flexible way and enable small changes in past clusters when more evidence is available.

heading: 3.5 Scaling up with form-based approaches
text: Regardless of the complexity of each presented method, it is difficult to scale an approach to the level of whole vocabulary in a large corpus. In addition, some senses remain stable for a long time before they potentially change meaning, others never change. Therefore, clustering the senses during the stability periods of words is superfluous. To reduce computational needs and scale to the entire vocabulary, form-based approaches (without senseinduction) can be used to monitor stability allowing
the use of more powerful sense-based approaches only when there is indication of change.
By considering change only in the general usage of a word, form-based approaches reduce the semantic change problem significantly. Thus, they serve for two important purposes: first, they can be used to quantify the degree of change at the vocabulary level, and thus give us the opportunity to quantify change during different time periods (e.g., before and after WWI v. WWII); secondly, they can be used to find words and periods of interest.
Such a kind of stability monitoring can be done via change point detection (Kulkarni et al., 2015) and be integrated with diachronic sense modeling as shown in Figure 6. In particular, step A involves quantifying semantic change through form-based assessment to detect change points across the entire time span covered by the corpus. Step B involves modeling each individual sense of the word around the detected change point(s) through approaches presented in Section 3.1-3.4.

heading: 4 Semantic change assessment
text: The diachronic word sense induction is independent from the assessment of change at the level of senses or words. While the modeling of word meaning relies on the notion of word senses, the assessment of change depends on the research questions that we want investigate. E.g., considering a perfect sense inventory we may want to ask how many meanings have been lost and gained, and if change is more evident in some time intervals compared to others. The answer to these depend on the way we assess change.
Assessment of change, like sense induction, has focused on two time intervals which is the smallest unit over which we can quantify change. However, generalizing from two intervals to multiple intervals is not trivial and needs considerations that depend heavily on the kind of research question that is being asked, as well as the kind of data available. Short-term data contra long-term data, or small contra large data require different strategies for quantifying change. Here we present some possible strategies that extend to multiple time periods.
Assessment over consecutive time intervals represents a general way to assess semantic change over time ⟨𝑡1, 𝑡2⟩, ⟨𝑡2, 𝑡3⟩, …, ⟨𝑡𝑛−1, 𝑡𝑛⟩. This kind of assessments can be affected by i) (random) fluctuations in the underlying corpus, where the coverage of topics can be heavily influenced by real-life
events; and ii) noisy artifacts of the computational modeling, e.g., influenced by frequency. The use of time series analysis or statistical tests can reduce the effect of potential artifacts from the data and capture only significant changes evident in the time series (Liu et al., 2021; Kulkarni et al., 2015).
This assessment represents a useful solution for scenarios where the focus is on detecting immediate changes, such as in rapidly evolving fields or during specific events that might impact language usage. When comparing ⟨𝑡𝑖−1, 𝑡𝑖⟩, the assumption is that all the active word meanings in 𝑡𝑖, except for the new or changed ones, are active also in 𝑡𝑖−1. However, some senses are periodic and an undesirable side-effect is that they may be detected as change each time they appear and disappear as they are not represented in 𝑡𝑖−1.
Pairwise assessment over time periods Sometimes research questions may be tailored to specific time intervals (e.g, before and after the time period 𝑡𝑖 of the corona pandemic). Thus, this assessment aims to quantify the change across specific time intervals ⟨𝑡𝑖−1, 𝑡𝑖⟩ and ⟨𝑡𝑗 , 𝑡𝑗+1⟩ such that 𝑖 < 𝑗. This assessment is also useful for identifying changes in periodic senses when the periodicity of the sense is known. For example, the meaning of the term gold related to the Olympic games that take place every forth year.
This assessment is also useful when research questions are tailored to specific types of change irrespective of when the change occurs. For example, when a diachronic sense inventory is available, broadening or narrowing can be investigated regardless of their time-specific appearance.
When all possible time intervals are considered, this assessment is associated with a computational complexity of (𝑛2) where 𝑛 is the number of considered periods. However, it provides a broader view of how meaning evolves over different spans, capturing trends that may not be apparent in consecutive intervals. For example, gradual changes over time would not appear with assessment over consecutive time intervals as too little evidence would be present, but could appear as radical changes when larger gaps between intervals are used.
By considering all the possible time intervals it is also possible to quantify the global level of change over the whole corpus. This method is insensitive to the order of the time periods and is useful for capturing overarching trends and patterns in semantic change across the entire timeline.
Cumulative assessment over time When research questions focus on the novel senses gained at time period 𝑡𝑖, the comprehensive overview of active senses from the past must be considered⋃𝑖−1
𝑗=1 𝑡𝑗 . Instead of considering only consecutive or specific time intervals, each new time period should be compared with the full diachronic sense inventory. Cumulative assessment emphasizes the overall evolution of meaning, providing a holistic view of changes from the beginning to the end of the timeline. It is useful for consolidating the evidence across multiple time periods which would not suffice on their own. For example, when research questions focus on the novelty introduced in time period 𝑡𝑖 compared to the past periods, the assessment of change should consider the cumulative evidence of the past as a single, large time period. Similar assessment can be employed when research questions want to compare a past time period 𝑡𝑖 with respect to the following ⋃𝑛−1 𝑗=𝑖+1 𝑡𝑗 , until the 𝑛𝑡ℎ time period.

heading: 5 Discussion and conclusion
text: Computational modeling of semantic change has long been done in a simplified way due to the challenges related to modeling senses across multiple time periods. However, sense inventories and the type of change a word exhibits, are fundamental aspects for text-based researchers like historians, linguists and lexicographers, and therefore, the full complexity of semantic change must be taken into consideration in the computational modeling. Now that we have powerful language models like GPT-4 (OpenAI, 2023) and XL-LEXEME (Cassotti et al., 2023) there are no excuses for taking a simplistic view on the modeling of semantic change.
In this paper, we have presented possible extensions to expand on the simplistic view. These extensions have equal implications both for the computational modeling as for the generation of manually annotated benchmarks which has also been done over two time periods due to the sheer volume of required annotations.
Crucial for the usefulness of semantic change studies is a diachronic sense inventory where the different senses are linked together to capture semantic change type and linguistic relation. It is using the diachronic sense inventory that the majority of the research questions can be answered. These pertain both to linguistic, language-level questions,
but also to societal and cultural enquiries where text can be used as evidence. How to best frame and store the diachronic sense inventory is still an open issue and requires involvement from the communities around computational modeling of semantic change, word sense induction and lexical semantics in general, as well as the text-based researchers that will use the outcome.
Human supervision is necessary to develop a reliable sense inventory. As diachronic corpora can span multiple time periods and contain millions of documents, automatic supervision support is mandatory to reduce manual efforts as much as is possible. In this regard, aligning similar clusters and detecting change types to speed up the interpretation process is as crucial as it is difficult. Employing different kinds of diachronic word sense induction and assessment as outlined here, will lead to different amounts of manual interaction.
Aligning clusters over time poses a very challenging task, as some clusters may represent outliers, time intervals may be characterized by different numbers of clusters, and multiple noisy (or nuanced) clusters denoting the same meaning may emerge. As a result, the cluster alignment often involves the discretization of a fuzzy problem (Kianmehr et al., 2010), that is the creation of new global clusters that encompass sets of fuzzy clusters. Furthermore, when cluster are aligned through a posteriori step rather than being linked and updated directly, the alignment process (worst case) involves comparing each cluster with every other cluster across all time periods. This risks amplifying the potential level of noise and require intricate decisions typically taken without any theoretical basis.
Thus far, the research community has focused more on the quantification of semantic change rather than the underlying word sense induction because form-based approaches consistently outperformed sense-based approaches. However, the clustering algorithms that have been employed do not take the temporal nature of documents into consideration, and we thus argue that they are not optimal for modeling word meaning over time.
In this paper, we have outlined several possible paths forward, both in terms of diachronic word sense induction and assessment of change. We have left methods for change type detection for future work. Each proposed path is suitable for different kinds of research questions and data. For example, by clustering embeddings over a whole corpus, smaller senses that would not appear in
sequential modeling can gain sufficient evidence in global clustering. Such a method is however computationally expensive. Other methods suffer from the problem that when only consecutive time periods are considered, slow and gradual shift risks being missed and over long time periods other strategies are more suitable. Among these methods, we strongly advocate for a shift towards incremental methods as these are currently the best fit to the LSC problem.

heading: 6 Limitations
text: This is a position paper and as such, we have not reported any experiments nor proposed concrete algorithms. Instead, we have outlined general weaknesses of the current methods in the field of computational modeling of semantic change and discussed possible ways forward. We believe that different kinds of solutions can be used for this purpose, spanning from different classes of clustering algorithms (e.g., evolutionary, Periti et al., 2023) to different classes of graphs and networks (e.g., temporal, Ma et al., 2024).
We have focused on unsupervised methods that induce senses through clustering of word representations. In particular, we have focused on contextualized representations, which represent the de facto standard, irrespective of the model that is used to generate the representations (e.g., Devlin et al., 2019; Hofmann et al., 2021; Cassotti et al., 2023). We only mention other methods such as word masking for lexical substitutions (Card, 2023; Arefyev and Zhikov, 2020) or previous paradigms such as the use of static embeddings (Shoemark et al., 2019). Typically, static embeddings, as well as methods based on SVD or PPMI (Hamilton et al., 2016), collapse all the meanings of a word into a single static vector, thus our proposals may not be considered suitable for such solutions even if dynamic word embeddings such as those presented by Bamler and Mandt (2017); Yao et al. (2018); Rudolph and Blei (2018) are used. However, we argue that the methods outlined in this paper are directly extendable to methods based on static embeddings where sense clusters are generated by looking at the top neighbors in the embedding space (Gonen et al., 2020).
We have not focused on how to detect the type of semantic change nor the cause of it, primarily due to space limitations. However, we believe that the methods outlined in this paper inherently offer
ways to detect type, but not necessarily cause, of change. When we begin to target change types, we need evaluation benchmarks. Creating such benchmarks entail consolidating and digitizing the types of change offered in taxonomies as, for example, (Blank, 1997; Ullmann, 1957; Bloomfield, 1933; Stern, 1931; Bréal, 1904; Darmesteter, 1893; Paul, 1880; Reisig, 1839), such as the work started by Cassotti et al. (2024).

heading: Acknowledgments
text: This work has in part been funded by the research program Change is Key! supported by Riksbankens Jubileumsfond (under reference number M21-0021).

heading: Future Work
text: We have left methods for change type detection for future work. Each proposed path is suitable for different kinds of research questions and data. For example, by clustering embeddings over a whole corpus, smaller senses that would not appear in
sequential modeling can gain sufficient evidence in global clustering. Such a method is however computationally expensive. Other methods suffer from the problem that when only consecutive time periods are considered, slow and gradual shift risks being missed and over long time periods other strategies are more suitable. Among these methods, we strongly advocate for a shift towards incremental methods as these are currently the best fit to the LSC problem